一次性查看每一帧：使用多轴梯度检查点的Video-Ma^2mba实现高效的长视频理解

Look Every Frame All at Once: Video-Ma^2mba for Efficient Long-form Video Understanding with Multi-Axis Gradient Checkpointing

November 29, 2024

作者: Hosu Lee, Junho Kim, Hyunjun Kim, Yong Man Ro

cs.AI

摘要

随着视频数据规模和复杂性的增长，有效处理长视频序列面临着重大挑战，因为现有基于Transformer的大型多模态模型（LMMs）带来的内存和计算需求呈二次增长。为解决这些问题，我们引入了Video-Ma^2mba，这是一种新颖的架构，它在Mamba-2框架中集成了状态空间模型（SSMs），取代了注意力机制。这使得LMMs在时间和内存需求方面呈线性扩展，从而使其能够处理长时间视频内容。此外，我们通过引入多轴梯度检查点（MA-GC）方法来增强内存效率，该方法通过在多个计算轴上仅保留必要的激活来策略性地管理内存。与标准梯度检查点相比，我们的方法显著减少了内存占用。实证分析表明，Video-Ma^2mba能够在单个GPU上处理大量视频序列，相当于数百万个标记或超过两小时的连续序列，帧率为1 FPS。通过保持对时间动态的详细捕获，我们的模型提高了长视频理解任务中响应的准确性和相关性，展示了与现有框架相比的显著优势。

English

With the growing scale and complexity of video data, efficiently processing long video sequences poses significant challenges due to the quadratic increase in memory and computational demands associated with existing transformer-based Large Multi-modal Models (LMMs). To address these issues, we introduce Video-Ma^2mba, a novel architecture that incorporates State Space Models (SSMs) within the Mamba-2 framework, replacing the attention mechanisms. This allows the LMMs to scale linearly in terms of time and memory requirements, making it feasible to handle long-duration video content. Furthermore, we enhance the memory efficiency introducing the Multi-Axis Gradient Checkpointing (MA-GC) method, which strategically manages memory by retaining only essential activations across multiple computational axes. Our approach significantly reduces the memory footprint compared to standard gradient checkpointing. Empirical analyses show that Video-Ma^2mba can process extensive video sequences-equivalent to millions of tokens or over two hours of continuous sequences at 1 FPS-on a single GPU. By maintaining a detailed capture of temporal dynamics, our model improves the accuracy and relevance of responses in long video understanding tasks, demonstrating substantial advantages over existing frameworks.