MR. Video：「MapReduce」是長視頻理解的核心原則

MR. Video: "MapReduce" is the Principle for Long Video Understanding

April 22, 2025

作者: Ziqi Pang, Yu-Xiong Wang

cs.AI

摘要

我們提出了MR. Video，這是一個具備自主性的長視頻理解框架，它展示了處理長視頻時簡單而有效的MapReduce原則：(1) Map：獨立且密集地感知短視頻片段，(2) Reduce：聯合聚合所有片段的信息。與序列到序列的視覺語言模型（VLMs）相比，MR. Video能夠進行細緻的短視頻感知，不受上下文長度的限制。與現有的通常依賴於順序關鍵片段選擇的視頻代理相比，Map操作實現了更簡單且更具擴展性的短視頻片段序列並行感知。其Reduce步驟允許更全面的上下文聚合與推理，超越了顯式的關鍵片段檢索。這一MapReduce原則既適用於VLMs也適用於視頻代理，我們利用LLM代理來驗證其有效性。 在實際應用中，MR. Video採用了兩個MapReduce階段：(A) 字幕生成：為短視頻片段生成字幕（map），然後將重複出現的角色和物體標準化為共享名稱（reduce）；(B) 分析：針對每個用戶問題，從各個短視頻中分析相關信息（map），並將其整合成最終答案（reduce）。在具有挑戰性的LVBench上，MR. Video相比於最先進的VLMs和視頻代理，實現了超過10%的準確率提升。 代碼可於以下網址獲取：https://github.com/ziqipang/MR-Video

English

We propose MR. Video, an agentic long video understanding framework that demonstrates the simple yet effective MapReduce principle for processing long videos: (1) Map: independently and densely perceiving short video clips, and (2) Reduce: jointly aggregating information from all clips. Compared with sequence-to-sequence vision-language models (VLMs), MR. Video performs detailed short video perception without being limited by context length. Compared with existing video agents that typically rely on sequential key segment selection, the Map operation enables simpler and more scalable sequence parallel perception of short video segments. Its Reduce step allows for more comprehensive context aggregation and reasoning, surpassing explicit key segment retrieval. This MapReduce principle is applicable to both VLMs and video agents, and we use LLM agents to validate its effectiveness. In practice, MR. Video employs two MapReduce stages: (A) Captioning: generating captions for short video clips (map), then standardizing repeated characters and objects into shared names (reduce); (B) Analysis: for each user question, analyzing relevant information from individual short videos (map), and integrating them into a final answer (reduce). MR. Video achieves over 10% accuracy improvement on the challenging LVBench compared to state-of-the-art VLMs and video agents. Code is available at: https://github.com/ziqipang/MR-Video