将专门的视觉编码器统一为视频语言模型

Unifying Specialized Visual Encoders for Video Language Models

January 2, 2025

作者: Jihoon Chung, Tyler Zhu, Max Gonzalez Saez-Diez, Juan Carlos Niebles, Honglu Zhou, Olga Russakovsky

cs.AI

摘要

最近出现的大型语言模型（LLMs）已经通过视频大型语言模型（VideoLLMs）将复杂的推理能力引入视频领域。然而，VideoLLMs目前依赖单个视觉编码器进行所有视觉处理，这限制了可以传达给LLM的视觉信息的数量和类型。我们的方法，MERV，即视频的多编码器表示（Multi-Encoder Representation of Videos），相反利用多个冻结的视觉编码器来创建视频的统一表示，为VideoLLM提供全面的专业视觉知识集。从每个编码器中空间-时间地对齐特征使我们能够处理更广泛的开放式和多选视频理解问题，并胜过先前的最先进作品。在标准套件视频理解基准测试中，MERV的准确率比Video-LLaVA高出多达3.7％，同时还具有更好的Video-ChatGPT得分。我们还提高了SeViLA在零样本感知测试准确率上的表现，提高了2.2％。MERV引入了最少的额外参数，比等效的单编码器方法训练更快，同时并行化视觉处理。最后，我们提供定性证据表明MERV成功地从每个编码器中捕获领域知识。我们的结果为利用多个视觉编码器进行全面视频理解提供了有前途的方向。

English

The recent advent of Large Language Models (LLMs) has ushered sophisticated reasoning capabilities into the realm of video through Video Large Language Models (VideoLLMs). However, VideoLLMs currently rely on a single vision encoder for all of their visual processing, which limits the amount and type of visual information that can be conveyed to the LLM. Our method, MERV, Multi-Encoder Representation of Videos, instead leverages multiple frozen visual encoders to create a unified representation of a video, providing the VideoLLM with a comprehensive set of specialized visual knowledge. Spatio-temporally aligning the features from each encoder allows us to tackle a wider range of open-ended and multiple-choice video understanding questions and outperform prior state-of-the-art works. MERV is up to 3.7% better in accuracy than Video-LLaVA across the standard suite video understanding benchmarks, while also having a better Video-ChatGPT score. We also improve upon SeViLA, the previous best on zero-shot Perception Test accuracy, by 2.2%. MERV introduces minimal extra parameters and trains faster than equivalent single-encoder methods while parallelizing the visual processing. Finally, we provide qualitative evidence that MERV successfully captures domain knowledge from each of its encoders. Our results offer promising directions in utilizing multiple vision encoders for comprehensive video understanding.