通过模型、数据和测试时间的扩展，拓展开源多模态模型的性能边界。

Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling

December 6, 2024

作者: Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuehui Wang, Qingyun Li, Yimin Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang, Tan Jiang, Bo Wang, Conghui He, Botian Shi, Xingcheng Zhang, Han Lv, Yi Wang, Wenqi Shao, Pei Chu, Zhongying Tu, Tong He, Zhiyong Wu, Huipeng Deng, Jiaye Ge, Kai Chen, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, Wenhai Wang

cs.AI

摘要

我们介绍了InternVL 2.5，这是一种先进的多模态大型语言模型（MLLM）系列，它在InternVL 2.0的基础模型架构上进行了改进，同时在训练和测试策略以及数据质量方面引入了显著的增强。在这项工作中，我们深入探讨了模型扩展和性能之间的关系，系统地探索了视觉编码器、语言模型、数据集大小和测试时间配置在性能趋势中的表现。通过在广泛的基准测试中进行全面评估，包括多学科推理、文档理解、多图像/视频理解、现实世界理解、多模态幻觉检测、视觉定位、多语言能力和纯语言处理等，InternVL 2.5展现出竞争力强劲的性能，与领先的商业模型如GPT-4o和Claude-3.5-Sonnet不相上下。值得注意的是，我们的模型是首个在MMMU基准测试中超过70%的开源MLLM，通过“思维链”（CoT）推理实现了3.7个百分点的改进，并展示了在测试时间扩展方面的强大潜力。我们希望这个模型通过为开源社区树立开发和应用多模态人工智能系统的新标准而做出贡献。HuggingFace演示请参见https://huggingface.co/spaces/OpenGVLab/InternVL

English

We introduce InternVL 2.5, an advanced multimodal large language model (MLLM) series that builds upon InternVL 2.0, maintaining its core model architecture while introducing significant enhancements in training and testing strategies as well as data quality. In this work, we delve into the relationship between model scaling and performance, systematically exploring the performance trends in vision encoders, language models, dataset sizes, and test-time configurations. Through extensive evaluations on a wide range of benchmarks, including multi-discipline reasoning, document understanding, multi-image / video understanding, real-world comprehension, multimodal hallucination detection, visual grounding, multilingual capabilities, and pure language processing, InternVL 2.5 exhibits competitive performance, rivaling leading commercial models such as GPT-4o and Claude-3.5-Sonnet. Notably, our model is the first open-source MLLMs to surpass 70% on the MMMU benchmark, achieving a 3.7-point improvement through Chain-of-Thought (CoT) reasoning and showcasing strong potential for test-time scaling. We hope this model contributes to the open-source community by setting new standards for developing and applying multimodal AI systems. HuggingFace demo see https://huggingface.co/spaces/OpenGVLab/InternVL