透過模型、資料和測試時間的擴展，拓展開源多模型模型的性能邊界。

Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling

December 6, 2024

作者: Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuehui Wang, Qingyun Li, Yimin Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang, Tan Jiang, Bo Wang, Conghui He, Botian Shi, Xingcheng Zhang, Han Lv, Yi Wang, Wenqi Shao, Pei Chu, Zhongying Tu, Tong He, Zhiyong Wu, Huipeng Deng, Jiaye Ge, Kai Chen, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, Wenhai Wang

cs.AI

摘要

我們介紹了 InternVL 2.5，這是一個先進的多模式大型語言模型（MLLM）系列，建立在 InternVL 2.0 的基礎上，保留其核心模型架構，同時引入了在訓練和測試策略以及數據質量方面的重大增強。在這項工作中，我們深入探討模型擴展和性能之間的關係，系統地探索視覺編碼器、語言模型、數據集大小和測試時間配置的性能趨勢。通過對廣泛範圍的基準測試進行全面評估，包括多學科推理、文件理解、多圖像/視頻理解、現實世界理解、多模式幻覺檢測、視覺基礎、多語言能力和純語言處理，InternVL 2.5 展現出競爭力強勁的性能，與領先的商業模型如 GPT-4o 和 Claude-3.5-Sonnet 不相上下。值得注意的是，我們的模型是第一個開源 MLLMs，在 MMMU 基準測試中超過 70%，通過「思維鏈」（CoT）推理實現了 3.7 個百分點的改進，展示了在測試時間擴展方面的強大潛力。我們希望這個模型通過為開源社區設定開發和應用多模式人工智慧系統的新標準而做出貢獻。HuggingFace 演示請參見 https://huggingface.co/spaces/OpenGVLab/InternVL

English

We introduce InternVL 2.5, an advanced multimodal large language model (MLLM) series that builds upon InternVL 2.0, maintaining its core model architecture while introducing significant enhancements in training and testing strategies as well as data quality. In this work, we delve into the relationship between model scaling and performance, systematically exploring the performance trends in vision encoders, language models, dataset sizes, and test-time configurations. Through extensive evaluations on a wide range of benchmarks, including multi-discipline reasoning, document understanding, multi-image / video understanding, real-world comprehension, multimodal hallucination detection, visual grounding, multilingual capabilities, and pure language processing, InternVL 2.5 exhibits competitive performance, rivaling leading commercial models such as GPT-4o and Claude-3.5-Sonnet. Notably, our model is the first open-source MLLMs to surpass 70% on the MMMU benchmark, achieving a 3.7-point improvement through Chain-of-Thought (CoT) reasoning and showcasing strong potential for test-time scaling. We hope this model contributes to the open-source community by setting new standards for developing and applying multimodal AI systems. HuggingFace demo see https://huggingface.co/spaces/OpenGVLab/InternVL