Chimera：利用領域專家來改進通用模型

摘要

最近在大型多模型模型（LMMs）方面的進展凸顯了通過增加圖像-文本配對數據來進行規模化的重要性，在通用任務上取得了令人印象深刻的性能。儘管這些通用模型在廣泛應用中非常有效，但它們主要是在以自然圖像為主導的網絡規模數據集上訓練的，這導致了對需要大量領域先驗知識的特定領域任務的專業能力的犧牲。此外，由於通用模型和專家模型之間的表示差距和不平衡優化，直接整合針對特定領域量身定制的專家模型是具有挑戰性的。為了應對這些挑戰，我們引入了Chimera，這是一個可擴展且低成本的多模管道，旨在通過領域專家來增強現有LMMs的能力。具體來說，我們設計了一種漸進式訓練策略，將專家模型的特徵集成到通用LMM的輸入中。為了應對由良好對齊的通用視覺編碼器引起的不平衡優化問題，我們引入了一種新穎的通用-專家協作遮罩（GSCM）機制。這導致了一個多才多藝的模型，在圖表、表格、數學和文檔領域表現出色，在多模推理和視覺內容提取任務上取得了最先進的性能，這兩個任務對於評估現有LMMs來說都是具有挑戰性的。

English

Recent advancements in Large Multi-modal Models (LMMs) underscore the importance of scaling by increasing image-text paired data, achieving impressive performance on general tasks. Despite their effectiveness in broad applications, generalist models are primarily trained on web-scale datasets dominated by natural images, resulting in the sacrifice of specialized capabilities for domain-specific tasks that require extensive domain prior knowledge. Moreover, directly integrating expert models tailored for specific domains is challenging due to the representational gap and imbalanced optimization between the generalist model and experts. To address these challenges, we introduce Chimera, a scalable and low-cost multi-modal pipeline designed to boost the ability of existing LMMs with domain-specific experts. Specifically, we design a progressive training strategy to integrate features from expert models into the input of a generalist LMM. To address the imbalanced optimization caused by the well-aligned general visual encoder, we introduce a novel Generalist-Specialist Collaboration Masking (GSCM) mechanism. This results in a versatile model that excels across the chart, table, math, and document domains, achieving state-of-the-art performance on multi-modal reasoning and visual content extraction tasks, both of which are challenging tasks for assessing existing LMMs.

Chimera：利用領域專家來改進通用模型

Chimera: Improving Generalist Model with Domain-Specific Experts

摘要

Summary

Support