DyMU：動態合併與虛擬分離的高效視覺語言模型

DyMU: Dynamic Merging and Virtual Unmerging for Efficient VLMs

April 23, 2025

作者: Zhenhailong Wang, Senthil Purushwalkam, Caiming Xiong, Silvio Savarese, Heng Ji, Ran Xu

cs.AI

摘要

我們提出了DyMU，這是一個高效且無需訓練的框架，能夠在保持高任務性能的同時，動態降低視覺-語言模型（VLMs）的計算負擔。我們的方法包含兩個關鍵組件。首先，動態令牌合併（DToMe）通過根據圖像複雜度合併相似的令牌來減少視覺令牌嵌入的數量，從而解決視覺變壓器中固定長度輸出的固有低效性。其次，虛擬令牌解合併（VTU）通過高效重建完整序列的注意力動態，模擬大型語言模型（LLMs）的預期令牌序列，從而無需額外微調即可保持下游性能。與以往方法不同，我們的方法根據圖像內容動態調整令牌壓縮，並且完全無需訓練，使其易於應用於大多數最先進的VLM架構。在圖像和視頻理解任務上的廣泛實驗表明，DyMU能夠將平均視覺令牌數量減少32%-85%，同時在多種VLM架構（包括最近流行的基於AnyRes的視覺編碼器）上實現與完整長度模型相當的性能。此外，通過定性分析，我們展示了DToMe能夠根據圖像複雜度有效調整令牌減少，並且與現有系統不同，為用戶提供了更多對計算成本的控制。項目頁面：https://mikewangwzhl.github.io/dymu/。

English

We present DyMU, an efficient, training-free framework that dynamically reduces the computational burden of vision-language models (VLMs) while maintaining high task performance. Our approach comprises two key components. First, Dynamic Token Merging (DToMe) reduces the number of visual token embeddings by merging similar tokens based on image complexity, addressing the inherent inefficiency of fixed-length outputs in vision transformers. Second, Virtual Token Unmerging (VTU) simulates the expected token sequence for large language models (LLMs) by efficiently reconstructing the attention dynamics of a full sequence, thus preserving the downstream performance without additional fine-tuning. Unlike previous approaches, our method dynamically adapts token compression to the content of the image and operates completely training-free, making it readily applicable to most state-of-the-art VLM architectures. Extensive experiments on image and video understanding tasks demonstrate that DyMU can reduce the average visual token count by 32%-85% while achieving comparable performance to full-length models across diverse VLM architectures, including the recently popularized AnyRes-based visual encoders. Furthermore, through qualitative analyses, we demonstrate that DToMe effectively adapts token reduction based on image complexity and, unlike existing systems, provides users more control over computational costs. Project page: https://mikewangwzhl.github.io/dymu/.