ILLUME：照亮您的LLMs，看見、繪製和自我增強

摘要

本文介紹了 ILLUME，一個統一的多模式大型語言模型（MLLM），通過統一的下一倗預測公式，在單一大型語言模型中無縫集成多模式理解和生成能力。為了應對通常需要大型數據集大小的圖像-文本對齊問題，我們提出通過設計一個包含語義信息的視覺分詞器和一個漸進式多階段訓練程序來增強數據效率。這種方法將預訓練的數據集大小減少到僅 15M，比通常需要的數據集大小少四倍以上，同時實現了與現有統一 MLLMs（如 Janus）相當甚至更優越的性能。此外，為了促進理解和生成能力之間的協同增強，這在先前的研究中尚未得到充分探索，我們引入了一種新穎的自我增強多模式對齊方案。該方案監督 MLLM 自我評估文本描述和自生成圖像之間的一致性，幫助模型更準確地解釋圖像，避免由於圖像生成中的不對齊而導致的不現實和不正確的預測。通過大量實驗，我們提出的 ILLUME 在各種多模式理解、生成和編輯的基準測試中脫穎而出，與最先進的統一 MLLMs 和專用模型競爭。

English

In this paper, we introduce ILLUME, a unified multimodal large language model (MLLM) that seamlessly integrates multimodal understanding and generation capabilities within a single large language model through a unified next-token prediction formulation. To address the large dataset size typically required for image-text alignment, we propose to enhance data efficiency through the design of a vision tokenizer that incorporates semantic information and a progressive multi-stage training procedure. This approach reduces the dataset size to just 15M for pretraining -- over four times fewer than what is typically needed -- while achieving competitive or even superior performance with existing unified MLLMs, such as Janus. Additionally, to promote synergistic enhancement between understanding and generation capabilities, which is under-explored in previous works, we introduce a novel self-enhancing multimodal alignment scheme. This scheme supervises the MLLM to self-assess the consistency between text descriptions and self-generated images, facilitating the model to interpret images more accurately and avoid unrealistic and incorrect predictions caused by misalignment in image generation. Based on extensive experiments, our proposed ILLUME stands out and competes with state-of-the-art unified MLLMs and specialized models across various benchmarks for multimodal understanding, generation, and editing.

ILLUME：照亮您的LLMs，看見、繪製和自我增強

ILLUME: Illuminating Your LLMs to See, Draw, and Self-Enhance

摘要

Summary

Support