ILLUME：照亮您的LLM，看见、绘制和自我增强

摘要

本文介绍了ILLUME，这是一个统一的多模态大型语言模型（MLLM），通过统一的下一个标记预测公式，在单个大型语言模型中无缝集成了多模态理解和生成能力。为了解决通常需要大规模数据集大小进行图像文本对齐的问题，我们提出通过设计一个融合语义信息的视觉分词器和渐进式多阶段训练程序来增强数据效率。这种方法将预训练的数据集大小减少到仅为1500万，是通常所需大小的四分之一，同时实现了与现有统一MLLM（如Janus）相媲美甚至更优越的性能。此外，为了促进理解和生成能力之间的协同增强，这在先前的研究中尚未得到充分探讨，我们引入了一种新颖的自我增强多模态对齐方案。该方案监督MLLM自我评估文本描述和自动生成图像之间的一致性，促使模型更准确地解释图像，并避免由于图像生成中的不对齐而导致的不现实和不正确的预测。基于大量实验，我们提出的ILLUME在各种多模态理解、生成和编辑基准测试中脱颖而出，与最先进的统一MLLM和专门模型竞争。

English

In this paper, we introduce ILLUME, a unified multimodal large language model (MLLM) that seamlessly integrates multimodal understanding and generation capabilities within a single large language model through a unified next-token prediction formulation. To address the large dataset size typically required for image-text alignment, we propose to enhance data efficiency through the design of a vision tokenizer that incorporates semantic information and a progressive multi-stage training procedure. This approach reduces the dataset size to just 15M for pretraining -- over four times fewer than what is typically needed -- while achieving competitive or even superior performance with existing unified MLLMs, such as Janus. Additionally, to promote synergistic enhancement between understanding and generation capabilities, which is under-explored in previous works, we introduce a novel self-enhancing multimodal alignment scheme. This scheme supervises the MLLM to self-assess the consistency between text descriptions and self-generated images, facilitating the model to interpret images more accurately and avoid unrealistic and incorrect predictions caused by misalignment in image generation. Based on extensive experiments, our proposed ILLUME stands out and competes with state-of-the-art unified MLLMs and specialized models across various benchmarks for multimodal understanding, generation, and editing.

ILLUME：照亮您的LLM，看见、绘制和自我增强

ILLUME: Illuminating Your LLMs to See, Draw, and Self-Enhance

摘要

Summary

Support