走向多模態智能的下一個標記預測：一項全面調查

Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Survey

December 16, 2024

作者: Liang Chen, Zekun Wang, Shuhuai Ren, Lei Li, Haozhe Zhao, Yunshui Li, Zefan Cai, Hongcheng Guo, Lei Zhang, Yizhe Xiong, Yichi Zhang, Ruoyu Wu, Qingxiu Dong, Ge Zhang, Jian Yang, Lingwei Meng, Shujie Hu, Yulong Chen, Junyang Lin, Shuai Bai, Andreas Vlachos, Xu Tan, Minjia Zhang, Wen Xiao, Aaron Yee, Tianyu Liu, Baobao Chang

cs.AI

摘要

在自然語言處理的語言建模基礎上，下一個標記預測（NTP）已演變為機器學習任務的多功能訓練目標，跨越各種形式，取得了相當大的成功。隨著大型語言模型（LLMs）不斷發展，統一了文本形式中的理解和生成任務，最近的研究表明，來自不同形式的任務也可以有效地封裝在NTP框架中，將多模態信息轉換為標記，並根據上下文預測下一個標記。本調查通過NTP的角度引入了一個統一的分類法，將理解和生成統一在多模態學習中。所提出的分類法涵蓋了五個關鍵方面：多模態標記化、MMNTP模型架構、統一任務表示、數據集和評估，以及開放挑戰。這個新的分類法旨在幫助研究人員探索多模態智能。一個相關的 GitHub 存儲庫，收集最新的論文和存儲庫，可在 https://github.com/LMM101/Awesome-Multimodal-Next-Token-Prediction 找到。

English

Building on the foundations of language modeling in natural language processing, Next Token Prediction (NTP) has evolved into a versatile training objective for machine learning tasks across various modalities, achieving considerable success. As Large Language Models (LLMs) have advanced to unify understanding and generation tasks within the textual modality, recent research has shown that tasks from different modalities can also be effectively encapsulated within the NTP framework, transforming the multimodal information into tokens and predict the next one given the context. This survey introduces a comprehensive taxonomy that unifies both understanding and generation within multimodal learning through the lens of NTP. The proposed taxonomy covers five key aspects: Multimodal tokenization, MMNTP model architectures, unified task representation, datasets \& evaluation, and open challenges. This new taxonomy aims to aid researchers in their exploration of multimodal intelligence. An associated GitHub repository collecting the latest papers and repos is available at https://github.com/LMM101/Awesome-Multimodal-Next-Token-Prediction