面向多模态智能的下一个标记预测：一项全面调查

Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Survey

December 16, 2024

作者: Liang Chen, Zekun Wang, Shuhuai Ren, Lei Li, Haozhe Zhao, Yunshui Li, Zefan Cai, Hongcheng Guo, Lei Zhang, Yizhe Xiong, Yichi Zhang, Ruoyu Wu, Qingxiu Dong, Ge Zhang, Jian Yang, Lingwei Meng, Shujie Hu, Yulong Chen, Junyang Lin, Shuai Bai, Andreas Vlachos, Xu Tan, Minjia Zhang, Wen Xiao, Aaron Yee, Tianyu Liu, Baobao Chang

cs.AI

摘要

在自然语言处理中的语言建模基础上，下一个标记预测（Next Token Prediction，NTP）已经发展成为机器学习任务中的一种多功能训练目标，跨越各种模态取得了相当大的成功。随着大型语言模型（Large Language Models，LLMs）不断发展，统一了文本模态内的理解和生成任务，最近的研究表明，来自不同模态的任务也可以有效地包含在NTP框架内，将多模态信息转换为标记并根据上下文预测下一个标记。本调查通过NTP的视角引入了一个统一理解和生成多模态学习的全面分类法。提出的分类法涵盖了五个关键方面：多模态标记化、MMNTP模型架构、统一任务表示、数据集与评估以及开放挑战。这一新分类法旨在帮助研究人员探索多模态智能。一个相关的 GitHub 仓库，收集了最新的论文和存储库，网址为https://github.com/LMM101/Awesome-Multimodal-Next-Token-Prediction。

English

Building on the foundations of language modeling in natural language processing, Next Token Prediction (NTP) has evolved into a versatile training objective for machine learning tasks across various modalities, achieving considerable success. As Large Language Models (LLMs) have advanced to unify understanding and generation tasks within the textual modality, recent research has shown that tasks from different modalities can also be effectively encapsulated within the NTP framework, transforming the multimodal information into tokens and predict the next one given the context. This survey introduces a comprehensive taxonomy that unifies both understanding and generation within multimodal learning through the lens of NTP. The proposed taxonomy covers five key aspects: Multimodal tokenization, MMNTP model architectures, unified task representation, datasets \& evaluation, and open challenges. This new taxonomy aims to aid researchers in their exploration of multimodal intelligence. An associated GitHub repository collecting the latest papers and repos is available at https://github.com/LMM101/Awesome-Multimodal-Next-Token-Prediction