关于多模态大型语言模型领域特定后训练

摘要

近年来，通用多模态大型语言模型（MLLMs）的快速发展备受关注。然而，将通用MLLMs调整到特定领域，如科学领域和工业应用，仍未得到充分探索。本文系统地研究了通过后期训练进行MLLMs领域自适应的方法，重点关注数据合成、训练流程和任务评估。（1）数据合成：利用开源模型，我们开发了一个视觉指导合成器，有效地从领域特定的图像说明对生成多样化的视觉指导任务。我们的合成任务在增强MLLMs的领域特定性能方面超越了通过手动规则、GPT-4和GPT-4V生成的任务。（2）训练流程：尽管通常采用两阶段训练——首先是图像说明对，然后是视觉指导任务——来开发通用MLLMs，我们应用单阶段训练流程来增强领域特定后期训练的任务多样性。（3）任务评估：我们在生物医学和食品两个领域进行实验，通过对不同来源和规模的MLLMs（例如Qwen2-VL-2B，LLaVA-v1.6-8B，Llama-3.2-11B）进行后期训练，然后评估MLLM在各种领域特定任务上的性能。为支持MLLM领域自适应的进一步研究，我们将开源我们的实现。

English

Recent years have witnessed the rapid development of general multimodal large language models (MLLMs). However, adapting general MLLMs to specific domains, such as scientific fields and industrial applications, remains less explored. This paper systematically investigates domain adaptation of MLLMs through post-training, focusing on data synthesis, training pipelines, and task evaluation. (1) Data Synthesis: Using open-source models, we develop a visual instruction synthesizer that effectively generates diverse visual instruction tasks from domain-specific image-caption pairs. Our synthetic tasks surpass those generated by manual rules, GPT-4, and GPT-4V in enhancing the domain-specific performance of MLLMs. (2) Training Pipeline: While the two-stage training--initially on image-caption pairs followed by visual instruction tasks--is commonly adopted for developing general MLLMs, we apply a single-stage training pipeline to enhance task diversity for domain-specific post-training. (3) Task Evaluation: We conduct experiments in two domains, biomedicine and food, by post-training MLLMs of different sources and scales (e.g., Qwen2-VL-2B, LLaVA-v1.6-8B, Llama-3.2-11B), and then evaluating MLLM performance on various domain-specific tasks. To support further research in MLLM domain adaptation, we will open-source our implementations.

关于多模态大型语言模型领域特定后训练

On Domain-Specific Post-Training for Multimodal Large Language Models

摘要

Summary

Support

Support