FuseChat-3.0：偏好优化与异构模型融合的完美结合

FuseChat-3.0: Preference Optimization Meets Heterogeneous Model Fusion

March 6, 2025

作者: Ziyi Yang, Fanqi Wan, Longguang Zhong, Canbin Huang, Guosheng Liang, Xiaojun Quan

cs.AI

摘要

我们推出了FuseChat-3.0，这是一套通过将异构源大型语言模型（LLMs）的优势整合到更为紧凑的目标LLMs中而开发的大型语言模型系列。我们的源模型包括强大的Gemma-2-27B-it、Mistral-Large-Instruct-2407、Qwen-2.5-72B-Instruct以及Llama-3.1-70B-Instruct。针对目标模型，我们聚焦于三种广泛使用的小型变体——Llama-3.1-8B-Instruct、Gemma-2-9B-it和Qwen-2.5-7B-Instruct，以及两种超紧凑选项——Llama-3.2-3B-Instruct和Llama-3.2-1B-Instruct。为了充分利用这些源模型的多样化能力，我们开发了一套专门针对不同任务和领域的数据构建协议。FuseChat-3.0的训练流程包含两个关键阶段：（1）监督微调（SFT）以对齐目标与源模型的分布，（2）直接偏好优化（DPO）以应用来自多个源LLMs的偏好来微调目标模型。最终得到的FuseChat-3.0模型在指令遵循、常识、数学和编程等任务上展现出显著的性能提升。如图1所示，以Llama-3.1-8B-Instruct作为目标模型，我们的融合方法在14个基准测试中平均提升了6.8分。此外，在指令遵循基准AlpacaEval-2和Arena-Hard上分别实现了37.1分和30.1分的显著增益。我们的代码、模型及数据集可在https://github.com/SLIT-AI/FuseChat-3.0获取。

English

We introduce FuseChat-3.0, a suite of large language models (LLMs) developed by integrating the strengths of heterogeneous source LLMs into more compact target LLMs. Our source models include the powerful Gemma-2-27B-it, Mistral-Large-Instruct-2407, Qwen-2.5-72B-Instruct, and Llama-3.1-70B-Instruct. For target models, we focus on three widely-used smaller variants-Llama-3.1-8B-Instruct, Gemma-2-9B-it, and Qwen-2.5-7B-Instruct-along with two ultra-compact options, Llama-3.2-3B-Instruct and Llama-3.2-1B-Instruct. To leverage the diverse capabilities of these source models, we develop a specialized data construction protocol tailored to various tasks and domains. The FuseChat-3.0 training pipeline consists of two key stages: (1) supervised fine-tuning (SFT) to align the target and source model distributions, and (2) Direct Preference Optimization (DPO) to apply preferences from multiple source LLMs to fine-tune the target model. The resulting FuseChat-3.0 models exhibit significant performance gains across tasks such as instruction following, general knowledge, mathematics, and coding. As illustrated in Figure 1, using Llama-3.1-8B-Instruct as the target model, our fusion approach achieves an average improvement of 6.8 points across 14 benchmarks. Moreover, it demonstrates remarkable gains of 37.1 points and 30.1 points on the instruction-following benchmarks AlpacaEval-2 and Arena-Hard, respectively. Our code, models, and datasets are available at https://github.com/SLIT-AI/FuseChat-3.0.