FuseChat-3.0:偏好优化与异构模型融合的完美结合
FuseChat-3.0: Preference Optimization Meets Heterogeneous Model Fusion
March 6, 2025
作者: Ziyi Yang, Fanqi Wan, Longguang Zhong, Canbin Huang, Guosheng Liang, Xiaojun Quan
cs.AI
摘要
我们推出了FuseChat-3.0,这是一套通过将异构源大型语言模型(LLMs)的优势整合到更为紧凑的目标LLMs中而开发的大型语言模型系列。我们的源模型包括强大的Gemma-2-27B-it、Mistral-Large-Instruct-2407、Qwen-2.5-72B-Instruct以及Llama-3.1-70B-Instruct。针对目标模型,我们聚焦于三种广泛使用的小型变体——Llama-3.1-8B-Instruct、Gemma-2-9B-it和Qwen-2.5-7B-Instruct,以及两种超紧凑选项——Llama-3.2-3B-Instruct和Llama-3.2-1B-Instruct。为了充分利用这些源模型的多样化能力,我们开发了一套专门针对不同任务和领域的数据构建协议。FuseChat-3.0的训练流程包含两个关键阶段:(1)监督微调(SFT)以对齐目标与源模型的分布,(2)直接偏好优化(DPO)以应用来自多个源LLMs的偏好来微调目标模型。最终得到的FuseChat-3.0模型在指令遵循、常识、数学和编程等任务上展现出显著的性能提升。如图1所示,以Llama-3.1-8B-Instruct作为目标模型,我们的融合方法在14个基准测试中平均提升了6.8分。此外,在指令遵循基准AlpacaEval-2和Arena-Hard上分别实现了37.1分和30.1分的显著增益。我们的代码、模型及数据集可在https://github.com/SLIT-AI/FuseChat-3.0获取。
English
We introduce FuseChat-3.0, a suite of large language models (LLMs) developed
by integrating the strengths of heterogeneous source LLMs into more compact
target LLMs. Our source models include the powerful Gemma-2-27B-it,
Mistral-Large-Instruct-2407, Qwen-2.5-72B-Instruct, and Llama-3.1-70B-Instruct.
For target models, we focus on three widely-used smaller
variants-Llama-3.1-8B-Instruct, Gemma-2-9B-it, and Qwen-2.5-7B-Instruct-along
with two ultra-compact options, Llama-3.2-3B-Instruct and
Llama-3.2-1B-Instruct. To leverage the diverse capabilities of these source
models, we develop a specialized data construction protocol tailored to various
tasks and domains. The FuseChat-3.0 training pipeline consists of two key
stages: (1) supervised fine-tuning (SFT) to align the target and source model
distributions, and (2) Direct Preference Optimization (DPO) to apply
preferences from multiple source LLMs to fine-tune the target model. The
resulting FuseChat-3.0 models exhibit significant performance gains across
tasks such as instruction following, general knowledge, mathematics, and
coding. As illustrated in Figure 1, using Llama-3.1-8B-Instruct as the target
model, our fusion approach achieves an average improvement of 6.8 points across
14 benchmarks. Moreover, it demonstrates remarkable gains of 37.1 points and
30.1 points on the instruction-following benchmarks AlpacaEval-2 and
Arena-Hard, respectively. Our code, models, and datasets are available at
https://github.com/SLIT-AI/FuseChat-3.0.Summary
AI-Generated Summary