白川對齊技術報告

Baichuan Alignment Technical Report

October 19, 2024

作者: Mingan Lin, Fan Yang, Yanjun Shen, Haoze Sun, Tianpeng Li, Tao Zhang, Chenzheng Zhu, Tao Zhang, Miao Zheng, Xu Li, Yijie Zhou, Mingyang Chen, Yanzhao Qin, Youquan Li, Hao Liang, Fei Li, Yadong Li, Mang Wang, Guosheng Dong, Kun Fang, Jianhua Xu, Bin Cui, Wentao Zhang, Zenan Zhou, Weipeng Chen

cs.AI

摘要

我們介紹了白川對齊（Baichuan Alignment），對白川系列模型中使用的對齊技術進行了詳細分析。這代表了行業首次對對齊方法論的全面闡述，為推進人工智慧研究提供了寶貴見解。我們研究了增強模型性能的關鍵組件，在對齊過程中包括優化方法、數據策略、能力增強和評估過程。該過程涵蓋三個關鍵階段：提示擴充系統（PAS）、監督微調（SFT）和偏好對齊。所遇到的問題、應用的解決方案和所做的改進都有詳細記錄。 通過與眾所周知的基準測試的比較，我們突出了白川對齊所啟用的技術進步。白川指導（Baichuan-Instruct）是一個內部模型，而Qwen2-Nova-72B和Llama3-PBM-Nova-70B是Qwen2-72B和Llama-3-70B基礎模型的指導版本，通過白川對齊進行了優化。白川指導展示了核心能力的顯著改進，用戶體驗提升範圍從17%到28%，在專業基準測試中表現優異。在開源基準評估中，無論是Qwen2-Nova-72B還是Llama3-PBM-Nova-70B，它們都在幾乎所有數據集上持續優於各自的官方指導版本。本報告旨在澄清對齊過程背後的關鍵技術，促進社區對此的更深入理解。 Llama3-PBM-Nova-70B模型可在以下網址找到：https://huggingface.co/PKU-Baichuan-MLSystemLab/Llama3-PBM-Nova-70B。

English

We introduce Baichuan Alignment, a detailed analysis of the alignment techniques employed in the Baichuan series of models. This represents the industry's first comprehensive account of alignment methodologies, offering valuable insights for advancing AI research. We investigate the critical components that enhance model performance during the alignment process, including optimization methods, data strategies, capability enhancements, and evaluation processes. The process spans three key stages: Prompt Augmentation System (PAS), Supervised Fine-Tuning (SFT), and Preference Alignment. The problems encountered, the solutions applied, and the improvements made are thoroughly recorded. Through comparisons across well-established benchmarks, we highlight the technological advancements enabled by Baichuan Alignment. Baichuan-Instruct is an internal model, while Qwen2-Nova-72B and Llama3-PBM-Nova-70B are instruct versions of the Qwen2-72B and Llama-3-70B base models, optimized through Baichuan Alignment. Baichuan-Instruct demonstrates significant improvements in core capabilities, with user experience gains ranging from 17% to 28%, and performs exceptionally well on specialized benchmarks. In open-source benchmark evaluations, both Qwen2-Nova-72B and Llama3-PBM-Nova-70B consistently outperform their respective official instruct versions across nearly all datasets. This report aims to clarify the key technologies behind the alignment process, fostering a deeper understanding within the community. Llama3-PBM-Nova-70B model is available at https://huggingface.co/PKU-Baichuan-MLSystemLab/Llama3-PBM-Nova-70B.