O1複製之旅--第2部分：透過簡單精煉超越O1預覽，取得重大進展還是痛苦教訓？

O1 Replication Journey -- Part 2: Surpassing O1-preview through Simple Distillation, Big Progress or Bitter Lesson?

November 25, 2024

作者: Zhen Huang, Haoyang Zou, Xuefeng Li, Yixiu Liu, Yuxiang Zheng, Ethan Chern, Shijie Xia, Yiwei Qin, Weizhe Yuan, Pengfei Liu

cs.AI

摘要

本文對目前複製 OpenAI 的 O1 模型能力的方法進行了批判性檢驗，特別關注廣泛但常常未公開使用知識蒸餾技術。雖然我們先前的工作探索了達到 O1 複製的基本技術途徑，這項研究揭示了如何從 O1 的 API 簡單蒸餾，結合監督微調，可以在複雜的數學推理任務上實現卓越表現。通過廣泛的實驗，我們展示了在僅對數千個樣本進行了 O1 蒸餾的基礎模型微調後，長期被認為無法超越的 O1 預覽，在美國邀請數學考試（AIME）上表現出色，並具有最小的技術複雜性。此外，我們的研究不僅限於數學推理，還探索了 O1 蒸餾模型在各種任務上的泛化能力：幻覺、安全性和開放領域問答。值得注意的是，儘管僅在數學問題解決數據上進行訓練，我們的模型展現出對開放式問答任務的強大泛化能力，並在微調後明顯減少對諂媚的敏感性。我們故意將這一發現公之於眾，以促進 AI 研究的透明度，挑戰領域中對技術主張的隱晦趨勢。我們的工作包括：(1) 對蒸餾過程及其有效性的詳細技術闡述，(2) 一個全面的基準框架，用於評估和分類基於其技術透明度和可重現性的 O1 複製嘗試，(3) 對過度依賴蒸餾方法的限制和潛在風險進行批判性討論，我們的分析最終得出一個重要的苦澀教訓：儘管追求更有能力的 AI 系統很重要，但培養植根於第一原則思維的研究人員的發展至關重要。

English

This paper presents a critical examination of current approaches to replicating OpenAI's O1 model capabilities, with particular focus on the widespread but often undisclosed use of knowledge distillation techniques. While our previous work explored the fundamental technical path to O1 replication, this study reveals how simple distillation from O1's API, combined with supervised fine-tuning, can achieve superior performance on complex mathematical reasoning tasks. Through extensive experiments, we show that a base model fine-tuned on simply tens of thousands of samples O1-distilled long-thought chains outperforms O1-preview on the American Invitational Mathematics Examination (AIME) with minimal technical complexity. Moreover, our investigation extends beyond mathematical reasoning to explore the generalization capabilities of O1-distilled models across diverse tasks: hallucination, safety and open-domain QA. Notably, despite training only on mathematical problem-solving data, our models demonstrated strong generalization to open-ended QA tasks and became significantly less susceptible to sycophancy after fine-tuning. We deliberately make this finding public to promote transparency in AI research and to challenge the current trend of obscured technical claims in the field. Our work includes: (1) A detailed technical exposition of the distillation process and its effectiveness, (2) A comprehensive benchmark framework for evaluating and categorizing O1 replication attempts based on their technical transparency and reproducibility, (3) A critical discussion of the limitations and potential risks of over-relying on distillation approaches, our analysis culminates in a crucial bitter lesson: while the pursuit of more capable AI systems is important, the development of researchers grounded in first-principles thinking is paramount.