O1 复制之旅 -- 第2部分：通过简单蒸馏超越 O1 预览，取得重大进展还是痛苦教训？

O1 Replication Journey -- Part 2: Surpassing O1-preview through Simple Distillation, Big Progress or Bitter Lesson?

November 25, 2024

作者: Zhen Huang, Haoyang Zou, Xuefeng Li, Yixiu Liu, Yuxiang Zheng, Ethan Chern, Shijie Xia, Yiwei Qin, Weizhe Yuan, Pengfei Liu

cs.AI

摘要

本文对当前复制OpenAI的O1模型能力的方法进行了批判性审查，特别关注广泛但常常未披露的知识蒸馏技术的使用。虽然我们先前的工作探讨了实现O1复制的基本技术路径，但这项研究揭示了如何通过简单地从O1的API中蒸馏，结合监督微调，在复杂数学推理任务上实现卓越性能。通过大量实验，我们展示了一个基础模型在仅微调了数万个样本的O1蒸馏长期思维链后，在美国邀请数学考试（AIME）上的表现优于O1预览，且技术复杂度极低。此外，我们的研究不仅限于数学推理，还探讨了O1蒸馏模型在各种任务上的泛化能力：幻觉、安全性和开放领域问答。值得注意的是，尽管仅在数学问题解决数据上进行训练，我们的模型展现出对开放式问答任务的强大泛化能力，并在微调后显著减少了对谄媚的敏感性。我们有意公开这一发现，以促进AI研究的透明度，并挑战该领域中对技术声明的模糊趋势。我们的工作包括：（1）对蒸馏过程及其有效性的详细技术阐述，（2）一个全面的基准框架，用于评估和分类基于其技术透明度和可重现性的O1复制尝试，（3）对依赖蒸馏方法的局限性和潜在风险的批判性讨论，我们的分析得出一个重要的苦涩教训：虽然追求更有能力的AI系统很重要，但培养以第一原则思维为基础的研究人员的发展至关重要。

English

This paper presents a critical examination of current approaches to replicating OpenAI's O1 model capabilities, with particular focus on the widespread but often undisclosed use of knowledge distillation techniques. While our previous work explored the fundamental technical path to O1 replication, this study reveals how simple distillation from O1's API, combined with supervised fine-tuning, can achieve superior performance on complex mathematical reasoning tasks. Through extensive experiments, we show that a base model fine-tuned on simply tens of thousands of samples O1-distilled long-thought chains outperforms O1-preview on the American Invitational Mathematics Examination (AIME) with minimal technical complexity. Moreover, our investigation extends beyond mathematical reasoning to explore the generalization capabilities of O1-distilled models across diverse tasks: hallucination, safety and open-domain QA. Notably, despite training only on mathematical problem-solving data, our models demonstrated strong generalization to open-ended QA tasks and became significantly less susceptible to sycophancy after fine-tuning. We deliberately make this finding public to promote transparency in AI research and to challenge the current trend of obscured technical claims in the field. Our work includes: (1) A detailed technical exposition of the distillation process and its effectiveness, (2) A comprehensive benchmark framework for evaluating and categorizing O1 replication attempts based on their technical transparency and reproducibility, (3) A critical discussion of the limitations and potential risks of over-relying on distillation approaches, our analysis culminates in a crucial bitter lesson: while the pursuit of more capable AI systems is important, the development of researchers grounded in first-principles thinking is paramount.