o1-Coder：用于编码的o1复制

o1-Coder: an o1 Replication for Coding

November 29, 2024

作者: Yuxiang Zhang, Shangxi Wu, Yuqi Yang, Jiangming Shu, Jinlin Xiao, Chao Kong, Jitao Sang

cs.AI

摘要

这份技术报告介绍了 O1-CODER，这是一个尝试复制 OpenAI 的 o1 模型，专注于编码任务的项目。它整合了强化学习（RL）和蒙特卡洛树搜索（MCTS）来增强模型的第二系统思维能力。该框架包括训练一个测试用例生成器（TCG）用于标准化代码测试，利用 MCTS 生成带有推理过程的代码数据，并通过迭代微调策略模型，最初生成伪代码，然后生成完整代码。报告还讨论了在实际应用中部署类似 o1 模型的机会和挑战，建议过渡到第二系统范式，并强调环境状态更新的必要性。更新后的模型进展和实验结果将在随后的版本中报告。所有源代码、筛选的数据集以及衍生模型将在 https://github.com/ADaM-BJTU/O1-CODER 上公开。

English

The technical report introduces O1-CODER, an attempt to replicate OpenAI's o1 model with a focus on coding tasks. It integrates reinforcement learning (RL) and Monte Carlo Tree Search (MCTS) to enhance the model's System-2 thinking capabilities. The framework includes training a Test Case Generator (TCG) for standardized code testing, using MCTS to generate code data with reasoning processes, and iteratively fine-tuning the policy model to initially produce pseudocode, followed by the generation of the full code. The report also addresses the opportunities and challenges in deploying o1-like models in real-world applications, suggesting transitioning to the System-2 paradigm and highlighting the imperative for environment state updates. Updated model progress and experimental results will be reported in subsequent versions. All source code, curated datasets, as well as the derived models will be disclosed at https://github.com/ADaM-BJTU/O1-CODER .