o1-Coder：一個針對編碼的o1複製。

o1-Coder: an o1 Replication for Coding

November 29, 2024

作者: Yuxiang Zhang, Shangxi Wu, Yuqi Yang, Jiangming Shu, Jinlin Xiao, Chao Kong, Jitao Sang

cs.AI

摘要

這份技術報告介紹了 O1-CODER，這是一個嘗試複製 OpenAI 的 o1 模型，專注於編碼任務。它整合了強化學習（RL）和蒙特卡羅樹搜索（MCTS）以增強模型的系統二思維能力。該框架包括訓練一個測試用例生成器（TCG）進行標準代碼測試，使用 MCTS 生成帶有推理過程的代碼數據，並通過迭代微調策略模型，最初生成偽代碼，然後生成完整代碼。報告還討論了在實際應用中部署類似 o1 模型的機會和挑戰，建議過渡到系統二範式，並強調環境狀態更新的必要性。更新的模型進展和實驗結果將在後續版本中報告。所有源代碼、策劃數據集以及衍生模型將在 https://github.com/ADaM-BJTU/O1-CODER 上公開。

English

The technical report introduces O1-CODER, an attempt to replicate OpenAI's o1 model with a focus on coding tasks. It integrates reinforcement learning (RL) and Monte Carlo Tree Search (MCTS) to enhance the model's System-2 thinking capabilities. The framework includes training a Test Case Generator (TCG) for standardized code testing, using MCTS to generate code data with reasoning processes, and iteratively fine-tuning the policy model to initially produce pseudocode, followed by the generation of the full code. The report also addresses the opportunities and challenges in deploying o1-like models in real-world applications, suggesting transitioning to the System-2 paradigm and highlighting the imperative for environment state updates. Updated model progress and experimental results will be reported in subsequent versions. All source code, curated datasets, as well as the derived models will be disclosed at https://github.com/ADaM-BJTU/O1-CODER .