START：具备工具使用能力的自学习推理器

START: Self-taught Reasoner with Tools

March 6, 2025

作者: Chengpeng Li, Mingfeng Xue, Zhenru Zhang, Jiaxi Yang, Beichen Zhang, Xiang Wang, Bowen Yu, Binyuan Hui, Junyang Lin, Dayiheng Liu

cs.AI

摘要

诸如OpenAI-o1和DeepSeek-R1等大型推理模型（LRMs）通过运用长链思维（CoT）在复杂推理任务中展现了卓越的能力。然而，这些模型由于仅依赖内部推理过程，常出现幻觉和效率低下的问题。本文介绍了一种新型工具集成长链思维推理大模型START（Self-Taught Reasoner with Tools），它通过利用外部工具显著增强了推理能力。通过代码执行，START能够进行复杂计算、自我检查、探索多种方法及自我调试，从而解决了LRMs的局限性。START的核心创新在于其自学习框架，该框架包含两项关键技术：1）提示推理（Hint-infer）：我们证明，在LRM的推理过程中插入人工设计的提示（例如，“等等，也许在这里使用Python是个好主意。”）能有效激发其利用外部工具的能力，而无需任何示范数据。提示推理还可作为一种简单有效的序列测试时间扩展方法；2）提示拒绝采样微调（Hint-RFT）：Hint-RFT结合了Hint-infer和RFT，通过对LRM通过Hint-infer生成的带有工具调用的推理轨迹进行评分、筛选和修改，随后对LRM进行微调。通过这一框架，我们微调了QwQ-32B模型，实现了START。在博士级科学问答（GPQA）、竞赛级数学基准测试（AMC23、AIME24、AIME25）以及竞赛级代码基准测试（LiveCodeBench）上，START分别达到了63.6%、95.0%、66.7%、47.1%和47.3%的准确率。它显著超越了基础QwQ-32B模型，并与最先进的开放权重模型R1-Distill-Qwen-32B及专有模型o1-Preview的性能相当。

English

Large reasoning models (LRMs) like OpenAI-o1 and DeepSeek-R1 have demonstrated remarkable capabilities in complex reasoning tasks through the utilization of long Chain-of-thought (CoT). However, these models often suffer from hallucinations and inefficiencies due to their reliance solely on internal reasoning processes. In this paper, we introduce START (Self-Taught Reasoner with Tools), a novel tool-integrated long CoT reasoning LLM that significantly enhances reasoning capabilities by leveraging external tools. Through code execution, START is capable of performing complex computations, self-checking, exploring diverse methods, and self-debugging, thereby addressing the limitations of LRMs. The core innovation of START lies in its self-learning framework, which comprises two key techniques: 1) Hint-infer: We demonstrate that inserting artificially designed hints (e.g., ``Wait, maybe using Python here is a good idea.'') during the inference process of a LRM effectively stimulates its ability to utilize external tools without the need for any demonstration data. Hint-infer can also serve as a simple and effective sequential test-time scaling method; 2) Hint Rejection Sampling Fine-Tuning (Hint-RFT): Hint-RFT combines Hint-infer and RFT by scoring, filtering, and modifying the reasoning trajectories with tool invocation generated by a LRM via Hint-infer, followed by fine-tuning the LRM. Through this framework, we have fine-tuned the QwQ-32B model to achieve START. On PhD-level science QA (GPQA), competition-level math benchmarks (AMC23, AIME24, AIME25), and the competition-level code benchmark (LiveCodeBench), START achieves accuracy rates of 63.6%, 95.0%, 66.7%, 47.1%, and 47.3%, respectively. It significantly outperforms the base QwQ-32B and achieves performance comparable to the state-of-the-art open-weight model R1-Distill-Qwen-32B and the proprietary model o1-Preview.