小型语言模型中的推理蒸馏与优化用于文档重排序

Distillation and Refinement of Reasoning in Small Language Models for Document Re-ranking

April 4, 2025

作者: Chris Samarinas, Hamed Zamani

cs.AI

摘要

我们提出了一种新颖的方法，用于训练小型语言模型进行推理密集型的文档排序，该方法将知识蒸馏与强化学习优化相结合。现有方法通常依赖于昂贵的人工标注或大型黑箱语言模型，而我们的方法则利用网络数据和教师大语言模型自动生成带有相关性解释的高质量训练样本。通过将文档排序问题转化为强化学习任务，并激励显式推理能力，我们训练了一个紧凑的30亿参数语言模型，该模型在BRIGHT基准测试中达到了最先进的性能。我们的模型在排行榜上位列第三，同时使用的参数数量远少于其他方法，超越了参数规模超过其20倍的模型。通过大量实验，我们证明在推理过程中生成解释而非直接预测相关性分数，能够使小型语言模型实现更有效的推理。我们方法的自监督特性为现代信息检索系统提供了一种可扩展且可解释的解决方案。

English

We present a novel approach for training small language models for reasoning-intensive document ranking that combines knowledge distillation with reinforcement learning optimization. While existing methods often rely on expensive human annotations or large black-box language models, our methodology leverages web data and a teacher LLM to automatically generate high-quality training examples with relevance explanations. By framing document ranking as a reinforcement learning problem and incentivizing explicit reasoning capabilities, we train a compact 3B parameter language model that achieves state-of-the-art performance on the BRIGHT benchmark. Our model ranks third on the leaderboard while using substantially fewer parameters than other approaches, outperforming models that are over 20 times larger. Through extensive experiments, we demonstrate that generating explanations during inference, rather than directly predicting relevance scores, enables more effective reasoning with smaller language models. The self-supervised nature of our method offers a scalable and interpretable solution for modern information retrieval systems.