LongReward：利用人工智能改进长上下文大型语言模型的反馈

LongReward: Improving Long-context Large Language Models with AI Feedback

October 28, 2024

作者: Jiajie Zhang, Zhongni Hou, Xin Lv, Shulin Cao, Zhenyu Hou, Yilin Niu, Lei Hou, Yuxiao Dong, Ling Feng, Juanzi Li

cs.AI

摘要

尽管在开发长文本大语言模型（LLMs）方面取得了重大进展，但LLM合成数据的质量通常会影响监督微调（SFT）的长文本性能，并导致固有限制。原则上，通过适当的奖励信号进行强化学习（RL）可以进一步增强模型的能力。然而，在长文本场景中如何获得可靠的奖励仍未被探索。为此，我们提出了LongReward，这是一种新颖方法，利用现成的LLM为长文本模型的响应提供来自四个人类价值维度的奖励：有用性、逻辑性、忠实度和完整性，每个维度都经过精心设计的评估流程。通过结合LongReward和离线RL算法DPO，我们能够有效改善长文本SFT模型。我们的实验表明，LongReward不仅显著提高了模型的长文本性能，还增强了它们遵循简短指令的能力。我们还发现，长文本DPO与LongReward以及传统的短文本DPO可以一起使用，而不会损害任何一方的性能。

English

Though significant advancements have been achieved in developing long-context large language models (LLMs), the compromised quality of LLM-synthesized data for supervised fine-tuning (SFT) often affects the long-context performance of SFT models and leads to inherent limitations. In principle, reinforcement learning (RL) with appropriate reward signals can further enhance models' capacities. However, how to obtain reliable rewards in long-context scenarios remains unexplored. To this end, we propose LongReward, a novel method that utilizes an off-the-shelf LLM to provide rewards for long-context model responses from four human-valued dimensions: helpfulness, logicality, faithfulness, and completeness, each with a carefully designed assessment pipeline. By combining LongReward and offline RL algorithm DPO, we are able to effectively improve long-context SFT models. Our experiments indicate that LongReward not only significantly improves models' long-context performance but also enhances their ability to follow short instructions. We also find that long-context DPO with LongReward and conventional short-context DPO can be used together without hurting either one's performance.