LongReward：利用人工智慧改進長文本大型語言模型的反饋

LongReward: Improving Long-context Large Language Models with AI Feedback

October 28, 2024

作者: Jiajie Zhang, Zhongni Hou, Xin Lv, Shulin Cao, Zhenyu Hou, Yilin Niu, Lei Hou, Yuxiao Dong, Ling Feng, Juanzi Li

cs.AI

摘要

儘管在發展長文本大型語言模型（LLMs）方面取得了重大進展，但LLM合成數據的品質通常會影響監督微調（SFT）的長文本性能，並導致固有限制。從原則上講，利用適當的獎勵信號進行強化學習（RL）可以進一步增強模型的能力。然而，在長文本情境中如何獲得可靠的獎勵仍未被探討。為此，我們提出了LongReward，一種新方法，利用現成的LLM從四個人類價值維度（幫助性、邏輯性、忠實度和完整性）提供長文本模型回應的獎勵，每個維度都有精心設計的評估流程。通過結合LongReward和離線RL算法DPO，我們能夠有效改善長文本SFT模型。我們的實驗表明，LongReward不僅顯著提高了模型的長文本性能，還增強了它們遵循簡短指令的能力。我們還發現，具有LongReward的長文本DPO和傳統的短文本DPO可以一起使用，而不會損害任何一個的性能。

English

Though significant advancements have been achieved in developing long-context large language models (LLMs), the compromised quality of LLM-synthesized data for supervised fine-tuning (SFT) often affects the long-context performance of SFT models and leads to inherent limitations. In principle, reinforcement learning (RL) with appropriate reward signals can further enhance models' capacities. However, how to obtain reliable rewards in long-context scenarios remains unexplored. To this end, we propose LongReward, a novel method that utilizes an off-the-shelf LLM to provide rewards for long-context model responses from four human-valued dimensions: helpfulness, logicality, faithfulness, and completeness, each with a carefully designed assessment pipeline. By combining LongReward and offline RL algorithm DPO, we are able to effectively improve long-context SFT models. Our experiments indicate that LongReward not only significantly improves models' long-context performance but also enhances their ability to follow short instructions. We also find that long-context DPO with LongReward and conventional short-context DPO can be used together without hurting either one's performance.