FLAG-Trader：融合大语言模型与梯度强化学习的智能金融交易代理

FLAG-Trader: Fusion LLM-Agent with Gradient-based Reinforcement Learning for Financial Trading

February 17, 2025

作者: Guojun Xiong, Zhiyang Deng, Keyi Wang, Yupeng Cao, Haohang Li, Yangyang Yu, Xueqing Peng, Mingquan Lin, Kaleb E Smith, Xiao-Yang Liu, Jimin Huang, Sophia Ananiadou, Qianqian Xie

cs.AI

摘要

基于多模态金融数据微调的大型语言模型（LLMs）已在多种金融任务中展现出卓越的推理能力。然而，在交互式金融市场中，如交易这类需要复杂代理策略以优化决策的多步骤、目标导向场景中，它们往往表现欠佳。为此，我们提出了FLAG-Trader，一种统一架构，它将语言处理（通过LLMs）与梯度驱动的强化学习（RL）策略优化相结合。在此架构中，部分微调的LLM充当策略网络，既利用预训练知识，又通过参数高效微调适应金融领域。通过交易奖励驱动的策略梯度优化，我们的框架不仅提升了LLM在交易中的表现，还改善了其在其他金融领域任务上的成果。我们提供了详尽的实证证据来验证这些改进。

English

Large language models (LLMs) fine-tuned on multimodal financial data have demonstrated impressive reasoning capabilities in various financial tasks. However, they often struggle with multi-step, goal-oriented scenarios in interactive financial markets, such as trading, where complex agentic approaches are required to improve decision-making. To address this, we propose FLAG-Trader, a unified architecture integrating linguistic processing (via LLMs) with gradient-driven reinforcement learning (RL) policy optimization, in which a partially fine-tuned LLM acts as the policy network, leveraging pre-trained knowledge while adapting to the financial domain through parameter-efficient fine-tuning. Through policy gradient optimization driven by trading rewards, our framework not only enhances LLM performance in trading but also improves results on other financial-domain tasks. We present extensive empirical evidence to validate these enhancements.