黃金試金石:用於評估金融大型語言模型的全面雙語基準
Golden Touchstone: A Comprehensive Bilingual Benchmark for Evaluating Financial Large Language Models
November 9, 2024
作者: Xiaojun Wu, Junxi Liu, Huanyi Su, Zhouchi Lin, Yiyan Qi, Chengjin Xu, Jiajun Su, Jiajie Zhong, Fuwei Wang, Saizhuo Wang, Fengrui Hua, Jia Li, Jian Guo
cs.AI
摘要
隨著大型語言模型在金融領域中變得日益普遍,迫切需要一種標準化方法來全面評估其性能。然而,現有的金融基準測試往往存在語言和任務範圍有限,以及低質量數據集和不足適應性用於大型語言模型評估等挑戰。為了解決這些限制,我們提出了「金色基準」,這是第一個針對金融語言模型的全面雙語基準測試,涵蓋了來自中文和英文的代表性數據集,涵蓋了八個核心金融自然語言處理任務。通過廣泛的開源數據收集和行業特定需求的開發,這個基準測試包括各種金融任務,旨在全面評估模型的語言理解和生成能力。通過在基準測試上對主要模型進行比較分析,如GPT-4o Llama3、FinGPT和FinMA,我們揭示了它們在處理複雜金融信息方面的優勢和限制。此外,我們開源了Touchstone-GPT,這是通過持續預訓練和金融指導調整訓練的金融語言模型,在雙語基準測試中表現出色,但在特定任務上仍存在限制。這項研究不僅為金融大型語言模型提供了一個實用的評估工具,還指導了未來研究的發展和優化。Golden Touchstone的源代碼和Touchstone-GPT的模型權重已經公開在https://github.com/IDEA-FinAI/Golden-Touchstone,有助於金融語言模型的持續演進,並促進這一關鍵領域的進一步研究。
English
As large language models become increasingly prevalent in the financial
sector, there is a pressing need for a standardized method to comprehensively
assess their performance. However, existing finance benchmarks often suffer
from limited language and task coverage, as well as challenges such as
low-quality datasets and inadequate adaptability for LLM evaluation. To address
these limitations, we propose "Golden Touchstone", the first comprehensive
bilingual benchmark for financial LLMs, which incorporates representative
datasets from both Chinese and English across eight core financial NLP tasks.
Developed from extensive open source data collection and industry-specific
demands, this benchmark includes a variety of financial tasks aimed at
thoroughly assessing models' language understanding and generation
capabilities. Through comparative analysis of major models on the benchmark,
such as GPT-4o Llama3, FinGPT and FinMA, we reveal their strengths and
limitations in processing complex financial information. Additionally, we
open-sourced Touchstone-GPT, a financial LLM trained through continual
pre-training and financial instruction tuning, which demonstrates strong
performance on the bilingual benchmark but still has limitations in specific
tasks.This research not only provides the financial large language models with
a practical evaluation tool but also guides the development and optimization of
future research. The source code for Golden Touchstone and model weight of
Touchstone-GPT have been made publicly available at
https://github.com/IDEA-FinAI/Golden-Touchstone, contributing to the
ongoing evolution of FinLLMs and fostering further research in this critical
area.Summary
AI-Generated Summary
1比特LLM時代:所有大型語言模型都在1.58比特。The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
1比特LLM時代:所有大型語言模型都在1.58比特。
The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Li Dong, Ruiping Wang, Jilong Xue, Furu Wei•Feb 27, 2024•612142
DeepSeek-R1:通過強化學習激勵LLM中的推理能力DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via
Reinforcement Learning
DeepSeek-R1:通過強化學習激勵LLM中的推理能力
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via
Reinforcement Learning
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, Zhen Zhang•Jan 22, 2025•3685
Qwen2.5 技術報告Qwen2.5 Technical Report
Qwen2.5 技術報告
Qwen2.5 Technical Report
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zihan Qiu•Dec 19, 2024•36311