Plutus：大型语言模型在希腊低资源金融领域的基准测试

Plutus: Benchmarking Large Language Models in Low-Resource Greek Finance

February 26, 2025

作者: Xueqing Peng, Triantafillos Papadopoulos, Efstathia Soufleri, Polydoros Giannouris, Ruoyu Xiang, Yan Wang, Lingfei Qian, Jimin Huang, Qianqian Xie, Sophia Ananiadou

cs.AI

摘要

尽管希腊在全球经济中扮演着关键角色，但由于希腊语的语言复杂性及领域特定数据集的稀缺，大型语言模型（LLMs）在希腊金融语境中的应用仍未被充分探索。以往的多语言金融自然语言处理（NLP）研究已揭示出显著的性能差异，然而，迄今为止，尚未开发出专门的希腊金融基准测试或希腊特定的金融LLMs。为填补这一空白，我们推出了Plutus-ben，首个希腊金融评估基准，以及Plutus-8B，首款基于希腊领域特定数据微调的希腊金融LLM。Plutus-ben涵盖了希腊金融NLP的五大核心任务：数值与文本命名实体识别、问答、摘要生成及主题分类，从而促进了LLM的系统化与可重复性评估。为支撑这些任务，我们提供了三个全新的高质量希腊金融数据集，这些数据集由希腊语母语专家精心标注，并补充了两个现有资源。我们对22个LLMs在Plutus-ben上的全面评估显示，希腊金融NLP因语言复杂性、领域特定术语及金融推理差距而仍具挑战性。这些发现凸显了跨语言迁移的局限性、希腊训练模型中金融专业知识的必要性，以及将金融LLMs适应于希腊文本的挑战。我们公开了Plutus-ben、Plutus-8B及所有相关数据集，以推动可重复性研究并促进希腊金融NLP的发展，从而在金融领域实现更广泛的多语言包容性。

English

Despite Greece's pivotal role in the global economy, large language models (LLMs) remain underexplored for Greek financial context due to the linguistic complexity of Greek and the scarcity of domain-specific datasets. Previous efforts in multilingual financial natural language processing (NLP) have exposed considerable performance disparities, yet no dedicated Greek financial benchmarks or Greek-specific financial LLMs have been developed until now. To bridge this gap, we introduce Plutus-ben, the first Greek Financial Evaluation Benchmark, and Plutus-8B, the pioneering Greek Financial LLM, fine-tuned with Greek domain-specific data. Plutus-ben addresses five core financial NLP tasks in Greek: numeric and textual named entity recognition, question answering, abstractive summarization, and topic classification, thereby facilitating systematic and reproducible LLM assessments. To underpin these tasks, we present three novel, high-quality Greek financial datasets, thoroughly annotated by expert native Greek speakers, augmented by two existing resources. Our comprehensive evaluation of 22 LLMs on Plutus-ben reveals that Greek financial NLP remains challenging due to linguistic complexity, domain-specific terminology, and financial reasoning gaps. These findings underscore the limitations of cross-lingual transfer, the necessity for financial expertise in Greek-trained models, and the challenges of adapting financial LLMs to Greek text. We release Plutus-ben, Plutus-8B, and all associated datasets publicly to promote reproducible research and advance Greek financial NLP, fostering broader multilingual inclusivity in finance.