S2S-Arena：基于副语言信息的指令跟随评估中的语音转语音协议研究

S2S-Arena, Evaluating Speech2Speech Protocols on Instruction Following with Paralinguistic Information

March 7, 2025

作者: Feng Jiang, Zhiyu Lin, Fan Bu, Yuhao Du, Benyou Wang, Haizhou Li

cs.AI

摘要

大型语言模型（LLMs）的快速发展，使得语音模型尤其是支持语音输入输出的speech2speech协议近期进展备受瞩目。然而，现有基准测试采用基于文本的自动评估器来评价这些模型的指令跟随能力，却忽视了语音理解与生成过程中副语言信息的考量。为解决这些问题，我们引入了S2S-Arena，一个创新的竞技场式S2S基准测试，它通过真实世界任务中的语音输入与输出，结合副语言信息来评估指令跟随能力。我们设计了154个样本，融合了TTS与现场录音，覆盖四个领域的21项任务，并以竞技场方式手动评估了现有热门语音模型。实验结果表明：（1）除GPT-4o表现卓越外，在speech2speech协议中，级联ASR、LLM与TTS的语音模型在文本-语音对齐后，其性能优于联合训练模型；（2）考虑到副语言信息，语音模型的知识性主要依赖于LLM主干，而其多语言支持则受限于语音模块；（3）优秀的语音模型已能理解语音输入中的副语言信息，但生成包含恰当副语言信息的音频仍是一大挑战。

English

The rapid development of large language models (LLMs) has brought significant attention to speech models, particularly recent progress in speech2speech protocols supporting speech input and output. However, the existing benchmarks adopt automatic text-based evaluators for evaluating the instruction following ability of these models lack consideration for paralinguistic information in both speech understanding and generation. To address these issues, we introduce S2S-Arena, a novel arena-style S2S benchmark that evaluates instruction-following capabilities with paralinguistic information in both speech-in and speech-out across real-world tasks. We design 154 samples that fused TTS and live recordings in four domains with 21 tasks and manually evaluate existing popular speech models in an arena-style manner. The experimental results show that: (1) in addition to the superior performance of GPT-4o, the speech model of cascaded ASR, LLM, and TTS outperforms the jointly trained model after text-speech alignment in speech2speech protocols; (2) considering paralinguistic information, the knowledgeability of the speech model mainly depends on the LLM backbone, and the multilingual support of that is limited by the speech module; (3) excellent speech models can already understand the paralinguistic information in speech input, but generating appropriate audio with paralinguistic information is still a challenge.