通往使用大型語言模型實現超人類語音理解的路線圖

Roadmap towards Superhuman Speech Understanding using Large Language Models

October 17, 2024

作者: Fan Bu, Yuhao Zhang, Xidong Wang, Benyou Wang, Qun Liu, Haizhou Li

cs.AI

摘要

大型語言模型（LLMs）的成功促使了整合語音和音頻數據的努力，旨在創建能夠處理文本和非文本輸入的通用基礎模型。最近的進展，如GPT-4o，突顯了端到端語音LLMs的潛力，該模型保留了非語義信息和世界知識，以實現更深入的語音理解。為了指導語音LLMs的發展，我們提出了一個五級路線圖，從基本的自動語音識別（ASR）到能夠將非語義信息與抽象聲學知識整合以處理複雜任務的高級超人模型。此外，我們設計了一個名為SAGI Benchmark的基準，標準化了這五個級別中各種任務的關鍵方面，揭示了使用抽象聲學知識和功能完整性時的挑戰。我們的研究結果揭示了在處理語音提示和抽象聲學知識方面存在的差距，並提出了未來的方向。本文概述了推進語音LLMs的路線圖，介紹了一個評估基準，並提供了有關它們目前限制和潛力的關鍵見解。

English

The success of large language models (LLMs) has prompted efforts to integrate speech and audio data, aiming to create general foundation models capable of processing both textual and non-textual inputs. Recent advances, such as GPT-4o, highlight the potential for end-to-end speech LLMs, which preserves non-semantic information and world knowledge for deeper speech understanding. To guide the development of speech LLMs, we propose a five-level roadmap, ranging from basic automatic speech recognition (ASR) to advanced superhuman models capable of integrating non-semantic information with abstract acoustic knowledge for complex tasks. Moreover, we design a benchmark, SAGI Bechmark, that standardizes critical aspects across various tasks in these five levels, uncovering challenges in using abstract acoustic knowledge and completeness of capability. Our findings reveal gaps in handling paralinguistic cues and abstract acoustic knowledge, and we offer future directions. This paper outlines a roadmap for advancing speech LLMs, introduces a benchmark for evaluation, and provides key insights into their current limitations and potential.