SongGen：用于文本到歌曲生成的一阶段自回归Transformer模型

SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation

February 18, 2025

作者: Zihan Liu, Shuangrui Ding, Zhixiong Zhang, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Dahua Lin, Jiaqi Wang

cs.AI

摘要

文本到歌曲生成，即从文本输入中创作人声与伴奏的任务，因领域复杂性和数据稀缺性而面临重大挑战。现有方法多采用多阶段生成流程，导致训练与推理管道繁琐。本文提出SongGen，一个完全开源、单阶段自回归变换器，专为可控歌曲生成设计。该模型支持对歌词及乐器配置、风格、情绪和音色等多样音乐属性的细粒度控制，同时提供可选的三秒参考片段用于声音克隆。在统一的自回归框架内，SongGen支持两种输出模式：混合模式直接生成人声与伴奏的混合，双轨模式则分别合成二者，为下游应用提供更大灵活性。我们探索了每种模式下的多样化标记模式策略，取得了显著改进并获得了宝贵洞见。此外，我们设计了一个自动化数据预处理流程，并实施了有效的质量控制。为促进社区参与和未来研究，我们将公开模型权重、训练代码、标注数据及预处理流程。生成样本展示于项目页面https://liuzh-19.github.io/SongGen/，代码将发布于https://github.com/LiuZH-19/SongGen。

English

Text-to-song generation, the task of creating vocals and accompaniment from textual inputs, poses significant challenges due to domain complexity and data scarcity. Existing approaches often employ multi-stage generation procedures, resulting in cumbersome training and inference pipelines. In this paper, we propose SongGen, a fully open-source, single-stage auto-regressive transformer designed for controllable song generation. The proposed model facilitates fine-grained control over diverse musical attributes, including lyrics and textual descriptions of instrumentation, genre, mood, and timbre, while also offering an optional three-second reference clip for voice cloning. Within a unified auto-regressive framework, SongGen supports two output modes: mixed mode, which generates a mixture of vocals and accompaniment directly, and dual-track mode, which synthesizes them separately for greater flexibility in downstream applications. We explore diverse token pattern strategies for each mode, leading to notable improvements and valuable insights. Furthermore, we design an automated data preprocessing pipeline with effective quality control. To foster community engagement and future research, we will release our model weights, training code, annotated data, and preprocessing pipeline. The generated samples are showcased on our project page at https://liuzh-19.github.io/SongGen/ , and the code will be available at https://github.com/LiuZH-19/SongGen .