MuCodec：超低比特率音樂編解碼器

MuCodec: Ultra Low-Bitrate Music Codec

September 20, 2024

作者: Yaoxun Xu, Hangting Chen, Jianwei Yu, Wei Tan, Rongzhi Gu, Shun Lei, Zhiwei Lin, Zhiyong Wu

cs.AI

摘要

音樂編碼器是音訊編碼研究中至關重要的一環，超低比特率壓縮對音樂傳輸和生成具有重要意義。由於音樂背景的複雜性和人聲豐富性，僅依賴建模語義或聲學信息無法有效地重建既包含人聲又包含背景音樂的音樂。為解決這一問題，我們提出了MuCodec，專門針對超低比特率的音樂壓縮和重建任務。MuCodec利用MuEncoder提取聲學和語義特徵，通過RVQ進行離散化，並通過流匹配獲得Mel-VAE特徵。然後使用預先訓練的MEL-VAE解碼器和HiFi-GAN重建音樂。MuCodec可以在超低（0.35kbps）或高比特率（1.35kbps）下重建高保真音樂，並在主觀和客觀指標上取得迄今為止最佳結果。代碼和演示：https://xuyaoxun.github.io/MuCodec_demo/.

English

Music codecs are a vital aspect of audio codec research, and ultra low-bitrate compression holds significant importance for music transmission and generation. Due to the complexity of music backgrounds and the richness of vocals, solely relying on modeling semantic or acoustic information cannot effectively reconstruct music with both vocals and backgrounds. To address this issue, we propose MuCodec, specifically targeting music compression and reconstruction tasks at ultra low bitrates. MuCodec employs MuEncoder to extract both acoustic and semantic features, discretizes them with RVQ, and obtains Mel-VAE features via flow-matching. The music is then reconstructed using a pre-trained MEL-VAE decoder and HiFi-GAN. MuCodec can reconstruct high-fidelity music at ultra low (0.35kbps) or high bitrates (1.35kbps), achieving the best results to date in both subjective and objective metrics. Code and Demo: https://xuyaoxun.github.io/MuCodec_demo/.