具有明确桥梁和检索增强的多模态音乐生成

摘要

多模态音乐生成旨在从多种输入模态（包括文本、视频和图像）中产生音乐。现有方法使用一个共同的嵌入空间进行多模态融合。尽管它们在其他模态中有效，但在多模态音乐生成中的应用面临着数据稀缺、跨模态对齐不足和可控性有限的挑战。本文通过使用文本和音乐的显式桥梁来解决这些问题。我们引入了一种名为视觉音乐桥梁（VMB）的新方法。具体来说，多模态音乐描述模型将视觉输入转换为详细的文本描述以提供文本桥梁；双轨音乐检索模块结合广泛和有针对性的检索策略以提供音乐桥梁并实现用户控制。最后，我们设计了一个明确条件的音乐生成框架，基于这两个桥梁生成音乐。我们在视频到音乐、图像到音乐、文本到音乐和可控音乐生成任务上进行实验，以及可控性实验。结果表明，与先前方法相比，VMB显著提高了音乐质量、模态和定制对齐。VMB为可解释和富有表现力的多模态音乐生成设定了新的标准，可应用于各种多媒体领域。演示和代码可在https://github.com/wbs2788/VMB 上找到。

English

Multimodal music generation aims to produce music from diverse input modalities, including text, videos, and images. Existing methods use a common embedding space for multimodal fusion. Despite their effectiveness in other modalities, their application in multimodal music generation faces challenges of data scarcity, weak cross-modal alignment, and limited controllability. This paper addresses these issues by using explicit bridges of text and music for multimodal alignment. We introduce a novel method named Visuals Music Bridge (VMB). Specifically, a Multimodal Music Description Model converts visual inputs into detailed textual descriptions to provide the text bridge; a Dual-track Music Retrieval module that combines broad and targeted retrieval strategies to provide the music bridge and enable user control. Finally, we design an Explicitly Conditioned Music Generation framework to generate music based on the two bridges. We conduct experiments on video-to-music, image-to-music, text-to-music, and controllable music generation tasks, along with experiments on controllability. The results demonstrate that VMB significantly enhances music quality, modality, and customization alignment compared to previous methods. VMB sets a new standard for interpretable and expressive multimodal music generation with applications in various multimedia fields. Demos and code are available at https://github.com/wbs2788/VMB.

具有明确桥梁和检索增强的多模态音乐生成

Multimodal Music Generation with Explicit Bridges and Retrieval Augmentation

摘要

Summary

Support