万物动起来：任意到运动生成

摘要

条件运动生成在计算机视觉领域已得到广泛研究，但仍面临两大关键挑战。首先，尽管掩码自回归方法近期超越了基于扩散的方法，现有掩码模型缺乏根据给定条件优先处理动态帧和身体部位的机制。其次，现有针对不同条件模态的方法往往难以有效整合多模态信息，限制了生成运动的控制性和连贯性。为解决这些挑战，我们提出了Motion Anything，一个多模态运动生成框架，引入了基于注意力的掩码建模方法，实现了对关键帧和动作的精细时空控制。我们的模型自适应地编码包括文本和音乐在内的多模态条件，提升了可控性。此外，我们推出了Text-Music-Dance (TMD)，一个包含2,153对文本、音乐和舞蹈的新运动数据集，其规模是AIST++的两倍，填补了该领域的重要空白。大量实验表明，Motion Anything在多个基准测试中超越了现有最先进方法，在HumanML3D上FID提升了15%，并在AIST++和TMD上展现出持续的性能优势。详情请访问我们的项目网站：https://steve-zeyu-zhang.github.io/MotionAnything。

English

Conditional motion generation has been extensively studied in computer vision, yet two critical challenges remain. First, while masked autoregressive methods have recently outperformed diffusion-based approaches, existing masking models lack a mechanism to prioritize dynamic frames and body parts based on given conditions. Second, existing methods for different conditioning modalities often fail to integrate multiple modalities effectively, limiting control and coherence in generated motion. To address these challenges, we propose Motion Anything, a multimodal motion generation framework that introduces an Attention-based Mask Modeling approach, enabling fine-grained spatial and temporal control over key frames and actions. Our model adaptively encodes multimodal conditions, including text and music, improving controllability. Additionally, we introduce Text-Music-Dance (TMD), a new motion dataset consisting of 2,153 pairs of text, music, and dance, making it twice the size of AIST++, thereby filling a critical gap in the community. Extensive experiments demonstrate that Motion Anything surpasses state-of-the-art methods across multiple benchmarks, achieving a 15% improvement in FID on HumanML3D and showing consistent performance gains on AIST++ and TMD. See our project website https://steve-zeyu-zhang.github.io/MotionAnything

万物动起来：任意到运动生成

Motion Anything: Any to Motion Generation

摘要

Summary

Support