MotionShop：在视频扩散模型中实现零样本动作迁移，采用混合评分引导。

MotionShop: Zero-Shot Motion Transfer in Video Diffusion Models with Mixture of Score Guidance

December 6, 2024

作者: Hidir Yesiltepe, Tuna Han Salih Meral, Connor Dunlop, Pinar Yanardag

cs.AI

摘要

在这项工作中，我们提出了扩散Transformer中的第一个运动迁移方法，通过混合评分指导（MSG），这是一个在扩散模型中进行运动迁移的理论基础框架。我们的关键理论贡献在于重新构造条件评分，以分解扩散模型中的运动评分和内容评分。通过将运动迁移构建为潜在能量的混合，MSG自然地保留了场景构成，并在保持传输的运动模式完整性的同时实现了创造性的场景转换。这种新颖的采样直接在预训练的视频扩散模型上运行，无需额外的训练或微调。通过大量实验，MSG展示了成功处理各种情景的能力，包括单个对象、多个对象和对象间运动迁移，以及复杂的摄像机运动迁移。此外，我们介绍了MotionBench，这是第一个运动迁移数据集，包括200个源视频和1000个迁移运动，涵盖了单个/多个对象的迁移和复杂的摄像机运动。

English

In this work, we propose the first motion transfer approach in diffusion transformer through Mixture of Score Guidance (MSG), a theoretically-grounded framework for motion transfer in diffusion models. Our key theoretical contribution lies in reformulating conditional score to decompose motion score and content score in diffusion models. By formulating motion transfer as a mixture of potential energies, MSG naturally preserves scene composition and enables creative scene transformations while maintaining the integrity of transferred motion patterns. This novel sampling operates directly on pre-trained video diffusion models without additional training or fine-tuning. Through extensive experiments, MSG demonstrates successful handling of diverse scenarios including single object, multiple objects, and cross-object motion transfer as well as complex camera motion transfer. Additionally, we introduce MotionBench, the first motion transfer dataset consisting of 200 source videos and 1000 transferred motions, covering single/multi-object transfers, and complex camera motions.