移动中的二维：基于二维条件的人体动作生成

Move-in-2D: 2D-Conditioned Human Motion Generation

December 17, 2024

作者: Hsin-Ping Huang, Yang Zhou, Jui-Hsien Wang, Difan Liu, Feng Liu, Ming-Hsuan Yang, Zhan Xu

cs.AI

摘要

生成逼真的人类视频仍然是一个具有挑战性的任务，目前最有效的方法通常依赖于人类运动序列作为控制信号。现有方法通常使用从其他视频中提取的现有运动，这限制了应用于特定运动类型和全局场景匹配。我们提出了Move-in-2D，这是一种新颖的方法，可以生成以场景图像为条件的人类运动序列，从而实现适应不同场景的多样化运动。我们的方法利用扩散模型，接受场景图像和文本提示作为输入，并生成适合场景的运动序列。为了训练这个模型，我们收集了一个大规模视频数据集，展示单人活动，为每个视频标注相应的人类运动作为目标输出。实验证明，我们的方法有效地预测了与场景图像投影后对齐的人类运动。此外，我们展示了生成的运动序列在视频合成任务中改善了人类运动质量。

English

Generating realistic human videos remains a challenging task, with the most effective methods currently relying on a human motion sequence as a control signal. Existing approaches often use existing motion extracted from other videos, which restricts applications to specific motion types and global scene matching. We propose Move-in-2D, a novel approach to generate human motion sequences conditioned on a scene image, allowing for diverse motion that adapts to different scenes. Our approach utilizes a diffusion model that accepts both a scene image and text prompt as inputs, producing a motion sequence tailored to the scene. To train this model, we collect a large-scale video dataset featuring single-human activities, annotating each video with the corresponding human motion as the target output. Experiments demonstrate that our method effectively predicts human motion that aligns with the scene image after projection. Furthermore, we show that the generated motion sequence improves human motion quality in video synthesis tasks.