MotionCanvas：具有可控图像到视频生成功能的电影镜头设计

MotionCanvas: Cinematic Shot Design with Controllable Image-to-Video Generation

February 6, 2025

作者: Jinbo Xing, Long Mai, Cusuh Ham, Jiahui Huang, Aniruddha Mahapatra, Chi-Wing Fu, Tien-Tsin Wong, Feng Liu

cs.AI

摘要

本文提出了一种方法，允许用户在图像到视频生成的背景下设计电影视频镜头。镜头设计是电影制作的关键方面，涉及精心规划场景中的摄像机移动和物体运动。然而，在现代图像到视频生成系统中实现直观的镜头设计面临两个主要挑战：首先，有效捕捉用户对运动设计的意图，在这里必须共同指定摄像机移动和场景空间物体运动；其次，表示可以被视频扩散模型有效利用以合成图像动画的运动信息。为了解决这些挑战，我们引入了MotionCanvas，这是一种将用户驱动控制集成到图像到视频（I2V）生成模型中的方法，允许用户以场景感知的方式控制场景中的物体和摄像机运动。通过结合经典计算机图形学和当代视频生成技术的见解，我们展示了在I2V合成中实现3D感知运动控制的能力，而无需昂贵的3D相关训练数据。MotionCanvas使用户能够直观地描绘场景空间运动意图，并将其转化为视频扩散模型的时空运动调节信号。我们在各种真实世界图像内容和镜头设计场景上展示了我们方法的有效性，突显了它在数字内容创作的创意工作流程中增强的潜力，并适应各种图像和视频编辑应用。

English

This paper presents a method that allows users to design cinematic video shots in the context of image-to-video generation. Shot design, a critical aspect of filmmaking, involves meticulously planning both camera movements and object motions in a scene. However, enabling intuitive shot design in modern image-to-video generation systems presents two main challenges: first, effectively capturing user intentions on the motion design, where both camera movements and scene-space object motions must be specified jointly; and second, representing motion information that can be effectively utilized by a video diffusion model to synthesize the image animations. To address these challenges, we introduce MotionCanvas, a method that integrates user-driven controls into image-to-video (I2V) generation models, allowing users to control both object and camera motions in a scene-aware manner. By connecting insights from classical computer graphics and contemporary video generation techniques, we demonstrate the ability to achieve 3D-aware motion control in I2V synthesis without requiring costly 3D-related training data. MotionCanvas enables users to intuitively depict scene-space motion intentions, and translates them into spatiotemporal motion-conditioning signals for video diffusion models. We demonstrate the effectiveness of our method on a wide range of real-world image content and shot-design scenarios, highlighting its potential to enhance the creative workflows in digital content creation and adapt to various image and video editing applications.