ObjCtrl-2.5D：无需训练的摄像头姿势物体控制

ObjCtrl-2.5D: Training-free Object Control with Camera Poses

December 10, 2024

作者: Zhouxia Wang, Yushi Lan, Shangchen Zhou, Chen Change Loy

cs.AI

摘要

本研究旨在实现图像到视频（I2V）生成中更精确、更灵活的目标控制。当前方法通常使用2D轨迹表示目标物体的空间移动，但往往无法捕捉用户意图并经常产生不自然的结果。为了增强控制能力，我们提出了ObjCtrl-2.5D，这是一种无需训练的目标控制方法，它使用包含深度信息的3D轨迹作为控制信号，扩展自2D轨迹。通过将目标物体的运动建模为摄像机运动，ObjCtrl-2.5D将3D轨迹表示为摄像机姿势序列，从而实现使用现有摄像机运动控制I2V生成模型（CMC-I2V）进行目标运动控制而无需训练。为了使最初设计用于全局运动控制的CMC-I2V模型适应处理局部目标运动，我们引入了一个模块来将目标物体与背景隔离，实现独立的局部控制。此外，我们设计了一种有效的方法，通过在帧间共享目标区域内的低频变形潜变量，实现更精确的目标控制。大量实验证明，与无需训练的方法相比，ObjCtrl-2.5D显著提高了目标控制精度，并比使用2D轨迹的基于训练的方法提供了更多样化的控制能力，实现了复杂效果，如目标旋转。代码和结果可在https://wzhouxiff.github.io/projects/ObjCtrl-2.5D/找到。

English

This study aims to achieve more precise and versatile object control in image-to-video (I2V) generation. Current methods typically represent the spatial movement of target objects with 2D trajectories, which often fail to capture user intention and frequently produce unnatural results. To enhance control, we present ObjCtrl-2.5D, a training-free object control approach that uses a 3D trajectory, extended from a 2D trajectory with depth information, as a control signal. By modeling object movement as camera movement, ObjCtrl-2.5D represents the 3D trajectory as a sequence of camera poses, enabling object motion control using an existing camera motion control I2V generation model (CMC-I2V) without training. To adapt the CMC-I2V model originally designed for global motion control to handle local object motion, we introduce a module to isolate the target object from the background, enabling independent local control. In addition, we devise an effective way to achieve more accurate object control by sharing low-frequency warped latent within the object's region across frames. Extensive experiments demonstrate that ObjCtrl-2.5D significantly improves object control accuracy compared to training-free methods and offers more diverse control capabilities than training-based approaches using 2D trajectories, enabling complex effects like object rotation. Code and results are available at https://wzhouxiff.github.io/projects/ObjCtrl-2.5D/.