轨迹重塑者：基于扩散模型的单目视频相机轨迹重定向

TrajectoryCrafter: Redirecting Camera Trajectory for Monocular Videos via Diffusion Models

March 7, 2025

作者: Mark YU, Wenbo Hu, Jinbo Xing, Ying Shan

cs.AI

摘要

我们提出了TrajectoryCrafter，一种针对单目视频重定向相机轨迹的新颖方法。通过将确定性视角变换与随机内容生成解耦，我们的方法实现了对用户指定相机轨迹的精确控制。我们提出了一种新颖的双流条件视频扩散模型，该模型同时整合点云渲染和源视频作为条件，确保了准确的视角变换和连贯的四维内容生成。我们没有依赖稀缺的多视角视频，而是通过创新的双重重投影策略，构建了一个结合网络规模单目视频与静态多视角数据集的混合训练数据集，显著提升了模型在多样化场景中的泛化能力。在多视角和大规模单目视频上的广泛评估验证了我们方法的卓越性能。

English

We present TrajectoryCrafter, a novel approach to redirect camera trajectories for monocular videos. By disentangling deterministic view transformations from stochastic content generation, our method achieves precise control over user-specified camera trajectories. We propose a novel dual-stream conditional video diffusion model that concurrently integrates point cloud renders and source videos as conditions, ensuring accurate view transformations and coherent 4D content generation. Instead of leveraging scarce multi-view videos, we curate a hybrid training dataset combining web-scale monocular videos with static multi-view datasets, by our innovative double-reprojection strategy, significantly fostering robust generalization across diverse scenes. Extensive evaluations on multi-view and large-scale monocular videos demonstrate the superior performance of our method.