SynCamMaster：從多元視角同步生成多攝影機視頻

SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse Viewpoints

December 10, 2024

作者: Jianhong Bai, Menghan Xia, Xintao Wang, Ziyang Yuan, Xiao Fu, Zuozhu Liu, Haoji Hu, Pengfei Wan, Di Zhang

cs.AI

摘要

最近在影片擴散模型方面的進展展現出卓越的能力，能夠模擬真實世界的動態並保持三維一致性。這些進展激發了我們對這些模型潛力的探究，以確保在各種視角下實現動態一致性，這對於虛擬拍攝等應用非常理想。與現有方法不同，現有方法專注於對單個物體進行多視角生成以進行四維重建，我們的興趣在於從任意視角生成開放世界影片，並納入六自由度相機姿勢。為了實現這一目標，我們提出了一個即插即用模組，用於增強預訓練的文本到影片模型，以進行多攝影機影片生成，確保在不同視角下內容保持一致。具體來說，我們引入了一個多視角同步模組，以保持這些視角下的外觀和幾何一致性。鑒於高質量訓練數據的稀缺性，我們設計了一種混合訓練方案，利用多攝影機圖像和單眼影片來補充虛幻引擎渲染的多攝影機影片。此外，我們的方法還能實現有趣的擴展，例如從新視角重新渲染影片。我們還發布了一個名為 SynCamVideo-Dataset 的多視角同步影片數據集。項目頁面：https://jianhongbai.github.io/SynCamMaster/。

English

Recent advancements in video diffusion models have shown exceptional abilities in simulating real-world dynamics and maintaining 3D consistency. This progress inspires us to investigate the potential of these models to ensure dynamic consistency across various viewpoints, a highly desirable feature for applications such as virtual filming. Unlike existing methods focused on multi-view generation of single objects for 4D reconstruction, our interest lies in generating open-world videos from arbitrary viewpoints, incorporating 6 DoF camera poses. To achieve this, we propose a plug-and-play module that enhances a pre-trained text-to-video model for multi-camera video generation, ensuring consistent content across different viewpoints. Specifically, we introduce a multi-view synchronization module to maintain appearance and geometry consistency across these viewpoints. Given the scarcity of high-quality training data, we design a hybrid training scheme that leverages multi-camera images and monocular videos to supplement Unreal Engine-rendered multi-camera videos. Furthermore, our method enables intriguing extensions, such as re-rendering a video from novel viewpoints. We also release a multi-view synchronized video dataset, named SynCamVideo-Dataset. Project page: https://jianhongbai.github.io/SynCamMaster/.