Vivid4D：通過視頻修復提升單目視頻的四維重建

Vivid4D: Improving 4D Reconstruction from Monocular Video by Video Inpainting

April 15, 2025

作者: Jiaxin Huang, Sheng Miao, BangBnag Yang, Yuewen Ma, Yiyi Liao

cs.AI

摘要

從隨意拍攝的單目影片中重建四維動態場景具有重要價值，但也極具挑戰性，因為每個時間點僅能從單一視角進行觀察。我們提出了Vivid4D，這是一種新穎的方法，通過增加觀察視角來增強四維單目影片的合成——從單目輸入中合成多視角影片。與現有方法不同，這些方法要么僅利用幾何先驗進行監督，要么使用生成先驗卻忽視幾何信息，我們將兩者結合起來。這將視角增強重新表述為一個影片修補任務，其中觀察到的視角基於單目深度先驗被扭曲到新的視角。為實現這一點，我們在未標定姿勢的網絡影片上訓練了一個影片修補模型，使用模擬扭曲遮擋的合成遮罩，確保缺失區域在空間和時間上的一致性補全。為了進一步減輕單目深度先驗中的不準確性，我們引入了迭代視角增強策略和魯棒的重建損失。實驗表明，我們的方法有效提升了單目四維場景的重建和補全效果。

English

Reconstructing 4D dynamic scenes from casually captured monocular videos is valuable but highly challenging, as each timestamp is observed from a single viewpoint. We introduce Vivid4D, a novel approach that enhances 4D monocular video synthesis by augmenting observation views - synthesizing multi-view videos from a monocular input. Unlike existing methods that either solely leverage geometric priors for supervision or use generative priors while overlooking geometry, we integrate both. This reformulates view augmentation as a video inpainting task, where observed views are warped into new viewpoints based on monocular depth priors. To achieve this, we train a video inpainting model on unposed web videos with synthetically generated masks that mimic warping occlusions, ensuring spatially and temporally consistent completion of missing regions. To further mitigate inaccuracies in monocular depth priors, we introduce an iterative view augmentation strategy and a robust reconstruction loss. Experiments demonstrate that our method effectively improves monocular 4D scene reconstruction and completion.