利用视频扩散先验进行新视角外推

Novel View Extrapolation with Video Diffusion Priors

November 21, 2024

作者: Kunhao Liu, Ling Shao, Shijian Lu

cs.AI

摘要

由于辐射场方法的发展，新视角合成领域取得了重大进展。然而，大多数辐射场技术在新视角插值方面表现优异，而在新视角外推方面则表现不佳，即合成的新视角远远超出了观察到的训练视角。我们设计了ViewExtrapolator，这是一种新颖的视角合成方法，利用稳定视频扩散（SVD）的生成先验进行逼真的新视角外推。通过重新设计SVD去噪过程，ViewExtrapolator改进了辐射场渲染的易出现伪影的视角，极大地提高了合成新视角的清晰度和逼真度。ViewExtrapolator是一种通用的新视角外推器，可以与不同类型的3D渲染一起使用，例如从点云渲染的视角，当只有单个视角或单目视频可用时。此外，ViewExtrapolator无需对SVD进行微调，既数据高效又计算高效。大量实验证明了ViewExtrapolator在新视角外推方面的优越性。项目页面：https://kunhao-liu.github.io/ViewExtrapolator/。

English

The field of novel view synthesis has made significant strides thanks to the development of radiance field methods. However, most radiance field techniques are far better at novel view interpolation than novel view extrapolation where the synthesis novel views are far beyond the observed training views. We design ViewExtrapolator, a novel view synthesis approach that leverages the generative priors of Stable Video Diffusion (SVD) for realistic novel view extrapolation. By redesigning the SVD denoising process, ViewExtrapolator refines the artifact-prone views rendered by radiance fields, greatly enhancing the clarity and realism of the synthesized novel views. ViewExtrapolator is a generic novel view extrapolator that can work with different types of 3D rendering such as views rendered from point clouds when only a single view or monocular video is available. Additionally, ViewExtrapolator requires no fine-tuning of SVD, making it both data-efficient and computation-efficient. Extensive experiments demonstrate the superiority of ViewExtrapolator in novel view extrapolation. Project page: https://kunhao-liu.github.io/ViewExtrapolator/.