VidPanos：從隨意移動的影片生成全景影片

VidPanos: Generative Panoramic Videos from Casual Panning Videos

October 17, 2024

作者: Jingwei Ma, Erika Lu, Roni Paiss, Shiran Zada, Aleksander Holynski, Tali Dekel, Brian Curless, Michael Rubinstein, Forrester Cole

cs.AI

摘要

全景圖像拼接提供了一個統一的、廣角的場景視圖，超出了相機的視野範圍。將一段全景攝影中的影格拼接成全景照片對於靜止場景是一個眾所周知的問題，但當物體移動時，靜態全景無法捕捉到場景。我們提出了一種從隨意拍攝的全景攝影中合成全景視頻的方法，就好像原始視頻是用廣角相機拍攝的一樣。我們將全景合成定義為一個時空外描繪問題，我們的目標是創建一個與輸入視頻相同長度的完整全景視頻。對時空體積的一致完成需要對視頻內容和運動進行強大、逼真的先驗，為此我們適應了生成式視頻模型。然而，現有的生成模型並不能立即擴展到全景完成，正如我們所展示的。相反，我們將視頻生成應用為全景合成系統的一部分，並展示如何利用模型的優勢同時最小化它們的局限性。我們的系統可以為各種野外場景創建視頻全景，包括人物、車輛和流動的水，以及靜止的背景特徵。

English

Panoramic image stitching provides a unified, wide-angle view of a scene that extends beyond the camera's field of view. Stitching frames of a panning video into a panoramic photograph is a well-understood problem for stationary scenes, but when objects are moving, a still panorama cannot capture the scene. We present a method for synthesizing a panoramic video from a casually-captured panning video, as if the original video were captured with a wide-angle camera. We pose panorama synthesis as a space-time outpainting problem, where we aim to create a full panoramic video of the same length as the input video. Consistent completion of the space-time volume requires a powerful, realistic prior over video content and motion, for which we adapt generative video models. Existing generative models do not, however, immediately extend to panorama completion, as we show. We instead apply video generation as a component of our panorama synthesis system, and demonstrate how to exploit the strengths of the models while minimizing their limitations. Our system can create video panoramas for a range of in-the-wild scenes including people, vehicles, and flowing water, as well as stationary background features.