Imagine360：從透視錨點生成沉浸式360度視頻

Imagine360: Immersive 360 Video Generation from Perspective Anchor

December 4, 2024

作者: Jing Tan, Shuai Yang, Tong Wu, Jingwen He, Yuwei Guo, Ziwei Liu, Dahua Lin

cs.AI

摘要

360度視頻提供了一種超級沉浸式體驗，讓觀眾可以從完整的360度探索動態場景。為了在360度視頻格式中實現更加用戶友好和個性化的內容創作，我們致力於將標準透視視頻轉換為360度等矩形視頻。為此，我們引入了Imagine360，這是第一個透視到360度視頻生成框架，可以從視頻錨點生成具有豐富和多樣運動模式的高質量360度視頻。Imagine360從有限的360度視頻數據中學習了精細的球面視覺和運動模式，具有幾個關鍵設計。首先，我們採用了雙分支設計，包括透視和全景視頻去噪分支，為360度視頻生成提供局部和全局約束，並在擴展的Web 360度視頻上對運動模塊和空間LoRA層進行微調。此外，還設計了一個對極遮罩，用於捕捉長距離運動依賴性，增強了跨半球的對極像素之間的相反相機運動。為了應對多樣的透視視頻輸入，我們提出了適應不同視頻遮罩的高程感知設計，以應對幀間高程變化。大量實驗表明，Imagine360在360度視頻生成方法中具有出色的圖形質量和運動連貫性。我們相信Imagine360有望推動個性化、沉浸式360度視頻創作的發展。

English

360^circ videos offer a hyper-immersive experience that allows the viewers to explore a dynamic scene from full 360 degrees. To achieve more user-friendly and personalized content creation in 360^circ video format, we seek to lift standard perspective videos into 360^circ equirectangular videos. To this end, we introduce Imagine360, the first perspective-to-360^circ video generation framework that creates high-quality 360^circ videos with rich and diverse motion patterns from video anchors. Imagine360 learns fine-grained spherical visual and motion patterns from limited 360^circ video data with several key designs. 1) Firstly we adopt the dual-branch design, including a perspective and a panorama video denoising branch to provide local and global constraints for 360^circ video generation, with motion module and spatial LoRA layers fine-tuned on extended web 360^circ videos. 2) Additionally, an antipodal mask is devised to capture long-range motion dependencies, enhancing the reversed camera motion between antipodal pixels across hemispheres. 3) To handle diverse perspective video inputs, we propose elevation-aware designs that adapt to varying video masking due to changing elevations across frames. Extensive experiments show Imagine360 achieves superior graphics quality and motion coherence among state-of-the-art 360^circ video generation methods. We believe Imagine360 holds promise for advancing personalized, immersive 360^circ video creation.