Imagine360：基于视角锚点的沉浸式360视频生成

Imagine360: Immersive 360 Video Generation from Perspective Anchor

December 4, 2024

作者: Jing Tan, Shuai Yang, Tong Wu, Jingwen He, Yuwei Guo, Ziwei Liu, Dahua Lin

cs.AI

摘要

360°视频提供了一种超级沉浸式体验，允许观众从完整的360度探索动态场景。为了在360°视频格式中实现更加用户友好和个性化的内容创作，我们致力于将标准透视视频转换为360°等距投影视频。为此，我们引入了Imagine360，这是第一个透视到360°视频生成框架，可以从视频锚点创建具有丰富和多样运动模式的高质量360°视频。Imagine360从有限的360°视频数据中学习细粒度的球面视觉和运动模式，具有几个关键设计。1) 首先，我们采用双分支设计，包括透视和全景视频去噪分支，为360°视频生成提供局部和全局约束，运动模块和空间LoRA层在扩展网络360°视频上进行了微调。2) 另外，设计了一个对极掩模来捕获长距离运动依赖关系，增强了跨半球的对极像素之间的反向相机运动。3) 为了处理多样的透视视频输入，我们提出了高程感知设计，适应由于帧间高程变化而产生的不同视频掩模。大量实验证明Imagine360在图形质量和运动连贯性方面优于现有360°视频生成方法。我们相信Imagine360有望推动个性化、沉浸式360°视频创作的发展。

English

360^circ videos offer a hyper-immersive experience that allows the viewers to explore a dynamic scene from full 360 degrees. To achieve more user-friendly and personalized content creation in 360^circ video format, we seek to lift standard perspective videos into 360^circ equirectangular videos. To this end, we introduce Imagine360, the first perspective-to-360^circ video generation framework that creates high-quality 360^circ videos with rich and diverse motion patterns from video anchors. Imagine360 learns fine-grained spherical visual and motion patterns from limited 360^circ video data with several key designs. 1) Firstly we adopt the dual-branch design, including a perspective and a panorama video denoising branch to provide local and global constraints for 360^circ video generation, with motion module and spatial LoRA layers fine-tuned on extended web 360^circ videos. 2) Additionally, an antipodal mask is devised to capture long-range motion dependencies, enhancing the reversed camera motion between antipodal pixels across hemispheres. 3) To handle diverse perspective video inputs, we propose elevation-aware designs that adapt to varying video masking due to changing elevations across frames. Extensive experiments show Imagine360 achieves superior graphics quality and motion coherence among state-of-the-art 360^circ video generation methods. We believe Imagine360 holds promise for advancing personalized, immersive 360^circ video creation.