SphereDiff：基於球形潛在表徵的免調參全向全景圖像與視頻生成

SphereDiff: Tuning-free Omnidirectional Panoramic Image and Video Generation via Spherical Latent Representation

April 19, 2025

作者: Minho Park, Taewoong Kang, Jooyeol Yun, Sungwon Hwang, Jaegul Choo

cs.AI

摘要

隨著AR/VR應用需求的日益增長，高品質360度全景內容的生成需求也愈發凸顯。然而，由於等距柱狀投影（ERP）所帶來的嚴重失真，生成高品質的360度全景圖像和視頻仍是一項具有挑戰性的任務。現有方法要麼在有限的ERP數據集上微調預訓練的擴散模型，要麼嘗試無需微調的方法，但仍依賴於ERP的潛在表示，這導致在極點附近出現不連續性。本文提出了一種名為SphereDiff的新方法，利用最先進的擴散模型實現無縫360度全景圖像和視頻的生成，無需額外微調。我們定義了一種球形潛在表示，確保所有視角的均勻分佈，從而緩解ERP固有的失真問題。我們將多擴散技術擴展至球形潛在空間，並提出了一種球形潛在採樣方法，以便直接使用預訓練的擴散模型。此外，我們引入了失真感知加權平均技術，進一步提升投影過程中的生成質量。我們的方法在生成360度全景內容方面優於現有方法，同時保持了高保真度，為沉浸式AR/VR應用提供了一個穩健的解決方案。代碼可在此處獲取：https://github.com/pmh9960/SphereDiff。

English

The increasing demand for AR/VR applications has highlighted the need for high-quality 360-degree panoramic content. However, generating high-quality 360-degree panoramic images and videos remains a challenging task due to the severe distortions introduced by equirectangular projection (ERP). Existing approaches either fine-tune pretrained diffusion models on limited ERP datasets or attempt tuning-free methods that still rely on ERP latent representations, leading to discontinuities near the poles. In this paper, we introduce SphereDiff, a novel approach for seamless 360-degree panoramic image and video generation using state-of-the-art diffusion models without additional tuning. We define a spherical latent representation that ensures uniform distribution across all perspectives, mitigating the distortions inherent in ERP. We extend MultiDiffusion to spherical latent space and propose a spherical latent sampling method to enable direct use of pretrained diffusion models. Moreover, we introduce distortion-aware weighted averaging to further improve the generation quality in the projection process. Our method outperforms existing approaches in generating 360-degree panoramic content while maintaining high fidelity, making it a robust solution for immersive AR/VR applications. The code is available here. https://github.com/pmh9960/SphereDiff