動態城市：從動態場景生成大規模LiDAR

DynamicCity: Large-Scale LiDAR Generation from Dynamic Scenes

October 23, 2024

作者: Hengwei Bian, Lingdong Kong, Haozhe Xie, Liang Pan, Yu Qiao, Ziwei Liu

cs.AI

摘要

最近，LiDAR場景生成技術發展迅速。然而，現有方法主要集中在生成靜態和單幀場景，忽略了現實世界行駛環境固有的動態特性。在這項工作中，我們介紹了DynamicCity，一個新穎的4D LiDAR生成框架，能夠生成大規模、高質量的LiDAR場景，捕捉動態環境的時間演變。DynamicCity主要由兩個關鍵模型組成。1) VAE模型用於學習HexPlane作為緊湊的4D表示。DynamicCity採用一個新穎的Projection Module，而非使用天真的平均操作，有效地將4D LiDAR特徵壓縮為六個2D特徵圖，用於HexPlane構建，從而顯著提高HexPlane的擬合質量（最高可達12.56 mIoU增益）。此外，我們利用擴展和壓縮策略並行重構3D特徵體積，比起天真地查詢每個3D點，進一步提高了網絡訓練效率和重構準確性（最高可達7.05 mIoU增益、2.06倍訓練加速和70.84%記憶體減少）。2) 基於DiT的擴散模型用於HexPlane生成。為了使HexPlane適合DiT生成，提出了一個Padded Rollout Operation，將HexPlane的所有六個特徵平面重新組織為一個方形的2D特徵圖。特別是，在擴散或採樣過程中可以引入各種條件，支持多樣化的4D生成應用，如軌跡和命令驅動生成、修補和佈局條件生成。對CarlaSC和Waymo數據集的大量實驗表明，DynamicCity在多個指標上顯著優於現有最先進的4D LiDAR生成方法。代碼將被釋放以促進未來研究。

English

LiDAR scene generation has been developing rapidly recently. However, existing methods primarily focus on generating static and single-frame scenes, overlooking the inherently dynamic nature of real-world driving environments. In this work, we introduce DynamicCity, a novel 4D LiDAR generation framework capable of generating large-scale, high-quality LiDAR scenes that capture the temporal evolution of dynamic environments. DynamicCity mainly consists of two key models. 1) A VAE model for learning HexPlane as the compact 4D representation. Instead of using naive averaging operations, DynamicCity employs a novel Projection Module to effectively compress 4D LiDAR features into six 2D feature maps for HexPlane construction, which significantly enhances HexPlane fitting quality (up to 12.56 mIoU gain). Furthermore, we utilize an Expansion & Squeeze Strategy to reconstruct 3D feature volumes in parallel, which improves both network training efficiency and reconstruction accuracy than naively querying each 3D point (up to 7.05 mIoU gain, 2.06x training speedup, and 70.84% memory reduction). 2) A DiT-based diffusion model for HexPlane generation. To make HexPlane feasible for DiT generation, a Padded Rollout Operation is proposed to reorganize all six feature planes of the HexPlane as a squared 2D feature map. In particular, various conditions could be introduced in the diffusion or sampling process, supporting versatile 4D generation applications, such as trajectory- and command-driven generation, inpainting, and layout-conditioned generation. Extensive experiments on the CarlaSC and Waymo datasets demonstrate that DynamicCity significantly outperforms existing state-of-the-art 4D LiDAR generation methods across multiple metrics. The code will be released to facilitate future research.