CityDreamer4D：無限4D城市的組合生成模型

CityDreamer4D: Compositional Generative Model of Unbounded 4D Cities

January 15, 2025

作者: Haozhe Xie, Zhaoxi Chen, Fangzhou Hong, Ziwei Liu

cs.AI

摘要

近年來，3D場景生成受到越來越多的關注並取得了顯著進展。生成4D城市比3D場景更具挑戰性，因為其中包含結構複雜、視覺多樣的物體，如建築和車輛，以及人類對城市環境扭曲更敏感。為應對這些問題，我們提出了CityDreamer4D，一種專門為生成無限4D城市而設計的組合生成模型。我們的主要見解是：1）4D城市生成應將動態物體（例如車輛）與靜態場景（例如建築和道路）分開，以及2）4D場景中的所有物體應由不同類型的神經場組成，包括建築物、車輛和背景物品。具體而言，我們提出了交通情境生成器和無限佈局生成器，以使用高度緊湊的BEV表示來生成動態交通情境和靜態城市佈局。4D城市中的物體是通過結合面向物品和實例的神經場來生成的，用於背景物品、建築物和車輛。為了適應背景物品和實例的不同特徵，神經場採用定制的生成哈希網格和周期性位置嵌入作為場景參數化。此外，我們提供了一套全面的城市生成數據集，包括OSM、GoogleEarth和CityTopia。OSM數據集提供各種真實世界的城市佈局，而Google Earth和CityTopia數據集提供了包含3D實例標註的大規模高質量城市影像。借助其組合設計，CityDreamer4D支持一系列下游應用，如實例編輯、城市風格化和城市模擬，同時在生成逼真4D城市方面表現出色。

English

3D scene generation has garnered growing attention in recent years and has made significant progress. Generating 4D cities is more challenging than 3D scenes due to the presence of structurally complex, visually diverse objects like buildings and vehicles, and heightened human sensitivity to distortions in urban environments. To tackle these issues, we propose CityDreamer4D, a compositional generative model specifically tailored for generating unbounded 4D cities. Our main insights are 1) 4D city generation should separate dynamic objects (e.g., vehicles) from static scenes (e.g., buildings and roads), and 2) all objects in the 4D scene should be composed of different types of neural fields for buildings, vehicles, and background stuff. Specifically, we propose Traffic Scenario Generator and Unbounded Layout Generator to produce dynamic traffic scenarios and static city layouts using a highly compact BEV representation. Objects in 4D cities are generated by combining stuff-oriented and instance-oriented neural fields for background stuff, buildings, and vehicles. To suit the distinct characteristics of background stuff and instances, the neural fields employ customized generative hash grids and periodic positional embeddings as scene parameterizations. Furthermore, we offer a comprehensive suite of datasets for city generation, including OSM, GoogleEarth, and CityTopia. The OSM dataset provides a variety of real-world city layouts, while the Google Earth and CityTopia datasets deliver large-scale, high-quality city imagery complete with 3D instance annotations. Leveraging its compositional design, CityDreamer4D supports a range of downstream applications, such as instance editing, city stylization, and urban simulation, while delivering state-of-the-art performance in generating realistic 4D cities.