GenEx：生成可探索的世界

GenEx: Generating an Explorable World

December 12, 2024

作者: Taiming Lu, Tianmin Shu, Junfei Xiao, Luoxin Ye, Jiahao Wang, Cheng Peng, Chen Wei, Daniel Khashabi, Rama Chellappa, Alan Yuille, Jieneng Chen

cs.AI

摘要

理解、導航和探索3D物理現實世界一直是人工智慧發展中的一個核心挑戰。在這項工作中，我們朝著這個目標邁出了一步，引入了名為GenEx的系統，能夠通過其生成想像力來引導複雜的具身世界探索，形成對周圍環境的先驗（期望）。GenEx能夠從單一RGB圖像中生成一個完整的3D一致的想像環境，並通過全景視頻流將其呈現出來。利用從虛幻引擎中精心策劃的可擴展3D世界數據，我們的生成模型根植於物理世界。它輕鬆捕捉連續的360度環境，為人工智慧代理提供了無限的探索和互動空間。GenEx實現了高質量的世界生成，長軌跡上的強健循環一致性，展現了強大的3D能力，如一致性和主動3D映射。在世界的生成想像力的驅動下，得到GPT輔助的代理人能夠執行複雜的具身任務，包括無目標探索和目標驅動導航。這些代理人利用對物理世界未見部分的預測期望來優化其信念，基於潛在決策模擬不同結果，並做出更明智的選擇。總之，我們展示了GenEx為推進具身人工智慧在想像空間中提供了一個革命性平台，並為將這些能力擴展到現實世界探索帶來了潛力。

English

Understanding, navigating, and exploring the 3D physical real world has long been a central challenge in the development of artificial intelligence. In this work, we take a step toward this goal by introducing GenEx, a system capable of planning complex embodied world exploration, guided by its generative imagination that forms priors (expectations) about the surrounding environments. GenEx generates an entire 3D-consistent imaginative environment from as little as a single RGB image, bringing it to life through panoramic video streams. Leveraging scalable 3D world data curated from Unreal Engine, our generative model is rounded in the physical world. It captures a continuous 360-degree environment with little effort, offering a boundless landscape for AI agents to explore and interact with. GenEx achieves high-quality world generation, robust loop consistency over long trajectories, and demonstrates strong 3D capabilities such as consistency and active 3D mapping. Powered by generative imagination of the world, GPT-assisted agents are equipped to perform complex embodied tasks, including both goal-agnostic exploration and goal-driven navigation. These agents utilize predictive expectation regarding unseen parts of the physical world to refine their beliefs, simulate different outcomes based on potential decisions, and make more informed choices. In summary, we demonstrate that GenEx provides a transformative platform for advancing embodied AI in imaginative spaces and brings potential for extending these capabilities to real-world exploration.