基因表达：生成可探索世界

GenEx: Generating an Explorable World

December 12, 2024

作者: Taiming Lu, Tianmin Shu, Junfei Xiao, Luoxin Ye, Jiahao Wang, Cheng Peng, Chen Wei, Daniel Khashabi, Rama Chellappa, Alan Yuille, Jieneng Chen

cs.AI

摘要

理解、导航和探索3D物理真实世界长期以来一直是人工智能发展中的核心挑战。在这项工作中，我们朝着这个目标迈出了一步，引入了名为GenEx的系统，能够规划复杂的体验式世界探索，其指导思想是通过生成想象力形成关于周围环境的先验（期望）。GenEx能够从单个RGB图像生成整个3D一致的想象环境，并通过全景视频流将其栩栩如生地展现出来。利用从虚幻引擎策划的可扩展3D世界数据，我们的生成模型根植于物理世界。它能够轻松捕捉连续的360度环境，为人工智能代理提供一个无边界的景观供其探索和互动。GenEx实现了高质量的世界生成，在长轨迹上具有强大的循环一致性，并展示了强大的3D能力，如一致性和主动3D映射。借助对世界的生成想象力，由GPT辅助的代理能够执行复杂的体验式任务，包括无目标探索和目标驱动导航。这些代理利用对物理世界未见部分的预测期望来优化其信念，基于潜在决策模拟不同结果，并做出更明智的选择。总之，我们证明了GenEx为推动具有想象空间的体验式人工智能提供了一个变革性平台，并为将这些能力扩展到真实世界探索带来了潜力。

English

Understanding, navigating, and exploring the 3D physical real world has long been a central challenge in the development of artificial intelligence. In this work, we take a step toward this goal by introducing GenEx, a system capable of planning complex embodied world exploration, guided by its generative imagination that forms priors (expectations) about the surrounding environments. GenEx generates an entire 3D-consistent imaginative environment from as little as a single RGB image, bringing it to life through panoramic video streams. Leveraging scalable 3D world data curated from Unreal Engine, our generative model is rounded in the physical world. It captures a continuous 360-degree environment with little effort, offering a boundless landscape for AI agents to explore and interact with. GenEx achieves high-quality world generation, robust loop consistency over long trajectories, and demonstrates strong 3D capabilities such as consistency and active 3D mapping. Powered by generative imagination of the world, GPT-assisted agents are equipped to perform complex embodied tasks, including both goal-agnostic exploration and goal-driven navigation. These agents utilize predictive expectation regarding unseen parts of the physical world to refine their beliefs, simulate different outcomes based on potential decisions, and make more informed choices. In summary, we demonstrate that GenEx provides a transformative platform for advancing embodied AI in imaginative spaces and brings potential for extending these capabilities to real-world exploration.