基因表达:生成可探索世界
GenEx: Generating an Explorable World
December 12, 2024
作者: Taiming Lu, Tianmin Shu, Junfei Xiao, Luoxin Ye, Jiahao Wang, Cheng Peng, Chen Wei, Daniel Khashabi, Rama Chellappa, Alan Yuille, Jieneng Chen
cs.AI
摘要
理解、导航和探索3D物理真实世界长期以来一直是人工智能发展中的核心挑战。在这项工作中,我们朝着这个目标迈出了一步,引入了名为GenEx的系统,能够规划复杂的体验式世界探索,其指导思想是通过生成想象力形成关于周围环境的先验(期望)。GenEx能够从单个RGB图像生成整个3D一致的想象环境,并通过全景视频流将其栩栩如生地展现出来。利用从虚幻引擎策划的可扩展3D世界数据,我们的生成模型根植于物理世界。它能够轻松捕捉连续的360度环境,为人工智能代理提供一个无边界的景观供其探索和互动。GenEx实现了高质量的世界生成,在长轨迹上具有强大的循环一致性,并展示了强大的3D能力,如一致性和主动3D映射。借助对世界的生成想象力,由GPT辅助的代理能够执行复杂的体验式任务,包括无目标探索和目标驱动导航。这些代理利用对物理世界未见部分的预测期望来优化其信念,基于潜在决策模拟不同结果,并做出更明智的选择。总之,我们证明了GenEx为推动具有想象空间的体验式人工智能提供了一个变革性平台,并为将这些能力扩展到真实世界探索带来了潜力。
English
Understanding, navigating, and exploring the 3D physical real world has long
been a central challenge in the development of artificial intelligence. In this
work, we take a step toward this goal by introducing GenEx, a system capable of
planning complex embodied world exploration, guided by its generative
imagination that forms priors (expectations) about the surrounding
environments. GenEx generates an entire 3D-consistent imaginative environment
from as little as a single RGB image, bringing it to life through panoramic
video streams. Leveraging scalable 3D world data curated from Unreal Engine,
our generative model is rounded in the physical world. It captures a continuous
360-degree environment with little effort, offering a boundless landscape for
AI agents to explore and interact with. GenEx achieves high-quality world
generation, robust loop consistency over long trajectories, and demonstrates
strong 3D capabilities such as consistency and active 3D mapping. Powered by
generative imagination of the world, GPT-assisted agents are equipped to
perform complex embodied tasks, including both goal-agnostic exploration and
goal-driven navigation. These agents utilize predictive expectation regarding
unseen parts of the physical world to refine their beliefs, simulate different
outcomes based on potential decisions, and make more informed choices. In
summary, we demonstrate that GenEx provides a transformative platform for
advancing embodied AI in imaginative spaces and brings potential for extending
these capabilities to real-world exploration.Summary
AI-Generated Summary
GenEx系统
概述
- GenEx是由约翰霍普金斯大学团队开发的系统,能够从单个RGB图像生成想象中的世界并呈现为视频。
- 通过生成环境来规划实体世界探索,为AI代理提供无限探索空间。
重点
- 从单幅图像生成想象世界
- 支持AI执行复杂任务
- 提供360°全景环境
- 模拟人类认知过程
- 多种探索模式
关键词
- GenEx
- 想象世界
- AI代理
- 360°全景
- 探索模式
- 人类认知
- 多智能体
GenEx框架
模式
- 互动探索
- GPT辅助自由探索
- 目标驱动导航
特点
- 提高环境理解
- 避免模型崩溃
- 精确控制和导航
- 想象增强政策
- 多智能体应用
关键词
- 探索模式
- GPT-4o
- 目标导航
- 想象政策
- 多智能体应用
应用与前景
潜力
- 视频生成质量
- 探索一致性
- 3D生成
- 决策制定
- 提升认知能力
发展
- 物理基础数据
- 具身人工智能
- 决策准确性
- 共同愿景贡献
关键词
- 视频质量
- 3D生成
- 决策准确性
- 具身AI
- 共同愿景
- 认知能力
相关工作
前沿
- WorldLabs动漫世界生成
- DeepMind交互式世界模型
- 想象增强政策概念
- 框架支持多代理交互
贡献
- 项目领导
- 数据引擎
- 模型研究
- 等距圆柱投影
- 图像变换
关键词
- WorldLabs
- DeepMind
- 想象政策
- 多代理交互
- 圆柱投影
- 图像变换
- 模型研究
1比特LLM时代:所有大型语言模型均为1.58比特。The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
1比特LLM时代:所有大型语言模型均为1.58比特。
The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Li Dong, Ruiping Wang, Jilong Xue, Furu Wei•Feb 27, 2024•612142
DeepSeek-R1:通过强化学习激励LLMs中的推理能力DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via
Reinforcement Learning
DeepSeek-R1:通过强化学习激励LLMs中的推理能力
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via
Reinforcement Learning
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, Zhen Zhang•Jan 22, 2025•3685
Qwen2.5 技术报告Qwen2.5 Technical Report
Qwen2.5 技术报告
Qwen2.5 Technical Report
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zihan Qiu•Dec 19, 2024•36311