能量宇宙：构想机器人操作的具象未来空间

EnerVerse: Envisioning Embodied Future Space for Robotics Manipulation

January 3, 2025

作者: Siyuan Huang, Liliang Chen, Pengfei Zhou, Shengcong Chen, Zhengkai Jiang, Yue Hu, Peng Gao, Hongsheng Li, Maoqing Yao, Guanghui Ren

cs.AI

摘要

我们介绍了EnerVerse，这是一个专为机器人操作任务设计的全面框架，用于实体化未来空间生成。EnerVerse 无缝集成了卷积和双向注意机制，用于内部区块空间建模，确保低层次的一致性和连续性。鉴于视频数据中固有的冗余性，我们提出了稀疏记忆上下文，结合区块式单向生成范式，实现无限长序列的生成。为了进一步增强机器人的能力，我们引入了自由锚定视图（FAV）空间，提供灵活的视角以增强观察和分析。FAV 空间减轻了运动建模的模糊性，在受限环境中消除了物理约束，并显著提高了机器人在各种任务和环境中的泛化和适应能力。为了解决获取多摄像头观察的成本和劳动强度过高的问题，我们提出了一个数据引擎管道，将生成模型与四维高斯飞溅（4DGS）相结合。该管道利用生成模型的强大泛化能力和 4DGS 提供的空间约束，实现数据质量和多样性的迭代增强，从而创造出一种数据飞轮效应，有效缩小模拟与真实之间的差距。最后，我们的实验表明，实体化未来空间生成先验显著增强了策略预测能力，从而提高了整体性能，特别是在长距离机器人操作任务中。

English

We introduce EnerVerse, a comprehensive framework for embodied future space generation specifically designed for robotic manipulation tasks. EnerVerse seamlessly integrates convolutional and bidirectional attention mechanisms for inner-chunk space modeling, ensuring low-level consistency and continuity. Recognizing the inherent redundancy in video data, we propose a sparse memory context combined with a chunkwise unidirectional generative paradigm to enable the generation of infinitely long sequences. To further augment robotic capabilities, we introduce the Free Anchor View (FAV) space, which provides flexible perspectives to enhance observation and analysis. The FAV space mitigates motion modeling ambiguity, removes physical constraints in confined environments, and significantly improves the robot's generalization and adaptability across various tasks and settings. To address the prohibitive costs and labor intensity of acquiring multi-camera observations, we present a data engine pipeline that integrates a generative model with 4D Gaussian Splatting (4DGS). This pipeline leverages the generative model's robust generalization capabilities and the spatial constraints provided by 4DGS, enabling an iterative enhancement of data quality and diversity, thus creating a data flywheel effect that effectively narrows the sim-to-real gap. Finally, our experiments demonstrate that the embodied future space generation prior substantially enhances policy predictive capabilities, resulting in improved overall performance, particularly in long-range robotic manipulation tasks.