EnerVerse：展望機器人操作的具體未來空間

EnerVerse: Envisioning Embodied Future Space for Robotics Manipulation

January 3, 2025

作者: Siyuan Huang, Liliang Chen, Pengfei Zhou, Shengcong Chen, Zhengkai Jiang, Yue Hu, Peng Gao, Hongsheng Li, Maoqing Yao, Guanghui Ren

cs.AI

摘要

我們介紹了 EnerVerse，這是一個專為機器人操作任務設計的全面框架，用於生成具體未來空間。EnerVerse 無縫地整合了卷積和雙向注意機制，用於內部塊空間建模，確保低級別的一致性和連續性。為了認識視頻數據中固有的冗餘性，我們提出了稀疏記憶上下文，結合塊狀單向生成範式，實現無限長序列的生成。為了進一步增強機器人的能力，我們引入了自由錨視圖（FAV）空間，提供靈活的觀察和分析視角。FAV 空間減輕了運動建模的模糊性，在受限環境中消除了物理限制，顯著提高了機器人在各種任務和環境中的泛化和適應能力。為了應對獲取多攝像機觀察的成本高昂和勞動強度，我們提出了一個數據引擎管道，將生成模型與 4D 高斯擴散（4DGS）相結合。這個管道利用生成模型的強大泛化能力和 4DGS 提供的空間約束，實現數據質量和多樣性的迭代增強，從而創造出一種有效縮小模擬與現實之間差距的數據飛輪效應。最後，我們的實驗表明，具體未來空間生成先前顯著增強了策略預測能力，從而提高了整體性能，特別是在長距離機器人操作任務中。

English

We introduce EnerVerse, a comprehensive framework for embodied future space generation specifically designed for robotic manipulation tasks. EnerVerse seamlessly integrates convolutional and bidirectional attention mechanisms for inner-chunk space modeling, ensuring low-level consistency and continuity. Recognizing the inherent redundancy in video data, we propose a sparse memory context combined with a chunkwise unidirectional generative paradigm to enable the generation of infinitely long sequences. To further augment robotic capabilities, we introduce the Free Anchor View (FAV) space, which provides flexible perspectives to enhance observation and analysis. The FAV space mitigates motion modeling ambiguity, removes physical constraints in confined environments, and significantly improves the robot's generalization and adaptability across various tasks and settings. To address the prohibitive costs and labor intensity of acquiring multi-camera observations, we present a data engine pipeline that integrates a generative model with 4D Gaussian Splatting (4DGS). This pipeline leverages the generative model's robust generalization capabilities and the spatial constraints provided by 4DGS, enabling an iterative enhancement of data quality and diversity, thus creating a data flywheel effect that effectively narrows the sim-to-real gap. Finally, our experiments demonstrate that the embodied future space generation prior substantially enhances policy predictive capabilities, resulting in improved overall performance, particularly in long-range robotic manipulation tasks.