GenSim2：利用多模態和推理擴展機器人數據生成LLM。

GenSim2: Scaling Robot Data Generation with Multi-modal and Reasoning LLMs

October 4, 2024

作者: Pu Hua, Minghuan Liu, Annabella Macaluso, Yunfeng Lin, Weinan Zhang, Huazhe Xu, Lirui Wang

cs.AI

摘要

如今，由於需要人力來創建多樣化的模擬任務和場景，機器人模擬仍然具有難以擴展的挑戰性。同時，由於許多模擬到真實方法專注於單一任務，受到規模化問題的影響，基於模擬訓練的策略也面臨著擴展性問題。為應對這些挑戰，本研究提出了GenSim2，這是一個可擴展的框架，利用具有多模態和推理能力的編碼LLMs來創建複雜且逼真的模擬任務，包括具有關節物體的長視程任務。為了自動地大規模生成這些任務的示範數據，我們提出了計劃和RL求解器，可以在物體類別內進行泛化。這個流程可以為多達100個有關節的任務生成數據，包含200個物體，並減少所需的人力。為了利用這些數據，我們提出了一種有效的多任務語言條件策略架構，名為本體感知點雲變換器（PPT），它可以從生成的示範中學習，展現出強大的模擬到真實的零-shot轉移。結合所提出的流程和策略架構，我們展示了GenSim2的一個有前途的應用，即生成的數據可以用於零-shot轉移或與真實收集的數據共同訓練，這樣相比僅在有限的真實數據上訓練，可以將策略性能提高20%。

English

Robotic simulation today remains challenging to scale up due to the human efforts required to create diverse simulation tasks and scenes. Simulation-trained policies also face scalability issues as many sim-to-real methods focus on a single task. To address these challenges, this work proposes GenSim2, a scalable framework that leverages coding LLMs with multi-modal and reasoning capabilities for complex and realistic simulation task creation, including long-horizon tasks with articulated objects. To automatically generate demonstration data for these tasks at scale, we propose planning and RL solvers that generalize within object categories. The pipeline can generate data for up to 100 articulated tasks with 200 objects and reduce the required human efforts. To utilize such data, we propose an effective multi-task language-conditioned policy architecture, dubbed proprioceptive point-cloud transformer (PPT), that learns from the generated demonstrations and exhibits strong sim-to-real zero-shot transfer. Combining the proposed pipeline and the policy architecture, we show a promising usage of GenSim2 that the generated data can be used for zero-shot transfer or co-train with real-world collected data, which enhances the policy performance by 20% compared with training exclusively on limited real data.