真实即仿真：通过动态数字孪生技术弥合仿真与现实差距，实现现实世界机器人策略评估

Real-is-Sim: Bridging the Sim-to-Real Gap with a Dynamic Digital Twin for Real-World Robot Policy Evaluation

April 4, 2025

作者: Jad Abou-Chakra, Lingfeng Sun, Krishan Rana, Brandon May, Karl Schmeckpeper, Maria Vittoria Minniti, Laura Herlant

cs.AI

摘要

近期行为克隆技术的进步已使机器人能够执行复杂的操作任务。然而，准确评估训练性能仍具挑战性，尤其是在现实世界应用中，因为行为克隆损失往往与实际任务成功率关联性较差。因此，研究人员不得不依赖于耗时且成本高昂的现实世界评估得出的成功率指标，这使得识别最优策略及检测过拟合或欠拟合变得不切实际。为解决这些问题，我们提出了real-is-sim，一种新颖的行为克隆框架，该框架在整个策略开发流程（数据收集、训练与部署）中整合了动态数字孪生体（基于Embodied Gaussians）。通过持续将模拟世界与物理世界对齐，可以在现实世界中收集演示，同时从模拟器中提取状态信息。模拟器通过渲染任意视角的图像输入或从场景中物体提取低级状态信息，实现了灵活的状态表示。在训练期间，策略可以直接在模拟器内以离线且高度并行化的方式进行评估。最后，在部署阶段，策略在模拟器内运行，真实机器人直接跟踪模拟机器人的关节，有效解耦了策略执行与真实硬件，缓解了传统的领域迁移挑战。我们在PushT操作任务上验证了real-is-sim，展示了模拟器内成功率与现实世界评估结果之间的强相关性。系统视频可访问https://realissim.rai-inst.com。

English

Recent advancements in behavior cloning have enabled robots to perform complex manipulation tasks. However, accurately assessing training performance remains challenging, particularly for real-world applications, as behavior cloning losses often correlate poorly with actual task success. Consequently, researchers resort to success rate metrics derived from costly and time-consuming real-world evaluations, making the identification of optimal policies and detection of overfitting or underfitting impractical. To address these issues, we propose real-is-sim, a novel behavior cloning framework that incorporates a dynamic digital twin (based on Embodied Gaussians) throughout the entire policy development pipeline: data collection, training, and deployment. By continuously aligning the simulated world with the physical world, demonstrations can be collected in the real world with states extracted from the simulator. The simulator enables flexible state representations by rendering image inputs from any viewpoint or extracting low-level state information from objects embodied within the scene. During training, policies can be directly evaluated within the simulator in an offline and highly parallelizable manner. Finally, during deployment, policies are run within the simulator where the real robot directly tracks the simulated robot's joints, effectively decoupling policy execution from real hardware and mitigating traditional domain-transfer challenges. We validate real-is-sim on the PushT manipulation task, demonstrating strong correlation between success rates obtained in the simulator and real-world evaluations. Videos of our system can be found at https://realissim.rai-inst.com.