Real-is-Sim：透過動態數位分身縮小模擬與現實差距，實現真實世界機器人策略評估

Real-is-Sim: Bridging the Sim-to-Real Gap with a Dynamic Digital Twin for Real-World Robot Policy Evaluation

April 4, 2025

作者: Jad Abou-Chakra, Lingfeng Sun, Krishan Rana, Brandon May, Karl Schmeckpeper, Maria Vittoria Minniti, Laura Herlant

cs.AI

摘要

近期行為克隆技術的進步使機器人能夠執行複雜的操作任務。然而，準確評估訓練表現仍然具有挑戰性，特別是在實際應用中，因為行為克隆的損失通常與實際任務的成功率關聯性較差。因此，研究人員依賴於從耗時且成本高昂的實際評估中得出的成功率指標，這使得識別最優策略以及檢測過擬合或欠擬合變得不切實際。為了解決這些問題，我們提出了real-is-sim，這是一種新穎的行為克隆框架，該框架在整個策略開發流程（數據收集、訓練和部署）中整合了一個動態數字孿生（基於Embodied Gaussians）。通過持續將模擬世界與物理世界對齊，可以在現實世界中收集示範數據，並從模擬器中提取狀態。模擬器能夠通過從任意視角渲染圖像輸入或從場景中物體提取低層次狀態信息，來實現靈活的狀態表示。在訓練期間，策略可以在模擬器中以離線且高度並行化的方式直接進行評估。最後，在部署階段，策略在模擬器中運行，其中真實機器人直接追蹤模擬機器人的關節，有效地將策略執行與真實硬件解耦，並緩解了傳統的領域轉移挑戰。我們在PushT操作任務上驗證了real-is-sim，展示了模擬器中獲得的成功率與實際評估之間的強相關性。我們系統的視頻可以在https://realissim.rai-inst.com找到。

English

Recent advancements in behavior cloning have enabled robots to perform complex manipulation tasks. However, accurately assessing training performance remains challenging, particularly for real-world applications, as behavior cloning losses often correlate poorly with actual task success. Consequently, researchers resort to success rate metrics derived from costly and time-consuming real-world evaluations, making the identification of optimal policies and detection of overfitting or underfitting impractical. To address these issues, we propose real-is-sim, a novel behavior cloning framework that incorporates a dynamic digital twin (based on Embodied Gaussians) throughout the entire policy development pipeline: data collection, training, and deployment. By continuously aligning the simulated world with the physical world, demonstrations can be collected in the real world with states extracted from the simulator. The simulator enables flexible state representations by rendering image inputs from any viewpoint or extracting low-level state information from objects embodied within the scene. During training, policies can be directly evaluated within the simulator in an offline and highly parallelizable manner. Finally, during deployment, policies are run within the simulator where the real robot directly tracks the simulated robot's joints, effectively decoupling policy execution from real hardware and mitigating traditional domain-transfer challenges. We validate real-is-sim on the PushT manipulation task, demonstrating strong correlation between success rates obtained in the simulator and real-world evaluations. Videos of our system can be found at https://realissim.rai-inst.com.