引導您的通才：通過價值指導改善機器人基礎模型

Steering Your Generalists: Improving Robotic Foundation Models via Value Guidance

October 17, 2024

作者: Mitsuhiko Nakamoto, Oier Mees, Aviral Kumar, Sergey Levine

cs.AI

摘要

通過在多樣化示範數據集上訓練的大型、通用型機器人策略已被證明在控制各種機器人在不同場景中以及獲取廣泛操作技能方面非常有效。然而，這些策略訓練所使用的數據通常質量不一 -- 人類收集的示範不太可能完美執行任務，而且數據集越大，就越難精選出最高質量的示例。另外，目前還不清楚從一個實體中獲取的最佳數據對於在另一個實體上進行訓練的效果如何。本文提出了一種通用且廣泛適用的方法，在部署時通過根據通過離線強化學習學習的值函數對其行動重新排序，從而提高這些通用型機器人策略的性能。這種方法被稱為價值引導策略引導（V-GPS），與各種不同的通用策略兼容，無需微調甚至訪問策略的權重。我們展示了相同的值函數如何提高五種不同架構的最新策略的性能，即使它們是在不同數據集上訓練的，也實現了在12個任務上多個機器人平台上的一致性性能改進。代碼和視頻可在以下網址找到：https://nakamotoo.github.io/V-GPS

English

Large, general-purpose robotic policies trained on diverse demonstration datasets have been shown to be remarkably effective both for controlling a variety of robots in a range of different scenes, and for acquiring broad repertoires of manipulation skills. However, the data that such policies are trained on is generally of mixed quality -- not only are human-collected demonstrations unlikely to perform the task perfectly, but the larger the dataset is, the harder it is to curate only the highest quality examples. It also remains unclear how optimal data from one embodiment is for training on another embodiment. In this paper, we present a general and broadly applicable approach that enhances the performance of such generalist robot policies at deployment time by re-ranking their actions according to a value function learned via offline RL. This approach, which we call Value-Guided Policy Steering (V-GPS), is compatible with a wide range of different generalist policies, without needing to fine-tune or even access the weights of the policy. We show that the same value function can improve the performance of five different state-of-the-art policies with different architectures, even though they were trained on distinct datasets, attaining consistent performance improvement on multiple robotic platforms across a total of 12 tasks. Code and videos can be found at: https://nakamotoo.github.io/V-GPS