統御一切：自然語言以統合溝通、感知和行動。

One to rule them all: natural language to bind communication, perception and action

November 22, 2024

作者: Simone Colombani, Dimitri Ognibene, Giuseppe Boccignone

cs.AI

摘要

近年來，人機互動領域的研究集中在開發能夠理解複雜人類指令並在動態多樣環境中執行任務的機器人上。這些系統具有廣泛的應用，從個人輔助到工業機器人，強調機器人靈活、自然且安全地與人類互動的重要性。本文提出了一種先進的機器人行動規劃架構，該架構將通信、感知和規劃與大型語言模型（LLMs）相結合。我們的系統旨在將用自然語言表達的指令轉換為可執行的機器人動作，並根據實時反饋動態更新計劃，同時納入環境信息。規劃模組是系統的核心，其中嵌入在修改後的ReAct框架中的LLMs被用來解釋並執行用戶指令。通過利用它們豐富的預訓練知識，LLMs能夠有效處理用戶請求，無需引入新的關於變化環境的知識。修改後的ReAct框架通過提供實時環境感知和物理行動結果進一步增強了執行空間。通過將堅固且動態的語義地圖表示作為圖形與控制組件和失敗解釋相結合，該架構增強了機器人在共享和動態環境中的適應性、任務執行和與人類用戶的無縫協作。通過將連續反饋環境的回路與系統整合，系統可以動態調整計劃以應對意外變化，從而優化機器人執行任務的能力。利用以前經驗的數據集，可以提供有關失敗的詳細反饋。更新下一次迭代的LLMs上下文，並提出如何克服問題的建議。

English

In recent years, research in the area of human-robot interaction has focused on developing robots capable of understanding complex human instructions and performing tasks in dynamic and diverse environments. These systems have a wide range of applications, from personal assistance to industrial robotics, emphasizing the importance of robots interacting flexibly, naturally and safely with humans. This paper presents an advanced architecture for robotic action planning that integrates communication, perception, and planning with Large Language Models (LLMs). Our system is designed to translate commands expressed in natural language into executable robot actions, incorporating environmental information and dynamically updating plans based on real-time feedback. The Planner Module is the core of the system where LLMs embedded in a modified ReAct framework are employed to interpret and carry out user commands. By leveraging their extensive pre-trained knowledge, LLMs can effectively process user requests without the need to introduce new knowledge on the changing environment. The modified ReAct framework further enhances the execution space by providing real-time environmental perception and the outcomes of physical actions. By combining robust and dynamic semantic map representations as graphs with control components and failure explanations, this architecture enhances a robot adaptability, task execution, and seamless collaboration with human users in shared and dynamic environments. Through the integration of continuous feedback loops with the environment the system can dynamically adjusts the plan to accommodate unexpected changes, optimizing the robot ability to perform tasks. Using a dataset of previous experience is possible to provide detailed feedback about the failure. Updating the LLMs context of the next iteration with suggestion on how to overcame the issue.