一统天下：自然语言以统一沟通、感知和行动。

One to rule them all: natural language to bind communication, perception and action

November 22, 2024

作者: Simone Colombani, Dimitri Ognibene, Giuseppe Boccignone

cs.AI

摘要

近年来，人机交互领域的研究集中在开发能够理解复杂人类指令并在动态多样环境中执行任务的机器人上。这些系统具有广泛的应用，从个人辅助到工业机器人，强调了机器人与人类灵活、自然和安全互动的重要性。本文提出了一种先进的机器人行动规划架构，将通信、感知和规划与大型语言模型（LLMs）相结合。我们的系统旨在将用自然语言表达的指令转化为可执行的机器人动作，融合环境信息，并根据实时反馈动态更新计划。规划模块是系统的核心，其中嵌入在修改后的ReAct框架中的LLMs被用于解释和执行用户指令。通过利用它们广泛的预训练知识，LLMs能够有效处理用户请求，无需引入新的关于变化环境的知识。修改后的ReAct框架通过提供实时环境感知和物理行动结果进一步增强了执行空间。通过将稳健且动态的语义地图表示作为图形与控制组件和失败解释相结合，该架构增强了机器人在共享和动态环境中的适应性、任务执行能力和与人类用户的无缝协作。通过将连续反馈循环与环境整合，系统可以动态调整计划以适应意外变化，优化机器人执行任务的能力。利用先前经验数据集可以提供有关失败的详细反馈。更新下一次迭代的LLMs上下文，并提出如何克服问题的建议。

English

In recent years, research in the area of human-robot interaction has focused on developing robots capable of understanding complex human instructions and performing tasks in dynamic and diverse environments. These systems have a wide range of applications, from personal assistance to industrial robotics, emphasizing the importance of robots interacting flexibly, naturally and safely with humans. This paper presents an advanced architecture for robotic action planning that integrates communication, perception, and planning with Large Language Models (LLMs). Our system is designed to translate commands expressed in natural language into executable robot actions, incorporating environmental information and dynamically updating plans based on real-time feedback. The Planner Module is the core of the system where LLMs embedded in a modified ReAct framework are employed to interpret and carry out user commands. By leveraging their extensive pre-trained knowledge, LLMs can effectively process user requests without the need to introduce new knowledge on the changing environment. The modified ReAct framework further enhances the execution space by providing real-time environmental perception and the outcomes of physical actions. By combining robust and dynamic semantic map representations as graphs with control components and failure explanations, this architecture enhances a robot adaptability, task execution, and seamless collaboration with human users in shared and dynamic environments. Through the integration of continuous feedback loops with the environment the system can dynamically adjusts the plan to accommodate unexpected changes, optimizing the robot ability to perform tasks. Using a dataset of previous experience is possible to provide detailed feedback about the failure. Updating the LLMs context of the next iteration with suggestion on how to overcame the issue.