协作实例导航：利用代理自我对话最小化用户输入

摘要

现有的基于具体实例目标导航任务，是由自然语言驱动的，假设人类用户在导航之前提供完整而细致的实例描述，然而在现实世界中，人类指令可能简短且含糊不清，这在实践中可能并不可行。为了弥合这一差距，我们提出了一项新任务，即协作式实例导航（CoIN），在导航过程中动态地进行智能体与人类的交互，以积极解决关于目标实例的不确定性，采用自然、无模板、开放式对话。为了解决CoIN问题，我们提出了一种新颖的方法，即具有不确定性感知的智能体-用户交互（AIUTA），利用视觉语言模型（VLMs）的感知能力和大型语言模型（LLMs）的能力。首先，在目标检测后，一个自我提问者模型启动自我对话，以获得完整准确的观察描述，同时一种新颖的不确定性估计技术减轻了VLM感知的不准确性。然后，一个交互触发模块确定是否向用户提问、继续或停止导航，从而最大程度地减少用户输入。为了评估，我们引入了CoIN-Bench，一个支持真实和模拟人类的基准测试。AIUTA在实例导航中取得了与最先进方法竞争力相当的表现，展示了处理用户输入时的极大灵活性。

English

Existing embodied instance goal navigation tasks, driven by natural language, assume human users to provide complete and nuanced instance descriptions prior to the navigation, which can be impractical in the real world as human instructions might be brief and ambiguous. To bridge this gap, we propose a new task, Collaborative Instance Navigation (CoIN), with dynamic agent-human interaction during navigation to actively resolve uncertainties about the target instance in natural, template-free, open-ended dialogues. To address CoIN, we propose a novel method, Agent-user Interaction with UncerTainty Awareness (AIUTA), leveraging the perception capability of Vision Language Models (VLMs) and the capability of Large Language Models (LLMs). First, upon object detection, a Self-Questioner model initiates a self-dialogue to obtain a complete and accurate observation description, while a novel uncertainty estimation technique mitigates inaccurate VLM perception. Then, an Interaction Trigger module determines whether to ask a question to the user, continue or halt navigation, minimizing user input. For evaluation, we introduce CoIN-Bench, a benchmark supporting both real and simulated humans. AIUTA achieves competitive performance in instance navigation against state-of-the-art methods, demonstrating great flexibility in handling user inputs.

协作实例导航：利用代理自我对话最小化用户输入

Collaborative Instance Navigation: Leveraging Agent Self-Dialogue to Minimize User Input

摘要

Summary

Support

Support