Paper Overview

Core Contribution

• Introduces MetaPlanOptimization (MPO) framework to enhance LLM agent planning capabilities.
• Provides a plug-and-play solution for improving task completion efficiency and generalization.
• Leverages high-level meta plans for explicit guidance and continuous optimization.

Research Context

• Addresses planning hallucinations and retraining challenges in LLM-based agents.
• Builds on prior work in implicit planning and explicit knowledge-guided approaches.

Keywords

• Meta Plan Optimization (MPO)
• Large Language Models (LLMs)
• Planning Hallucination
• Task Completion Efficiency
• Generalization

Background

Research Gap

• Existing methods suffer from planning hallucinations and require retraining for new agents.
• Lack of high-quality, generalizable explicit guidance for agent planning.

Technical Challenges

• Ensuring meta plan quality and adaptability across diverse tasks.
• Integrating meta plans into agent workflows without disrupting reasoning.

Prior Approaches

• Implicit planning methods (e.g., ReAct, Reflexion).
• Explicit knowledge-guided approaches (e.g., KnowAgent, WKM).
• Trajectory tuning methods (e.g., AgentTuning, ETO).

Methodology

Technical Architecture

• Meta Planner: Generates high-level meta plans.
• Agent: Executes tasks and provides feedback for meta plan optimization.

Implementation Details

• Supervised Fine-Tuning (SFT) for meta planner initialization.
• Monte Carlo (MC) sampling for meta plan quality evaluation.
• Direct Preference Optimization (DPO) for meta planner refinement.

Innovation Points

• Decouples meta plans from specific environmental details.
• Enables continuous optimization of meta plans based on agent feedback.
• Provides a plug-and-play solution compatible with existing frameworks.

Results

Experimental Setup

• Benchmarks: ALFWorld (embodied household tasks) and ScienceWorld (textual science experiment tasks).
• Base Models: GPT-4o, Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct, etc.

Key Findings

• MPO significantly outperforms baselines, with up to 100% performance improvement.
• Enhances task completion efficiency and generalization in unseen scenarios.
• Compatible with various agent frameworks, delivering additional performance gains.

Limitations

• Meta planner initialization relies on GPT-4o, which may introduce biases.
• Limited exploration of lightweight models for meta planner construction.
• Sampling methods for DPO training could be further optimized.