PlotGen：基于多智能体LLM的科学数据可视化通过多模态反馈

PlotGen: Multi-Agent LLM-based Scientific Data Visualization via Multimodal Feedback

February 3, 2025

作者: Kanika Goswami, Puneet Mathur, Ryan Rossi, Franck Dernoncourt

cs.AI

摘要

科学数据可视化对于将原始数据转化为可理解的视觉表达至关重要，实现模式识别、预测以及呈现基于数据的见解。然而，新手用户常常面临困难，因为选择适当工具和掌握可视化技术的复杂性。大型语言模型（LLMs）最近展示了在辅助代码生成方面的潜力，尽管它们在准确性方面存在困难，并需要迭代调试。在本文中，我们提出了PlotGen，这是一个旨在自动化创建精确科学可视化的新型多代理框架。PlotGen协调多个基于LLM的代理，包括一个查询规划代理，将复杂用户请求分解为可执行步骤，一个代码生成代理，将伪代码转换为可执行的Python代码，以及三个检索反馈代理 - 数值反馈代理、词汇反馈代理和视觉反馈代理 - 利用多模式LLMs通过自我反思迭代地改进生成图的数据准确性、文本标签和视觉正确性。大量实验证明，PlotGen优于强基线，在MatPlotBench数据集上实现了4-6%的改进，提高了用户对LLM生成的可视化的信任，并由于减少了用于处理图表错误所需的调试时间，提高了新手用户的生产效率。

English

Scientific data visualization is pivotal for transforming raw data into comprehensible visual representations, enabling pattern recognition, forecasting, and the presentation of data-driven insights. However, novice users often face difficulties due to the complexity of selecting appropriate tools and mastering visualization techniques. Large Language Models (LLMs) have recently demonstrated potential in assisting code generation, though they struggle with accuracy and require iterative debugging. In this paper, we propose PlotGen, a novel multi-agent framework aimed at automating the creation of precise scientific visualizations. PlotGen orchestrates multiple LLM-based agents, including a Query Planning Agent that breaks down complex user requests into executable steps, a Code Generation Agent that converts pseudocode into executable Python code, and three retrieval feedback agents - a Numeric Feedback Agent, a Lexical Feedback Agent, and a Visual Feedback Agent - that leverage multimodal LLMs to iteratively refine the data accuracy, textual labels, and visual correctness of generated plots via self-reflection. Extensive experiments show that PlotGen outperforms strong baselines, achieving a 4-6 percent improvement on the MatPlotBench dataset, leading to enhanced user trust in LLM-generated visualizations and improved novice productivity due to a reduction in debugging time needed for plot errors.