SketchAgent：基于语言驱动的序列草图生成

SketchAgent: Language-Driven Sequential Sketch Generation

November 26, 2024

作者: Yael Vinker, Tamar Rott Shaham, Kristine Zheng, Alex Zhao, Judith E Fan, Antonio Torralba

cs.AI

摘要

素描作为一种多才多艺的工具，用于外化思想，实现快速探索和跨越各个学科领域的视觉交流。虽然人工系统在内容创作和人机交互方面取得了重大进展，但捕捉人类素描的动态和抽象特性仍然具有挑战性。在这项工作中，我们介绍了SketchAgent，这是一种基于语言驱动的、顺序素描生成方法，使用户能够通过动态的对话交互来创建、修改和完善素描。我们的方法不需要训练或微调。相反，我们利用现成的多模态大型语言模型（LLMs）的顺序特性和丰富的先验知识。我们提出了一种直观的素描语言，通过上下文示例引入模型，使其能够使用基于字符串的操作来“绘图”。这些操作被处理成矢量图形，然后呈现为在像素画布上创建的素描，可以再次访问以进行进一步的任务。通过逐笔绘制，我们的代理捕捉了素描固有的不断发展的动态特性。我们证明了SketchAgent能够从不同的提示中生成素描，进行基于对话的绘图，并与人类用户进行有意义的合作。

English

Sketching serves as a versatile tool for externalizing ideas, enabling rapid exploration and visual communication that spans various disciplines. While artificial systems have driven substantial advances in content creation and human-computer interaction, capturing the dynamic and abstract nature of human sketching remains challenging. In this work, we introduce SketchAgent, a language-driven, sequential sketch generation method that enables users to create, modify, and refine sketches through dynamic, conversational interactions. Our approach requires no training or fine-tuning. Instead, we leverage the sequential nature and rich prior knowledge of off-the-shelf multimodal large language models (LLMs). We present an intuitive sketching language, introduced to the model through in-context examples, enabling it to "draw" using string-based actions. These are processed into vector graphics and then rendered to create a sketch on a pixel canvas, which can be accessed again for further tasks. By drawing stroke by stroke, our agent captures the evolving, dynamic qualities intrinsic to sketching. We demonstrate that SketchAgent can generate sketches from diverse prompts, engage in dialogue-driven drawing, and collaborate meaningfully with human users.