BlenderGym：图形编辑基础模型系统基准测试平台

BlenderGym: Benchmarking Foundational Model Systems for Graphics Editing

April 2, 2025

作者: Yunqi Gu, Ian Huang, Jihyeon Je, Guandao Yang, Leonidas Guibas

cs.AI

摘要

在电影制作和游戏设计等应用中，3D图形编辑至关重要，然而这一过程依然耗时且需要高度专业化的领域知识。自动化这一过程颇具挑战，因为图形编辑涉及执行多种任务，每种任务都需要不同的技能组合。最近，视觉-语言模型（VLMs）作为一种强大的框架崭露头角，用于自动化编辑流程，但其开发与评估却因缺乏一个要求人类级别感知并呈现真实世界编辑复杂性的综合基准而受阻。在本研究中，我们推出了BlenderGym，这是首个针对3D图形编辑的全面VLM系统基准。BlenderGym通过基于代码的3D重建任务来评估VLM系统。我们对闭源和开源的VLM系统进行了评估，发现即便是最先进的VLM系统，在处理对人类Blender用户相对容易的任务时也显得力不从心。借助BlenderGym，我们研究了推理扩展技术如何影响VLM在图形编辑任务上的表现。值得注意的是，我们的发现表明，用于指导生成扩展的验证器本身也能通过推理扩展得到改进，这补充了最近关于LLM生成在编码和数学任务中推理扩展的见解。我们进一步展示了推理计算并非均匀有效，可以通过在生成与验证之间策略性地分配计算资源来优化其效果。

English

3D graphics editing is crucial in applications like movie production and game design, yet it remains a time-consuming process that demands highly specialized domain expertise. Automating this process is challenging because graphical editing requires performing a variety of tasks, each requiring distinct skill sets. Recently, vision-language models (VLMs) have emerged as a powerful framework for automating the editing process, but their development and evaluation are bottlenecked by the lack of a comprehensive benchmark that requires human-level perception and presents real-world editing complexity. In this work, we present BlenderGym, the first comprehensive VLM system benchmark for 3D graphics editing. BlenderGym evaluates VLM systems through code-based 3D reconstruction tasks. We evaluate closed- and open-source VLM systems and observe that even the state-of-the-art VLM system struggles with tasks relatively easy for human Blender users. Enabled by BlenderGym, we study how inference scaling techniques impact VLM's performance on graphics editing tasks. Notably, our findings reveal that the verifier used to guide the scaling of generation can itself be improved through inference scaling, complementing recent insights on inference scaling of LLM generation in coding and math tasks. We further show that inference compute is not uniformly effective and can be optimized by strategically distributing it between generation and verification.