AutoVFX：从自然语言实现物理真实视频编辑 指南

AutoVFX: Physically Realistic Video Editing from Natural Language Instructions

November 4, 2024

作者: Hao-Yu Hsu, Zhi-Hao Lin, Albert Zhai, Hongchi Xia, Shenlong Wang

cs.AI

摘要

现代视觉效果（VFX）软件使熟练艺术家能够创作几乎任何图像。然而，创作过程仍然费时费力、复杂，并且大多数普通用户难以接触。在这项工作中，我们提出了AutoVFX，这是一个框架，可以从单个视频和自然语言指令自动创建逼真且动态的VFX视频。通过精心整合神经场景建模、基于LLM的代码生成和物理模拟，AutoVFX能够提供基于物理的、逼真的编辑效果，可以直接使用自然语言指令进行控制。我们进行了大量实验证实AutoVFX在各种视频和指令上的有效性。定量和定性结果表明，AutoVFX在生成质量、指令对齐、编辑多样性和物理合理性方面远远优于所有竞争方法。

English

Modern visual effects (VFX) software has made it possible for skilled artists to create imagery of virtually anything. However, the creation process remains laborious, complex, and largely inaccessible to everyday users. In this work, we present AutoVFX, a framework that automatically creates realistic and dynamic VFX videos from a single video and natural language instructions. By carefully integrating neural scene modeling, LLM-based code generation, and physical simulation, AutoVFX is able to provide physically-grounded, photorealistic editing effects that can be controlled directly using natural language instructions. We conduct extensive experiments to validate AutoVFX's efficacy across a diverse spectrum of videos and instructions. Quantitative and qualitative results suggest that AutoVFX outperforms all competing methods by a large margin in generative quality, instruction alignment, editing versatility, and physical plausibility.