AutoVFX：從自然語言實現物理寫實的影片編輯指令

AutoVFX: Physically Realistic Video Editing from Natural Language Instructions

November 4, 2024

作者: Hao-Yu Hsu, Zhi-Hao Lin, Albert Zhai, Hongchi Xia, Shenlong Wang

cs.AI

摘要

現代視覺效果（VFX）軟體使熟練的藝術家能夠創造幾乎任何事物的影像。然而，創作過程仍然費時、複雜，並且對於一般用戶來說很難接觸。在這項研究中，我們提出了AutoVFX，這是一個框架，可以從單個視頻和自然語言指示自動創建逼真且動態的VFX視頻。通過精心整合神經場景建模、基於LLM的代碼生成和物理模擬，AutoVFX能夠提供具有物理基礎的、照片般逼真的編輯效果，並且可以直接使用自然語言指示進行控制。我們進行了大量實驗，以驗證AutoVFX在各種視頻和指示中的有效性。定量和定性結果表明，AutoVFX在生成質量、指示對齊、編輯多功能性和物理合理性方面遠遠優於所有競爭方法。

English

Modern visual effects (VFX) software has made it possible for skilled artists to create imagery of virtually anything. However, the creation process remains laborious, complex, and largely inaccessible to everyday users. In this work, we present AutoVFX, a framework that automatically creates realistic and dynamic VFX videos from a single video and natural language instructions. By carefully integrating neural scene modeling, LLM-based code generation, and physical simulation, AutoVFX is able to provide physically-grounded, photorealistic editing effects that can be controlled directly using natural language instructions. We conduct extensive experiments to validate AutoVFX's efficacy across a diverse spectrum of videos and instructions. Quantitative and qualitative results suggest that AutoVFX outperforms all competing methods by a large margin in generative quality, instruction alignment, editing versatility, and physical plausibility.