DreamWaltz-G：從骨架引導的2D擴散生成具表現力的3D高斯化身

DreamWaltz-G: Expressive 3D Gaussian Avatars from Skeleton-Guided 2D Diffusion

September 25, 2024

作者: Yukun Huang, Jianan Wang, Ailing Zeng, Zheng-Jun Zha, Lei Zhang, Xihui Liu

cs.AI

摘要

借助預訓練的2D擴散模型和分數蒸餾採樣（SDS），最近的方法展示了在從文本生成3D頭像方面取得了令人期待的結果。然而，生成具有表現力動畫能力的高質量3D頭像仍然具有挑戰性。在這項工作中，我們提出了一種名為DreamWaltz-G的新型學習框架，用於從文本生成可動畫的3D頭像。該框架的核心在於骨架引導的分數蒸餾和混合3D高斯頭像表示。具體而言，所提出的骨架引導的分數蒸餾將3D人類模板的骨架控制整合到2D擴散模型中，增強了SDS監督在視角和人體姿勢方面的一致性。這有助於生成高質量的頭像，減輕了多個臉部、額外肢體和模糊等問題。所提出的混合3D高斯頭像表示建立在高效的3D高斯基礎上，結合了神經隱式場和參數化的3D網格，實現了實時渲染、穩定的SDS優化和表現力豐富的動畫。大量實驗表明，DreamWaltz-G在生成和動畫化3D頭像方面非常有效，在視覺質量和動畫表現方面優於現有方法。我們的框架進一步支持各種應用，包括人類視頻再現和多主題場景合成。

English

Leveraging pretrained 2D diffusion models and score distillation sampling (SDS), recent methods have shown promising results for text-to-3D avatar generation. However, generating high-quality 3D avatars capable of expressive animation remains challenging. In this work, we present DreamWaltz-G, a novel learning framework for animatable 3D avatar generation from text. The core of this framework lies in Skeleton-guided Score Distillation and Hybrid 3D Gaussian Avatar representation. Specifically, the proposed skeleton-guided score distillation integrates skeleton controls from 3D human templates into 2D diffusion models, enhancing the consistency of SDS supervision in terms of view and human pose. This facilitates the generation of high-quality avatars, mitigating issues such as multiple faces, extra limbs, and blurring. The proposed hybrid 3D Gaussian avatar representation builds on the efficient 3D Gaussians, combining neural implicit fields and parameterized 3D meshes to enable real-time rendering, stable SDS optimization, and expressive animation. Extensive experiments demonstrate that DreamWaltz-G is highly effective in generating and animating 3D avatars, outperforming existing methods in both visual quality and animation expressiveness. Our framework further supports diverse applications, including human video reenactment and multi-subject scene composition.