TalkinNeRF：可動畫神經場用於全身說話人類

TalkinNeRF: Animatable Neural Fields for Full-Body Talking Humans

September 25, 2024

作者: Aggelina Chatziagapi, Bindita Chaudhuri, Amit Kumar, Rakesh Ranjan, Dimitris Samaras, Nikolaos Sarafianos

cs.AI

摘要

我們提出了一個新穎的框架，從單眼視頻中學習動態神經輻射場（NeRF）以呈現全身說話的人類。先前的研究僅表示身體姿勢或臉部。然而，人類通過全身溝通，結合身體姿勢、手勢以及面部表情。在這項工作中，我們提出了TalkinNeRF，一個統一的基於NeRF的網絡，代表整體的4D人體運動。給定一個主題的單眼視頻，我們學習相應的身體、臉部和手部模塊，將它們結合在一起生成最終結果。為了捕捉複雜的手指關節運動，我們為手部學習了一個額外的變形場。我們的多身份表示使得能夠同時訓練多個主題，並在完全看不見的姿勢下實現強大的動畫。它還可以推廣到新的身份，僅需一個簡短的視頻作為輸入。我們展示了在動畫全身說話的人類時具有最先進性能，具有細緻的手部關節運動和面部表情。

English

We introduce a novel framework that learns a dynamic neural radiance field (NeRF) for full-body talking humans from monocular videos. Prior work represents only the body pose or the face. However, humans communicate with their full body, combining body pose, hand gestures, as well as facial expressions. In this work, we propose TalkinNeRF, a unified NeRF-based network that represents the holistic 4D human motion. Given a monocular video of a subject, we learn corresponding modules for the body, face, and hands, that are combined together to generate the final result. To capture complex finger articulation, we learn an additional deformation field for the hands. Our multi-identity representation enables simultaneous training for multiple subjects, as well as robust animation under completely unseen poses. It can also generalize to novel identities, given only a short video as input. We demonstrate state-of-the-art performance for animating full-body talking humans, with fine-grained hand articulation and facial expressions.