可重新照明的全身高斯编解码化身

Relightable Full-Body Gaussian Codec Avatars

January 24, 2025

作者: Shaofei Wang, Tomas Simon, Igor Santesteban, Timur Bagautdinov, Junxuan Li, Vasu Agrawal, Fabian Prada, Shoou-I Yu, Pace Nalbone, Matt Gramlich, Roman Lubachersky, Chenglei Wu, Javier Romero, Jason Saragih, Michael Zollhoefer, Andreas Geiger, Siyu Tang, Shunsuke Saito

cs.AI

摘要

我们提出了一种新方法，称为可重照全身高斯编解码化身，用于建模具有面部和手部等细致细节的可重照全身化身。可重照全身化身面临的独特挑战在于由身体关节运动引起的大变形，以及光传输造成的外观影响。身体姿势的变化可以显著改变身体表面相对于光源的方向，导致局部外观变化，这是由于局部光传输函数的变化，以及由于身体部位之间的遮挡而导致的非局部变化。为了解决这个问题，我们将光传输分解为局部和非局部效应。局部外观变化使用可学习的区域谐波来建模漫反射辐射传输。与球谐波不同，区域谐波在关节运动下旋转非常高效。这使我们能够在局部坐标系中学习漫反射辐射传输，从而将局部辐射传输与身体的关节运动分离开来。为了考虑非局部外观变化，我们引入了一个阴影网络，根据基础网格上预先计算的入射辐照度来预测阴影。这有助于学习身体部位之间的非局部阴影。最后，我们采用延迟着色方法来建模镜面辐射传输，并更好地捕捉反射和高光，如眼睛的闪光。我们展示了我们的方法成功地模拟了可重照全身化身所需的局部和非局部光传输，具有在新颖照明条件和未见姿势下的卓越泛化能力。

English

We propose Relightable Full-Body Gaussian Codec Avatars, a new approach for modeling relightable full-body avatars with fine-grained details including face and hands. The unique challenge for relighting full-body avatars lies in the large deformations caused by body articulation and the resulting impact on appearance caused by light transport. Changes in body pose can dramatically change the orientation of body surfaces with respect to lights, resulting in both local appearance changes due to changes in local light transport functions, as well as non-local changes due to occlusion between body parts. To address this, we decompose the light transport into local and non-local effects. Local appearance changes are modeled using learnable zonal harmonics for diffuse radiance transfer. Unlike spherical harmonics, zonal harmonics are highly efficient to rotate under articulation. This allows us to learn diffuse radiance transfer in a local coordinate frame, which disentangles the local radiance transfer from the articulation of the body. To account for non-local appearance changes, we introduce a shadow network that predicts shadows given precomputed incoming irradiance on a base mesh. This facilitates the learning of non-local shadowing between the body parts. Finally, we use a deferred shading approach to model specular radiance transfer and better capture reflections and highlights such as eye glints. We demonstrate that our approach successfully models both the local and non-local light transport required for relightable full-body avatars, with a superior generalization ability under novel illumination conditions and unseen poses.