DressRecon：從單眼視頻進行自由形式的4D人體重建

DressRecon: Freeform 4D Human Reconstruction from Monocular Video

September 30, 2024

作者: Jeff Tan, Donglai Xiang, Shubham Tulsiani, Deva Ramanan, Gengshan Yang

cs.AI

摘要

我們提出了一種從單眼視頻中重建時間一致的人體模型的方法，重點放在極鬆散的服裝或手持物體的互動上。先前的人體重建工作要麼僅限於緊身服裝且無物體互動，要麼需要校準的多視圖捕獲或個性化模板掃描，這在大規模收集時成本高昂。我們實現高質量且靈活重建的關鍵在於將有關人體骨架的通用先驗（從大規模訓練數據中學習）與視頻特定的關節“骨袋”變形（通過測試時優化適合單個視頻）巧妙結合。我們通過學習一個神經隱式模型來實現這一點，該模型將身體和服裝變形解開為獨立的運動模型層。為了捕捉服裝微妙的幾何形狀，我們在優化過程中利用基於圖像的先驗，如人體姿勢、表面法線和光流。生成的神經場可以提取為時間一致的網格，或進一步優化為明確的3D高斯函數，以進行高保真的交互式渲染。在具有極具挑戰性的服裝變形和物體互動的數據集上，DressRecon比先前的技術產生了更高保真度的3D重建。項目頁面：https://jefftan969.github.io/dressrecon/

English

We present a method to reconstruct time-consistent human body models from monocular videos, focusing on extremely loose clothing or handheld object interactions. Prior work in human reconstruction is either limited to tight clothing with no object interactions, or requires calibrated multi-view captures or personalized template scans which are costly to collect at scale. Our key insight for high-quality yet flexible reconstruction is the careful combination of generic human priors about articulated body shape (learned from large-scale training data) with video-specific articulated "bag-of-bones" deformation (fit to a single video via test-time optimization). We accomplish this by learning a neural implicit model that disentangles body versus clothing deformations as separate motion model layers. To capture subtle geometry of clothing, we leverage image-based priors such as human body pose, surface normals, and optical flow during optimization. The resulting neural fields can be extracted into time-consistent meshes, or further optimized as explicit 3D Gaussians for high-fidelity interactive rendering. On datasets with highly challenging clothing deformations and object interactions, DressRecon yields higher-fidelity 3D reconstructions than prior art. Project page: https://jefftan969.github.io/dressrecon/