SPAR3D：从单个图像稳定地感知重建3D物体

SPAR3D: Stable Point-Aware Reconstruction of 3D Objects from Single Images

January 8, 2025

作者: Zixuan Huang, Mark Boss, Aaryaman Vasishta, James M. Rehg, Varun Jampani

cs.AI

摘要

我们研究单图像三维物体重建问题。最近的研究分为两个方向：基于回归的建模和生成式建模。回归方法能够高效推断可见表面，但在遮挡区域方面表现不佳。生成方法通过建模分布更好地处理不确定区域，但计算成本高且生成结果常常与可见表面不对齐。本文提出了SPAR3D，一种新颖的两阶段方法，旨在兼收并蓄两个方向的优点。SPAR3D的第一阶段利用轻量级点扩散模型生成稀疏的三维点云，具有快速采样速度。第二阶段利用采样的点云和输入图像创建高度详细的网格。我们的两阶段设计实现了对单图像三维任务的概率建模，同时保持了高计算效率和出色的输出保真度。进一步使用点云作为中间表示还允许进行交互式用户编辑。在多样数据集上评估，SPAR3D在0.7秒的推断速度下展现出优于先前最先进方法的性能。项目页面包含代码和模型：https://spar3d.github.io

English

We study the problem of single-image 3D object reconstruction. Recent works have diverged into two directions: regression-based modeling and generative modeling. Regression methods efficiently infer visible surfaces, but struggle with occluded regions. Generative methods handle uncertain regions better by modeling distributions, but are computationally expensive and the generation is often misaligned with visible surfaces. In this paper, we present SPAR3D, a novel two-stage approach aiming to take the best of both directions. The first stage of SPAR3D generates sparse 3D point clouds using a lightweight point diffusion model, which has a fast sampling speed. The second stage uses both the sampled point cloud and the input image to create highly detailed meshes. Our two-stage design enables probabilistic modeling of the ill-posed single-image 3D task while maintaining high computational efficiency and great output fidelity. Using point clouds as an intermediate representation further allows for interactive user edits. Evaluated on diverse datasets, SPAR3D demonstrates superior performance over previous state-of-the-art methods, at an inference speed of 0.7 seconds. Project page with code and model: https://spar3d.github.io