SAMPart3D：在3D物體中分割任何部分

SAMPart3D: Segment Any Part in 3D Objects

November 11, 2024

作者: Yunhan Yang, Yukun Huang, Yuan-Chen Guo, Liangjun Lu, Xiaoyang Wu, Edmund Y. Lam, Yan-Pei Cao, Xihui Liu

cs.AI

摘要

3D部件分割是三維感知中一項至關重要且具有挑戰性的任務，在機器人技術、三維生成和三維編輯等應用中發揮著關鍵作用。最近的方法利用強大的視覺語言模型（VLMs）進行2D到3D知識蒸餾，實現了零樣本3D部件分割。然而，這些方法受限於對文本提示的依賴，這限制了對大規模未標記數據集的可擴展性以及處理部件模糊性的靈活性。在本研究中，我們介紹了SAMPart3D，一個可擴展的零樣本3D部件分割框架，可以將任何3D物體分割為多個粒度的語義部件，而無需預定義部件標籤集作為文本提示。為了實現可擴展性，我們使用文本無關的視覺基礎模型來蒸餾3D特徵提取骨幹，實現對大型未標記3D數據集的擴展以學習豐富的3D先驗知識。為了實現靈活性，我們蒸餾了尺度條件下的部件感知3D特徵，用於多個粒度的3D部件分割。一旦從尺度條件下的部件感知3D特徵中獲得分割的部件，我們使用VLMs基於多視圖渲染為每個部件分配語義標籤。相對於先前的方法，我們的SAMPart3D可以擴展到最新的大規模3D物體數據集Objaverse並處理複雜的非常規物體。此外，我們貢獻了一個新的3D部件分割基準，以解決現有基準中對象和部件缺乏多樣性和複雜性的問題。實驗表明，我們的SAMPart3D明顯優於現有的零樣本3D部件分割方法，並且可以促進各種應用，如部件級編輯和交互式分割。

English

3D part segmentation is a crucial and challenging task in 3D perception, playing a vital role in applications such as robotics, 3D generation, and 3D editing. Recent methods harness the powerful Vision Language Models (VLMs) for 2D-to-3D knowledge distillation, achieving zero-shot 3D part segmentation. However, these methods are limited by their reliance on text prompts, which restricts the scalability to large-scale unlabeled datasets and the flexibility in handling part ambiguities. In this work, we introduce SAMPart3D, a scalable zero-shot 3D part segmentation framework that segments any 3D object into semantic parts at multiple granularities, without requiring predefined part label sets as text prompts. For scalability, we use text-agnostic vision foundation models to distill a 3D feature extraction backbone, allowing scaling to large unlabeled 3D datasets to learn rich 3D priors. For flexibility, we distill scale-conditioned part-aware 3D features for 3D part segmentation at multiple granularities. Once the segmented parts are obtained from the scale-conditioned part-aware 3D features, we use VLMs to assign semantic labels to each part based on the multi-view renderings. Compared to previous methods, our SAMPart3D can scale to the recent large-scale 3D object dataset Objaverse and handle complex, non-ordinary objects. Additionally, we contribute a new 3D part segmentation benchmark to address the lack of diversity and complexity of objects and parts in existing benchmarks. Experiments show that our SAMPart3D significantly outperforms existing zero-shot 3D part segmentation methods, and can facilitate various applications such as part-level editing and interactive segmentation.