在3D中查找任意部分

Find Any Part in 3D

November 20, 2024

作者: Ziqi Ma, Yisong Yue, Georgia Gkioxari

cs.AI

摘要

我们研究3D开放世界部分分割：基于任何文本查询，在任何物体中分割任何部分。以往的方法在物体类别和部分词汇方面存在局限性。人工智能领域的最新进展展示了在2D中有效的开放世界识别能力。受到这一进展的启发，我们提出了一种用于3D部分分割的开放世界直接预测模型，可以零样本应用于任何物体。我们的方法名为Find3D，通过在互联网上大规模3D资产上训练通用类别点嵌入模型，而无需任何人工标注。它结合了一个数据引擎，由基础模型驱动以注释数据，并采用对比训练方法。我们在多个数据集上实现了强大的性能和泛化能力，相比下一个最佳方法，mIoU提高了最多3倍。我们的模型比现有基准模型快6倍至300倍以上。为鼓励开展通用类别开放世界3D部分分割研究，我们还发布了一个通用物体和部分的基准测试。项目网站：https://ziqi-ma.github.io/find3dsite/

English

We study open-world part segmentation in 3D: segmenting any part in any object based on any text query. Prior methods are limited in object categories and part vocabularies. Recent advances in AI have demonstrated effective open-world recognition capabilities in 2D. Inspired by this progress, we propose an open-world, direct-prediction model for 3D part segmentation that can be applied zero-shot to any object. Our approach, called Find3D, trains a general-category point embedding model on large-scale 3D assets from the internet without any human annotation. It combines a data engine, powered by foundation models for annotating data, with a contrastive training method. We achieve strong performance and generalization across multiple datasets, with up to a 3x improvement in mIoU over the next best method. Our model is 6x to over 300x faster than existing baselines. To encourage research in general-category open-world 3D part segmentation, we also release a benchmark for general objects and parts. Project website: https://ziqi-ma.github.io/find3dsite/