TAPIP3D：持久3D幾何中的任意點追蹤

TAPIP3D: Tracking Any Point in Persistent 3D Geometry

April 20, 2025

作者: Bowei Zhang, Lei Ke, Adam W. Harley, Katerina Fragkiadaki

cs.AI

摘要

我們介紹了TAPIP3D，這是一種用於單目RGB和RGB-D視頻中長期3D點追踪的新方法。TAPIP3D將視頻表示為相機穩定的時空特徵雲，利用深度和相機運動信息將2D視頻特徵提升到3D世界空間中，從而有效消除相機運動。TAPIP3D在這種穩定表示中迭代地優化多幀3D運動估計，實現了長時間的穩健追踪。為了解決3D點分佈的固有非規則性，我們提出了一種局部對注意力機制。這種3D上下文策略有效地利用了3D中的空間關係，形成信息豐富的特徵鄰域，以實現精確的3D軌跡估計。我們以3D為中心的方法顯著優於現有的3D點追踪方法，甚至在準確深度可用時，相比傳統的2D像素追踪器，還提高了2D追踪的準確性。它支持在相機坐標（即未穩定）和世界坐標中進行推理，我們的結果表明，補償相機運動可以提高追踪性能。我們的方法取代了先前2D和3D追踪器中使用的傳統2D方形相關鄰域，從而在各種3D點追踪基準測試中獲得更穩健和準確的結果。項目頁面：https://tapip3d.github.io

English

We introduce TAPIP3D, a novel approach for long-term 3D point tracking in monocular RGB and RGB-D videos. TAPIP3D represents videos as camera-stabilized spatio-temporal feature clouds, leveraging depth and camera motion information to lift 2D video features into a 3D world space where camera motion is effectively canceled. TAPIP3D iteratively refines multi-frame 3D motion estimates within this stabilized representation, enabling robust tracking over extended periods. To manage the inherent irregularities of 3D point distributions, we propose a Local Pair Attention mechanism. This 3D contextualization strategy effectively exploits spatial relationships in 3D, forming informative feature neighborhoods for precise 3D trajectory estimation. Our 3D-centric approach significantly outperforms existing 3D point tracking methods and even enhances 2D tracking accuracy compared to conventional 2D pixel trackers when accurate depth is available. It supports inference in both camera coordinates (i.e., unstabilized) and world coordinates, and our results demonstrate that compensating for camera motion improves tracking performance. Our approach replaces the conventional 2D square correlation neighborhoods used in prior 2D and 3D trackers, leading to more robust and accurate results across various 3D point tracking benchmarks. Project Page: https://tapip3d.github.io