TAPTRv3：空間和時間背景促進對長視頻中任何點的穩健追蹤

摘要

本文介紹了TAPTRv3，它是在TAPTRv2的基礎上構建的，旨在提高長視頻中的點跟踪魯棒性。TAPTRv2是一個簡單的DETR-like框架，可以在現實世界的視頻中精確跟踪任何點，而無需成本體積。TAPTRv3通過解決TAPTRv2在從長視頻中查詢高質量特徵方面的不足來改進TAPTRv2，在這些視頻中，目標跟踪點通常隨時間增加而變化。在TAPTRv3中，我們提出利用空間和時間上下文，以在空間和時間維度上帶來更好的特徵查詢，從而實現對長視頻的更強魯棒跟踪。為了更好地進行空間特徵查詢，我們提出了上下文感知交叉注意力（CCA），它利用周圍的空間上下文來增強在查詢圖像特徵時的注意力分數質量。為了更好地進行時間特徵查詢，我們引入了可見性感知長時間注意力（VLTA），以在考慮其相應可見性的情況下對所有過去幀進行時間注意力，這有效地解決了TAPTRv2中由其類似RNN的長時間建模帶來的特徵漂移問題。TAPTRv3在大多數具有挑戰性的數據集上遠遠超越了TAPTRv2，並獲得了最先進的性能。即使與使用大規模額外內部數據訓練的方法相比，TAPTRv3仍然具有競爭力。

English

In this paper, we present TAPTRv3, which is built upon TAPTRv2 to improve its point tracking robustness in long videos. TAPTRv2 is a simple DETR-like framework that can accurately track any point in real-world videos without requiring cost-volume. TAPTRv3 improves TAPTRv2 by addressing its shortage in querying high quality features from long videos, where the target tracking points normally undergo increasing variation over time. In TAPTRv3, we propose to utilize both spatial and temporal context to bring better feature querying along the spatial and temporal dimensions for more robust tracking in long videos. For better spatial feature querying, we present Context-aware Cross-Attention (CCA), which leverages surrounding spatial context to enhance the quality of attention scores when querying image features. For better temporal feature querying, we introduce Visibility-aware Long-Temporal Attention (VLTA) to conduct temporal attention to all past frames while considering their corresponding visibilities, which effectively addresses the feature drifting problem in TAPTRv2 brought by its RNN-like long-temporal modeling. TAPTRv3 surpasses TAPTRv2 by a large margin on most of the challenging datasets and obtains state-of-the-art performance. Even when compared with methods trained with large-scale extra internal data, TAPTRv3 is still competitive.

TAPTRv3：空間和時間背景促進對長視頻中任何點的穩健追蹤

TAPTRv3: Spatial and Temporal Context Foster Robust Tracking of Any Point in Long Video

摘要

Summary

Support