通过点轨迹处理从日常视频快速基于编码器的三维重建

Fast Encoder-Based 3D from Casual Videos via Point Track Processing

April 10, 2024

作者: Yoni Kasten, Wuyue Lu, Haggai Maron

cs.AI

摘要

本文解决了从具有动态内容的视频中重建3D结构的长期挑战。目前针对这一问题的方法并非旨在处理由标准摄像机录制的非正式视频，或需要长时间的优化。 为了显著提高以前方法的效率，我们提出了TracksTo4D，这是一种基于学习的方法，可以通过单次高效的前向传递推断出源自非正式视频的动态内容的3D结构和摄像机位置。为实现这一目标，我们建议直接处理2D点轨迹作为输入，并设计了一个专门用于处理2D点轨迹的架构。我们设计的架构考虑了两个关键原则：（1）考虑输入点轨迹数据中存在的固有对称性，以及（2）假设可以使用低秩逼近有效地表示运动模式。TracksTo4D在一个非监督方式的数据集上进行训练，该数据集利用了仅从视频中提取的2D点轨迹，而没有任何3D监督。我们的实验表明，TracksTo4D可以重建出底层视频的时间点云和摄像机位置，其准确性可与最先进的方法相媲美，同时将运行时间大幅缩短高达95%。我们进一步展示，TracksTo4D在推断时对未见过的语义类别的未见视频具有良好的泛化能力。

English

This paper addresses the long-standing challenge of reconstructing 3D structures from videos with dynamic content. Current approaches to this problem were not designed to operate on casual videos recorded by standard cameras or require a long optimization time. Aiming to significantly improve the efficiency of previous approaches, we present TracksTo4D, a learning-based approach that enables inferring 3D structure and camera positions from dynamic content originating from casual videos using a single efficient feed-forward pass. To achieve this, we propose operating directly over 2D point tracks as input and designing an architecture tailored for processing 2D point tracks. Our proposed architecture is designed with two key principles in mind: (1) it takes into account the inherent symmetries present in the input point tracks data, and (2) it assumes that the movement patterns can be effectively represented using a low-rank approximation. TracksTo4D is trained in an unsupervised way on a dataset of casual videos utilizing only the 2D point tracks extracted from the videos, without any 3D supervision. Our experiments show that TracksTo4D can reconstruct a temporal point cloud and camera positions of the underlying video with accuracy comparable to state-of-the-art methods, while drastically reducing runtime by up to 95\%. We further show that TracksTo4D generalizes well to unseen videos of unseen semantic categories at inference time.