通过自监督预训练在自然视频上出现的直觉物理理解

Intuitive physics understanding emerges from self-supervised pretraining on natural videos

February 17, 2025

作者: Quentin Garrido, Nicolas Ballas, Mahmoud Assran, Adrien Bardes, Laurent Najman, Michael Rabbat, Emmanuel Dupoux, Yann LeCun

cs.AI

摘要

我们研究了在训练用于预测自然视频中遮蔽区域的通用深度神经网络模型时直观物理理解的出现。利用违反期望框架，我们发现训练在学习表示空间中预测结果的视频预测模型展现了对各种直观物理属性的理解，如物体恒常性和形状一致性。相比之下，在像素空间和通过文本推理的多模态大语言模型中进行视频预测的表现更接近于随机。我们对这些架构的比较揭示了，在联合学习抽象表示空间的同时预测感官输入的缺失部分，类似于预测编码，就足以获得对直观物理的理解，即使是在仅训练一周的独特视频上的模型也能取得高于随机的表现。这挑战了核心知识的观念 —— 一套帮助理解世界的先天系统需要被硬编码以发展对直观物理的理解的想法。

English

We investigate the emergence of intuitive physics understanding in general-purpose deep neural network models trained to predict masked regions in natural videos. Leveraging the violation-of-expectation framework, we find that video prediction models trained to predict outcomes in a learned representation space demonstrate an understanding of various intuitive physics properties, such as object permanence and shape consistency. In contrast, video prediction in pixel space and multimodal large language models, which reason through text, achieve performance closer to chance. Our comparisons of these architectures reveal that jointly learning an abstract representation space while predicting missing parts of sensory input, akin to predictive coding, is sufficient to acquire an understanding of intuitive physics, and that even models trained on one week of unique video achieve above chance performance. This challenges the idea that core knowledge -- a set of innate systems to help understand the world -- needs to be hardwired to develop an understanding of intuitive physics.