聚类与预测潜在图像块以提升掩码图像建模效果

Cluster and Predict Latents Patches for Improved Masked Image Modeling

February 12, 2025

作者: Timothée Darcet, Federico Baldassarre, Maxime Oquab, Julien Mairal, Piotr Bojanowski

cs.AI

摘要

掩码图像建模（MIM）为自监督表示学习提供了一种极具前景的方法，然而现有的MIM模型仍落后于当前最先进水平。本文中，我们系统性地分析了目标表示、损失函数及架构，进而提出了CAPI——一种基于潜在聚类预测的全新纯MIM框架。我们的方法采用了一种基于聚类的损失函数，该函数训练稳定，并展现出良好的扩展性。我们的ViT-L骨干网络CAPI，在ImageNet上实现了83.8%的准确率，在ADE20K上达到了32.1%的mIoU，仅使用简单的线性探测便显著超越了以往的MIM方法，并接近当前最先进的DINOv2的性能。我们已公开所有代码与模型。

English

Masked Image Modeling (MIM) offers a promising approach to self-supervised representation learning, however existing MIM models still lag behind the state-of-the-art. In this paper, we systematically analyze target representations, loss functions, and architectures, to introduce CAPI - a novel pure-MIM framework that relies on the prediction of latent clusterings. Our approach leverages a clustering-based loss, which is stable to train, and exhibits promising scaling properties. Our ViT-L backbone, CAPI, achieves 83.8% accuracy on ImageNet and 32.1% mIoU on ADE20K with simple linear probes, substantially outperforming previous MIM methods and approaching the performance of the current state-of-the-art, DINOv2. We release all our code and models.