感知编码器：最佳视觉嵌入并非位于网络输出层

摘要

我们推出了感知编码器（Perception Encoder, PE），这是一种通过简单的视觉-语言学习训练而成、用于图像和视频理解的最先进编码器。传统上，视觉编码器依赖于多种预训练目标，每种目标都针对特定的下游任务（如分类、字幕生成或定位）进行了定制。令人惊讶的是，在扩展了我们精心调整的图像预训练方案并通过我们强大的视频数据引擎进行优化后，我们发现仅对比视觉-语言训练就能为所有这些下游任务生成强大且通用的嵌入表示。唯一需要注意的是：这些嵌入隐藏在网络的中间层中。为了提取它们，我们引入了两种对齐方法：用于多模态语言建模的语言对齐，以及用于密集预测的空间对齐。结合核心对比检查点，我们的PE模型家族在广泛的任务上实现了最先进的性能，包括零样本图像和视频分类与检索；文档、图像和视频问答；以及检测、深度估计和跟踪等空间任务。为了促进进一步研究，我们将发布我们的模型、代码以及一个包含合成和人工标注视频的新颖数据集。

English

We introduce Perception Encoder (PE), a state-of-the-art encoder for image and video understanding trained via simple vision-language learning. Traditionally, vision encoders have relied on a variety of pretraining objectives, each tailored to specific downstream tasks such as classification, captioning, or localization. Surprisingly, after scaling our carefully tuned image pretraining recipe and refining with our robust video data engine, we find that contrastive vision-language training alone can produce strong, general embeddings for all of these downstream tasks. There is only one caveat: these embeddings are hidden within the intermediate layers of the network. To draw them out, we introduce two alignment methods, language alignment for multimodal language modeling, and spatial alignment for dense prediction. Together with the core contrastive checkpoint, our PE family of models achieves state-of-the-art performance on a wide variety of tasks, including zero-shot image and video classification and retrieval; document, image, and video Q&A; and spatial tasks such as detection, depth estimation, and tracking. To foster further research, we are releasing our models, code, and a novel dataset of synthetically and human-annotated videos.

感知编码器：最佳视觉嵌入并非位于网络输出层

Perception Encoder: The best visual embeddings are not at the output of the network

摘要

Summary

Support

Support