解释性指导：走向统一的视觉任务理解和零样本泛化

摘要

计算机视觉（CV）尚未完全实现自然语言处理（NLP）中观察到的零-shot任务泛化，尽管遵循了NLP中建立的许多里程碑，如大型变压器模型、广泛的预训练和自回归范式等。在本文中，我们探讨了一个观点，即CV采用离散和术语化的任务定义（例如，“图像分割”），这可能是零-shot任务泛化的关键障碍。我们的假设是，由于这些术语化定义，深度模型在没有真正理解先前见过的任务的情况下很难推广到新任务。为了验证这一点，我们引入了解释性指令，通过从输入图像到输出的详细语言转换提供了一种直观定义CV任务目标的方式。我们创建了一个包含1200万个“图像输入到解释性指令到输出”三元组的大规模数据集，并训练了一个以自回归为基础的视觉-语言模型（AR-based VLM），该模型将图像和解释性指令作为输入。通过学习遵循这些指令，AR-based VLM实现了先前见过任务的指令级零-shot能力，并展示了对未见CV任务的强大零-shot泛化能力。代码和数据集将在我们的GitHub存储库上公开提供。

English

Computer Vision (CV) has yet to fully achieve the zero-shot task generalization observed in Natural Language Processing (NLP), despite following many of the milestones established in NLP, such as large transformer models, extensive pre-training, and the auto-regression paradigm, among others. In this paper, we explore the idea that CV adopts discrete and terminological task definitions (\eg, ``image segmentation''), which may be a key barrier to zero-shot task generalization. Our hypothesis is that without truly understanding previously-seen tasks--due to these terminological definitions--deep models struggle to generalize to novel tasks. To verify this, we introduce Explanatory Instructions, which provide an intuitive way to define CV task objectives through detailed linguistic transformations from input images to outputs. We create a large-scale dataset comprising 12 million ``image input to explanatory instruction to output'' triplets, and train an auto-regressive-based vision-language model (AR-based VLM) that takes both images and explanatory instructions as input. By learning to follow these instructions, the AR-based VLM achieves instruction-level zero-shot capabilities for previously-seen tasks and demonstrates strong zero-shot generalization for unseen CV tasks. Code and dataset will be openly available on our GitHub repository.

解释性指导：走向统一的视觉任务理解和零样本泛化

Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization

摘要

Summary

Support