解釋性指示：朝向統一的視覺任務理解和零樣本泛化

摘要

電腦視覺（CV）尚未完全實現自然語言處理（NLP）中觀察到的零-shot任務泛化，儘管採用了許多NLP中建立的里程碑，如大型Transformer模型、廣泛的預訓練和自回歸範式等。在本文中，我們探討了一個觀點，即CV採用離散和術語化的任務定義（例如，“圖像分割”），這可能是零-shot任務泛化的一個關鍵障礙。我們的假設是，由於這些術語化定義，深度模型在沒有真正理解先前見過的任務的情況下，很難對新任務進行泛化。為了驗證這一點，我們引入了解釋性指令，通過從輸入圖像到輸出的詳細語言轉換提供了一種直觀定義CV任務目標的方式。我們創建了一個包含1200萬個“圖像輸入到解釋性指令到輸出”三元組的大規模數據集，並訓練了一個基於自回歸的視覺語言模型（AR-based VLM），該模型將圖像和解釋性指令作為輸入。通過學習遵循這些指令，基於AR的VLM實現了對先前見過的任務的指令級零-shot能力，並展示了對未見CV任務的強大零-shot泛化。代碼和數據集將在我們的GitHub存儲庫中公開提供。

English

Computer Vision (CV) has yet to fully achieve the zero-shot task generalization observed in Natural Language Processing (NLP), despite following many of the milestones established in NLP, such as large transformer models, extensive pre-training, and the auto-regression paradigm, among others. In this paper, we explore the idea that CV adopts discrete and terminological task definitions (\eg, ``image segmentation''), which may be a key barrier to zero-shot task generalization. Our hypothesis is that without truly understanding previously-seen tasks--due to these terminological definitions--deep models struggle to generalize to novel tasks. To verify this, we introduce Explanatory Instructions, which provide an intuitive way to define CV task objectives through detailed linguistic transformations from input images to outputs. We create a large-scale dataset comprising 12 million ``image input to explanatory instruction to output'' triplets, and train an auto-regressive-based vision-language model (AR-based VLM) that takes both images and explanatory instructions as input. By learning to follow these instructions, the AR-based VLM achieves instruction-level zero-shot capabilities for previously-seen tasks and demonstrates strong zero-shot generalization for unseen CV tasks. Code and dataset will be openly available on our GitHub repository.

解釋性指示：朝向統一的視覺任務理解和零樣本泛化

Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization

摘要

Summary

Support