GeoPixel：遥感中的像素基础大型多模态模型

GeoPixel: Pixel Grounding Large Multimodal Model in Remote Sensing

January 23, 2025

作者: Akashah Shabbir, Mohammed Zumri, Mohammed Bennamoun, Fahad S. Khan, Salman Khan

cs.AI

摘要

最近在大型多模态模型（LMMs）方面取得的进展已经认识到，细粒度的接地作为视觉理解和对话的一个必要因素。然而，这种表示在LMMs中的好处仅限于自然图像领域，这些模型在遥感（RS）方面表现不佳。高分辨率RS图像中独特的俯视角度、尺度变化以及小物体的存在提出了区域级理解中的独特挑战。此外，在RS领域内发展LMMs的接地对话能力受到缺乏细粒度、RS领域特定接地数据的阻碍。为了解决这些限制，我们提出了GeoPixel - 第一个端到端的高分辨率RS-LMM，支持像素级接地。这种能力通过在对话中生成交错的掩模来实现细粒度的视觉感知。GeoPixel支持任何长宽比的4K高清分辨率，非常适合高精度RS图像分析。为了支持RS图像中的接地对话生成（GCG），我们通过一种半自动化流程策划了一个视觉上接地的数据集GeoPixelD，该流程利用了专为RS数据量身定制的标记提示和空间先验来系统地控制数据生成过程。GeoPixel在像素级理解方面表现出优越性，超越了现有LMMs在单目标和多目标分割任务中的表现。我们的方法论消融研究验证了整体架构中每个组件的有效性。我们的代码和数据将会公开发布。

English

Recent advances in large multimodal models (LMMs) have recognized fine-grained grounding as an imperative factor of visual understanding and dialogue. However, the benefits of such representation in LMMs are limited to the natural image domain, and these models perform poorly for remote sensing (RS). The distinct overhead viewpoint, scale variation, and presence of small objects in high-resolution RS imagery present a unique challenge in region-level comprehension. Moreover, the development of the grounding conversation capability of LMMs within RS is hindered by the lack of granular, RS domain-specific grounded data. Addressing these limitations, we propose GeoPixel - the first end-to-end high resolution RS-LMM that supports pixel-level grounding. This capability allows fine-grained visual perception by generating interleaved masks in conversation. GeoPixel supports up to 4K HD resolution in any aspect ratio, ideal for high-precision RS image analysis. To support the grounded conversation generation (GCG) in RS imagery, we curate a visually grounded dataset GeoPixelD through a semi-automated pipeline that utilizes set-of-marks prompting and spatial priors tailored for RS data to methodically control the data generation process. GeoPixel demonstrates superior performance in pixel-level comprehension, surpassing existing LMMs in both single-target and multi-target segmentation tasks. Our methodological ablation studies validate the effectiveness of each component in the overall architecture. Our code and data will be publicly released.