将视觉基础模型调整为在遥感图像中实现稳健的云分割

Adapting Vision Foundation Models for Robust Cloud Segmentation in Remote Sensing Images

November 20, 2024

作者: Xuechao Zou, Shun Zhang, Kai Li, Shiying Wang, Junliang Xing, Lei Jin, Congyan Lang, Pin Tao

cs.AI

摘要

在遥感图像解释中，云分割是一个关键挑战，其准确性直接影响后续数据处理和分析的有效性。最近，视觉基础模型（VFM）展示了在各种视觉任务中强大的泛化能力。本文提出了一种名为Cloud-Adapter的参数高效自适应方法，旨在增强云分割的准确性和稳健性。我们的方法利用了在通用领域数据上预训练的VFM，该模型保持冻结状态，消除了额外训练的需求。Cloud-Adapter包含一个轻量级的空间感知模块，最初利用卷积神经网络（ConvNet）提取密集的空间表示。这些多尺度特征然后被聚合，并作为上下文输入传递给一个适应模块，该模块调节VFM内的冻结变换器层。实验结果表明，Cloud-Adapter方法仅利用冻结骨干网络可训练参数的0.6%，就实现了显著的性能提升。Cloud-Adapter在多个卫星数据源、传感器系列、数据处理级别、土地覆盖情景和注释细粒度的各种云分割数据集上始终保持最先进的性能。我们已在https://github.com/XavierJiezou/Cloud-Adapter发布了源代码和预训练模型，以支持进一步研究。

English

Cloud segmentation is a critical challenge in remote sensing image interpretation, as its accuracy directly impacts the effectiveness of subsequent data processing and analysis. Recently, vision foundation models (VFM) have demonstrated powerful generalization capabilities across various visual tasks. In this paper, we present a parameter-efficient adaptive approach, termed Cloud-Adapter, designed to enhance the accuracy and robustness of cloud segmentation. Our method leverages a VFM pretrained on general domain data, which remains frozen, eliminating the need for additional training. Cloud-Adapter incorporates a lightweight spatial perception module that initially utilizes a convolutional neural network (ConvNet) to extract dense spatial representations. These multi-scale features are then aggregated and serve as contextual inputs to an adapting module, which modulates the frozen transformer layers within the VFM. Experimental results demonstrate that the Cloud-Adapter approach, utilizing only 0.6% of the trainable parameters of the frozen backbone, achieves substantial performance gains. Cloud-Adapter consistently attains state-of-the-art (SOTA) performance across a wide variety of cloud segmentation datasets from multiple satellite sources, sensor series, data processing levels, land cover scenarios, and annotation granularities. We have released the source code and pretrained models at https://github.com/XavierJiezou/Cloud-Adapter to support further research.