# Mind the Modality Gap: Towards a Remote Sensing Vision-Language Model via
Cross-modal Alignment

Angelos Zavras, Dimitrios Michail, Begüm Demir, Ioannis Papoutsis A. Zavras is
with Orion Lab, National Observatory of Athens, National Technical University
of Athens and Harokopio University of AthensD. Michail is with Harokopio
University of AthensB. Demir is with Technische Universität BerlinI. Papoutsis
is with Orion Lab, National Observatory of Athens and National Technical
University of Athens

###### Abstract

Deep Learning (DL) is undergoing a paradigm shift with the emergence of
foundation models, aptly named by their crucial, yet incomplete nature. In
this work, we focus on Contrastive Language-Image Pre-training (CLIP), an
open-vocabulary foundation model, which achieves high accuracy across many
image classification tasks and is often competitive with a fully supervised
baseline without being explicitly trained. Nevertheless, there are still
domains where zero-shot CLIP performance is far from optimal, such as Remote
Sensing (RS) and medical imagery. These domains do not only exhibit
fundamentally different distributions compared to natural images, but also
commonly rely on complementary modalities, beyond RGB, to derive meaningful
insights. To this end, we propose a methodology for the purpose of aligning
distinct RS imagery modalities with the visual and textual modalities of CLIP.
Our two-stage procedure, comprises of robust fine-tuning CLIP in order to deal
with the distribution shift, accompanied by the cross-modal alignment of a RS
modality encoder, in an effort to extend the zero-shot capabilities of CLIP.
We ultimately demonstrate our method on the tasks of RS imagery classification
and cross-modal retrieval. We empirically show that both robust fine-tuning
and cross-modal alignment translate to significant performance gains, across
several RS benchmark datasets. Notably, these enhancements are achieved
without the reliance on textual descriptions, without introducing any task-
specific parameters, without training from scratch and without catastrophic
forgetting.

###### Index Terms:

Remote sensing, foundation model, vision-language model, multi-modal learning,
cross-modal alignment, cross-modal retrieval, cross-modal distillation,
satellite representation learning.

##  I Introduction

Traditional earth surface mapping and monitoring, has been constrained to
expensive and time-consuming, methods (e.g field surveys), with data acquired
over lengthy intervals. For real-time global and regional mapping, these
methods are quite inefficient and impractical. EO is not new, but
historically, the utilization of EO data was constrained to the specialized
scientific community, due to the limitations posed by the satellite
infrastructure available and the accessibility of data and analytical tools.
However, recent advancements in these areas, coupled with breakthroughs in
data processing techniques, have democratized the use of EO data, enabling the
widespread use of this information at scale. Because of the massive volume of
data transmitted every day from many sensors monitoring earth, Remote Sensing
(RS) provides fertile ground for the application of modern machine learning
(ML) algorithms. Notably, as of today, Europe’s Copernicus satellite
constellation alone is producing more than 25 terabytes of satellite data
every day, which as a matter of fact outstrips our capacity to extract
meaningful information.

DL models for RS imagery classification are typically either trained from
scratch using supervised learning methods [1] or fine-tuned for the downstream
task, after self-supervised pre-training [2] or transfer learning from another
task [3]. Unfortunately, those models are capable of simply enumerating
numeric classes behind the scenes, completely disregarding the semantic
information encapsulated within each class label. As a result, these models
are exceptionally fragile, designed to excel only within the narrow
classification space for which they were specifically trained. This
discrepancy, is of great interest in the EO community since various
nomenclatures and classification schemes are used, while researchers and
practitioners continue to explore new schemes and refine existing ones to
enhance the understanding of Earth’s surface and its dynamic behavior. These
schemes play a crucial role in classifying and interpreting the data, enabling
effective analysis and information extraction, depending on the objectives of
the study, available data, and the desired level of detail and accuracy
required for the analysis.

In contrast to typical image classifiers, open-vocabulary models are not
constrained to a fixed classification space and are able to perform any image
classification task, using textual descriptions of the class names. One the
other hand, foundation models [4], commonly trained via self-supervision at
scale, on large amounts of unlabeled data obtained through web-crawling
approaches, are large and versatile deep learning models that can be adapted
to a wide range of downstream tasks. Open-vocabulary foundation models,
represent a powerful combination of large-scale pre-training and the ability
to handle words outside of a fixed vocabulary. These models have gathered
considerable attention due to their exceptional performance and ability to
generalize across different domains and are particularly useful in scenarios
where the textual input may include domain-specific terms.

In this work, we investigate OpenAI’s Contrastive Language-Image Pre-Training
(CLIP) model, one of the so-called open-vocabulary foundation models, which is
able to comprehend concepts of both visual and textual modalities and predict
the most relevant text description without being explicitly optimized for the
task. Nevertheless, there are still domains where zero-shot CLIP performance
is far from optimal, such as RS and medical imagery. Satellite scene
classification, in particular, is one of the few tasks presented by the CLIP
authors, where zero-shot CLIP significantly under-performs on the EuroSAT
dataset, resulting in the biggest delta (37,1%) among all tasks assessed, when
compared to a fully supervised ResNet50 baseline model. To this end, we
identify three major gaps for the task of satellite scene classification which
we address in this work. The first gap regards the distribution shift [5],
i.e. a change in the underlying data distribution that can occur when the data
used during training, significantly differs from the data the model encounters
when deployed, leading to significant performance degradation. Satellite
imagery does not only demonstrate a fundamentally different distribution
compared to natural images, but it also poses unique challenges in terms of
spatial resolution, atmospheric conditions, and varying perspectives due to
different orbits, which can significantly impact the performance of computer
vision models. The second gap arises from the information constraint induced
by relying solely on the RGB modality. Satellite imagery commonly exploit
complementary modalities, beyond RGB, to derive meaningful insights. These
modalities include multi-spectral, hyper-spectral and radar data, as well as
byproducts of those modalities such as InSAR [6] data. Leveraging these
diverse modalities grants us with critical and complementary information that
contribute to a deeper understanding of our planet. The third gap, pertains to
the scarcity of datasets containing pairs of satellite imagery and
corresponding textual descriptions. Typically used RS image-text paired
datasets [7, 8, 9, 10], are limited to aerial and Very High-Resolution (VHR)
commercial satellite imagery. There are two interrelated drawbacks in these
datasets. First, they do not rely on free and open-access data, such as the
Copernicus Sentinel satellite constellation data. Second, the spatial
resolution of these datasets, which directly relates to the level of detail
that can be retrieved from a scene, leads to a different underlying data
distribution compared to non-commercial satellite data. Both factors limit
their usability in downstream applications.

In order to address these gaps, we propose a novel methodology comprised of
two consecutive stages, working towards cross-modal alignment of RS imagery
modalities in the context of CLIP. Our study is based on OpenAI’s CLIP
collection of pre-trained image-text models. For the first step, we define a
patching process to robustly fine-tune CLIP using RS data RGB composites, in
order to deal with the aforementioned distribution shift without impacting
CLIP zero-shot performance on natural image classification tasks. As for the
second step, we extend CLIP’s zero-shot capabilities by cross-modal aligning a
pre-trained RS encoder with the visual and textual modalities of CLIP. We
ultimately demonstrate our method on the task of RS imagery classification. We
empirically show that both robust fine-tuning and cross-modal alignment
translate to notable performance gains, over several RS imagery classification
benchmark datasets.

Our main contributions can be summarized as follows:

  * •

We propose a novel methodology for cross-modal alignment of RS imagery
modalities in the context of CLIP, without the reliance on textual
descriptions, without introducing any task-specific parameters, without
training from scratch and without catastrophic forgetting.

  * •

We evaluate the generalization of both patched and aligned models, and provide
an extensive benchmark on a series of prominent RS imagery datasets for the
task of RS imagery classification and cross-modal retrieval.

  * •

We set the baseline for cross-modal retrieval and text-based zero-shot tasks,
in comparison to a representative CLIP supported task (i.e ImageNet) proving
that our final Vision-Language model (VLM) extends CLIP pre-training.

  * •

We present an improved CLIP model that now attains good zero-shot performance
on RS imagery classification and cross-modal retrieval tasks. This enhancement
is achieved by effectively capturing the semantic content of RS, transcending
nomenclature limitations and leveraging complementary information, beyond RGB,
utilizing available RS imagery modalities. We have made our code
implementation and weights for all experiments publicly available, aiming to
facilitate the development of novel applications based on RS data.

##  II Related Work

###  II-A Contrastive Language-Image Pre-training

Recent developments in the multi-modal contrastive learning era have
significantly enhanced the capabilities of CLIP-like models. Such models are
notorious for their ability to extract rich visual and textual features, with
exceptional discriminative competence [11, 12, 13] and are suitable for a
variety of downstream tasks, such as image classification [14], action
recognition [15], semantic segmentation [16], text-guided image generation
[17], image and video captioning [18, 19, 20]. Lately, such models have been
also utilized either in part or in their entirety, as versatile building
blocks for the next generation of foundation models [21, 22, 23, 24]. It is
evident that CLIP due to its large scale pre-training, as well as its proven
efficacy on a wide range of downstream tasks and applications, emerges as an
advantageous frontier in the ongoing quest towards foundation models for EO.

LAION-AI with OpenCLIP [25], the open-source implementation of OpenAI’s CLIP
[14], has demonstrated impressive results. They managed to replicate OpenAI’s
proprietary pre-training dataset [26] and subsequently trained and published
several models [27], using various architectures, on a variety of data sources
and compute budgets, ranging from small to large-scale experiments. Recent
advancements in the context of CLIP pre-training have showcased remarkable
achievements along the axes of pre-train data filtering [28, 29, 30], model
architecture [31] and computational efficiency [32, 33, 34], leading to
substantial improvements and eventually establishing new standards within the
era of pre-trained CLIP models. These advancements, formulate an imperative
demand for an extensive set of benchmarking experiments, aiming to assess the
ability of those newly released pre-trained CLIP models to facilitate novel
applications, in the context of RS.

###  II-B Domain Specialized CLIP Models

Data is regarded as the cornerstone for foundation model training and CLIP is
no exception. Existing CLIP models are trained on large amounts of web-crawled
data, to encode general knowledge, without emphasizing on specialized domains
such as RS and medical imagery, which as a matter of fact demonstrate
fundamentally different distributions compared to natural images encountered
during pre-training.

Several efforts have been made to overcome this phenomenon and adapt CLIP
aiming to yield a robust foundation model with proficient expertise in
specialized domains. Unlike natural image-text datasets, which can easily
reach billion-scale [26, 28], specialized datasets hold a relatively limited
scale due to their innate demand for highly targeted knowledge. The medical
imaging community, for instance, has witnessed a significant surge in
contributions of CLIP-centered studies [35, 36, 37, 38]. They commonly
encompass biomedical image-caption pairs, collected and filtered from open-
access research papers [39, 40, 41, 42], medical reports [43, 44, 45] and
social media platforms [46, 47]. On the contrary, the RS domain lags behind,
in terms of CLIP-oriented developments, primarily due to the in-domain image-
text paired data scarcity. To this end, recent developments predominantly
revolve around the exploitation of existing limited extent RS datasets and the
data efficient adaptation of CLIP models to the problem at hand.

Arutiunian et al. [48] fine-tuned CLIP by leveraging three small existing RS
image captioning datasets and demonstrated their results on retrieval-related
tasks. Czerkawski et al. [49] highlighted that in zero-shot setting, CLIP
exhibits difficulty detecting cloud-free images and mitigated this limitation
through a cost-effective training stage consisting of a few hundred
optimization steps of a single linear layer on top of CLIP image encodings,
demonstrating improved performance and transferability across various sensor
types and spectral bands. Singha et al. [50] proposed APPLeNet, an image-
conditioned prompt learning strategy for few-shot RS image generalization
using CLIP models. Their approach focuses on multi-scale feature learning and
disentangles visual style and content primitives for domain generalization
tasks in RS, outperforming zero-shot CLIP in several RS benchmark datasets.
Liu et al. [51] presented RemoteCLIP, a RS domain specialized CLIP model,
fine-tuned using existing remote sensing image captioning datasets, extended
using heterogeneous satellite and UAV imagery annotations (e.g bounding boxes
and masks) after turning them into image-caption pairs, achieving remarkable
results across various datasets and downstream tasks. Zhang et al. [52]
presented RS5M, a 5 million RS image captioning dataset, obtained by filtering
publicly available image-text paired datasets and captioning label-only RS
datasets with pre-trained VLM, with the purpose of fine-tuning CLIP. They
experimented with full fine-tuning and several Parameter-Efficient Fine-Tuning
methods and ultimately demonstrated their final model GeoRSCLIP, on zero-shot
classification, cross-modal image-text retrieval and semantic localization
tasks in comparison to state-of-the-art RS-tailored CLIP models. Yuan et al.
[53] introduced a Parameter-Efficient Transfer Learning (PETL) [54] method for
RS Image-Text Retrieval, leveraging a pre-trained CLIP model along a multi-
modal adapter and a Hybrid Multi-modal Contrastive learning objective,
outperforming traditional methods and showcasing a drastic reduction in
training costs compared to full fine-tuning. Mo et al. [55] proposed S-CLIP, a
semi-supervised learning method for fine-tuning CLIP that utilizes additional
unpaired images by employing pseudo-labeling strategies specifically designed
for contrastive learning, significantly enhancing fine-tuning results using
fewer image-text pairs than typically required. Bhattacharya et al. [56]
proposed C-SAW, a self-supervised prompt learning technique which incorporates
a reconstruction task for better image generalization in RS applications.
During fine-tuning, they kept the CLIP backbone frozen and introduced a small
set of projectors for both CLIP encoders to train contrastively using C-SAW.
Dhakal et al. [57] introduced a novel weakly supervised method for creating
maps based on free-form textual descriptions, termed zero-shot mapping. They
utilized a contrastive learning framework named Sat2Cap, trained on paired
overhead and ground-level images, to predict CLIP embeddings of ground-level
scenery from satellite images. They managed to map a wide variety of
attributes without text-labeled data, overcoming the limitations of previous
models that could only map pre-defined attributes. Similarly, Mall et al. [58]
addressed the scarcity of textual descriptions, by training contrastively an
image encoder that maps RGB satellite imagery to the same representation space
of a frozen CLIP image encoder, using a large amount of paired internet images
and RGB satellite composites. They managed to yield state-of-the-art results
across numerous downstream tasks and indicated that leveraging a large number
of ground-satellite image pairs without accompanying text is more beneficial
than fine-tuning on small datasets, in a supervised manner.

It is evident that the main focus in tailoring CLIP for RS revolves around
overcoming the scarcity of in-domain image-text paired data. Efforts span from
leveraging limited-scale and synthetic datasets for fine-tuning to innovative
data-efficient and self-supervised strategies, all aimed at optimizing CLIP
for RS. Our work stands out by exploiting existing labeled RS imagery data to
deal with the RS domain distribution shift, without introducing any task-
specific parameters to the pre-trained CLIP models and without catastrophic
forgetting.

###  II-C Multi-modal CLIP-inspired models in Remote Sensing

Multi-modal learning in the RS domain [59] has evolved significantly, marking
a transition from traditional single-source data analysis to the integration
of diverse RS data types, such as optical imagery and radar, to attain a more
accurate and holistic understanding of the Earth. Over time, advancements in
machine learning and data fusion [60] techniques have led to sophisticated
models that can handle the complexity and variability of multi-modal RS data
[61, 62, 63]. Recently, the introduction of CLIP [14] has significantly
influenced the development of RS models capable of seamlessly integrating
various modalities, demonstrating an intuitive and versatile approach that
extends well beyond the initial image-text framework. This advancement has
opened up new possibilities for more intuitive and semantically rich analyses
of RS imagery, allowing models to complement and correlate the same Earth
features as being observed by different sensors, demonstrating the growing
potential of multi-modal learning in capturing the intricate features of our
planet.

In this context, Allen et al. [64] pre-trained a ViT-based CLIP model using
three different RS imagery modalities (Sentinel-2 RGB optical and Sentinel-1
SAR radar amplitude and interferometric coherence), across five AOIs covering
a small percentage of Earth’s total landmass. The model consists of three
separate single channel ViT-based input encoders, operating on whatever
channel was randomly selected for each modality, during pre-training, aiming
to create a shared embedding space between all three modalities, while
similarity is measured for each pair of modalities and then averaged. Klemmer
et al. [65] demonstrated SatCLIP, a global, geographic location encoder that
learns general-purpose, implicit representations using globally available
satellite imagery, pre-trained by matching satellite images and their
respective coordinates using the CLIP objective. Along the same lines, Cepeda
et al. [66] introduced GeoCLIP, a CLIP-inspired Image-to-GPS retrieval
approach for worldwide geo-localization. Unlike SatCLIP, GeoCLIP leverages a
pre-trained CLIP image encoder and enforces alignment with the corresponding
GPS locations. Using hierarchical learning and random Fourier features,
GeoCLIP demonstrated impressive effectiveness in limited-data settings and
potential application in a plethora of downstream tasks. Khanal et al. [67]
proposed GeoCLAP, a contrastive-learning framework for the task of soundscape
mapping based on the relationship between sound and the visual characteristics
of geographic locations. They leveraged a CLIP-based model to encode three
types of data: geo-tagged audio recordings, textual descriptions of audio, and
overhead images of their capture locations. As a result they enabled a unified
embedding space, for all three modalities, that can predict the most probable
sounds at any given geographic location, outperforming previous state-of-the-
art models.

A fundamental notion that distinguishes our work from related research, is the
cross-modal alignment of complementary RS imagery modalities, beyond RGB,
within the shared embedding space of CLIP, leveraging its large-scale pre-
training. Our endeavors focus on aligning a multi-spectral Sentinel-2 encoder
with the RGB-image and text encoders of CLIP, providing a blueprint for the
cross-modal alignment of more RS imagery modalities and ultimately enabling a
rich set of cross-modal retrieval and text-based zero-shot downstream tasks.

##  III Method

Figure 1: Patching with interpolation (PAINT) [68] is utilized in the first
stage of our method, in order to deal with the underlying distribution shift
of the satellite data, without compromising the performance of CLIP on
supported tasks, i.e tasks where performance is already adequate.

Let
$D=\\{(\mathcal{I}_{SAT}^{1},y_{1}),\dots,(\mathcal{I}_{SAT}^{n},y_{n})\\}$ be
a labeled RS imagery archive consisting of $n$ image-label pairs, where
$(\mathcal{I}_{SAT}^{n},y_{n})$ is the $n$th image-label pair in the archive.
Each image $\mathcal{I}_{SAT}^{i}$ has a corresponding RGB composite image
$\mathcal{I}_{RGB}^{i}\subsetneq\mathcal{I}_{SAT}^{i}$, produced using a
dedicated function $T_{RGB}()$ depending on the RS modality of
$\mathcal{I}_{SAT}$. Given the set of RS imagery $\mathcal{I}_{SAT}$ and the
corresponding set of RGB composites $\mathcal{I}_{RGB}$, along with associated
labels $y$, we aim to learn a shared embedding space $E$, by utilizing RGB
composites and labels as anchors, in order to effectively align these
modalities. The objective is to find a mapping
$f:\mathcal{I}_{SAT}\cup\mathcal{I}_{RGB}\cup y\rightarrow E$ such that for
each triplet $(\mathcal{I}_{SAT}^{i},\mathcal{I}_{RGB}^{i},{y}_{i})$, the
embeddings $f(\mathcal{I}_{SAT}^{i}),f(\mathcal{I}_{RGB}^{i})$, and
$f({y}_{i})$ are closely aligned in $E$. The final embedding space $E$ enables
multi-modal downstream tasks by facilitating the association between RS
imagery modalities, RGB images, and text through learned representations.

To this end, we take a two-stage approach, explicitly defined in Algorithm 1.
The first step addresses the distribution shift in satellite imagery by
robustly fine-tuning the CLIP image encoder. Subsequently, in the second stage
the satellite multi-spectral modality embeddings are aligned with the RGB
image and text embeddings, i.e. Sentinel-2 multi-spectral imagery to RGB
composites and text. The resulting embedding space, effectively associates
pairs of modalities, enabling a plethora of cross-modal retrieval and text-
based zero-shot downstream tasks.

###  III-A First Stage: Patching CLIP by interpolating weights

Large pre-trained models such as CLIP [14], ALIGN[69] and BASIC [70] have
demonstrated unprecedented robustness to a plethora of challenging
distribution shifts. However, there are still settings where their zero-shot
performance is far from optimal. To this end, Patching with Interpolation
(PAINT) [68] has emerged as a method that substantially improves accuracy
under distribution shift, while maintaining high performance on the target
distribution. PAINT employs a two-step procedure, which consists of fine-
tuning the model and then linearly interpolating between the weights of the
model before and after fine-tuning. This approach enables the expansion of the
set of tasks on which models achieve high accuracy, without introducing any
task-specific parameters, without re-training them from scratch, and without
catastrophic forgetting [71].

In order to deal with the distribution shift between natural images and
satellite imagery we initially robust fine-tune CLIP, as shown in Fig. 1,
following the patching protocol described in Ilharco et al. [68]. Eventually,
we end up with a refined embedding space, thus providing a solid foundation to
facilitate the satellite cross-modal alignment stage. Given an open-vocabulary
model, i.e. CLIP, with weights $W_{zs}$, we fine-tune $W_{zs}$ on training
data from a patching task $D_{patch}$, aiming to produce weights $W_{ft}$ that
achieve high accuracy on $D_{patch}$, without decreasing the performance on
supported tasks $D_{supp}$, where performance is already adequate. Thus, a
mixing coefficient $\alpha\in[0,1]$ is determined via held-out validation sets
for $D_{supp}$ and $D_{patch}$, in order to linearly interpolate $W_{zs}$ and
$W_{ft}$ and produce $W_{patch}=(1-\alpha)\times W_{zs}+\alpha\times W_{ft}$.
Essentially, $W_{zs}$, $W_{ft}$ and $W_{patch}$ are the weights of a CLIP
image encoder. Instead of introducing a learnable classification layer,
Ilharco et al. [68] used the frozen output of CLIP’s text, as the output layer
of the image encoder in order to map image features to the space of classes,
during the course of fine-tuning.

###  III-B Second Stage: Aligning RS imagery modalities with natural images
and text

The main objective of contrastive representation learning is to create a
shared embedding space where similar sample pairs are close together and
dissimilar pairs are far apart. This technique can be applied in both
supervised and unsupervised learning scenarios, with particular effectiveness
in self-supervised learning. Contrastive learning also allows for aligning
specific pairs of modalities, such as images and text or audio and text. By
using pairs of related and unrelated examples, contrastive learning enables
the alignment of these modalities in a joint embedding space. However, it’s
important to note that the resulting embeddings are specific to the aligned
modalities and may not directly transfer to other modalities. For instance,
embeddings obtained from aligning image and audio modalities may not be
directly applicable to text-based tasks. CLIP leverages the aligned (image-
text) embedding space for the zero-shot classification task. Given a list of
textual descriptions generated using the candidate classes, an input image is
classified based on their similarity in the embedding space. Extending the
zero-shot classification to other modalities usually requires additional
training using paired text data, e.g (audio, text).

Figure 2: The second stage of our method concerns the cross-modal alignment of
a satellite modality encoder. The image depicts: (A) the standard CLIP [14]
architecture, (B) our satellite modality alignment with the visual and textual
modalities of CLIP and (C) the emergent cross-modal alignment.

In the cross-modal alignment stage (as shown in Fig. 2), we leverage the
patched CLIP image encoder $\mathcal{M}_{\text{patched}}$, used in the first
stage, as the teacher network, and a pre-trained satellite modality encoder
$\mathcal{M}_{\text{sat}}$ as the student network. To effectively align these
two networks, we adopt a straightforward approach grounded in the assumption
that diverse modalities associated with the same sample should yield similar
embeddings within the shared CLIP embedding space. The process involves a pair
of modalities $\mathcal{I}_{RGB}$ and $\mathcal{I}_{SAT}$ corresponding to RGB
composites and some other satellite modality. For a given image
$x_{i}\in\mathcal{I}_{RGB}$ and its corresponding sample
$\widetilde{x}_{i}\in\mathcal{I}_{SAT}$ from the two modalities, we obtain
their respective embeddings $E_{i}=\mathcal{M}_{\text{patched}}(x_{i})$ and
$\widetilde{E}_{i}=\mathcal{M}_{\text{sat}}(\widetilde{x}_{i})$. In cases
where the embedding dimensions of $E_{i}$ and $\widetilde{E}_{i}$ differ, we
introduce a linear projection head for the student network
$\mathcal{M}_{\text{sat}}$ to ensure matching output embedding sizes.

Inspired by the knowledge distillation[72] loss function, as well as previous
distillation works [73, 74, 75, 76], the student is guided to mimic the
teacher’s visual and textual embeddings via a joint objective function
formulated as the linear combination of cross-entropy $\mathcal{L}_{CE}$ and
mean squared error loss $\mathcal{L}_{MSE}$.

$\mathcal{L}_{I,M}=\mathcal{L}_{MSE}(E_{i},\widetilde{E}_{i})+\lambda\times\mathcal{L}_{CE}(y_{pred},y_{true})$
(1)

$\mathcal{L}_{CE}$ is determined by labeled data supervision using the frozen
output of CLIP’s text encoder as an anchor to map image features to the space
of classes, while $\mathcal{L}_{MSE}$ encourages the student to imitate the
output embeddings of the teacher, in contrast to the softened class scores in
the case of the original knowledge distillation loss.

##  IV Experiments

###  IV-A Datasets

We utilize three benchmark datasets for RS image classification, featuring a
total of 5 different nomenclatures. These datasets vary in terms of the input
satellite data specifications, but more importantly, they have adopted
different labeling nomenclatures to describe the semantic content of the
satellite images (see fig.3).

BigEarthNet[77] comprises of 590326 atmospherically corrected Sentinel-2
(S2-L2A) patches acquired between June 2017 and May 2018 over 10 European
countries. The dataset is designed for multi-label LULC scene classification,
each image patch is a section of i) $120\times 120$ pixels for 10m bands; ii)
$60\times 60$ pixels for 20m bands; and iii) $20\times 20$ pixels for 60m
bands and is annotated with multiple land-cover classes sourced from the
CORINE Land Cover (CLC) database of the year 2018 [78]. The annotations are
based on the detailed CORINE Level-3 class nomenclature, supplemented by an
alternative 19-class LULC nomenclature [79] introduced by the authors,
specifically for use in the machine-learning domain. The dataset has been
ultimately expanded to include Synthetic Aperture Radar Sentinel-1 (S1)
patches, resulting in Sentinel-1 and Sentinel-2 patch pairs [1]. Consequently,
the term BigEarthNet now encompasses both the BigEarthNet-S1 and
BigEarthNet-S2 datasets.

TABLE I: Synopsis of the fundamental traits of the satellite datasets employed during the course of our experiments. Dataset | # of samples | Patch size | Modalities | Spatial Coverage | Temporal Coverage | Atmospherically Corrected | Nomenclature
---|---|---|---|---|---|---|---
BigEarthNet | 590.326 | 120$\times$120 | S1, S2 | Europe | Single season per patch | True | CORINE LULC L1, L2, L3
SEN12MS | 541.986 | 256$\times$256 | S1, S2 | Global | Single season per patch | False | MODIS IGBP, LCCS
EuroSAT | 27.000 | 64$\times$64 | RGB, S2 | Europe | Single season per patch | False | 10 generic LULC classes

EuroSAT [80] consists of 27000 single-date non-atmospherically corrected
Sentinel-2 (S2-L1C) patches, distributed all over Europe, of size $64\times
64$ pixels. The dataset is designed for multi-class LULC classification tasks
and features a simple 10-class nomenclature. All 27000 patches have been
manually checked.

SEN12MS[81] comprises of 180662 patch triplets that include Sentinel-1 dual-
polarization SAR data, non-atmospherically corrected Sentinel-2 (S2-L1C)
multi-spectral images and MODIS-derived land cover maps. These patches are of
size 256$\times$256 pixels and are geographically distributed across Earth’s
inhabited continents while encompassing all four meteorological seasons. The
dataset is designed for multi-label, multi-class LULC scene classification and
semantic segmentation tasks. The four-band MODIS land cover patches have been
created using 2016 data, at an upsampled pixel spacing of 10m. The first band
contains land cover following the International Geosphere-Biosphere Programme
(IGBP) classification scheme [82], while the remaining bands contain the Land
Cover Classification System (LCCS) land cover, land use and hydrology layers
[83]. The authors have ultimately redesigned their dataset [84], featuring a
simplified IGBP scheme [85] for single-label and multi-label scene
classification tasks, in order to ensure comparability with other land cover
schemes and mitigate, to some extent, the class imbalance of their dataset.

Figure 3: Comparison of image-label pairs, sampled from the satellite datasets
employed during the course of our experiments. Several innate features of the
datasets, summarized in Table I, can be observed. Specifically, the effect of
atmospheric corrections, which mitigate atmospheric interference thus
enhancing true surface reflectance and reducing color distortion, can be
highlighted when comparing BigEarthNet with EuroSAT and SEN12MS datasets.
Moreover the spatial resolution differences are evident, as well as the
variance with regards to the level of nomenclature detail used.

1

Input: Pre-trained CLIP model $\mathcal{M}$ and tokenizer $\mathcal{T}$

Input: Dataset $D=\\{(x_{i},y_{i})\\}^{n}_{i=1},D_{classnames},D_{prompts}\\}$

Input: Pre-trained satellite modality encoder $\mathcal{M}_{\text{sat}}$

Output: Aligned satellite encoder $\mathcal{M}_{\text{sat}}$

2

3

/* Create classification head. */

4 Function  _cls_head(__)_ :

5 set $\mathcal{M}$ in inference mode

6 $W_{cls}\leftarrow[]$

7 for  _$cls\in D_{classnames}$_ do

8 $T\leftarrow[]$

9 for  _$p\in D_{prompts}$_ do

10 $T\leftarrow T\cup\\{p.format(cls)$}

11

12 end for

$T\leftarrow\mathcal{T}(T)$

// tokenize

$\widetilde{E}\leftarrow\mathcal{M}(T)$

// encode text

13 Normalize & average text embeddings $\widetilde{E}$

14 $W_{cls}\leftarrow W_{cls}\cup\widetilde{E}$

15

16 end for

17

18 Stack $W_{cls}$, transpose and apply logit scaling

19 $h\leftarrow$ linear $classification\\_head$ using $W_{cls}$

20 return _$h$_

21

22

23

/* Align models $\mathcal{M}_{\text{teach}}$ and $\mathcal{M}_{\text{stud}}$.
*/

24 Function  _align(_$\mathcal{M}_{\text{teach}}$ ,
$\mathcal{M}_{\text{stud}}$_)_:

25 initialize frozen classification head $h\leftarrow cls\\_head()$

26 set $\mathcal{M}_{\text{stud}}$ to trainable and freeze
$\mathcal{M}_{\text{teach}}$

27 for  _$\\{x_{i},\widetilde{x}_{i}\\},y_{i}\in D$_ do

28 let $x_{i},\widetilde{x}_{i}$ be the two modalities for sample $i$

29 $E_{i}\leftarrow\mathcal{M}_{\text{teach}}(x_{i})$

30 $\widetilde{E}_{i}\leftarrow\mathcal{M}_{\text{stud}}(\widetilde{x_{i}})$

31 $y_{pred}\leftarrow h(\widetilde{E}_{i})$

32 $loss\leftarrow MSE(E_{i},\widetilde{E}_{i})+\lambda\times
CE(y_{pred},y_{i})$

33 …

34

35 end for

36 return _$\mathcal{M}_{\text{stud}}$_

37

38

39

40

41 Function  _main(__)_ :

/* Step 1: CLIP model Patching */

42 initialize CLIP model $\mathcal{M}$ and tokenizer $\mathcal{T}$

43 $clip\\_visual\leftarrow$ image encoder of $\mathcal{M}$

44 $clip\\_visual.fc\leftarrow cls\\_head()$

45 $W_{ft}\leftarrow$ finetune $clip\\_visual$ weights $W_{zs}$ on dataset $D$

46 $W_{paint}\leftarrow(1-\alpha)\times W_{zs}+\alpha\times W_{ft}$

47

/* Step 2: Cross-modal alignment */

48 initialize satellite modality encoder $\mathcal{M}_{\text{sat}}$

49 $clip\\_visual\leftarrow clip\\_visual.load(W_{paint})$

50 $\mathcal{M}_{\text{sat}}\leftarrow
align(clip\\_visual,\mathcal{M}_{\text{sat}},D)$

51

52 return _$\mathcal{M}_{\text{sat}}$_

53

54

55

Algorithm 1 Cross-modal alignment method.

###  IV-B Experimental Setup

Tasks. Following Ilharco et al. [68], tasks are categorized as patching and
supported tasks. Out of the diverse set of image classification tasks from
Radford et al. [14], we use ImageNet [86] as a representative supported task
and BigEarthNet-S2 [1] as a representative patching task for satellite scene
classification.

Prompts. For each downstream dataset, we use a set of pre-defined prompts for
each class, which we collected from prior works. We primarily use the prompts
assembled by Radford et al. [14] for each downstream dataset, so that we
provide a fair comparison of our results. In the case of satellite imagery
datasets that were not present in the CLIP collection of downstream datasets
(e.g. BigEarthNet), we use the alternative set of prompts proposed by Zhai et
al. [87] for satellite datasets. Our approach follows the methodology
introduced by Radford et al. [14], commonly referred to as ”prompt ensembling”
and includes the generation of multiple candidate captions for each class,
followed by the averaging of their representations prior to computing
similarities.

Models. During the course of our experiments, we employ CLIP [14] pre-trained
vision transformer (ViT) models [88]. We deliberately disregarded the ResNet-
based models, in favor of the ViT-based models, predicated on prior research
indicating that ResNets are more susceptible to catastrophic forgetting [89,
68], while such interpolated networks suffer from variance collapse [90]. For
the satellite modality encoder we utilize the SSL4EO-S12 [91] ViT-S-16 model
architecture and weights, pre-trained via self-supervision using MoCo-v3 [92].

Fine-tuning on patching task. We fine-tune CLIP, as shown in Fig. 1, using RGB
composites from the BigEarthNet-S2 dataset, with a batch size of 128 for 1
epoch using learning rate 1e-5 with 200 warm-up steps with a cosine annealing
learning rate schedule and the AdamW optimizer [93, 94] with weight decay [95]
of 0.5. Additionally, we randomly apply ColorJitter, RandomGrayScale and
GaussianBlur as data augmentations, aiming to compensate for the color
variability between true-color images of surface reflectance and top-of-the-
atmosphere reflectance products. During fine-tuning, we use CLIP’s text tower
output, as the frozen final classification layer, so that we do not introduce
any additional learnable parameters.

Cross-modal alignment. For the cross-modal alignment stage, we use a teacher-
student framework, to align a pre-trained satellite (Sentinel-2) encoder with
the CLIP image and text encoders, where the satellite encoder is the trainable
student network and CLIP is the frozen teacher network, as shown in Fig. 2. We
use again CLIP’s text tower output, as a frozen final classification layer, so
that we do not introduce any additional learnable parameters. We guide the
student to mimic teacher features, using paired Sentinel-2 multi-spectral data
and RGB composites from the BigEarthNet-S2 dataset, with a batch size of 256
for 5 epochs using learning rate 1e-4 with 2000 warm-up steps with a cosine
annealing learning rate schedule and the AdamW optimizer with weight decay of
0.01. We use the loss function listed in Eq. 1, where we empirically set
$\lambda=0.05$ in order to compensate for the difference in terms of magnitude
between the two loss terms, while also purposely prioritizing slightly the MSE
term over CE for the first few epochs, as discussed in Section V.

Evaluation. We follow the setup of Radford et al. [14]. For the assessment of
text-based zero-shot tasks, Mean Average Precision (mAP) and Accuracy (acc)
are employed as the primary evaluation metrics, for multi-label and multi-
class datasets, respectively. For each dataset, a set of pre-defined prompts
for each class is utilized, as discussed in Section IV-B. In order to compute
the embedding of each class, the embeddings of the pre-defined prompts are
first computed using the text encoder, then they are averaged and
L2-normalized. Each image is classified based on the cosine similarity of the
L2-normalized image embedding with the class embeddings. For the evaluation of
cross-modal retrieval tasks, we utilize Recall@K (R@k) as our performance
metric. For two modalities, $A$ and $B$, we compute $A\rightarrow B$ scores
using the cosine similarity between modalities $A$ and $B$ and rank the top-k
samples of modality $B$ for each sample of modality $A$.

###  IV-C Results

The first stage of Algorithm 1 consists of patching CLIP by linearly
interpolating weights (PAINT) [68]. The goal is to improve accuracy on
satellite scene classification tasks where the model performs poorly (patching
tasks), without degrading performance on tasks where accuracy is already
adequate (supported tasks).

Fig. 4 summarizes the effect of patching using the BigEarthNet-19 dataset by
highlighting the patching and supported tasks (ImageNet) performance trade-off
for different mixing coefficients. We notice that the performance for both
patching and supported tasks maximizes for $\alpha=0.5$. We report in detail
the patching results for all four CLIP ViT-based models for the patching task
(i.e. BigEarthNet-19) and the representative supported task (i.e. ImageNet)
for the different $\alpha$ weight interpolation coefficients in Appendix A.
For completeness, we also include the zero-shot performance evaluation results
of the patched models, for the different $\alpha$ coefficients, on a set of
diverse satellite scene classification nomenclatures and datasets.

Fig. 5 compares and contrasts the pre- and post-patching performance for the
patching and the representative supported tasks and the various nomenclatures
and datasets across all candidate models. Best performance trade-off across
patching and supported tasks is achieved using the larger ViT-L-14 and
ViT-L-14-336 models. These results, are in-line with those of Ramasesh et al.
[89] and Mehta et al. [96] who observed that larger models are less
susceptible to catastrophic forgetting and Ilharco et al. [68] who highlighted
that PAINT is more effective for larger models.

Figure 4:  Fine-tuning of CLIP ViT-based variants on BigEarthNet-19 and the interpolation of the weights of the fine-tuned and zero-shot model in order to find an intermediate solution where accuracy improves on the patching task without reducing accuracy on supported tasks. Accuracy is measured on BigEarthNet-19 as well as on ImageNet (representative supported task). Each plot shows the effect on the performance of the two tasks as $\alpha$ increases. Figure 5:  Zero-shot performance on a set of diverse satellite scene classification nomenclatures and datasets. Left figure shows the performance of CLIP ViT-based variants before patching. The figure in the middle shows the performance on each dataset after patching the CLIP ViT-based encoders with BigEarthNet-19. The right figure shows the performance of the ViT-S-16 satellite encoder after the cross-modal alignment stage.  TABLE II: We evaluate the cross-modal retrieval performance of our aligned models on a set of diverse satellite imagery datasets. We present the cross-modal retrieval results for all four CLIP [14] ViT-based models, for both RGB$\rightarrow$MS and MS$\rightarrow$RGB retrieval. ViT-B-32 | RGB$\rightarrow$MS | MS$\rightarrow$RGB
---|---|---
R@1 | R@5 | R@10 | R@20 | R@1 | R@5 | R@10 | R@20
BigEarthNet | 3,61 | 10,61 | 15,86 | 22,90 | 13,44 | 30,44 | 39,95 | 50,35
ImageNet | – | – | – | – | – | – | – | –
EuroSAT | 0,07 | 0,30 | 0,57 | 1,09 | 0,11 | 0,41 | 0,69 | 1,19
SEN12MS | 0,31 | 1,34 | 2,42 | 4,07 | 0,31 | 1,40 | 2,55 | 4,08
ViT-B-16 | RGB$\rightarrow$MS | MS$\rightarrow$RGB
R@1 | R@5 | R@10 | R@20 | R@1 | R@5 | R@10 | R@20
BigEarthNet | 2,81 | 8,60 | 13,16 | 19,42 | 11,21 | 27,14 | 36,67 | 47,57
ImageNet | – | – | – | – | – | – | – | –
EuroSAT | 0,07 | 0,20 | 0,44 | 1,00 | 0,11 | 0,50 | 0,81 | 1,63
SEN12MS | 0,26 | 1,09 | 1,85 | 3,24 | 0,47 | 1,78 | 3,06 | 4,88
ViT-L-14 | RGB$\rightarrow$MS | MS$\rightarrow$RGB
R@1 | R@5 | R@10 | R@20 | R@1 | R@5 | R@10 | R@20
BigEarthNet | 6,84 | 17,93 | 25,03 | 33,75 | 22,78 | 44,26 | 53,87 | 63,43
ImageNet | – | – | – | – | – | – | – | –
EuroSAT | 0,11 | 0,30 | 0,59 | 1,17 | 0,13 | 0,81 | 1,52 | 2,41
SEN12MS | 0,28 | 1,20 | 2,22 | 3,96 | 0,93 | 3,65 | 6,13 | 9,76
ViT-L-14-336 | RGB$\rightarrow$MS | MS$\rightarrow$RGB
R@1 | R@5 | R@10 | R@20 | R@1 | R@5 | R@10 | R@20
BigEarthNet | 7,61 | 19,45 | 26,96 | 35,97 | 23,09 | 44,89 | 54,73 | 64,42
ImageNet | – | – | – | – | – | – | – | –
EuroSAT | 0,02 | 0,17 | 0,26 | 0,54 | 0,17 | 0,72 | 1,31 | 2,56
SEN12MS | 0,11 | 0,57 | 1,15 | 2,05 | 0,47 | 2,12 | 3,84 | 6,55

During the cross-modal alignment stage, we are distilling each one of the four
CLIP ViT-based models into a pre-trained ViT-S-16 satellite (Sentinel-2)
modality encoder. We summarize our results for zero-shot classification in
Fig. 5. Appendix B contains more performance results per dataset and per
class. When compared to both the non-patched and patched RGB CLIP variants,
our results demonstrate notable performance improvements across all multi-
spectral datasets used for evaluation, with the exception of the EuroSAT
dataset. We argue that this is due to the objective function used during the
alignment stage and further justify this claim in Section V. For completeness,
the cross-modal retrieval results are reported in Table II, setting the
baselines for the RGB$\rightarrow$MS and MS$\rightarrow$RGB retrieval tasks
evaluation. It is evident that significantly better performance is achieved
when using a more informative modality for the cross-modal retrieval, i.e.,
multi-spectral imagery aiming to retrieve the respective RGB imagery.

###  IV-D Ablation Study

In this section we systematically investigate various design choices for each
one of the two stages of the proposed method. Unless explicitly stated, the
experimental setup is similar to Section IV-B. We conduct the entirety of the
ablation study using the smallest pre-trained CLIP model variant, namely
ViT-B-32.

Patching dataset selection. The dataset used for patching is the most
important parameter in the first stage of Algorithm 1. To this end, we
evaluate the effect of patching CLIP and the resulting zero-shot performance,
for each one of the datasets considered in Section IV-A. The experimental
results stacked for each patching dataset are presented in Fig. 6. It can be
observed that patching CLIP using any variation of the BigEarthNet dataset
(5/19/43-class nomenclature) as the proxy task, outperforms the EuroSAT and
the SEN12MS datasets in terms of both supported and patching tasks evaluation.
At the same time, BigEarthNet-19 yields the best results among the alternative
BigEarthNet class nomenclatures. The impact on the performance of the
representative supported task (i.e ImageNet) is negligible among the patching
datasets. More specifically, EuroSAT and BigEarthNet-43 resulting in the upper
and lower performance bounds, respectively, with the rest of the datasets
translating into marginal performance differences in between them.

Figure 6: Patching dataset selection ablation study. We evaluate the effect of
patching CLIP and the resulting zero-shot performance, for each one of the
datasets considered in Section IV-A.

Alignment loss variants. The alignment across all modalities can be measured
using cross-modal retrieval and text-based zero-shot tasks. To that end, we
evaluate the effectiveness of various alignment loss variants on the cross-
modal alignment. To cover every aspect, we also evaluate the ability of the
patched CLIP model, as the teacher network during the satellite modality
encoder alignment stage. We summarize our results in Table III and Table IV
for classification and retrieval tasks respectively.

The results demonstrate both the necessity and the effectiveness of the
initial patching stage. More specifically, we notice significant performance
gains on the derived cross-modal alignment when using the patched CLIP model,
for the alignment of the satellite modality encoder, instead of the original
CLIP model weights. This can be observed both in the classification task (see
Table III) and the retrieval task (see Table IV). In terms of the loss
variants evaluated, we yield the best results for both classification and
retrieval tasks, using the proposed loss of Eq. 1. The exclusive utilization
of either cross entropy (CE) or mean squared error (MSE) loss, for the
satellite modality encoder alignment, results in a tendency for alignment with
either the CLIP text or image encoder. More specifically, the aligned
satellite modality encoder gravitates towards the CLIP text encoder when using
the CE loss and towards the CLIP image encoder when using the MSE loss.This
behavior disrupts the alignment between the satellite modality encoder and the
residual modality.

TABLE III: Ablation study results, concerning the effectiveness of various alignment loss variants on the classification task. The left side of the table shows performance without patching (with the BigEarthNet-19 dataset) while the right side includes patching. The CE loss yields better results on the classification task. Classification | Non patched | Patched
---|---|---
CE | MSE | MSE & CE | CE | MSE | MSE & CE
BigEarthNet-5 | 74,07 | 35,16 | 70,27 | 74,17 | 62,86 | 78,72
BigEarthNet-19 | 81,39 | 15,18 | 67,81 | 81,38 | 51,06 | 74,58
BigEarthNet-43 | 33,49 | 6,82 | 29,09 | 33,49 | 22,5 | 33,03
ImageNet | – | – | – | – | – | –
EuroSAT | 20,13 | 20,28 | 22,83 | 20,37 | 20.57 | 23,75
SEN12MS | 46,45 | 22,83 | 41,39 | 46,45 | 41,66 | 49,06
TABLE IV: The cross-modal retrieval results for all four CLIP ViT-based
models, for both RGB$\rightarrow$MS and MS$\rightarrow$RGB retrieval. Metric
used is R@5. The left side of table shows performance for different loss
functions without patching while the right side shows performance with
patching. The results highlight the importance of the patching stage.
Additionally, the MSE loss during alignment yields better results in the
retrieval tasks.

Retrieval | Non patched | Patched
---|---|---
CE | MSE | MSE & CE | CE | MSE | MSE & CE
RGB$\rightarrow$MS | MS$\rightarrow$RGB | RGB$\rightarrow$MS | MS$\rightarrow$RGB | RGB$\rightarrow$MS | MS$\rightarrow$RGB | RGB$\rightarrow$MS | MS$\rightarrow$RGB | RGB$\rightarrow$MS | MS$\rightarrow$RGB | RGB$\rightarrow$MS | MS$\rightarrow$RGB
BigEarthNet | 0,02 | 0,01 | 8,13 | 28,72 | 5,15 | 19,03 | 0,14 | 0,02 | 17,72 | 45,26 | 10,61 | 30,44
ImageNet | – | – | – | – | – | – | – | – | – | – | – | –
EuroSAT | 0,22 | 0,13 | 0,22 | 0,44 | 0,22 | 0,37 | 0,09 | 0,17 | 0,33 | 0,69 | 0,30 | 0,41
SEN12MS | 0,07 | 0,07 | 1,00 | 1,79 | 1,15 | 1,52 | 0,14 | 0,09 | 1,81 | 2,39 | 1,34 | 1,40

##  V Discussion and Conclusion

We present a straightforward method aiming to create a joint embedding space
for distinct RS imagery modalities. Our proposed method, leverages CLIP large
scale pre-training, working towards a RS Vision-Language model (VLM), without
relying on textual descriptions, without training from scratch, without
introducing any task-specific parameters and without catastrophic forgetting.
We enable, in a computationally efficient manner, a rich set of multi-modal
tasks across different modalities and set preliminary baselines for cross-
modal retrieval and text-based zero-shot tasks using RS imagery. We aspire for
our method to serve as a stepping stone for the EO community and facilitate
novel applications on RS imagery.

###  V-A A blueprint towards satellite vision-language models

We view our proposed methodology as a blueprint for the resource-efficient
development of a RS Vision-Language model (VLM). Our experimental results
highlight two major challenges that warrant further attention: the
generalizability of our generated VLM and the need for a clearer understanding
of state-of-the-art VLMs in comparison to well-established standards and
benchmarks.

In the pursuit of enhancing the generalizability of our method, it becomes
essential to explore specific considerations and refinements that transcend
the immediate scope of our experimental setup. We deliberately patch CLIP
using a single proxy dataset, as shown in Fig. 1. Nevertheless, it should be
obvious that it is extremely difficult for a single dataset to address
multiple training objectives (i.e multi-class vs multi-label classification),
as well as the unique challenges posed by remote sensing imagery in terms of
spatial coverage, atmospheric conditions, varying perspectives, temporal and
spatial resolution, which can significantly impact the performance of DL
models. We summarize the characteristics of the datasets used during the
course of our experiments, in Table I. For completeness, we also provide a
direct comparison of per dataset sampled image-label pairs in Fig. 3, where
several differences become apparent, such as: i) the spatial resolution
differences, ii) the diversity with regards to the level of nomenclature
detail used, as well as iii) the effect of atmospheric corrections which in
the case of BigEarthNet mitigate atmospheric interference thus enhancing true
surface reflectance and reducing color distortion. A straightforward
modification to the CLIP model patching method tailored on multiple datasets
[68], should remedy the shortcomings observed in our experimental outcomes.
Such a modification is expected to significantly improve the performance of
patched models, thereby leading also to notable improvements at the cross-
modal alignment stage.

A similar modification could be also introduced to our cross-modal alignment
stage, seen in Fig. 2. We are confident that such a modification, would
substantially improve our experimental results, summarized in Fig. 4 and Table
II. Especially for multi-class classification tasks, such as EuroSAT, where
our results are currently lacking in terms of performance, since this stems
from our current way of aligning the Self-supervised Learning (SSL) pre-
trained satellite encoder using solely the BigEarthNet dataset and as a
consequence a multi-label optimization objective for the label loss part of
our proposed loss function Eq. 1. Another possible improvement could also be
the use of a larger SSL pre-trained satellite modality encoder. As of today,
Wang et al. [91] have published a series of pre-trained Sentinel-2 encoder
weights, featuring different SSL methods, yet only for ViT-S-16 architecture.
This could possibly pose a limiting factor for our cross-modal alignment
outcome, since it might outstrip the capacity of our aligned encoder to
acquire concepts, especially when the knowledge is transferred from
considerably larger CLIP encoders.

Turning our attention to benchmarking, a comprehensive examination of our
approach’s performance in comparison to established standards is essential to
validate its robustness and potential applications in diverse contexts. To
this end, recent works [97, 98, 99] have been focusing on enabling a reliable
assessment of progress, in the field of EO foundation models, by fabricating
EO benchmark suites alongside robust methodologies for evaluating models and
reporting results. Such evaluation suites, along the one used for the
evaluation of CLIP [14], could ultimately provide the community with ranked
candidate CLIP models and streamline future application within the EO era.

###  V-B Bridging modalities with distinct intrinsic features

The cross-modal alignment of complementary RS imagery modalities, beyond RGB,
is very important in the context of the EO community. Working towards this
end, we pinpoint a pivotal juncture for the successful outcome of our cross-
modal alignment stage, specifically the information gap between heterogenetic
modalities [100, 101]. Prompted by prior works [102], which determined that
contrastive learning preserves the gap between different modalities, we
suggest an alternative loss 1 formulated as the linear combination of Mean
Squared Error (MSE) and Cross Entropy (CE) losses. Contrary to Girdhar et al.
[103], which trains a joint embedding space for multiple modalities using
solely image alignment, we leverage guidance from the scaled CE signal in an
effort to alleviate the modality gap of RGB composites and RS imagery and
allow the RS modality encoder to better align within the shared embedding
space while being constrained to produce similar image embedddings, due to the
MSE loss. All in all, our alignment procedure resembles a plain cross-modal
knowledge distillation setup, where the knowledge distillation (KD) loss has
been altered to imitate the output embeddings of the teacher network, in
contrast to the softened class scores. CLIP knowledge distillation has been
receiving a lot of attention lately and has already proven its capabilities on
a series of works [104, 76, 105, 106, 107, 108], which aspire to exploit the
large scale pre-training of CLIP in an computationally efficient and
interoperable way.

## Acknowledgment

This work has received funding from the European Union’s
Horizon Europe research and innovation project ThinkingEarth under grant
agreement number 101130544.

## References

  * [1] G. Sumbul, A. De Wall, T. Kreuziger, F. Marcelino, H. Costa, P. Benevides, M. Caetano, B. Demir, and V. Markl, “Bigearthnet-mm: A large-scale, multimodal, multilabel benchmark archive for remote sensing image classification and retrieval [software and data sets],” _IEEE Geoscience and Remote Sensing Magazine_ , vol. 9, no. 3, pp. 174–180, 2021.
  * [2] Y. Wang, C. M. Albrecht, N. A. A. Braham, L. Mou, and X. X. Zhu, “Self-supervised learning in remote sensing: A review,” _IEEE Geoscience and Remote Sensing Magazine_ , vol. 10, no. 4, pp. 213–247, 2022.
  * [3] F. Hu, G.-S. Xia, J. Hu, and L. Zhang, “Transferring deep convolutional neural networks for the scene classification of high-resolution remote sensing imagery,” _Remote Sensing_ , vol. 7, no. 11, pp. 14 680–14 707, 2015.
  * [4] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill _et al._ , “On the opportunities and risks of foundation models,” _arXiv preprint arXiv:2108.07258_ , 2021.
  * [5] P. W. Koh, S. Sagawa, H. Marklund, S. M. Xie, M. Zhang, A. Balsubramani, W. Hu, M. Yasunaga, R. L. Phillips, I. Gao _et al._ , “Wilds: A benchmark of in-the-wild distribution shifts,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 5637–5664.
  * [6] N. I. Bountos, I. Papoutsis, D. Michail, A. Karavias, P. Elias, and I. Parcharidis, “Hephaestus: A large scale multitask dataset towards insar understanding,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 1453–1462.
  * [7] B. Qu, X. Li, D. Tao, and X. Lu, “Deep semantic understanding of high resolution remote sensing image,” in _2016 International conference on computer, information and telecommunication systems (Cits)_. IEEE, 2016, pp. 1–5.
  * [8] X. Lu, B. Wang, X. Zheng, and X. Li, “Exploring models and data for remote sensing image caption generation,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 56, no. 4, pp. 2183–2195, 2017.
  * [9] Z. Yuan, W. Zhang, K. Fu, X. Li, C. Deng, H. Wang, and X. Sun, “Exploring a fine-grained multiscale method for cross-modal remote sensing image retrieval,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 60, pp. 1–19, 2022.
  * [10] Y. Zhan, Z. Xiong, and Y. Yuan, “Rsvg: Exploring data and models for visual grounding on remote sensing data,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 61, pp. 1–13, 2023.
  * [11] S. Shen, L. H. Li, H. Tan, M. Bansal, A. Rohrbach, K.-W. Chang, Z. Yao, and K. Keutzer, “How much can clip benefit vision-and-language tasks?” _arXiv preprint arXiv:2107.06383_ , 2021.
  * [12] M. Barraco, M. Cornia, S. Cascianelli, L. Baraldi, and R. Cucchiara, “The unreasonable effectiveness of clip features for image captioning: an experimental analysis,” in _proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 4662–4670.
  * [13] W. Li, L. Zhu, L. Wen, and Y. Yang, “Decap: Decoding clip latents for zero-shot captioning via text-only training,” _arXiv preprint arXiv:2303.03032_ , 2023.
  * [14] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark _et al._ , “Learning transferable visual models from natural language supervision,” in _International conference on machine learning_. PMLR, 2021, pp. 8748–8763.
  * [15] M. Wang, J. Xing, and Y. Liu, “Actionclip: A new paradigm for video action recognition,” _arXiv preprint arXiv:2109.08472_ , 2021.
  * [16] Z. Wang, Y. Lu, Q. Li, X. Tao, Y. Guo, M. Gong, and T. Liu, “Cris: Clip-driven referring image segmentation,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 11 686–11 695.
  * [17] O. Patashnik, Z. Wu, E. Shechtman, D. Cohen-Or, and D. Lischinski, “Styleclip: Text-driven manipulation of stylegan imagery,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 2085–2094.
  * [18] M. Cornia, L. Baraldi, G. Fiameni, and R. Cucchiara, “Universal captioner: Inducing content-style separation in vision-and-language model training,” _arXiv preprint arXiv:2111.12727_ , 2021.
  * [19] R. Mokady, A. Hertz, and A. H. Bermano, “Clipcap: Clip prefix for image captioning,” _arXiv preprint arXiv:2111.09734_ , 2021.
  * [20] M. Tang, Z. Wang, Z. Liu, F. Rao, D. Li, and X. Li, “Clip4caption: Clip for video caption,” in _Proceedings of the 29th ACM International Conference on Multimedia_ , 2021, pp. 4858–4862.
  * [21] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever, “Zero-shot text-to-image generation,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 8821–8831.
  * [22] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, “Hierarchical text-conditional image generation with clip latents,” _arXiv preprint arXiv:2204.06125_ , vol. 1, no. 2, p. 3, 2022.
  * [23] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds _et al._ , “Flamingo: a visual language model for few-shot learning,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 23 716–23 736, 2022.
  * [24] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo _et al._ , “Segment anything,” _arXiv preprint arXiv:2304.02643_ , 2023.
  * [25] G. Ilharco, M. Wortsman, R. Wightman, C. Gordon, N. Carlini, R. Taori, A. Dave, V. Shankar, H. Namkoong, J. Miller, H. Hajishirzi, A. Farhadi, and L. Schmidt, “Openclip,” Jul. 2021, if you use this software, please cite it as below. [Online]. Available: https://doi.org/10.5281/zenodo.5143773
  * [26] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman _et al._ , “Laion-5b: An open large-scale dataset for training next generation image-text models,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 25 278–25 294, 2022.
  * [27] M. Cherti, R. Beaumont, R. Wightman, M. Wortsman, G. Ilharco, C. Gordon, C. Schuhmann, L. Schmidt, and J. Jitsev, “Reproducible scaling laws for contrastive language-image learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 2818–2829.
  * [28] S. Y. Gadre, G. Ilharco, A. Fang, J. Hayase, G. Smyrnis, T. Nguyen, R. Marten, M. Wortsman, D. Ghosh, J. Zhang _et al._ , “Datacomp: In search of the next generation of multimodal datasets,” _arXiv preprint arXiv:2304.14108_ , 2023.
  * [29] A. Fang, A. M. Jose, A. Jain, L. Schmidt, A. Toshev, and V. Shankar, “Data filtering networks,” _arXiv preprint arXiv:2309.17425_ , 2023.
  * [30] H. Xu, S. Xie, X. E. Tan, P.-Y. Huang, R. Howes, V. Sharma, S.-W. Li, G. Ghosh, L. Zettlemoyer, and C. Feichtenhofer, “Demystifying clip data,” _arXiv preprint arXiv:2309.16671_ , 2023.
  * [31] Q. Sun, Y. Fang, L. Wu, X. Wang, and Y. Cao, “Eva-clip: Improved training techniques for clip at scale,” _arXiv preprint arXiv:2303.15389_ , 2023.
  * [32] X. Li, Z. Wang, and C. Xie, “An inverse scaling law for clip training,” _arXiv preprint arXiv:2305.07017_ , 2023.
  * [33] ——, “Clipa-v2: Scaling clip training with 81.1% zero-shot imagenet accuracy within a $10,000 budget; an extra $4,000 unlocks 81.8% accuracy,” _arXiv preprint arXiv:2306.15658_ , 2023.
  * [34] X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer, “Sigmoid loss for language image pre-training,” in _2023 IEEE/CVF International Conference on Computer Vision (ICCV)_. Los Alamitos, CA, USA: IEEE Computer Society, oct 2023, pp. 11 941–11 952. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/ICCV51070.2023.01100
  * [35] Z. Wang, Z. Wu, D. Agarwal, and J. Sun, “MedCLIP: Contrastive learning from unpaired medical images and text,” in _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , Y. Goldberg, Z. Kozareva, and Y. Zhang, Eds. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics, Dec. 2022, pp. 3876–3887. [Online]. Available: https://aclanthology.org/2022.emnlp-main.256
  * [36] S. Eslami, C. Meinel, and G. De Melo, “Pubmedclip: How much does clip benefit visual question answering in the medical domain?” in _Findings of the Association for Computational Linguistics: EACL 2023_ , 2023, pp. 1151–1163.
  * [37] S. Zhang, Y. Xu, N. Usuyama, J. Bagga, R. Tinn, S. Preston, R. Rao, M. Wei, N. Valluri, C. Wong _et al._ , “Large-scale domain-specific pretraining for biomedical vision-language processing,” _arXiv preprint arXiv:2303.00915_ , 2023.
  * [38] Z. Zhao, Y. Liu, H. Wu, Y. Li, S. Wang, L. Teng, D. Liu, X. Li, Z. Cui, Q. Wang _et al._ , “Clip in medical imaging: A comprehensive survey,” _arXiv preprint arXiv:2312.07353_ , 2023.
  * [39] O. Pelka, S. Koitka, J. Rückert, F. Nensa, and C. M. Friedrich, “Radiology objects in context (roco): a multimodal image dataset,” in _Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis: 7th Joint International Workshop, CVII-STENT 2018 and Third International Workshop, LABELS 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16, 2018, Proceedings 3_. Springer, 2018, pp. 180–189.
  * [40] S. Subramanian, L. L. Wang, S. Mehta, B. Bogin, M. van Zuylen, S. Parasa, S. Singh, M. Gardner, and H. Hajishirzi, “Medicat: A dataset of medical images, captions, and textual references,” _arXiv preprint arXiv:2010.06000_ , 2020.
  * [41] W. Lin, Z. Zhao, X. Zhang, C. Wu, Y. Zhang, Y. Wang, and W. Xie, “Pmc-clip: Contrastive language-image pre-training using biomedical documents,” _arXiv preprint arXiv:2303.07240_ , 2023.
  * [42] J. Liu, Z. Wang, Q. Ye, D. Chong, P. Zhou, and Y. Hua, “Qilin-med-vl: Towards chinese large vision-language model for general healthcare,” _arXiv preprint arXiv:2310.17956_ , 2023.
  * [43] A. E. Johnson, T. J. Pollard, S. J. Berkowitz, N. R. Greenbaum, M. P. Lungren, C.-y. Deng, R. G. Mark, and S. Horng, “Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports,” _Scientific data_ , vol. 6, no. 1, p. 317, 2019.
  * [44] A. Bustos, A. Pertusa, J.-M. Salinas, and M. De La Iglesia-Vaya, “Padchest: A large chest x-ray image dataset with multi-label annotated reports,” _Medical image analysis_ , vol. 66, p. 101797, 2020.
  * [45] M. Li, W. Cai, R. Liu, Y. Weng, X. Zhao, C. Wang, X. Chen, Z. Liu, C. Pan, M. Li _et al._ , “Ffa-ir: Towards an explainable and reliable medical report generation benchmark,” in _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_ , 2021.
  * [46] Z. Huang, F. Bianchi, M. Yuksekgonul, T. J. Montine, and J. Zou, “A visual–language foundation model for pathology image analysis using medical twitter,” _Nature medicine_ , vol. 29, no. 9, pp. 2307–2316, 2023.
  * [47] W. O. Ikezogwo, M. S. Seyfioglu, F. Ghezloo, D. S. C. Geva, F. S. Mohammed, P. K. Anand, R. Krishna, and L. Shapiro, “Quilt-1m: One million image-text pairs for histopathology,” _arXiv preprint arXiv:2306.11207_ , 2023.
  * [48] A. Arutiunian, D. Vidhani, G. Venkatesh, M. Bhaskar, R. Ghosh, and S. Pal, “Fine tuning clip with remote sensing (satellite) images and captions,” 2021.
  * [49] M. Czerkawski, R. Atkinson, and C. Tachtatzis, “Detecting cloud presence in satellite images using the rgb-based clip vision-language model,” in _International Geoscience and Remote Sensing Symposium_ , 2023.
  * [50] M. Singha, A. Jha, B. Solanki, S. Bose, and B. Banerjee, “Applenet: Visual attention parameterized prompt learning for few-shot remote sensing image generalization using clip,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023.
  * [51] F. Liu, D. Chen, Z. Guan, X. Zhou, J. Zhu, and J. Zhou, “Remoteclip: A vision language foundation model for remote sensing,” _arXiv preprint arXiv:2306.11029_ , 2023.
  * [52] Z. Zhang, T. Zhao, Y. Guo, and J. Yin, “Rs5m: A large scale vision-language dataset for remote sensing vision-language foundation model,” _arXiv preprint arXiv:2306.11300_ , 2023.
  * [53] Y. Yuan, Y. Zhan, and Z. Xiong, “Parameter-efficient transfer learning for remote sensing image-text retrieval,” _IEEE Transactions on Geoscience and Remote Sensing_ , 2023.
  * [54] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly, “Parameter-efficient transfer learning for nlp,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 2790–2799.
  * [55] S. Mo, M. Kim, K. Lee, and J. Shin, “S-clip: Semi-supervised vision-language learning using few specialist captions,” in _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.
  * [56] A. Bhattacharya, M. Singha, A. Jha, and B. Banerjee, “C-saw: Self-supervised prompt learning for image generalization in remote sensing,” _arXiv preprint arXiv:2311.15812_ , 2023.
  * [57] A. Dhakal, A. Ahmad, S. Khanal, S. Sastry, and N. Jacobs, “Sat2cap: Mapping fine-grained textual descriptions from satellite images,” _arXiv preprint arXiv:2307.15904_ , 2023.
  * [58] U. Mall, C. P. Phoo, M. K. Liu, C. Vondrick, B. Hariharan, and K. Bala, “Remote sensing vision-language foundation models without annotations via ground remote alignment,” _arXiv preprint arXiv:2312.06960_ , 2023.
  * [59] E. Rolf, K. Klemmer, C. Robinson, and H. Kerner, “Mission critical–satellite data is a distinct modality in machine learning,” _arXiv preprint arXiv:2402.01444_ , 2024.
  * [60] J. Li, D. Hong, L. Gao, J. Yao, K. Zheng, B. Zhang, and J. Chanussot, “Deep learning in multimodal remote sensing data fusion: A comprehensive review,” _International Journal of Applied Earth Observation and Geoinformation_ , vol. 112, p. 102926, 2022.
  * [61] Z. Zhang, G. Vosselman, M. Gerke, D. Tuia, and M. Y. Yang, “Change detection between multimodal remote sensing data using siamese cnn,” 2018.
  * [62] B. Uzkent, E. Sheehan, C. Meng, Z. Tang, M. Burke, D. Lobell, and S. Ermon, “Learning to interpret satellite images in global scale using wikipedia,” _arXiv preprint arXiv:1905.02506_ , 2019.
  * [63] A. Toker, Q. Zhou, M. Maximov, and L. Leal-Taixé, “Coming down to earth: Satellite-to-street view synthesis for geo-localization,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 6488–6497.
  * [64] M. Allen, F. Dorr, J. A. Gallego-Mejia, L. Martínez-Ferrer, A. Jungbluth, F. Kalaitzis, and R. Ramos-Pollán, “Fewshot learning on global multimodal embeddings for earth observation tasks,” _arXiv preprint arXiv:2310.00119_ , 2023.
  * [65] K. Klemmer, E. Rolf, C. Robinson, L. Mackey, and M. Rußwurm, “Satclip: Global, general-purpose location embeddings with satellite imagery,” _arXiv preprint arXiv:2311.17179_ , 2023.
  * [66] V. V. Cepeda, G. K. Nayak, and M. Shah, “Geoclip: Clip-inspired alignment between locations and images for effective worldwide geo-localization,” _arXiv preprint arXiv:2309.16020_ , 2023.
  * [67] S. Khanal, S. Sastry, A. Dhakal, and N. Jacobs, “Learning tri-modal embeddings for zero-shot soundscape mapping,” _arXiv preprint arXiv:2309.10667_ , 2023.
  * [68] G. Ilharco, M. Wortsman, S. Y. Gadre, S. Song, H. Hajishirzi, S. Kornblith, A. Farhadi, and L. Schmidt, “Patching open-vocabulary models by interpolating weights,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 29 262–29 277, 2022.
  * [69] C. Jia, Y. Yang, Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. Le, Y.-H. Sung, Z. Li, and T. Duerig, “Scaling up visual and vision-language representation learning with noisy text supervision,” in _International conference on machine learning_. PMLR, 2021, pp. 4904–4916.
  * [70] H. Pham, Z. Dai, G. Ghiasi, K. Kawaguchi, H. Liu, A. W. Yu, J. Yu, Y.-T. Chen, M.-T. Luong, Y. Wu _et al._ , “Combined scaling for zero-shot transfer learning,” _Neurocomputing_ , vol. 555, p. 126658, 2023.
  * [71] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska _et al._ , “Overcoming catastrophic forgetting in neural networks,” _Proceedings of the national academy of sciences_ , vol. 114, no. 13, pp. 3521–3526, 2017.
  * [72] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” _arXiv preprint arXiv:1503.02531_ , 2015.
  * [73] Z. Fang, J. Wang, X. Hu, L. Wang, Y. Yang, and Z. Liu, “Compressing visual-linguistic model via knowledge distillation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 1428–1438.
  * [74] J. Yang, B. Martinez, A. Bulat, and G. Tzimiropoulos, “Knowledge distillation via softmax regression representation learning,” in _International Conference on Learning Representations_ , 2020.
  * [75] T. Kim, J. Oh, N. Kim, S. Cho, and S.-Y. Yun, “Comparing kullback-leibler divergence and mean squared error loss in knowledge distillation,” _arXiv preprint arXiv:2105.08919_ , 2021.
  * [76] C. Yang, Z. An, L. Huang, J. Bi, X. Yu, H. Yang, and Y. Xu, “Clip-kd: An empirical study of distilling clip models,” 2023.
  * [77] G. Sumbul, M. Charfuelan, B. Demir, and V. Markl, “Bigearthnet: A large-scale benchmark archive for remote sensing image understanding,” in _IGARSS 2019-2019 IEEE International Geoscience and Remote Sensing Symposium_. IEEE, 2019, pp. 5901–5904.
  * [78] Copernicus. (2018) Corine land cover 2018. [Online]. Available: https://land.copernicus.eu/pan-european/corine-land-cover/clc2018
  * [79] G. Sumbul, J. Kang, T. Kreuziger, F. Marcelino, H. Costa, P. Benevides, M. Caetano, and B. Demir, “Bigearthnet dataset with a new class-nomenclature for remote sensing image understanding,” 2020.
  * [80] P. Helber, B. Bischke, A. Dengel, and D. Borth, “Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification,” _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , 2019.
  * [81] M. Schmitt, L. H. Hughes, C. Qiu, and X. X. Zhu, “Sen12ms – a curated dataset of georeferenced multi-spectral sentinel-1/2 imagery for deep learning and data fusion,” in _ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences_ , vol. IV-2/W7, 2019, pp. 153–160.
  * [82] T. R. Loveland and A. Belward, “The international geosphere biosphere programme data and information system global land cover data set (discover),” _Acta Astronautica_ , vol. 41, no. 4-10, pp. 681–689, 1997.
  * [83] A. Di Gregorio, _Land cover classification system: classification concepts and user manual: LCCS_. Food & Agriculture Org., 2005, vol. 2.
  * [84] M. Schmitt and Y.-L. Wu, “Remote sensing image classification with the sen12ms dataset,” in _ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences_ , vol. V-2-2021, 2021, pp. 101–106.
  * [85] N. Yokoya, P. Ghamisi, R. Hansch, and M. Schmitt, “Report on the 2020 ieee grss data fusion contest-global land cover mapping with weak supervision [technical committees],” _IEEE Geoscience and Remote Sensing Magazine_ , vol. 8, no. 4, pp. 134–137, 2020.
  * [86] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _2009 IEEE conference on computer vision and pattern recognition_. Ieee, 2009, pp. 248–255.
  * [87] X. Zhai, X. Wang, B. Mustafa, A. Steiner, D. Keysers, A. Kolesnikov, and L. Beyer, “Lit: Zero-shot transfer with locked-image text tuning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 18 123–18 133.
  * [88] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” _arXiv preprint arXiv:2010.11929_ , 2020.
  * [89] V. V. Ramasesh, A. Lewkowycz, and E. Dyer, “Effect of scale on catastrophic forgetting in neural networks,” in _International Conference on Learning Representations_ , 2021.
  * [90] K. Jordan, H. Sedghi, O. Saukh, R. Entezari, and B. Neyshabur, “Repair: Renormalizing permuted activations for interpolation repair,” _arXiv preprint arXiv:2211.08403_ , 2022.
  * [91] Y. Wang, N. A. A. Braham, Z. Xiong, C. Liu, C. M. Albrecht, and X. X. Zhu, “Ssl4eo-s12: A large-scale multi-modal, multi-temporal dataset for self-supervised learning in earth observation,” _arXiv preprint arXiv:2211.07044_ , 2022.
  * [92] X. Chen, S. Xie, and K. He, “An empirical study of training self-supervised vision transformers,” in _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_. Los Alamitos, CA, USA: IEEE Computer Society, oct 2021, pp. 9620–9629. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/ICCV48922.2021.00950
  * [93] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [94] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” _arXiv preprint arXiv:1711.05101_ , 2017.
  * [95] M. Andriushchenko, F. D’Angelo, A. Varre, and N. Flammarion, “Why do we need weight decay in modern deep learning?” 2023.
  * [96] S. V. Mehta, D. Patil, S. Chandar, and E. Strubell, “An empirical investigation of the role of pre-training in lifelong learning,” _Journal of Machine Learning Research_ , vol. 24, no. 214, pp. 1–50, 2023.
  * [97] J. Roberts, K. Han, and S. Albanie, “Satin: A multi-task metadataset for classifying satellite imagery using vision-language models,” _arXiv preprint arXiv:2304.11619_ , 2023.
  * [98] A. Lacoste, N. Lehmann, P. Rodriguez, E. D. Sherwin, H. Kerner, B. Lütjens, J. A. Irvin, D. Dao, H. Alemohammad, A. Drouin _et al._ , “Geo-bench: Toward foundation models for earth monitoring,” _arXiv preprint arXiv:2306.03831_ , 2023.
  * [99] N. I. Bountos, A. Ouaknine, and D. Rolnick, “Fomo-bench: a multi-modal, multi-scale and multi-task forest monitoring benchmark for remote sensing foundation models,” _arXiv preprint arXiv:2312.10114_ , 2023.
  * [100] P. Shi, M. C. Welle, M. Björkman, and D. Kragic, “Towards understanding the modality gap in clip,” in _ICLR 2023 Workshop on Multimodal Representation Learning: Perks and Pitfalls_ , 2023.
  * [101] H. Fu, R. Wu, C. Liu, and J. Sun, “Mcen: Bridging cross-modal gap between cooking recipes and dish images with latent variable model,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 14 570–14 580.
  * [102] V. W. Liang, Y. Zhang, Y. Kwon, S. Yeung, and J. Y. Zou, “Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 17 612–17 625, 2022.
  * [103] R. Girdhar, A. El-Nouby, Z. Liu, M. Singh, K. V. Alwala, A. Joulin, and I. Misra, “Imagebind: One embedding space to bind them all,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 15 180–15 190.
  * [104] K. Wu, H. Peng, Z. Zhou, B. Xiao, M. Liu, L. Yuan, H. Xuan, M. Valenzuela, X. S. Chen, X. Wang _et al._ , “Tinyclip: Clip distillation via affinity mimicking and weight inheritance,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023, pp. 21 970–21 980.
  * [105] R. Pei, J. Liu, W. Li, B. Shao, S. Xu, P. Dai, J. Lu, and Y. Yan, “Clipping: Distilling clip-based models with a student base for video-language retrieval,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 18 983–18 992.
  * [106] Z. Wang, N. Codella, Y.-C. Chen, L. Zhou, J. Yang, X. Dai, B. Xiao, H. You, S.-F. Chang, and L. Yuan, “Clip-td: Clip targeted distillation for vision-language tasks,” _arXiv preprint arXiv:2201.05729_ , 2022.
  * [107] H.-H. Wu, P. Seetharaman, K. Kumar, and J. P. Bello, “Wav2clip: Learning robust audio representations from clip,” in _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2022, pp. 4563–4567.
  * [108] A. Guzhov, F. Raue, J. Hees, and A. Dengel, “Audioclip: Extending clip to image, text and audio,” in _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2022, pp. 976–980.

##  Appendix A Patched models zero-shot performance

In this section we report in detail the patching results for all four CLIP
ViT-based models for the patching task (i.e. BigEarthNet-19) and the
representative supported task (i.e. ImageNet) for the different $\alpha$
weight interpolation coefficients. For completeness, we also include the zero-
shot performance evaluation results of the patched models, for the different
$\alpha$ coefficients, on a set of diverse satellite scene classification
nomenclatures and datasets. See Table V.

TABLE V: We evaluate the zero-shot performance of the patched models on a set
of diverse satellite scene classification nomenclatures and datasets. We
consider BigEarthNet-19 as a proxy dataset for the satellite scene
classification task, and ImageNet as a representative supported task. We
present the patching results for all four CLIP [14] ViT-based models, in
conjunction with the different $\alpha$ weight interpolation values.

ViT-B-32 | $\boldsymbol{\alpha}$
---|---
0,00 | 0,10 | 0,20 | 0,30 | 0,40 | 0,50 | 0,60 | 0,70 | 0,80 | 0,90 | 1,00
BigEarthNet-19 (mAP) | 15,18 | 15,18 | 15,18 | 16,38 | 29,55 | 48,58 | 67,1 | 73,04 | 75,15 | 76,01 | 76,28
ImageNet (acc) | 63,25 | 63,29 | 62,78 | 62,0 | 60,66 | 59,62 | 58,31 | 56,41 | 54,51 | 52,73 | 50.45
EuroSAT (acc) | 44,53 | 50,59 | 55,52 | 57,81 | 58,54 | 58,42 | 54,69 | 48,48 | 39,49 | 28,39 | 17,2
SEN12MS (mAP) | 22,83 | 22,83 | 22,83 | 22,84 | 30,46 | 41,32 | 47,85 | 50,23 | 50,29 | 50,14 | 50,06
BigEarthNet-5 (mAP) | 35,16 | 35,16 | 35,16 | 37,74 | 52,29 | 60,56 | 68,04 | 69,10 | 69,20 | 69,48 | 69,62
BigEarthNet-43 (mAP) | 6,82 | 6,82 | 6,82 | 7,57 | 13,11 | 21,24 | 29,43 | 32,36 | 33,15 | 33,28 | 33,14
ViT-B-16 | $\boldsymbol{\alpha}$
0,00 | 0,10 | 0,20 | 0,30 | 0,40 | 0,50 | 0,60 | 0,70 | 0,80 | 0,90 | 1,00
BigEarthNet-19 (mAP) | 15,18 | 15,18 | 15,18 | 15,59 | 21,62 | 41,37 | 64,05 | 73,35 | 76,04 | 77,28 | 77,76
ImageNet (acc) | 68,38 | 68,3 | 67,92 | 67,27 | 65,78 | 63,8 | 58,68 | 51,01 | 40,54 | 28,45 | 16,88
EuroSAT (acc) | 53,76 | 55,72 | 55,79 | 53,54 | 50,47 | 45,12 | 37,62 | 39,65 | 41,76 | 41,99 | 40,42
SEN12MS (mAP) | 22,83 | 22,83 | 22,83 | 22,83 | 23,84 | 30,03 | 38,52 | 45,24 | 46,10 | 44,91 | 43,82
BigEarthNet-5 (mAP) | 35,16 | 35,16 | 35,16 | 35,88 | 45,13 | 56,79 | 67,07 | 73,86 | 74,46 | 74,04 | 73,62
BigEarthNet-43 (mAP) | 6,82 | 6,82 | 6,82 | 7,10 | 9,73 | 16,72 | 26,65 | 31,55 | 32,36 | 32,52 | 32,48
ViT-L-14 | $\boldsymbol{\alpha}$
0,00 | 0,10 | 0,20 | 0,30 | 0,40 | 0,50 | 0,60 | 0,70 | 0,80 | 0,90 | 1,00
BigEarthNet-19 (mAP) | 15,18 | 15,2 | 15,75 | 24,5 | 42,93 | 65,92 | 76,84 | 79,66 | 80,78 | 81,33 | 81,54
ImageNet (acc) | 75,49 | 75,33 | 75,19 | 74,96 | 74,72 | 74,21 | 73,59 | 72,62 | 71,36 | 69,37 | 66,24
EuroSAT (acc) | 63,72 | 69,0 | 70,96 | 72,05 | 72,27 | 72,25 | 72,04 | 71,36 | 69,81 | 67,35 | 64,15
SEN12MS (mAP) | 22,83 | 22,83 | 22,87 | 24,79 | 34,17 | 42,03 | 48,43 | 51,19 | 51,04 | 50,44 | 50,06
BigEarthNet-5 (mAP) | 35,16 | 35,17 | 35,74 | 50,59 | 58,86 | 69,90 | 76,72 | 77,14 | 76,55 | 76,15 | 75,97
BigEarthNet-43 (mAP) | 6,83 | 6,83 | 7,06 | 10,98 | 18,92 | 30,14 | 37,84 | 39,87 | 40,14 | 39,91 | 39,43
ViT-L-14-336 | $\boldsymbol{\alpha}$
0,00 | 0,10 | 0,20 | 0,30 | 0,40 | 0,50 | 0,60 | 0,70 | 0,80 | 0,90 | 1,00
BigEarthNet-19 (mAP) | 15,18 | 15,18 | 16,21 | 34,79 | 59,47 | 73,59 | 78,12 | 80,18 | 81,27 | 81,83 | 82,05
ImageNet (acc) | 76,49 | 76,42 | 76,3 | 76,13 | 75,83 | 75,32 | 74,68 | 73,81 | 72,51 | 70,21 | 66,54
EuroSAT (acc) | 62,88 | 68,11 | 68,89 | 66,23 | 69,07 | 68,2 | 64,83 | 61,97 | 59,78 | 57,52 | 54,82
SEN12MS (mAP) | 22,83 | 22,83 | 22,83 | 22,98 | 27,98 | 39,75 | 48,33 | 50,50 | 50,49 | 49,72 | 48,85
BigEarthNet-5 (mAP) | 35,16 | 35,16 | 36,05 | 60,25 | 70,59 | 76,95 | 78,68 | 78,82 | 78,88 | 79,05 | 79,23
BigEarthNet-43 (mAP) | 6,82 | 6,82 | 7,12 | 14,96 | 27,45 | 36,01 | 38,64 | 39,38 | 39,50 | 39,40 | 39,13

##  Appendix B Per-class performance

In this section we present the per-class performance of our best CLIP-L-14
model. As always we present the performance of the model pre- and post-
patching with BigEarthNet-19. Additionally the plots contain the performance
of the ViT-S-16 after alignment with the CLIP-L-14 model. Figures 7 and 8
present per-class performance on BigEarthNet-5 and BigEarthNet-19
respectively, while Figure 9 present per-class performance on BigEarthNet-43.
Finally, Figures 10 and 11 contain per-class performance on SEN12MS and
EuroSAT.

In the case of SEN12MS, we replicated the experimental setup reported by the
authors and used their published PyTorch checkpoints in order to calculate mAP
for their SOTA ResNet50 model and provide a fair comparison with our results.
Unfortunately, there were no publicly available PyTorch checkpoints for
EuroSAT and BigEarthNet, to evaluate per class performance.

Figure 7: Zero-shot performance comparison of our best performing CLIP-L-14
model on BigEarthNet-43 RGB, pre- and post-patching on BigEarthNet-19 RGB.
Additionally, performance of the pre-trained ViT-S-16 after alignment with the
patched CLIP-L-14 model. BigEarthNet-5 features a much simpler 5-class
nomenclature, in comparison to the BigEarthNet-19 19-class one. Figure 8:
Performance comparison of our best performing CLIP-L-14 model, pre- and post-
patching on BigEarthNet-19 RGB. Additionally, performance of the pre-trained
ViT-S-16 after alignment with the patched CLIP-L-14 model. Figure 9: Zero-shot
performance comparison of our best performing CLIP-L-14 model on
BigEarthNet-43 RGB, pre- and post-patching on BigEarthNet-19 RGB.
Additionally, performance of the pre-trained ViT-S-16 after alignment with the
patched CLIP-L-14 model. BigEarthNet-43 features a detailed 43-class
nomenclature, in comparison to the BigEarthNet-19 19-class one. Figure 10:
Zero-shot performance comparison of our best performing CLIP-L-14 model on
SEN12MS RGB, pre- and post-patching on BigEarthNet-19 RGB. Additionally,
performance of the pre-trained ViT-S-16 after alignment with the patched
CLIP-L-14 model. We also include the evaluation results of the ResNet50 SOTA
model published by the dataset authors. SEN12MS features a 9-class
nomenclature. Figure 11: Zero-shot performance comparison of our best
performing CLIP-L-14 model on EuroSAT RGB, pre- and post-patching on
BigEarthNet-19. Additionally, performance of the pre-trained ViT-S-16 after
alignment with the patched CLIP-L-14 model. EuroSAT features a 10-class
nomenclature.