#  SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite
Imagery

Yezhen Cong
<EMAIL_ADDRESS>
&Samar Khanna11footnotemark: 1
<EMAIL_ADDRESS>
&Chenlin Meng
&Patrick Liu
&Erik Rozi
&Yutong He
&Marshall Burke
&David B. Lobell
&Stefano Ermon
&Stanford University Equal contribution. Order determined via coin flip.

###### Abstract

Unsupervised pre-training methods for large vision models have shown to
enhance performance on downstream supervised tasks. Developing similar
techniques for satellite imagery presents significant opportunities as
unlabelled data is plentiful and the inherent temporal and multi-spectral
structure provides avenues to further improve existing pre-training
strategies. In this paper, we present SatMAE, a pre-training framework for
temporal or multi-spectral satellite imagery based on Masked Autoencoder
(MAE). To leverage temporal information, we include a temporal embedding along
with independently masking image patches across time. In addition, we
demonstrate that encoding multi-spectral data as groups of bands with distinct
spectral positional encodings is beneficial. Our approach yields strong
improvements over previous state-of-the-art techniques, both in terms of
supervised learning performance on benchmark datasets (up to $\uparrow$ 7%),
and transfer learning performance on downstream remote sensing tasks,
including land cover classification (up to $\uparrow$ 14%) and semantic
segmentation.

##  1 Introduction

Figure 1: Our proposed SSL framework on temporal and multi-spectral satellite
imagery. With carefully-designed masking strategy across image layers of
different time or spectrum, and temporal and spectral encoding, our SatMAE
serves as a powerful vision learner for remote sensing tasks.

In recent years, self-supervised learning techniques have quickly become the
norm for pre-training models on large-scale natural image datasets [1, 2, 3,
4, 5, 6, 7, 8], and have demonstrated strong performance on downstream tasks
including image classification [3, 4, 9, 10], image segmentation [3, 11],
representation learning [12, 13, 14], image compression [12, 15], and image
reconstruction [1]. Unlike supervised learning approaches, self-supervised
learning techniques do not require human labeling, making them appealing in
settings where unlabeled data are abundant but labeled data are scarce, such
as remote sensing data (e.g., satellite imagery). While several large-scale
satellite image datasets have been carefully curated in the past few years,
including Functional Map of the World (fMoW) [16], BigEarthNet [17], xView
[18], SpaceNet [19], annotating these datasets requires specialized skills and
is more expensive than traditional computer vision datasets. Moreover,
automatic analysis of satellite imagery is often needed for tasks with large
societal impact such as poverty or crop yield prediction[20, 21, 22, 23, 24,
25], where acquiring large amounts of labeled data through surveys is
impossible or prohibitively expensive. This suggests that self-supervised
learning approaches for satellite imagery could be especially valuable.

However, existing self-supervised learning approaches [1, 2, 3, 4, 5, 6] are
mainly designed for natural images. As opposed to natural images such as
ImageNet [26], satellite imagery is usually associated with meaningful
geographical and temporal information, and can consist of multiple spectral
bands representing various sensor readings besides visible light (i.e., RGB
channels typical in natural images). Depending on the data source, satellite
imagery can also vary significantly in resolution. While self-supervised
learning methods for satellite imagery exist [27, 28], these approaches cannot
learn general representations for both temporal and multi-spectral remote
sensing data.

To address this issue, in this paper, we propose SatMAE, a self-supervised
learning framework based on masked autoencoders (MAEs) [1] which naturally
handles temporal and multi-spectral input data. We show that introducing a
positional encoding for the temporal/spectral dimension and independently
masking patches across the temporal/spectral dimension benefits pre-training,
allowing the model to learn representations of the data that are more
conducive to fine-tuning. Specifically, our contributions are as follows

  1. 1.

We propose a novel method to leverage temporal or multi-spectral information
in satellite imagery to improve self-supervised pre-training with masked
autoencoders (see 4).

  2. 2.

We introduce a new dataset called fMoW-Sentinel based on Sentinel-2 imagery,
cross-referenced with fMoW, to serve as a benchmark for training models on
multi-spectral satellite imagery (see 5.1).

  3. 3.

We demonstrate the effectiveness of pre-training transformers [29] on
satellite imagery, achieving significant improvement over previous state-of-
the-art methods on benchmark datasets as well as downstream remote sensing
tasks (see 5)

##  2 Related Work

##### SSL for Satellite Imagery

Self-supervised learning approaches [4, 3, 2, 5, 6] have emerged as promising
approaches in remote sensing domains. For instance, [27] and [28] propose to
incorporate spatially aligned images over time into contrastive self-
supervised learning. Despite the promising results, these two contrastive
learning approaches rely heavily on the quality of positive pairs, which is
often hard to control. Contrastive Sensor Fusion [30] constructs positive
pairs by using co-located images generated by combining different sensor
channels. [31, 32, 33] apply off-the-shelf contrastive learning algorithms to
satellite images. [33] utilizes image inpainting and transformation prediction
as additional pretext tasks. [34] leverages geographical knowledge to aid SSL,
which, however, can be difficult to obtain as annotations.

##### Masked Autoencoder

MAE [1] is a powerful self-supervised learning method built upon the Vision
Transformer [29] architecture. Instead of constructing a contrastive
objective, it proposes the pretext task of reconstructing masked patches of
the input, which hardly needs any data augmentation. MAE has achieved state-
of-the-arts performance on a wide collection of vision benchmarks. Inspired by
the success of [1], many follow-up works extend MAE to different data
modalities. VideoMAE [35] proposes video tube masking and reconstruction as a
pretext task for video analysis. It masks the same region across all frames of
a given video. GMAE [36] adapts MAE to the domain of graphs. MultiMAE [37]
enhance the flexibility of MAE by enabling it to take optional input of
different modality and correspondingly adding other training objectives to
facilitate multi-modality learning. However, these works fail to handle
temporal and multi-spectral input. VideoMAE takes image series as input but
requires the frames to be equally-spaced in the temporal dimension, which is
not the case for remote sensing data due to the temporal irregularity and
discontinuity. In this work, we incorporate temporal and spectral information
into a masked autoencoder architecture, and propose a novel self-supervised
framework for remote sensing domains.

##  3 Background

##### Masked Autoencoder

The Masked Autoencoder (MAE) [1] operates on images of the form
$I\in\mathbb{R}^{C\times H\times W}$, where $H,W$ are the height and width of
the image, respectively, and $C$ is the number of channels (e.g., 3 for RGB
images).

The MAE is an autoencoder with asymmetrical encoding and decoding stages. The
input image $I$ is resized to a sequence of non-overlapping patches,
$S\in\mathbb{R}^{L\times P^{2}C}$, where $P$ is the height and width of the
patch, and $L=(H/P)\cdot(W/P)$ is the number of patches. Each patch is passed
through a patch embedding $f_{p}:\mathbb{R}^{P^{2}C}\mapsto\mathbb{R}^{D}$ to
create a sequence $S^{\prime}\in\mathbb{R}^{L\times D}$ of embedded patch
“tokens”. A fraction $p_{m}$ of the $L$ tokens are masked (usually
$p_{m}=0.75$) and only the remaining $(1-p_{m})L$ tokens corresponding to
“visible" patches are fed to the encoder, a Vision Transformer (ViT) [29] with
positional embeddings to denote the spatial location of the patch in the
image. The decoder is a series of transformer blocks that operates on all $L$
tokens (with positional embeddings added), where the $p_{m}L$ encoded visible
patches are placed in their original sequence position among $(1-p_{m})L$
masked patches represented by a learnable mask token. The decoder then outputs
a reconstructed image $\hat{I}\in\mathbb{R}^{C\times H\times W}$, which is
compared to the original input image using the mean-squared error (MSE) loss,
computed per-pixel only on the masked patches [1].

##### Positional encoding

Positional encoding is an important feature of transformers that makes the
learned representation position-aware. In MAE [1] and many transformers [38,
39], the positional encoding is formulated as:

$\texttt{Encode}(k,2i)=\sin\frac{k}{n^{\frac{2i}{d}}},~{}\texttt{Encode}(k,2i+1)=\cos\frac{k}{n^{\frac{2i}{d}}}$

Here, $k$ is the position, $i$ is the index of feature dimension in the
encoding, $d$ is the number of possible positions, and $n$ is a large constant
(normally set to 10000). In MAE, position is defined as the index of the patch
along the x and y axes, respectively. Therefore, $k$ ranges from 0 to $H/P$
(or $W/P$). The final encoding is generated by concatenating the encodings of
the x and y coordinates.

##  4 Method

In this section, we describe the design of SatMAE, and how we utilize temporal
(4.1) and multi-spectral (4.2) satellite images as input.

###  4.1 Temporal SatMAE

We now consider input tensors of the form $I_{T}\in\mathbb{R}^{T\times C\times
H\times W}$, where $T$ denotes the number of images in a temporal sequence. In
video data, $T$ is the number of frames, which are usually equally spaced.
However, temporal satellite imagery rarely has images at regular intervals.
More commonly, several snapshots, or versions, of a given location are taken
at irregular times. The length and sample frequency of these sequences of
satellite images vary drastically over years and across different regions. For
a given spatial location $loc$, we define $T_{loc}$ as the number of
temporally distinct snapshots present in the dataset.

The naïve way to handle an extra dimension would be to reshape $I_{T}$ to
$I_{T}^{\prime}\in\mathbb{R}^{TC\times H\times W}$, effectively concatenating
the temporal sequence of images along the spectral (i.e. channel) dimension,
and then apply the MAE machinery verbatim. This method poses a few
difficulties:  (i) the model may be unable to generalise to a temporal
ordering different to the one used in pre-training, since it can only
understand order through the position of images in the stacked-timeseries
(ii) the model cannot reason about the length of time separating two
consecutive images in a time sequence, which may be variable when images of a
location are sampled at irregular intervals  (iii) the model loses access to
temporal fine-grained information in deeper layers, as its only direct
exposure to encode temporal information is through the initial patch embedding
$f_{p}$ (iv) the resulting model is not temporally-shift invariant (i.e. the
model would need to separately learn to detect the same event in two different
segments of the temporal sequence).

To address these challenges and to avoid losing temporal information, we
resize the temporal sequence $I_{T}$ to $S_{T}\in\mathbb{R}^{L_{T}\times
P_{T}P^{2}C}$, where $L_{T}=L\cdot(T/P_{T})=(H/P)\cdot(W/P)\cdot(T/P_{T})$,
$P_{T}$ is the “patch size” in the temporal dimension, and $L$ and $P$ are
defined in 3. Prior works using transformers for video data suggest using
$P_{T}=2$, where each “patch” is a cube of shape $2\times 16\times 16$ [35,
40, 41]. Since our data has much shorter temporal sequence lengths [16], we
let $P_{T}=1$ such that $L_{T}=L\cdot T$. In order to operate on inputs of any
temporal order, we re-use the same patch embedding
$f_{p}:\mathbb{R}^{P^{2}C}\mapsto\mathbb{R}^{D}$ for each image in the time
series, giving us an embedded sequence of tokens
$S_{T}^{\prime}\in\mathbb{R}^{L_{T}\times D}$.

####  4.1.1 Temporal Encoding

Figure 2: Top: Encoding each temporal patch with a shared patch embedding
$f_{p}$. Bottom: Encoding each spectral patch with a different patch embedding
$f_{p_{j}}$ for each group $j$.

For each embedded token in the $L_{T}$ length sequence, we need to ensure the
model retains information about its spatial and temporal position. As shown in
many prior works [27, 28], the timestamp of a satellite image is useful for
many pre-training or downstream vision tasks. We propose a temporal encoding
scheme compatible with the masked autoencoder architecture by treating the
temporal dimension similarly to the positional dimensions (see 3).

The timestamp of a satellite image is represented as “year-month-day-hour-
minute-second”. Instead of passing the entire numerized timestamp into a
feature encoder, we propose only keeping the useful parts. Intuitively, the
day, minute, and second should be unrelated to the visual appearance of a
region. Thus, including these components in the temporal encoding may not be
beneficial, and can even be detrimental. In contrast, a landscape may evolve
over years due to weather, geology, and human activity. The month reflects
season and climate, and the hour reflects daylight and temperature.

Then, the temporal encoding is formulated as:

$t_{k,i}=\textsc{CONCAT}[\texttt{Encode}(k_{\text{year}},i),\texttt{Encode}(k_{\text{month}},i),\texttt{Encode}(k_{\text{hour}},i)]$

And the final encoding is generated by concatenating the temporal encoding to
the positional encoding defined in 3 such that the total length of the
encoding is $D$.

####  4.1.2 Masking Strategies

(a) Temporal data

(b) Spectral data

Figure 3: 3(a) Temporal masking: When masking the patches for images in a
timeseries, we can choose to keep a patch fully visible or fully masked across
time (consistent masking), or independently mask all patches (independent
masking). In both cases, a fraction $p_{m}$ patches are masked. Here, $T=3$,
and the leftmost column orders the temporal sequence according to the
timestamp features. For example, “y-12, m-12, h-15” denotes 12 years from the
minimum year (2002), the zero-indexed month 2, and the 15th hour of the day;
i.e., roughly 2014, March, 15:00. 3(b) Spectral Masking: The same masking
strategies are adapted to groups of the 13 spectral bands in Sentinel-2
images.

With an additional temporal dimension, masking a subset of the $L_{T}$ tokens
needs to be treated with care. As seen in figure 3, there are different ways
to mask a temporal series of satellite images.

##### Consistent Masking

Each image is “patchified” separately, but the masked regions are kept
consistent across all images (fig. 3(a)). This approach is also used in
VideoMAE [35], with video input.

##### Independent Masking

Each image is “patchified” separately, and masked regions may not be the same
across every image. Instead, a fraction $p_{m}$ of the full sequence of all
patch tokens are masked. Another variant is to independently mask the regions
of each image, but keep the ratio $p_{m}$ of masked regions fixed per image.
Both variants are equivalent in expectation. Effectively, the model can look
at unmasked values of a region that is masked in one image but not in others.
While this setting may lead to an easier task for video data since the model
can “cheat” and exploit temporal redundancy in videos with high framerates
[35], we argue that this form of “cheating” is less feasible in temporal
satellite imagery, given the strong impact of seasonal variation and changing
human activity over periods of time and the much larger time deltas between
temporally consecutive images (see fig. 3(a)).

##### Independent Masking + Inconsistent Cropping

During data pre-processing, we can crop square regions for input
inconsistently so that images in the same temporal sequence may be spatially-
unaligned. This strategy may help the model learn better representations as it
may learn to align images in the sequence across the spatial and temporal
dimensions.

###  4.2 Multi-spectral SatMAE

While MAE does operate on images $I\in\mathbb{R}^{C\times H\times W}$, usually
$C=3$ for RGB images. Satellite data, on the other hand, can often have
multiple spectral bands. For example, Sentinel-2 imagery has $C=13$ bands of
10m, 20m and 60m spatial resolution, each of different wavelengths (see
A.2.2). In the following, we discuss and later experimentally compare various
ways to encode spectral information.

##### Stack Channels

This method treats the images $I$ with multiple bands as-is, and encodes each
patch of $C$ channels to a $D$ dimensional token embedding. Therefore, the
sequence of patches $S\in\mathbb{R}^{L\times P^{2}C}$ is embedded to a
sequence of tokens $S^{\prime}\in\mathbb{R}^{L\times D}$. We denote this
method SatMAE+Stack.

##### Group Channels

There are limitations to naively stacking the spectral information, especially
that a single convolutional patch embedding may be insufficient to fully
capture fine-grained information present in multiple bands of different
wavelengths and spatial resolution. We would like the model to preserve
information about the different bands through the encoding and decoding
stages.

To address this limitation, we propose grouping subsets of spectral bands.
Given $C$ channels, we form $G$ groups $g_{1},g_{2},\dots,g_{G}$ such that
$g_{1}+g_{2}+\dots+g_{G}=C$. This is analogous to slicing the image $I$ in the
channel dimension, creating images $I_{1},\dots,I_{G}$, where
$I_{j}\in\mathbb{R}^{g_{j}\times H\times W}$. Since for some $i,j$, it is
possible that $g_{i}\neq g_{j}$, we use a separate patch embedding
$f_{p_{j}}:\mathbb{R}^{P^{2}G_{j}}\mapsto\mathbb{R}^{D}$ for each group $j$,
thus allowing the model to best represent each group of channels as token
embeddings. Therefore, each group $j$ is first resized from
$I_{j}\in\mathbb{R}^{G_{j}\times H\times W}$ to $S_{j}\in\mathbb{R}^{L\times
P^{2}G_{j}}$, and then each patch is embedded with $f_{p_{j}}$ to produce a
sequence of embedded tokens $S^{\prime}_{j}\in\mathbb{R}^{L\times D}$. The
sequences $S^{\prime}_{1},\dots,S^{\prime}_{G}$ are concatenated to produce
the final set of tokens $S^{\prime}\in\mathbb{R}^{GL\times D}$.

##### Spectral Encoding

Since the tokens in $S^{\prime}$ now correspond both to a patch location
$(i,j)$ in the original input image, as well as a group of channels $g$, we
include an encoding for the group id as well following a similar form as
4.1.1. For a given group index $k_{g}$, we have the group encoding

$g_{k,i}=\texttt{Encode}(k_{g},i)$

The final encoding is a concatenation of the positional encoding $x_{k,i}$,
$y_{k,i}$ and the spectral encoding $g_{k,i}$ such that the total encoding
dimension is $D$ (see figure 2). This positional encoding is added to
$S^{\prime}$ before inputting it to the encoder. We denote the combined
setting of grouping channels and using a group encoding as SatMAE+Group.

##### Masking Strategies

We consider consistent masking (denoted SatMAE+Group+CM) and independent
masking (SatMAE+Group+IM) as defined in 4.1.2 and as visualized in figure
3(b).

##  5 Experiments

In this section, we start by introducing the datasets we considered for our
experiments. Specifically, we collect a new multi-spectral remote sensing
image dataset for downstream task evaluation (5.1). We then present our
results on various remote sensing transfer-learning and downstream tasks. In
summary, our approach demonstrates strong performance on all the tasks we
considered, yielding improvements over previous state-of-the-art techniques by
up to 6% on supervised learning benchmarks, and up to 14% on remote sensing
transfer-learning downstream remote sensing tasks.

For all experiments, we compare with both the current state-of-the-art methods
[27, 28] and with supervised learning from scratch using the ViT backbone of
our model.

###  5.1 Datasets for Pre-training

##### fMoW RGB

Functional Map of the World (fMoW) [16] is a dataset of high-resolution
satellite image time series across the world, with a task of classification
among 62 categories.

##### fMoW Sentinel

We create a new dataset based on the fMoW RGB dataset. We collect all 13
frequency bands provided by Sentinel-2 (B1-12 and B8A) for the original fMoW
locations, at some of the same times as fMoW images plus some extra times, for
a total of 712,874 training images, 84,939 validation images, and 84,966 test
images. More details of the two datasets are included in A.1.

###  5.2 fMoW RGB (non-temporal)

In this section, we perform experiments on fMoW single image classification
task. Following [27], we report both the performance of linear probing and
finetuning setting. Table 4 shows that compared to the previous state-of-the-
art self-supervised method using a contrastive momentum encoding approach [27,
3], our SatMAE achieved a 6.29% improvement in top 1 classification accuracy.
An interesting fact is that without SatMAE pre-training, the ViT-large model
could only reach 62.48% at convergence after 50 epochs of finetuning compared
to 69.05% achieved by training a ResNet-50 model from scratch . This is likely
because the ViT [29] backbone is harder to train than ResNet50 [42], which
makes the pre-trained model more valuable in this case.

###  5.3 fMoW RGB (temporal)

Method | Backbone | Frozen/Finetune
---|---|---
Sup. (Scratch) | ResNet50 | -/69.05
Sup. (IN init.) | ResNet50 | -/69.07
GASSL [27] | ResNet50 | 68.32/71.55
Sup. (Scratch) | ViT-Large | -/62.48
Sup. (IN init.) | ViT-Large | -/76.91
SatMAE (Ours) | ViT-Large | 65.94/77.84
|  |

Table 1: Top 1 Accuracy on fMoW classification. Frozen means only performing
linear classification on frozen features of the pre-trained model. Finetune
corresponds to end-to-end finetuning the whole model. Scratch means training
from scratch, and IN init. means using ImageNet pre-trained weights.

Method | Backbone | Top Acc. (1/5)
---|---|---
Sup. (Scratch) | ResNet50 | 73.24/-
GASSL [27] | ResNet50 | 74.11/-
SatMAE+Stack | ViT-Large | 75.85/88.68
Sup. (Scratch) | ViT-Large | 61.89/84.23
SatMAE* | ViT-Large | 76.78/92.01
SatMAE | ViT-Large |  81.49/93.26
|  |

Table 2:  Classification results on the temporal fMoW RGB dataset. * means
copying the same image for three times instead of using image sequences as
input. SatMAE+Stack here means stacking the image sequence along the channel
space.

Method | Backbone | Top Acc. (1/5)
---|---|---
Sup. Learning* | ResNet152 | 49.12/75.73
Sup. Learning$\dagger$ | ResNet152 | 54.46/78.99
SatMAE+Group* | ViT-Large | 53.03/77.14
SatMAE+Group$\dagger$ | ViT-Large | 51.61/77.26
SatMAE+Group$\ddagger$ | ViT-Large | 49.49/76.30
SatMAE+Stack | ViT-Large | 57.37/81.63
SatMAE+Group+CM | ViT-Large | 57.76/80.49
SatMAE+Group+IM | ViT-Large |  59.30/82.81
|  |

Table 3:  Top 1 and Top 5 Accuracy on the fMoW Sentinel validation set. In
each row, the full model is finetuned. * denotes finetuning from scratch.
$\dagger$ is finetuning from ImageNet weights, $\ddagger$ is finetuning from
fMoW RGB weights. The remaining rows use fMoW Sentinel for pre-training.

| Temp.
---
Enc.
| Indep.
---
Mask.
| Cons.
---
Crop.
| Test
---
Aug.
Top Acc. (1/5)
| ✓ | ✓ |  | 78.07/92.73
✓ |  | ✓ |  | 78.45/91.64
✓ | ✓ |  |  | 79.90/92.75
✓ | ✓ | ✓ |  | 79.69/92.09
✓ | ✓ | ✓ | ✓ |  81.49/93.26
|  |  |  |

Table 4:  Ablation studies on different components of temporal SatMAE on the
temporal fMoW classification task. The first column is whether using temporal
encoding, the second is whether using independent masking, the third is
whether cropping consistently, and the last one is whether applying test-time
augmentation.

##### Main experiments

We perform image-sequence classification on the temporal version of fMoW RGB
to evaluate our temporal SatMAE. The temporal fMoW consists of co-located
image sequences with a length of 3. As shown in Table 4, SatMAE surpasses the
previous state-of-the-art by 4.48%, and also improves the non-temporal result
by 2.06% in terms of top 1 classification accuracy. We can observe from the
second last row that this gain is not from the larger model to handle
sequences of data. Naively stacking the image sequences in the channel
dimension performs even worse than the non-temporal SatMAE. Again, the pre-
trained model is crucial for the ViT to achieve high performance over
ResNet50.

##### Ablation studies

Table 4 provides a comprehensive ablation study on the components of our
temporal SatMAE. We can see that the performance mainly comes from using
temporal encoding and adopting the independent masking strategy instead of
consistent masking as suggested in VideoMAE [35]. Interestingly, the
consistent cropping option slightly hurts the performance, indicating the
model does not rely on image sequences to be perfectly spatially-aligned. In
addition, applying test-time augmentations similar to [27] brings considerable
improvement.

###  5.4 fMoW Sentinel (Multi-spectral)

In this section, we describe experiments pre-training on fMoW-Sentinel imagery
and finetuning models on the image classification task of the fMoW-Sentinel
dataset. We pre-train SatMAE+Stack 4.2 and investigate consistent masking
4.1.2 and independent masking 4.1.2, with SatMAE+Group+CM and SatMAE+Group+IM,
respectively (see 4.2, 4.2, 4.2). The full models are then finetuned on the
fMoW-Sentinel image classification task. For comparison, we also consider
finetuning the ResNet-152 model [42] from scratch and from an ImageNet
initialisation. We pick the largest model, ResNet-152, for fairer comparison
with ViTs.

##### Model configuration

As not all of the 13 Sentinel-2 bands may be useful, in our experiments we
drop bands B1, B9 and B10, which correspond to a spatial resolution of 60m. Of
the remaining 10 bands, we form three groups:  (i) RGB+NIR: B2, B3, B4, B8
(ii) Red Edge: B5, B6, B7, B8A  (iii) SWIR: B11, B12  . We choose this
grouping to ensure each group has bands of the same spatial resolution as well
as similar wavelength (see A.2.2). See A.3.3 for pre-training and finetuning
details.

##### Results

We present the results in table 4. Our method SatMAE+Group+IM is able to
achieve the highest accuracy, outperforming supervised training from scratch
by 6.27%. Interestingly, independent masking performs the best, suggesting
that it is beneficial to allow the model to “peek” at regions of unmasked band
groups when reconstructing the same region in a masked band group. While
ImageNet-initialised backbones appear to be effective, we demonstrate that our
method designed specifically for multi-spectral satellite imagery strongly
outperforms ImageNet-trained models.

###  5.5 Transfer Learning Experiments

In this section, we use our pre-trained weights to finetune SatMAE on a few
downstream tasks on remote-sensing datasets, including land cover
classification (5.5), multi-label classification (5.5), and building
segmentation (5.5).

Method | Backbone | Top 1 Acc.
---|---|---
| Sup. (Scratch)
---
ResNet50 | 54.46
GASSL [27] | ResNet50 | 57.63
| Sup. (Scratch)
---
ViT-Large | 69.65
SatMAE | ViT-Large | 71.77
|  |

Table 5:  NAIP land cover classification results.

Method | Backbone |  xxxmIoUxxx
---|---|---
| Sup. (Scratch)
---
ResNet50 | 75.57
GASSL [27] | ResNet50 | 78.51
| Sup. (Scratch)
---
ViT-Large | 74.71
SatMAE | ViT-Large | 78.07
|  |

Table 6:  SpaceNet v2 building segmentation results.

Method | Backbone | Top 1 Acc.
---|---|---
| Sup. (Scratch)
---
ResNet18 | 63.21
| Sup. (IN init.)
---
ResNet18 | 86.44
GASSL [27] | ResNet18 | 89.51
SeCo [28] | ResNet18 | 93.14
SatMAE* | ViT-Large | 95.74
SatMAE* | ViT-Large | 98.94
SatMAE+Group | ViT-Large | 96.54
SatMAE+Group+IM | ViT-Large | 98.98
|  |

Table 7:  EuroSAT land cover classification results. * means we only use the
RGB channels of the data.

Method | Backbone |  xxxmAPxxxx
---|---|---
| Sup. (Scratch)
---
ResNet50 | 69.49
| Sup. (IN init.)
---
ResNet50 | 80.04
GASSL [27] | ResNet50 | 80.20
SeCo [28] | ResNet50 | 82.62
| Sup. (Scratch)
---
ViT-Large | 80.07
SatMAE | ViT-Large | 82.13
|  |

Table 8:  BigEarthNet multi-label classification results. Following [28], we
use mean Average Precision (mAP) as the metric, and use a newer set of class
labels.

##### Land Cover Classification

We perform transfer learning experiments on land cover classification using
the NAIP and EuroSAT [43] dataset. NAIP consists of RGB+CIR images of 66 land
cover classes obtained by the USDA’s National Agricultural Imagery Program,
which are split into 100,000 training and 50,000 validation images. EuroSAT is
a small dataset containing 27,000 13-band satellite images of 10 classes based
on Sentinel-2. We follow [28, 44] for the train/val splits on EuroSAT.

Table 8 and table 8 shows the remarkable improvement of our SatMAE over the
state-of-the-arts. Although using the ViT-Large backbone already achieved good
results, initiating the model with SAT-MAE pre-trained weights further
increased the accuracy by 2%-3%.

##### Multi-label Land Cover Classification

We also use the BigEarthNet [17] dataset for multi-label classification.
BigEarthNet consists of 13-band Sentinel-2 images of multiple labels from 19
classes. There are 354,196 images for training and 118,065 images for
validation. Following [28], we use a 10% subset of the train set.

Table 8 shows our SatMAE pre-training improves upon the model trained from
scratch by over 2%, and achieves comparable results to the state-of-the-art.
GASSL and SeCo were actually trained on a larger pre-train dataset (1M
Sentinel-2 images v.s. 713k) than our fMoW Sentinel. Therefore we expect
further improvement when we pre-train SatMAE with more data and for longer.

##### Building Segmentation

In this section, we evaluate SatMAE on the semantic segmentation downstream
task of the SpaceNet v2 dataset [19]. The SpaceNet v2 datasets consists of
6940 high resolution satellite images with segmentation masks for buildings,
which are divided into train and test sets of 5000 and 1940 images,
respectively.

The results in table 8 show that our method achieves a larger performance gain
from supervised learning from scratch setting compared to [27]. The
incompatibility of the ViT backbone with PSANet could explain why the baseline
performance is not as good as that of using a ResNet50 backbone.

###  5.6 Visualizing reconstruction quality for SatMAE

Figure 4: Comparison of reconstruction quality between SatMAE+IM (left) and
SatMAE+CM (right).

We show the visualization of the reconstruction quality of two different
SatMAE masking strategies in Figure 4. The SatMAE with Independent Masking
successfully reconstructs all the airplanes even though their number varies
across time. In contrast, the SatMAE with Consistent Masking missed some
airplanes in the reconstruction.

##  6 Conclusion

In this paper, we propose a new SSL framework based on the MAE architecture
[1] tailored to remote-sensing data (satellite imagery). Our novel masking
strategy in a joint positional, temporal and spectral space, along with the
temporal and spectral encoding, enables our model to handle temporal and
multi-spectral satellite images as input and learn useful representations.
Experiments on the datasets for pre-training and multiple downstream datasets
demonstrate the effectiveness of our pre-trained SatMAE model, outperforming
previous state-of-the-art results by large margins.

In the future, it would be useful to design more efficient transformer
architectures. While SatMAE has a similar number of parameters for both the
temporal and multi-spectral setting as a regular ViT, the increased sequence
length can strain computational resources. Moreover, it is also worth
exploring optimal positional encodings for spectral and temporal data, as well
as optimal groups of spectral bands, either by neural-based search methods, or
using prior knowledge. Lastly, investigating better architectures for object
detection and semantic segmentation using ViTs will be important in
generalising SatMAE to further downstream tasks.

## Broader Impact

Accurate measurements of economic, social, and environmental phenomena are key
inputs into policy decisions made around the world, but the sparsity of
labelled data on many outcomes means that such decisions are often not guided
by timely or accurate data. We demonstrate how a pre-training framework could
relieve the dependence on labelled data for many downstream tasks that use
satellite imagery as input. We hope our SatMAE method will help close the gap
between SSL performance on natural imagery and on the more challenging
satellite imagery, and prompt further attention from the ML community on the
usefulness of SSL in satellite-imagery-related tasks.

Better extraction of information from satellite imagery has profound
implications for our ability to measure and understand a broad array of
social, economic and environmental phenomena that are critical for decision
making. Our approach further amplifies the usefulness of the sparse amount of
labelled data that exist on key human outcomes, and could enable rapid and
accurate extraction of imagery features relevant for critical downstream
tasks, including poverty prediction, infrastructure development, and
population estimation. Such information could aid governments in more rapid
and data-informed decision making and ultimately bring large societal
benefits.

##  7 Acknowledgements

This research is based upon work supported in part by the Office of the
Director of National Intelligence (ODNI), Intelligence Advanced Research
Projects Activity (IARPA), via 2021-2011000004, HAI, NSF(#1651565), AFOSR
(FA95501910024), ARO (W911NF-21-1-0125) and Sloan Fellowship. The views and
conclusions contained herein are those of the authors and should not be
interpreted as necessarily representing the official policies, either
expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S.
Government is authorized to reproduce and distribute reprints for governmental
purposes not-withstanding any copyright annotation therein.

## Checklist

  1. 1.

For all authors…

    1. (a)

Do the main claims made in the abstract and introduction accurately reflect
the paper’s contributions and scope? [Yes] We tried our best to be precise.

    2. (b)

Did you describe the limitations of your work? [Yes] We described the
limitations in the conclusion section.

    3. (c)

Did you discuss any potential negative societal impacts of your work? [Yes] We
discussed in Appendix B

    4. (d)

Have you read the ethics review guidelines and ensured that your paper
conforms to them? [Yes]

  2. 2.

If you are including theoretical results…

    1. (a)

Did you state the full set of assumptions of all theoretical results? [N/A]

    2. (b)

Did you include complete proofs of all theoretical results? [N/A]

  3. 3.

If you ran experiments…

    1. (a)

Did you include the code, data, and instructions needed to reproduce the main
experimental results (either in the supplemental material or as a URL)? [Yes]
We list the url to our code repo in the supplementary material.

    2. (b)

Did you specify all the training details (e.g., data splits, hyperparameters,
how they were chosen)? [Yes] We list part of them in the experiments section
and part of them in Appendix A.

    3. (c)

Did you report error bars (e.g., with respect to the random seed after running
experiments multiple times)? [No] We trained on large datasets with small
variation across different runs expected. Limited by computation resources, we
only ran each experiment once.

    4. (d)

Did you include the total amount of compute and the type of resources used
(e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We list that
in Appendix A.

  4. 4.

If you are using existing assets (e.g., code, data, models) or
curating/releasing new assets…

    1. (a)

If your work uses existing assets, did you cite the creators? [Yes] Yes, we do
that by citing creators of datasets and authors of prior works.

    2. (b)

Did you mention the license of the assets? [Yes] We do that in Appendix A

    3. (c)

Did you include any new assets either in the supplemental material or as a
URL? [Yes] We are releasing a new dataset. The instructions and links will be
released in our codebase mentioned above.

    4. (d)

Did you discuss whether and how consent was obtained from people whose data
you’re using/curating? [Yes] All data we used are released publicly.

    5. (e)

Did you discuss whether the data you are using/curating contains personally
identifiable information or offensive content? [Yes] We concluded that no data
we are using has such concerns.

  5. 5.

If you used crowdsourcing or conducted research with human subjects…

    1. (a)

Did you include the full text of instructions given to participants and
screenshots, if applicable? [N/A]

    2. (b)

Did you describe any potential participant risks, with links to Institutional
Review Board (IRB) approvals, if applicable? [N/A]

    3. (c)

Did you include the estimated hourly wage paid to participants and the total
amount spent on participant compensation? [N/A]

## References

  * [1] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick.  Masked autoencoders are scalable vision learners.  CoRR, abs/2111.06377, 2021.
  * [2] Yonglong Tian, Dilip Krishnan, and Phillip Isola.  Contrastive multiview coding.  In European conference on computer vision, pages 776–794. Springer, 2020.
  * [3] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729–9738, 2020.
  * [4] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  * [5] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap your own latent-a new approach to self-supervised learning.  Advances in Neural Information Processing Systems, 33:21271–21284, 2020.
  * [6] Xinlei Chen and Kaiming He.  Exploring simple siamese representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15750–15758, 2021.
  * [7] Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia Makedon.  A survey on contrastive self-supervised learning.  CoRR, abs/2011.00362, 2020.
  * [8] Longlong Jing and Yingli Tian.  Self-supervised visual feature learning with deep neural networks: A survey.  IEEE transactions on pattern analysis and machine intelligence, 43(11):4037–4058, 2020.
  * [9] Ishan Misra and Laurens van der Maaten.  Self-supervised learning of pretext-invariant representations.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6707–6717, 2020.
  * [10] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  Advances in Neural Information Processing Systems, 33:9912–9924, 2020.
  * [11] Yude Wang, Jie Zhang, Meina Kan, Shiguang Shan, and Xilin Chen.  Self-supervised equivariant attention mechanism for weakly supervised semantic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12275–12284, 2020.
  * [12] Yann Dubois, Benjamin Bloem-Reddy, Karen Ullrich, and Chris J Maddison.  Lossy compression for lossless prediction.  Advances in Neural Information Processing Systems, 34, 2021.
  * [13] Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer.  Revisiting self-supervised visual representation learning.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1920–1929, 2019.
  * [14] Zeyu Feng, Chang Xu, and Dacheng Tao.  Self-supervised representation learning by rotation feature decoupling.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10364–10374, 2019.
  * [15] Soroush Abbasi Koohpayegani, Ajinkya Tejankar, and Hamed Pirsiavash.  Compress: Self-supervised learning by compressing representations.  Advances in Neural Information Processing Systems, 33:12980–12992, 2020.
  * [16] Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee.  Functional map of the world.  In CVPR, 2018.
  * [17] Gencer Sumbul, Marcela Charfuelan, Begüm Demir, and Volker Markl.  Bigearthnet: A large-scale benchmark archive for remote sensing image understanding.  In IGARSS 2019-2019 IEEE International Geoscience and Remote Sensing Symposium, pages 5901–5904. IEEE, 2019.
  * [18] Darius Lam, Richard Kuzma, Kevin McGee, Samuel Dooley, Michael Laielli, Matthew Klaric, Yaroslav Bulatov, and Brendan McCord.  xview: Objects in context in overhead imagery.  arXiv preprint arXiv:1802.07856, 2018.
  * [19] Adam Van Etten, Dave Lindenbaum, and Todd M Bacastow.  Spacenet: A remote sensing dataset and challenge series.  arXiv preprint arXiv:1807.01232, 2018.
  * [20] Marshall Burke, Anne Driscoll, David B Lobell, and Stefano Ermon.  Using satellite imagery to understand and promote sustainable development.  Science, 371(6535):eabe8628, 2021.
  * [21] Kumar Ayush, Burak Uzkent, Kumar Tanmay, Marshall Burke, David Lobell, and Stefano Ermon.  Efficient poverty mapping from high resolution remote sensing images.  In Proc. AAAI Conf. Artif. Intell, volume 35, pages 12–20, 2021\.
  * [22] Kumar Ayush, Burak Uzkent, Marshall Burke, David Lobell, and Stefano Ermon.  Generating interpretable poverty maps using object detection in satellite images.  arXiv preprint arXiv:2002.01612, 2020.
  * [23] Neal Jean, Marshall Burke, Michael Xie, W Matthew Davis, David B Lobell, and Stefano Ermon.  Combining satellite imagery and machine learning to predict poverty.  Science, 353(6301):790–794, 2016.
  * [24] Jiaxuan You, Xiaocheng Li, Melvin Low, David Lobell, and Stefano Ermon.  Deep gaussian process for crop yield prediction based on remote sensing data.  In Thirty-First AAAI conference on artificial intelligence, 2017\.
  * [25] Anna X Wang, Caelin Tran, Nikhil Desai, David Lobell, and Stefano Ermon.  Deep transfer learning for crop yield prediction with remote sensing data.  In Proceedings of the 1st ACM SIGCAS Conference on Computing and Sustainable Societies, pages 1–5, 2018.
  * [26] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009.
  * [27] Kumar Ayush, Burak Uzkent, Chenlin Meng, Kumar Tanmay, Marshall Burke, David Lobell, and Stefano Ermon.  Geography-aware self-supervised learning.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10181–10190, 2021.
  * [28] Oscar Mañas, Alexandre Lacoste, Xavier Giro-i Nieto, David Vazquez, and Pau Rodriguez.  Seasonal contrast: Unsupervised pre-training from uncurated remote sensing data.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9414–9423, 2021.
  * [29] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  ICLR, 2021.
  * [30] Aidan M Swope, Xander H Rudelis, and Kyle T Story.  Representation learning for remote sensing: An unsupervised sensor fusion approach.  arXiv preprint arXiv:2108.05094, 2021.
  * [31] Vladan Stojnic and Vladimir Risojevic.  Self-supervised learning of remote sensing scene representations using contrastive multiview coding.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1182–1191, 2021.
  * [32] Pallavi Jain, Bianca Schoen-Phelan, and Robert Ross.  Multi-modal self-supervised representation learning for earth observation.  In 2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS, pages 3241–3244. IEEE, 2021.
  * [33] Wenyuan Li, Hao Chen, and Zhenwei Shi.  Semantic segmentation of remote sensing images with self-supervised multitask representation learning.  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 14:6438–6450, 2021.
  * [34] Wenyuan Li, Keyan Chen, Hao Chen, and Zhenwei Shi.  Geographical knowledge-driven representation learning for remote sensing images.  IEEE Transactions on Geoscience and Remote Sensing, 2021.
  * [35] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang.  Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training.  arXiv preprint arXiv:2203.12602, 2022.
  * [36] Hongxu Chen, Sixiao Zhang, and Guandong Xu.  Graph masked autoencoder.  arXiv preprint arXiv:2202.08391, 2022.
  * [37] Roman Bachmann, David Mizrahi, Andrei Atanov, and Amir Zamir.  Multimae: Multi-modal multi-task masked autoencoders.  arXiv preprint arXiv:2204.01678, 2022.
  * [38] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [39] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In International Conference on Machine Learning, pages 10347–10357. PMLR, 2021.
  * [40] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid.  Vivit: A video vision transformer.  In International Conference on Computer Vision (ICCV), 2021.
  * [41] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu.  Video swin transformer.  arXiv preprint arXiv:2106.13230, 2021.
  * [42] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [43] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth.  Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217–2226, 2019.
  * [44] Maxim Neumann, Andre Susano Pinto, Xiaohua Zhai, and Neil Houlsby.  In-domain representation learning for remote sensing.  arXiv preprint arXiv:1911.06721, 2019.
  * [45] Hengshuang Zhao, Yi Zhang, Shu Liu, Jianping Shi, Chen Change Loy, Dahua Lin, and Jiaya Jia.  Psanet: Point-wise spatial attention network for scene parsing.  In Proceedings of the European conference on computer vision (ECCV), pages 267–283, 2018.

##  Appendix A Appendix

###  A.1 Datasets

##### fMoW RGB

Functional Map of the World (fMoW) [16] is a dataset of high-resolution
satellite image time series across the world, with a task of classification
among 62 architecture categories such as airport, shipyard, and zoo. fMoW
provides RGB images as well as metadata including location, time, sun angles,
etc. The license is provided here 111fMoW license:
https://github.com/fMoW/dataset/raw/master/LICENSE.

Co-located images of different timestamps, or sequences, are provided in fMoW.
They are of different length, and around 60% of the samples have length larger
than 2. Readers can refer to the fMoW paper [16] for statistics on the
distribution of sequence lengths. We construct a temporal version of fMoW by
randomly associating every single image with two images of the same location
but of different timestamps if possible.

##### fMoW Sentinel

We collect a new dataset based on the fMoW RGB dataset. We crop surface
reflectance images from the Sentinel-2 (ESA) satellite (courtesy of the U.S.
Geological Survey), consisting of 90-day composites of images at the same
locations as fMoW images (to reduce the impacts of cloud coverage). At each
fMoW datapoint location, we collect a time series of Sentinel-2 images, using
the provided geo-coordinate bounding boxes. For locations where all fMoW
images are before the Sentinel-2 time range, we discard the location.
Otherwise, we collect a composite centered at the same time as each fMoW image
within Sentinel-2 time range. Because many fMoW images occur before
Sentinel-2, we augment the time series by adding extra images in 6-month
intervals that do not have an image in the fMoW dataset.

We collect all 13 frequency bands provided by Sentinel-2 (B1-12 and B8A), at
some of the same times as fMoW images plus some extra times, for a total of
712,874 training images, 84,939 validation images, and 84,966 test images. Out
of these 155,446 training images, 22,602 validation images, and 22,824 test
images occur at the same time as a corresponding fMoW image.

Figure 5: Distribution of images and locations across the categories over the
fMoW Sentinel training set.

###  A.2 fMoW Sentinel

We provide information about the fMoW-Sentinel dataset, collected using
Sentinel-2 222Sentinel-2 license:
https://scihub.copernicus.eu/twiki/pub/SciHubWebPortal/TermsConditions/Sentinel_Data_Terms_and_Conditions.pdf.

####  A.2.1 Geographic Distribution

Figure 6: Geographic distribution of fMoW Sentinel images by country

####  A.2.2 fMoW Sentinel Bands

Channel | Resolution | Central wavelength | Mean | Standard deviation
---|---|---|---|---
B1: Aerosols | 60m | 443nm | 1370.192 | 633.152
B2: Blue | 10m | 490nm | 1184.382 | 650.284
B3: Green | 10m | 560nm | 1120.771 | 965.231
B4: Red | 10m | 665nm | 1136.260 | 948.982
B5: Red Edge 1 | 20m | 705nm | 1263.739 | 1108.067
B6: Red Edge 2 | 20m | 740nm | 1645.403 | 1258.364
B7: Red Edge 3 | 20m | 783nm | 1846.870 | 1233.149
B8: NIR | 10m | 842nm | 1762.595 | 1364.387
B8A: Red Edge 4 | 20m | 865nm | 1972.624 | 3545.66
B9: Water Vapor | 60m | 940nm | 582.726 | 472.380
B10: Cirrus | 60m | 1375nm | 14.771 | 14.311
B11: SWIR 1 | 20m | 1610nm | 1732.164 | 1310.370
B12: SWIR 2 | 20m | 2190nm | 1247.919 | 1087.602
Table 9: Mean and standard deviation of pixel values for each channel across
the fMoW Sentinel training dataset. Note that channel B10 does not contain
bottom-of-atmosphere information, and is no longer accessible on Google Earth
Engine. Further details can be found here. Figure 7: Distribution of pixel
counts per band across the fMoW Sentinel training set.

###  A.3 Training Details

Here we describe the settings used for pre-training and finetuning our models
on fMoW RGB (non-temporal) (A.3.1), fMoW RGB (temporal) (A.3.2), fMoW Sentinel
(A.3.3), NAIP (A.4), EuroSAT (A.5), BigEarthNet (A.6), and SpaceNet (A.7).

####  A.3.1 fMoW RGB (non-temporal)

##### Pre-training

We use ViT-Large [29] as the backbone. The model configuration is the same as
in [1], e.g. the input image size is $224$ and the patch size is $P=16$. Since
the original image size of fMoW varies greatly, we first resize the image so
that the shorter side is $224$ pixels and the aspect ratio is maintained, then
randomly crop a $224\times 224$ region from the resized image. We also
normalize the image according to the mean and standard deviation calculated on
the whole dataset. We use 8 NVIDIA V100 GPUs on Google Cloud to train the
model for $800$ epochs with a learning rate of $2.4\times 10^{-3}$ and batch
size of $4096$. The optimizer and learning rate scheduler setting are kept the
same as in [1].

##### Finetuning

We load the pre-trained weights below the ViT head and finetune the ViT-Large
model in an end-to-end manner. We adopt the same learning rate decay and
weight decay strategy during finetuning as in [1]. We apply the same data
augmentation during pre-training and additionally use Mixup and Cutmix
augmentation. We use 8 NVIDIA V100 GPUs to train the model for $50$ epochs
with a learning rate of $2\times 10^{-3}$ and batch size of $512$. Other
paramters including Mixup coefficients are kept the same as in [1].

####  A.3.2 fMoW RGB (temporal)

##### Dataset

We iterate over the dataset in the same way as in non-temporal fMoW, except
that we randomly find 2 co-located images with different timestamps (if
possible) for every image sample, so every sample becomes a image sequence of
length 3. If there are not enough co-located images with different timestamps
we simply duplicate the original image. The fMoW dataset train/val split
guarantees that co-located images belong to the same split so there is no
leakage involved.

##### Pre-training

We use the same model as above, though the number of input patches triples. To
incorporate temporal encoding, the positional encoding of a spatial location
of a patch shortens to a $320+320=640$ dimensional vector, and the temporal
encoding is a $384$ dimensional vector, divided equally among the year, month,
and hour. We apply constraints on the mask indices to implement different mask
strategies. For independent masking (4.1.2), we pick the variant where we keep
the ratio of masked patches fixed to $p_{m}=0.75$ for each image in the
sequence. For the consistent cropping option, we first resize the image
sequence to the same size and then apply cropping to all three images instead
of randomly cropping each image separately. We use 8 NVIDIA V100 GPUs to train
the model for $100$ epochs with a learning rate of $6\times 10^{-4}$ and batch
size of $1024$.

##### Finetuning

We use 4 or 8 NVIDIA V100 GPUs to train the model for $50$ epochs with a
learning rate of $5\times 10^{-4}$ and batch size of $128$.

##### Test-time Augmentation

Unlike the test-time augmentation used in [27], we average the prediction
score of 9 random samples of image sequences for every single image as the
final prediction score. To be consistent with previous experiments, we
calculate the mean classification accuracy on the whole validation set instead
of evaluating on the subset with unique locations. These two metrics give very
similar numbers.

####  A.3.3 fMoW Sentinel

We choose the ViT-Large backbone [29] with $D=1024$. The positional encoding
of the spatial location of a patch is a $768$ dimensional vector, and the
spectral group encoding is a $256$ dimensional vector. Given the relatively
smaller size of Sentinel-2 imagery, we resize all images to $96\times 96$
pixels and use a patch size $P=8$. This results in $L=(96/8)^{2}=144$ patches
which are passed to SatMAE+Stack. For SatMAE+Group, since we pick 3 groups of
channels, we have $3L=432$ patches. We did experiment with letting each
channel be its own group. However, this resulted in a very large memory
footprint with $10L=1440$ patches and unstable training which would frequently
result in NaN loss. We thus decided to group bands in terms of spatial
resolution and wavelength similarity (see 5.4).

##### Pre-training

We use 8 NVIDIA v100 GPUs, an effective batch size of 4096, a base learning
rate of $10^{-4}$ and the same warmup and half-cyle cosine decay schedule used
by [1]. For each image, we use standard normalisation (see statistics in
A.2.2), randomly crop 0.2-1.0x of the area of the image, resize it to
$96\times 96$ pixels, and randomly flip the image horizontally. We use a
masking ratio of $p_{m}=0.75$, as was found to be optimal in [1]. We pre-train
each model for 50 epochs.

##### Finetuning

We use an effective batch size of 4096, a base learning rate of $10^{-3}$ and
a warmup and decay schedule. We use standard normalisation, and resize each
image to $96\times 96$ pixels. We fine-tune for 30 epochs, but report results
on the best validation set Top 1 accuracy achieved.

####  A.3.4 Choosing important multi-spectral bands

As mentioned in 5.4, all 13 bands of the Sentinel-2 data may not be useful. In
our experiments, we drop bands B1, B9, and B10. To correctly identify the
utility of each band, one would need to pre-train a model with all bands
except the one in question, and then measure the performance after fine-tuning
without that band. However, this is prohibitively expensive in terms of
computational resources. Instead, we pre-trained a SatMAE+Stack model with a
ViT Base backbone on all 13 Sentinel-2 bands of fMoW Sentinel, and then fine-
tuned the model on the image classification task of fMoW Sentinel using all 13
bands. Using the finetuned model, we ran an ablation masking out subsets of
bands with the mean value for those bands and measuring the drop in validation
set accuracy. Since the model was trained to rely on information of all 13
bands, if masking out some bands doesn’t result in a big drop in accuracy,
this gives us an indication for which bands might be least useful for the
model to perform well.

Bands Masked | Top 1 Acc. | Top 5 Acc.
---|---|---
None | 57.80 | 80.07
B1 | 45.83 | 69.46
B2, B3, B4 | 17.46 | 35.05
B5, B6, B7 | 13.25 | 35.33
B8, B8A | 16.03 | 35.36
B9, B10 | 55.06 | 78.15
B11, B12 | 27.83 | 54.41
|  |
Table 10: An ablation to determine which of the 13 bands are least useful to a
SatMAE+Stack model pre-trained and finetuned on all 13 bands of fMoW Sentinel.
During evaluation, for each image, the relevant bands are masked with their
mean value recorded in table 9 and then passed as is to the finetuned
SatMAE+Stack model.

As seen in table 10, we notice the smallest drop in accuracy when masking
bands B9 and B10. The drop in accuracy when masking B1 is larger, but this may
be because the model relies on potentially unimportant signals from B1 during
fine-tuning. We therefore also drop B1 in our experiments in section 5.4. We
note that the other multi-spectral bands in addition to RGB seem highly
relevant to our model.

###  A.4 NAIP Land Cover Classification

We use the finetuning setting as in the fMoW RGB (non-temporal) finetuning
experiment (A.3.1).

###  A.5 EuroSAT Land Cover Classification

We use the exact finetuning setting as in the fMoW RGB (non-temporal)
finetuning experiment for RGB-only input (A.3.1). We also use the exact
finetuning setting as in the fMoW-Sentinel finetuning experiment (A.3.3)
except for training longer (150 epochs) for multi-spectral (13-band) input. It
took the model longer to converge most probably because EuroSAT is
comparatively a much smaller dataset. The license333EuroSAT license:
https://creativecommons.org/licenses/by/4.0/ is provided in the footnote.

###  A.6 BigEarthNet Land Cover Multi-label Classification

We use the exact finetuning setting as in the fMoW-Sentinel finetuning
experiment (A.3.3). Since the task is multi-label classification instead of
single-label classification, we changed the training objective to multi-label
soft margin loss. We use the mean Average Precision metric as provided in
[28].

The license 444BigEarthNet license:
https://bigearth.net/downloads/documents/License.pdf is provided in the
footnote.

###  A.7 SpaceNet V2 Building Segmentation

We use PSANet [45] for the binary image segmentation and replace the backbone
with ViT-Large. Following [27], we set the learning rate to $1\times 10^{-3}$
for ViT encoder and to $1\times 10^{-2}$ for ViT head and PSA module. We train
the model for $100$ epochs with batch size $128$ using an SGD optimizer of
momentum $0.9$ and weight decay $1\times 10^{-4}$ and a polynomial learning
rate decay scheduler of power $0.9$. Also following [27], we resize and crop
the input image to $400\times 400$ for fair comparison. This indicates our
model will take more patches per image (625). We use the positional encoding
interpolation algorithm provided by [1] to adjust the pre-trained weights.

The license555SpaceNet V2 license: http://creativecommons.org/licenses/by-
sa/4.0/ is provided in the footnote.

##  Appendix B Societal Impact

Measurements of economic, social, and environmental indicators are critical to
policy-making across the world. However, such measurements are constantly
lacking, hindering the process of decision making. Instead of using
traditional measurements (_e.g_. ground survey), our method exploits abundant,
globally-available and frequently-updated unlabelled satellite data. Our model
is capable of capturing representations from remote sensing imagery that are
beneficial for critical downstream tasks, including poverty prediction,
infrastructure development, and population estimation. Governments could make
good use of such information in decision making and consequently bring
significant societal benefits.

Although the use of satellite imagery could potentially lead to data abuse and
privacy violations from malicious actors, we contend that applications of our
model trained on publicly available satellite imagery respects privacy and
avoids exposing sensitive information. For example, individually identifiable
information cannot easily be retrieved from these images. Thus, we believe
that the imagery we used does not directly constitute a privacy concern.
However, we note that representations learned from SatMAE could potentially
suffer from biases if the training data is biased. For instance, SatMAE
trained on geographically imbalanced data could bias the model towards certain
regions, especially those in Northern America and Europe (see 6). Thus, we
advise researchers to be aware of directly applying our SatMAE models to
datasets with a geographical distribution different to that of fMoW RGB and
fMoW Sentinel. In our code release, we will also specify allowable uses with
appropriate licenses.

##  Appendix C Visualizations

In this section, we include visualisations in the temporal (C.1) and multi-
spectral settings (C.2).

###  C.1 Temporal SatMAE

Figure 8: More visualization examples of the reconstruction quality of image
sequences from the fMoW temporal dataset across multiple settings, including
using no temporal encoding + IM (Independent Masking), default + CM, default +
IM.

As shown in fig. 8, SatMAE+IM achieved relatively satisfying reconstruction
quality. Without the temporal encoding, the patches across all three images
cannot be distinguished, and we observe a mixture of the three images in the
reconstruction outcome of the second column. As explained earlier, using
independent masking can allow SatMAE to reconstruct an image in the time
series using information from other temporal patches. Our experiments show
that this helps SatMAE learn better representations for satellite imagery.

###  C.2 Spectral SatMAE

We also consider visualising the in-painting quality for the different multi-
spectral settings, including SatMAE+Group+IM, SatMAE+Group+CM (see 4.2, 4.2),
and SatMAE+Stack (see 4.2) in fig. 9.

Figure 9: Visualizations of SatMAE in-painting under different settings. RGB
represents bands B4, B3, B2, from group B2, B3, B4, B8. NIR represents bands
B7, B6, B5, from group B5, B6, B7, B8A. SWIR represents bands B11 in
grayscale, from group B11, B12. For each method, we show the input band group
masked and reconstructed side-by-side. We note that the reconstruction for
visible patches is worse than for the masked patches, since no loss is
computed on visible patches. The top-left quadrant represents a multi-spectral
image of an airport, the top-right quadrants is of a recreational facility,
the bottom-left is of an airport and the bottom-right is of an amusement park.

We can see a clear improvement in the quality of reconstruction under
SatMAE+Group+IM compared to SatMAE+Group+CM and SatMAE+Stack. Independent
masking results in sharper reconstructions, whereas the results from
consistent masking and stacking the channels are much fuzzier. We also note
that the model is able to learn correlations between bands; in the top-left
for the SWIR band group, even though the bottom right corner of the image is
masked, SatMAE+Group+IM is able to reconstruct the bright spot based on
information from the other band groups.

We hypothesize that further improvements in reconstruction quality and learned
representations can be achieved with longer pre-training.