# SEnSeI: A Deep Learning Module for Creating Sensor Independent Cloud Masks

Alistair Francis, John Mrziglod, Panagiotis Sidiropoulos , and Jan-Peter
Muller Manuscript received May 24, 2021; revised July 22, 2021 and October 29,
2021; accepted November 9, 2021A. Francis was with Mullard Space Science
Laboratory, University College London, and is now with $\Phi$-lab, ESRIN.
e-mail<EMAIL_ADDRESS>Sidiropoulos and J.-P. Muller are with
Mullard Space Science Laboratory, University College LondonJ. Mrziglod was
previously with the World Food Programme, Rome

###### Abstract

We introduce a novel neural network architecture—Spectral ENcoder for SEnsor
Independence (SEnSeI)—by which several multispectral instruments, each with
different combinations of spectral bands, can be used to train a generalised
deep learning model. We focus on the problem of cloud masking, using several
pre-existing datasets, and a new, freely available dataset for Sentinel-2. Our
model is shown to achieve state-of-the-art performance on the satellites it
was trained on (Sentinel-2 and Landsat 8), and is able to extrapolate to
sensors it has not seen during training such as Landsat 7, PerúSat-1, and
Sentinel-3 SLSTR. Model performance is shown to improve when multiple
satellites are used in training, approaching or surpassing the performance of
specialised, single-sensor models. This work is motivated by the fact that the
remote sensing community has access to data taken with a hugely variety of
sensors. This has inevitably led to labelling efforts being undertaken
separately for different sensors, which limits the performance of deep
learning models, given their need for huge training sets to perform optimally.
Sensor independence can enable deep learning models to utilise multiple
datasets for training simultaneously, boosting performance and making them
much more widely applicable. This may lead to deep learning approaches being
used more frequently for on-board applications and in ground segment data
processing, which generally require models to be ready at launch or soon
afterwards.

###### Index Terms:

Cloud masking, deep learning, multispectral, interoperability, sensor
independence, earth observation

©2021 IEEE. Personal use of this material is permitted. Permission from IEEE
must be obtained for all other uses, in any current or future media, including
reprinting/republishing this material for advertising or promotional purposes,
creating new collective works, for resale or redistribution to servers or
lists, or reuse of any copyrighted component of this work in other works.

##  I Introduction

There is an ever-increasing quantity of remote sensing data—supplied by
hundreds of satellites—being captured and downlinked every day. To preprocess
and analyse this satellite data, automated techniques are essential. Deep
learning has been of particular interest to remote sensing practitioners
recently [1, 2], given its proven abilities in a wide range of computer vision
problems. Cloud masking is one such instance, which we will focus on
specifically in this paper. However, many others exist, e.g. crop-type
classification [3], sea ice detection [4], forest fire detection [5], urban
structure analysis [6], and sea vessel tracking [7].

When these tasks are approached through the lens of supervised learning,
labelled datasets are needed for the dual purpose of training and testing
predictive models (often deep learning models, which are our focus here, but
not exclusively). The labour required to annotate datasets is substantial in
all fields of computer vision [8], but is perhaps particularly relevant in
remote sensing, because each dataset is specific to not only a given problem,
but also a given sensor. In practical terms, this has led to a somewhat
fractured set of data collection projects, each producing relatively modest
datasets that target a specific satellite. These cannot easily be combined to
add cumulative value to a predictive model, although exceptions to this do
exist, for example the many high-resolution satellites that use similar sensor
designs with RGB channels only, often using Bayer color filter patterns, which
lend themselves to direct interoperability.

These atomised data collection efforts mean that when a researcher approaches
a problem, one of two things often happen. Many decide to work on the largest,
highest quality dataset available for a given problem, which can lead to an
over-focus of studies on a few satellites and datasets. Alternatively, if a
researcher deems it necessary, they will resort to creating their own purpose-
built dataset for their sensor of choice. This is by no means an entirely
negative consequence, as those datasets can add value to others’ work if made
publicly available. However, it does mean that the prior efforts of other
researchers—who may have created labelled datasets and designed models for the
same task but on different sensors—are not adding value to the project. In
this way, the quantity of labelled datasets used in model training seems
unlikely to dramatically increase, even as the total quantity of existing
labelled data does continue to grow.

Given that deep learning’s potential is most easily realised with huge
quantities of training data, we argue that researchers should continue to
examine ways to build interoperability into their work, in order to make the
quantity of usable labelled data increase in proportion to the total labour
expended on annotation, and lead to larger and larger amounts of publicly
available training data.

In pursuit of this interoperability, we define sensor independence as the
ability of a model to ingest data from different multispectral sensors,
without retraining, and without the need to preprocess the data to be in the
same format. A sensor independent model is one which can take data without a
fixed number of spectral bands. This is related to—but distinct from—the
concept of multimodality, which concerns the combination of data from multiple
sensors into a single input (e.g. [9, 10]), whereas a sensor independent model
tries to perform a task with data from one sensor, regardless of which sensor
it is.

Sensor independence is a skill that humans largely take for granted given the
cognitive ease with which we can assimilate and compare visual inputs of very
different kinds, e.g. our ability to simultaneously understand contour plots
and heatmaps. Our ability to do this hinges on our understanding of how these
different styles can be translated to express the same underlying concepts.
Similarly, for a deep learning model to emulate this capability, a translation
from the individual satellite representations into a shared, general
representation must be made.

Besides an expansion in the available training data for a model, sensor
independence also makes deep learning approaches more practicable, and
immediately usable on a wider variety of sensors. Algorithms used for data
processing by satellite operators are usually designed before launch, so that
they can be applied immediately when the satellite begins to downlink data.
Machine learning and deep learning techniques—despite commonly exhibiting
better results than other methods—are rarely used in this setting. This is
often because without access to training data prior to launch, it is not easy
to develop a model which relies on supervised learning. A similar situation
exists for on-board deployment of machine learning. Both cases would be made
more practical if models could be developed prior to launch, using pre-
existing datasets from other satellites, which sensor independence would
allow.

This paper proposes a versatile framework, Spectral ENcoder for SEnsor
Independence (SEnSeI), which enables deep learning architectures to become
sensor independent. SEnSeI is a pre-processing module, which acts as the
translator between the “languages” of individual sensors (their individual
channels’ reflectances and wavelengths) and a shared feature space that can
then be inputted into an existing deep learning model (or indeed, any other
machine learning model). SEnSeI’s design does not depend on the model it is
used with, allowing it to be bolted onto any pre-existing machine learning or
deep learning architecture, which can then be trained to be sensor
independent. SEnSeI is designed as a permutation invariant neural network, the
theoretical background and practical development of which is set out in
Section II-C.

Ideally, the addition of SEnSeI to a model will have no deleterious effects on
its performance, whilst also making it sensor independent. In order to distil
down the implications of SEnSeI, we tie all our experimental results back to
the following hypotheses:

  1. Hyp. (A):

If SEnSeI is added to a deep learning model, there is no reduction in
performance when trained and tested on data from a single sensor.

  2. Hyp. (B):

If SEnSeI is added to a deep learning model, the model can be trained on
multiple sensors without a drop in performance relative to when it is trained
on an individual sensor.

  3. Hyp. (C):

SEnSeI can enable a model to perform adequately on a previously unseen sensor,
given training data that provide close equivalents to the unseen sensor’s
spectral bands.

In order to measure SEnSeI’s performance and ability to generalise to
different sensors, we have chosen cloud masking as a case-study. Cloud masking
is a central problem in Earth Observation, as rejecting cloudy areas is a
vital pre-processing stage in many other applications. It is well studied, and
has large datasets from several sensors, but these datasets have not yet
allowed researchers to employ deep learning to cloud masking on other sensors
which don’t currently have large manually labelled datasets. This is a key
limitation in current approaches, because whilst high performance can be
demonstrated on individual sensors, the techniques are usually not applicable
to other multispectral instruments.

Both a single-pixel neural network, and a large convolutional model
(DeepLabv3+ [11]) are combined with SEnSeI, using various training set
configurations. These two models show how SEnSeI interacts with as wide a
range of models as possible, from the very simple to the very complex. Our key
findings in this paper include:

  * •

A novel model architecture, SEnSeI, is proposed. SEnSeI is a permutation
invariant neural network, with unique features specifically designed for
remote sensing data.

  * •

DeepLabv3+ is shown to work extremely well as a cloud masking model,
exhibiting higher performance than other existing techniques for several
multispectral satellites.

  * •

By combining SEnSeI with DeepLabv3+, training datasets can be combined to make
models which work across many instruments, with a range of resolutions and
spectral band combinations.

  * •

Whilst in some instances, sensor independence leads to higher numerical
performance (e.g. in our tests on Landsat 8 data), the primary benefit of
adopting sensor independence with SEnSeI is the increase in a model’s
usability. With SEnSeI, it is now possible to create reliable cloud masks for
a huge family of multispectral instruments, even those which do not currently
have labelled datasets.

After a section exploring previous studies pertinent to our work (Section II),
our experimental setup, including model design, datasets and their
preprocessing, and model training is detailed in Section III. Then,
quantitative test results on four satellites’ data, Sentinel-2, Landsat 8,
Landsat 7 and PerúSat-1 are given in IV, along with visual results on
Sentinel-3 SLSTR. In Sections V and VI we discuss how these results fit with
the aforementioned hypotheses, the possible implications this work has on deep
learning in remote sensing, some of the future avenues of research this
entails, and its limitations. The code associated with this work is publicly
available [12].

##  II Related Work

###  II-A Satellite Interoperability

Here we consider the similarities and differences between satellite sensors,
and prior studies which successfully use combinations of satellites with
predictive models. Comparison of data between Sentinel-2 and Landsat 8 has
shown good correlation [13]. By comparing coincident observations between
Sentinel-2 and Landsat 8, their top of atmosphere reflectances were shown to
be within 1% (Blue and Coastal Aerosol bands showed slightly more deviation at
1.5%). A similar analysis over desert sites also found results that suggested
good agreement between the instruments [14].

A prominent example of data calibration across satellites is NASA’s
harmonization project [15], calibrating Sentinel-2 with Landsat 8. This has
allowed for time series analyses with more frequent observations, as more
cloud-free images can be found when using multiple sensors. Applications of
this harmonized data include detection of irrigated land using several
classifiers [16], high-resolution land phenology [17], and winter wheat yield
estimation [18]. Whilst interoperability of contemporaneous missions can lead
to more frequent observations, calibration of satellites past and present can
lead to longer baseline studies, as [19] covers in the context of forest
monitoring.

These approaches show that interoperability can be a highly successful
strategy for expanding available Earth Observation data, by lowering revisit
times, or expanding the period of study. A general trend that is seen in
current applications of interoperability is shared feature selection; derived
values, e.g. the Normalized Difference Vegetation Index (NDVI) [20] are
calculated for the different sensors. This effectively translates the
instruments into a shared representation which a predictive model can use. We
propose instead to make this translation into a general, non-linear process,
with SEnSeI. This comes with the hypothetical advantage of the predictive
model’s utilisation of all spectral information in each available satellite,
not only those bands used to create derived values such as NDVI.

###  II-B Cloud Masking

A wide variety of modelling approaches have been studied in the detection of
clouds [21]. Some models have used multitemporal information to predict cloud
cover (e.g. [22, 23, 24]). By leveraging the differences between images in a
time series, one can ascertain which images contain cloud. However, we focus
primarily on those which take single images as their input, as that is the
approach we prefer.

Many algorithms are based on reflectance or radiance thresholds, which are
specified by experts and are usually fixed. For example, Landsat 7’s ACCA [25]
estimates the cloud cover over a scene by passing the band values through a
rules-based system, which predict whether each pixel is a cloud. [26] builds
and improves upon this design with AT-ACCA. Similarly, [27] use a thresholding
method on the high-resolution visible channel of Meteosat SEVIRI. Sen2Cor,
which serves to correct for atmospheric effects—including clouds—in Sentinel-2
Level 2 products also uses a rules-based approach[28]. Fmask [29, 30] is an
algorithm originally designed for Landsat imagery, which uses situational
thresholding based on whether the cloud is thought to be above water or land,
and can assign confidence values to its predictions, allowing users to balance
their needs for sensitivity and specificity of detections. Unfortunately, in
studies comparing the relative performance of different methods, thresholding
based approaches tend to fall short of techniques which embrace some kind of
optimisation through labelled training data [21, 31].

Convolutional Neural Networks (CNNs) have come to dominate research in many
Computer Vision tasks, largely triggered by the publication of AlexNet [32,
33]. For image segmentation, many architectures have been proposed. For
example, U-Net [34] was developed for biomedical image segmentation, but
variations of it have also been used with success in cloud masking [35, 36,
37, 38]. More bespoke architectures have also been developed, e.g. [39, 40,
41, 42, 43]. Given that our work concerns sensor independence, and our
contribution, SEnSeI, could theoretically be used with any and all machine
learning architectures, we forgo a discussion of their relative benefits,
(though such a discussion is available in [21]). Instead, we now turn to look
at studies which have focused on using models across different sensors, as we
have done.

Many studies have applied deep learning models to cloud masking across
multiple sensors. Here, we define some broad categories of multi-satellite
model application that have been previously studied, and differentiate them
from our proposed framework of sensor independence. First, we characterise
passive interoperability as the ability to use the same model architecture
across different sensors, but with training datasets for each being needed.
This is the most common form of interoperability within deep learning-based
cloud masking.

Next, we define transfer learning (in the context of cloud masking) as the
opportunistic use of similarities between sensors to train models on specific,
shared spectral band combinations. This approach still provides a model which
is not generalised, in that it cannot then be used on any arbitrary set of
spectral bands. This has two drawbacks, (i) the number of sensors a model can
work with is limited to only those with all of the selected bands, and (ii)
the information from other spectral bands not included in the specific
combination of selected bands is discarded. [44] use PlanetScope data, and
Sentinel-2 data restricted to the RGB and NIR bands, to simulate the same
spectral bands as PlanetScope, allowing for a model which is interoperable
with 4-band RGB+NIR satellite data. Similarly, [45] shows a single model
working across RGB images taken from 8 different sensors and a range of
resolutions, from 4–50 m. [36] notes that not only are RGB and NIR bands
shared, but also that the two SWIR bands across Sentinel-2, and Landsat 5, 7,
and 8 are similar, using this to train a 6-band model shared across the four
sensors.

In contrast to passive interoperability and transfer learning, we propose the
term sensor independence to describe the property of a model which can be used
across different sensors without selecting a specific spectral band
combination, and without retraining. The bands that a sensor independent model
can use are still limited to those available to the model during training, but
any combination of those bands can be taken as input. This affords us greater
flexibility in sensor choice, more training data, and a more comprehensive
leveraging of the available information from the spectral bands those sensors’
data include, by not throwing away those bands which are not shared. We now
introduce the machine learning concepts which make this possible.

###  II-C Permutation Invariant Neural Networks

A distinct formulation of neural network architectures has arisen, in recent
years, for tasks which do not have a fixed number of input variables.
Permutation Invariant Neural Networks (PINNs) are a broad family of model
designs, which have the property that the output of the network does not
change given any re-ordering (permutation) of the set of inputs. The PINN is
therefore symmetric with respect to the order of the inputs. This section
first introduces the theoretical basis for permutation invariance, and then
gives some examples pertinent to our work.

Permutation invariance can be achieved through any symmetric pooling operation
(e.g. summation, maximisation, or mean averaging) over the outputs of some
function acting on a set of inputs. Broadly, a PINN can be seen as applying a
transformation (which in the case of a PINN is a neural network),
$\mathcal{F}$, across the inputs at each index $i$ of a set of $N$ inputs $X$,
then pooling those $N$ transformed inputs with an operation $\mathcal{P}$.

$y=\mathcal{P}(\\{\mathcal{F}(X,\;i)\;\;\forall\;\;i\;\in\;1,2,...\>,N\\})$
(1)

This formula has the useful property that the size of the vector y is
independent of the number of input elements, N, allowing for any arbitrary
group of inputs to be represented in the same vector space, which a standard
neural network can then use.

In simple cases, $\mathcal{F}$ acts upon each element $x_{i}$ individually.
However, more complex operations can be performed which rely on other members
of the set of inputs $X$, such that elements other than $x_{i}$ affect the
$i^{th}$ output. Therefore, in the general case, if the output $y$ is always
to be symmetric with respect to permutations on the members of $X$, then
$\mathcal{F}$ must be equivariant with respect to any index swapping
transformation, $\mathcal{T}$, on a pair on input indices i and j.
Equivariance, then, means that any reordering of inputs results in exactly the
same reordering of outputs:

$\mathcal{F}(\mathcal{T}_{ij}(X),\;i)=\mathcal{F}(X,\;j)$ (2)

Several studies have proposed PINN models which satisfy this definition.
PointNet [46] and PointNet++ [47] are prominent examples, designed for the
classification or segmentation of point cloud data. They process each
3-dimensional point (x, y, z) into a 1024-dimensional feature vector. These
are then combined across all $N$ points using a max-pooling operation, to
produce a single 1024-dimensional vector which holds information on all points
in the point cloud simultaneously.

Permutational layers—which combine each pair of $N$ inputs, and then pool them
so as to return $N$ outputs—are permutationally equivariant (satisfying (2))
and have been shown to work well in modelling dynamics [48]. Each
permutational layer consists of a neural network and allows for information
about the interactions between inputs to be considered by the network, by
looking at each pair. This allowed the model to successfully predict the
velocities and positions of idealised discs moving and colliding in 2
dimensions.

Unfortunately, pairwise operations have $\mathcal{O}(n^{2})$ complexity, and
so their application may not be practical in tasks with a large number of
input points. To alleviate this, PointNet++ [47] limits pairwise processing to
only those points within the vicinity of the target point, limiting the
computational complexity whilst retaining the fusion of information from
nearby points. In our work, this is not a concern, given that the maximum
number of spectral bands in multispectral imagery is comparatively low.
However, future work concerning sensor independence with hyperspectral
data—which contains many more bands—may benefit from an approach which
intelligently reduces the number of band pairings considered.

##  III Methods

###  III-A SEnSeI’s Design

SEnSeI is an example of a PINN. By this, we simply mean that the output of
SEnSeI is independent of the order of the inputs. In practical terms, this
invariance is achieved through a simple averaging over the outputted features
corresponding to each input. Imagine a neural network ingesting $N$ input
vectors, and thus outputting $N$ feature vectors, which are then pooled to
make a single feature vector. This averaged feature space is the shared
representation that SEnSeI passes to a standard deep learning model.
Importantly, this can only be done because of the fixed size of the
representation outputted by SEnSeI; an output which changed shape based on the
inputs would not be compatible with standard deep learning models.

The structure of satellite data is the primary driver for the design of
SEnSeI. In essence, satellite data can be considered as a set of channels,
each with an extended spatial map of reflectances, and a spectral profile.
This set of bands is our set of input points to SEnSeI, and its output is a
matrix of the same spatial dimensions as the original bands, but with a fixed
number of channels. The precise nature of this representation is, of course,
non-linear and determined through stochastic training of the network, meaning
the output channels do not necessarily have a well-defined physical analogue,
or strong associations with any one input channel.

Essentially then, we want SEnSeI to be able to take each channel, and
represent them all in a way that preserves the information about their
reflectance and their wavelength, and also ensures that when pooled, that
information is not degraded or confused with other channels. If trained on a
fixed set of input channels, it is easy to see how this architecture could
overfit, given that the spectral profiles it fits to would always be
identical. To avoid this, random subsets of the available channels are used to
increase variation, as described further in Section III-B.

We parameterise the spectral profile of each band as a ‘descriptor vector’.
This consists of a vector with three values: The lower, central and upper
wavelengths of the spectral band in question. The wavelength values are
normalised, with further details of the procedure found in Section III-E. Fig.
1 visualises the spectral bands of the sensors used in this experiment.

Figure 1: The band spectra of the instruments used in this paper. This shows
the diversity of sensors used, and underlines the need for sensor
independence. The values here represent the FWHM of the bands, but should not
be taken as exact. The descriptor vector corresponding to each band comprises
its minimum, central, and maximum wavelengths. SLSTR’s thermal bands are not
shown as we do not use them in our experiments. Colours and small vertical
offsets are used arbitrarily, to help make neighbouring and overlapping bands
more distinct. Figure 2: Schematic of SEnSeI. Inputs are $N$ images with $N$
corresponding descriptor vectors (circles), with only 3 (red, green and blue)
shown here for simplicity. Band multiplication creates feature maps which have
the same extent as the input image, wherein each pixel is represented by a
64-dimension vector, scaled by the band’s reflectance in that position, plus
an offset of 0.5 such that low reflectances are still represented. The entire
operation is permutation invariant, given that the final mean averaging over
the feature maps does not depend on the inputs’ order.

The architecture of SEnSeI can be split into 4 sub-modules, detailed in Fig.
2. First, the ‘DESCRIPTOR BLOCK’ comprises a neural network with several fully
connected layers, with ReLU and Group Normalization layers between them. In
our case, 9 layers were used, with a final output vector of 64 features. At
this point, there are $N$ vectors being outputted from the ‘DESCRIPTOR BLOCK’,
each corresponding to one of the $N$ bands at input. No information from the
reflectance values, nor other bands’ descriptor vectors, are used in these
calculations. Simply, each of the $N$ descriptor vectors (of length 3) enter a
9-layer neural network, which transforms them each into vectors of length 64.

Next, the ‘PERMUTATION BLOCK’ takes each possible pair of the $N$ outputs from
the previous module, and concatenates them together into $N^{2}$
128-dimensional feature vectors, where each band is paired with all $N$ bands,
including itself. The 128 dimensions simply come from the two 64-dimensional
vectors concatenated together. These are then fed through 5 fully connected
layers, and outputted as 128-dimensional vectors. To reduce the number of
vectors from $N^{2}$ back down to $N$, a pooling operation must be performed.
By averaging across the $N$ pairs associated with each of the bands, we then
reduce the number of feature vectors back down to $N$.

Third, a ‘COMBINED DESCRIPTOR BLOCK’ does essentially the same as the
‘DESCRIPTOR BLOCK’ module, but this time acts on the outputs of the previous
module, again producing a 64-dimensional feature vector for each of the $N$
inputs. This block is the final set of neural network layers in SEnSeI.

At this stage, SEnSeI has not used any of the reflectance band information. To
do so within a neural network would require us to compute the outputs of each
layer across each pixel of the input image, as the pixels do not all have the
same reflectances. This could become very computationally intensive for large
images. To get around this, we instead introduce the reflectance information
with the ‘BAND MULTIPLICATION BLOCK’. This module takes the 64-dimensional
output feature vectors of the ‘COMBINED DESCRIPTOR BLOCK’ and combines it with
the X-by-Y matrices of reflectance values for each band. At each position in
the original reflectance map, the 64-dimensional feature vector is multiplied
by the corresponding reflectance value, with an added offset of 0.5 (so that
information from low reflectance values are still included in the output).
This produces an N-by-X-by-Y-by-64 feature map. This multiplication is
computationally cheap and so can be carried out easily over each pixel.

The entire operation, up to this point, can be thought of as our
transformation $\mathcal{F}$ from (1). If we were to swap any two inputs, say,
Red and Green, in Fig. 2, then we can see that the order of the outputs would
be changed in the same way, hence satisfying (2). Finally, we apply our
pooling operation, $\mathcal{P}$, to our set of feature maps: A mean averaging
over the $N$ feature maps, producing an X-by-Y-by-64 feature space that can be
directly inputted into any other model, in our case DeepLabv3+. Whilst SEnSeI
does not fuse information between pixels, the spatial features one finds in
the input image are also present in the outputted feature map, because each
pixel’s feature vector is affected by its input reflectances. This means that
models attached to SEnSeI still have access to that spatial information.

Although this approach is applied specifically to multispectral data, SEnSeI
could easily be adapted to work with a range of different data types.
Essentially, any data structure for which there is a set of images, each with
a corresponding group of descriptors (in this case wavelengths) that can be
parameterised.

Whilst SEnSeI has many neural network layers, computation is incredibly cheap,
because the neural networks act only on the descriptor vectors, rather than
the much larger image data. The final band multiplication operation is the
only one which scales with the spatial dimensions of the image data. During
supervised training we saw only a 5% slow down per iteration compared to the
standard DeepLabv3+ model. When using the models to make predictions after
training, SEnSeI did lower the speed by a small amount, but only when many
bands were used (see Appendix A).

###  III-B SEnSeI’s Pre-training

The weights for SenSeI were pre-trained before the supervised training of
cloud masks, as part of an autoencoder architecture. There were two primary
aims in this exercise: (i) optimise SEnSeI’s design, by testing many different
architectures, and (ii) pre-train the weights of SEnSeI for use later in the
supervised training of SEnSeI with DeepLabv3+.

Autoencoders are commonly used to train models in an unsupervised manner, as
we do here. The autoencoder setup involves encoding the inputs with SEnSeI,
and then decoding those encoded features with a decoder. In our case, there
were two, parallel decoder modules. At this point, SEnSeI is not being trained
for cloud masking, but simply its ability to represent the data from different
spectral bands in a coherent manner. The autoencoder training setup is
outlined in Fig. 3.

The first decoder module, known as the discriminator, predicts whether or not
a given reflectance band was included in the input. It outputs a confidence
that a band was in the original input, doing so for twice as many candidate
descriptors as were in the original, such that half its answers should be true
(for those which were in the inputs), and half false (for those which were
not). The loss for the discriminator, $\mathcal{L}_{discriminator}$, is the
mean squared error of the predictions. The $i^{th}$ truth value is 1 if that
descriptor was present in the original input, making its contribution to the
loss $(1-p_{i})^{2}$ where $p_{i}$ is the prediction. For fake descriptors
which weren’t in the original, the truth value is 0, making the term in the
loss $p_{i}^{2}$. These two cases lead to the function for
$\mathcal{L}_{discriminator}$ seen in Fig. 3.

Second, the estimator takes the inputted descriptors, and determines what the
reflectance value of each band was, based on the output of SEnSeI. These two
different decoders lead SEnSeI to be optimized both in its ability to
represent many spectral bands simultaneously without overlap and confusion,
and to retain information about the reflectance values in each of those bands.
The loss for this branch, $\mathcal{L}_{estimator}$, is also the mean square
error. This is calculated as the square of the difference between each band’s
true reflectance, and the predicted reflectance outputted by the estimator
(Fig. 3).

As input, SEnSeI was given a set of between 3 and 14 randomly generated
descriptor vectors, alongside randomly generated reflectance values associated
to each one. The descriptor vectors were created so that they represented
realistic spectral bands: central wavelengths between 400nm–20$\mu$m, covering
the range of typical multispectral instrument bands (see Fig. 1), and
bandwidths also tuned to cover typical cases. Having outputted a combined
feature vector, the two decoder modules (simple neural networks with several
fully connected layers) performed their discrimination and estimation. Once
trained, the discriminator and estimator are no longer needed and are
discarded, leaving only SEnSeI to be used in the supervised training of the
cloud masking model.

The specific architecture of SEnSeI was chosen during this unsupervised
training, by minimizing the loss with respect to the number of layers in each
block, number of hidden units in each layer, types of pooling operation, and
so on. However, like many neural network architectures, performance was not
hugely sensitive to changes in these hyperparameters, and so the final version
of SEnSeI was chosen from a wide set of possible configurations with similar
performance. Fig. 2 shows that architecture, as selected by this unsupervised
training.

Figure 3: Flowchart for unsupervised autoencoder training of SEnSeI. A set of
$N$ inputs comprising pairs of random descriptor vectors and reflectance
values are processed by SEnSeI. Then, two decoder branches, the discriminator
and estimator, attempt to recover information about the inputs based on
SEnSeI’s output. The discriminator learns to identify which descriptor vectors
were present in the input, whilst the estimator predicts the reflectance
values associated with each descriptor. Both branches are trained using the
mean square error function. When used with a convolutional neural network,
SEnSeI takes images, rather than single reflectance values, however during
unsupervised training single values are preferable, as they allow for larger
batch sizes.

###  III-C DeepLabv3+

DeepLabv3+ [11] is a CNN designed for image segmentation, building on prior
efforts [49, 50]. DeepLabv3+ uses a portion of a large CNN (pre-trained on
other tasks) as a feature extractor. Image segmentation performance often
benefits from fusing information from different spatial scales, from the
single pixel, to the whole image. DeepLabv3+ achieves this through atrous
convolutions. Atrous convolutions are like a standard convolution, but with a
dilated kernel. The rate of dilation can then lead to features being extracted
at different scales, by taking information from wider and wider fields of
view. The resulting feature maps are combined using spatial pyramid pooling
[51].

Whilst class predictions are required at every pixel, many neighbouring
predictions are highly correlated with one another. Therefore, DeepLabv3+
outputs downsampled predictions, at a rate known as the output stride, and
then upsamples the predictions to the full image resolution through a bilinear
interpolation. The output stride is a power of two, and is often set to 8, 16,
or 32. In our work, we use an output stride of 8, given that cloud masking
often requires very small areas to be segmented, unlike in other image
segmentation domains where the output mask is less spatially complex. This
relatively low output stride increases the computational requirements in
training. For this reason, we use a relatively computationally cheap CNN as
our feature extractor; MobileNetv2 [52], which enabled us to use a higher
batch size in training than would be possible with larger CNNs. Even with this
smaller CNN, DeepLabv3+ has 10 times the number of trainable parameters as
SEnSeI (see Appendix A). Overfitting is a common problem when the number of
parameters is large, but none of the models exhibited diverging validation and
training losses during training, suggesting it was not a problem for them,
with or without SEnSeI.

Our implementation of DeepLabv3+ uses TensorFlow 2 [53] and is based heavily
on [54]. We alter the first convolutional layer of the model to accept inputs
with more than the standard 3 RGB channels—64 when used with SEnSeI, or the
number of bands on a satellite if used without SEnSeI.

Supervised training of our DeepLabv3+ models remained identical across
experiments, other than whether or not SEnSeI was included as a preprocessing
module, and which datasets were used. We used a desktop machine with an NVidia
RTX 2080 Super (8GB) GPU. The models were initialized with pre-trained weights
in SEnSeI, derived using the unsupervised training described in Section III-B,
and MobileNetv2 with weights from a pre-trained version of DeepLabv3+ on the
PASCAL VOC training set [55]. All images were preprocessed according to
Section III-E, and entered the model at 257x257 pixels across, with a batch
size of 8. The model was trained using stochastic gradient descent with an
initial learning rate of $1\mathrm{e}{-3}$ and a momentum of 0.99. The initial
learning rate was reduced by a factor of 4 when the loss converged, down to a
minimum learning rate of $2\mathrm{e}{-5}$. Each epoch was 1000 steps, with a
maximum of 200 epochs (although most models were stopped after around 100
epochs as they had converged on a stable loss rate). Training of the models
took several hours, with each epoch taking roughly 5 minutes to complete.
Models with SEnSeI took around 5% longer per epoch, but also tended to take
10–20 more epochs to fully converge.

###  III-D Datasets

TABLE I: Overview of the labelled datasets used in our work. Whilst our own Sentinel-2 dataset has data from the most scenes, the Landsat 7 CCA and Landsat 8 CCA datasets cover many more pixels, as they use whole scenes. No-data pixels are not included in these counts. Landsat 8 SPARCS is something of an outlier regarding cloud cover percentage, with much less than other datasets. Dataset | No. of images | No. of pixels (millions) | Surface area (km2) | Cloud cover
---|---|---|---|---
Sentinel-2 Cloud Mask Catalogue [56] | 513 | 535 | 214’000 | 53.05%
Landsat 8 SPARCS [57] | 80 | 80 | 72’000 | 19.37%
Landsat 8 Cloud Cover Assessment [58] | 96 | 4’000 | 3’600’000 | 47.91%
Landsat 7 Cloud Cover Assessment [59] | 197 | 7’560 | 6’800’000 | 33.23%
CloudPeru2 [60] | 153 | 5’700 | 44’400 | 48.88%

####  III-D1 Sentinel-2 Cloud Mask Catalogue

We produced a dataset [56] comprising 513 subscenes taken from the 2018
archive of Sentinel-2 Level-1C imagery. Sampling of the scenes was completely
random, so that the data approximated the real-world distribution of
Sentinel-2 data. Each subscene and associated cloud mask is 1022 pixels
squared, at 20 m resolution for all bands. In order to label a large amount of
data, we designed the semi-automated tool called IRIS (Intelligent
Reinforcement for Image Segmentation), available on GitHub [61]. This tool
uses a Random Forest model to extrapolate the user’s paintbrush annotations
across the whole image, whilst still allowing the user to make follow-up
manual corrections and adjustments where needed after each run of the Random
Forest.

Alongside each pixel-wise segmentation mask, each image was given several
classification tags, describing the surface type, cloud thickness, cloud
structure, relative cloud height, and cloud type. These tags could then be
used to filter the test results, allowing us to measure performance over a
range of different conditions:

  1. Surface type:

forest/jungle, snow/ice, agricultural, urban/developed, coastal,
hills/mountains, desert/barren, shrublands/plains, wetland/bog/marsh,
open_water, enclosed_water

  2. Relative cloud height:

low, high

  3. Cloud extent:

isolated, extended

  4. Cloud type:

cumulus, cumulonimbus, stratocumulus, cirrus, haze/fog, ice_clouds, contrails

The first two authors processed the individual cloud masks, including a set of
50 which were marked by both individuals. By comparing the two annotators’
masks, we can estimate an inter-annotator agreement score, which informs users
of the data what sort of accuracy their model should achieve if human-level
performance is reached. The total agreement between the annotators was
95.0$\%$. The dataset is publicly available, and more information can be found
in its documentation [56]. In our experiments, we split the dataset in a
40:10:50 ratio, for training, validation and testing respectively.

####  III-D2 Landsat 8 SPARCS

The Spatial Procedures for Automated Removal of Clouds and Shadows (SPARCS)
validation dataset [57] consists of 80 manually created masks, each over a
1000 pixel squared subscene taken from a Landsat 8 product. As well as cloud
and cloud shadow, the masks differentiate between land, water, snow/ice, and
flooded surfaces. We combine these different surface types in the mask, to
make the labels simply clear vs. cloud.

SPARCS is distributed in GeoTIFF format, with all bands and the mask sampled
at 30m. The dataset does not contain Landsat’s Panchromatic band, but includes
all other 10 bands from Landsat 8. Of the 80 scenes, many show complex and
varied cloud structures, suggesting the images were hand-picked. The lack of
entirely cloudy or clear scenes means the dataset is not representative of
real-world use on the entire Landsat 8 catalogue, but provides a challenging
and diverse benchmark for models to be tested on.

####  III-D3 Landsat 8 Cloud Cover Assessment

The USGS Landsat 8 Cloud Cover Assessment (CCA) dataset [58] is designed for
validation and inter-comparison of cloud masking algorithms [31]. In total, 96
full Landsat 8 scenes were annotated by hand. The scenes are sampled so as to
get an equal number of examples from 8 different surface types (referred to as
biomes), with 12 samples each. These are; Barren, Forest, Grass/Crops,
Shrubland, Snow/Ice, Urban, Water, Wetlands. As well as annotating clear vs.
cloud vs. cloud shadow, they also differentiate thick from thin cloud, however
these different classes are merged in this work, in order to make it
compatible with our other datasets.

####  III-D4 Landsat 7 Cloud Cover Assessment

This dataset [59], also produced by the USGS, follows a similar format to the
Landsat 8 Cloud Cover Assessment dataset. 206 scenes are included, but are
grouped by latitude, rather than by surface type. The classes used in
annotation are also the same as those used for the Landsat 8 Cloud Cover
Assessment, however the percentage of scenes annotated with cloud shadow is
somewhat lower than the Landsat 8 dataset. Additionally, we found through
visual inspection that some thin clouds were not marked, compared to similarly
thin clouds that were marked in other datasets. During preprocessing of the
dataset, we were unable to recover 9 of the 206 scenes, leaving a total of 197
for our experiments.

####  III-D5 CloudPeru2

PerúSat-1 is a cubesat mission launched by the Peruvian space agency (CONIDA).
The sensor samples data at 2.8 m resolution, in 4 spectral bands; red, green,
blue and NIR. The CloudPeru2 dataset [60] comprises 22’000 images, each
512x512 pixels across, taken from 153 scenes over Peru. Alongside each image
is a hand-drawn mask with cloud vs. non-cloud pixels. This dataset has been
used to train and test cloud masking algorithms before [62], although in our
experiments we treat it only as a test set, and do not use it for training.
This dataset—and those discussed previously—are summarised in Table I.

###  III-E Preprocessing and Formatting

For training, each image is split into 263-by-263 cropped patches and saved.
Whilst DeepLabv3+ was configured to use 257-by-257 images as input, we made
the crops slightly larger to introduce some variation to the samples through
random translation, whilst still being small enough to be read in quickly
during training. All reflectance values were converted from their native
ranges to double precision floating point values between 0–1, although some
pixels have reflectances higher than 1, if there is specular reflection, or if
the surface is at an angle which reflects more than the normal plane could.

For the thermal bands of Landsat 7 and 8, we converted the radiances to
Brightness Temperatures (BTs), using (3) and the parameters $K_{1}$ and
$K_{2}$ in their product specifications. Then, we normalised these BTs between
0 and 1, such that a value of 1 corresponded to the 95th percentile of the
total available data (4). This was done in order to create a similar
distribution of values between thermal and non-thermal bands.

$BT=\dfrac{K_{2}}{ln(\dfrac{K_{1}}{Radiance}+1)}$ (3)

$BT_{normalised}=\dfrac{BT-BT_{min}}{BT_{95\%}-BT_{min}}$ (4)

Alongside each image and mask pair, we also saved the wavelengths of the
spectral bands held in the image, to be used as the descriptor vectors which
SEnSeI takes as inputs. This was simply an array with the lower, central, and
upper wavelengths of each band in nanometres, as defined in their product
specifications. For input into SEnSeI, we normalise these from nanometres to a
logarithmic scale, as the linear differences between e.g. red and green are
far smaller than those in the SWIR or TIR range.

$\lambda_{normalised}=log_{10}(\lambda-300)-2$ (5)

During training, data augmentation was applied as it has been shown to help
prevent over-fitting, and improve model performance [63]. We included random
flipping, rotation, translation, addition of white noise, salt-and-pepper
noise, and small random shifts in overall reflectances across the image’s
bands.

###  III-F Performance Metrics

Trying to summarise the performance of models in an image segmentation task
with a single number is problematic. Whilst it can be useful to judge all
models with a single variable, this obscures the reality that models perform
in different ways, with different strengths and weaknesses. The experiments in
this study use five well-known metrics so that a more nuanced account of the
models’ performance can be made.

All five metrics are defined by the populations of True Positive (correctly
identified cloud), True Negative (correctly identified non-cloud), False
Positive (non-cloud labelled incorrectly as cloud) and False Negative (cloud
incorrectly labelled as non-cloud).

Overall Accuracy (OA) is the total percentage of correctly identified pixels.
It is an intuitive metric, but can be unrepresentative of performance when
class prevalences are very imbalanced:

$OA=100*\frac{TP+TN}{TP+TN+FP+FN}$ (6)

Recall tells us how successful the model is in finding the population of
cloudy pixels. It does not take into account the fact that the model might
incorrectly label pixels as cloud (False Positive):

$Recall=100*\frac{TP}{TP+FN}$ (7)

Precision, as opposed to recall, determines the percentage of pixels which
were identified as cloudy that are correct. It does not depend on the number
of missed cloudy pixels (False Negatives):

$Precision=100*\frac{TP}{TP+FP}$ (8)

The $F_{1}$ score is the harmonic average between precision and recall. This
means that it always lies between their values, but it is heavily penalised if
one is much lower than the other. Of the metrics commonly found in deep
learning, $F_{1}$ is one of the most widely used as a singular measure of a
model’s performance, given that it takes into account both precision and
recall. However, the $F_{1}$ score still does not take into account the True
Negative population, which can lead to very low scores when the negative class
is much more common than the positive one:

$F_{1}=100*\frac{2*Precision*Recall}{Precision+Recall}$ (9)

Balanced Accuracy (BA) is similar to OA, but it is weighted by the relative
prevalence of each class (in this case, cloud or non-cloud). It can also be
thought of as the average of each class’ recall:

$BA=100*0.5*\left(\frac{TP}{TP+FN}+\frac{TN}{TN+FP}\right)$ (10)

##  IV Experimental Results

###  IV-A Sentinel-2

This experiment was conducted to measure the performance of both the standard
DeepLabv3+ model, and the sensor independent variants with SEnSeI prepending
DeepLabv3+, alongside some other cloud masking options that are widely used.
The three main goals of this experiment were:

  1. 1.

Demonstrate a state-of-the-art cloud masking model for Sentinel-2.

  2. 2.

Investigate whether Hyp. (A) and Hyp. (B) from Section I would hold, or if the
addition of SEnSeI, and an increasingly generalised training configuration,
would cause performance to suffer against the specialised, non-sensor
independent model.

  3. 3.

Determine how a deep learning-based approach compared to other popular
Sentinel-2 cloud masking algorithms.

The Sentinel-2 dataset was split randomly into training, validation, and
testing sets, such that 40% (205 subscenes) were in training, 10% (51
subscenes) in validation, and 50% (257 subscenes) in testing. The training for
each model was then carried out as Section III-E details.

Two versions of DeepLabv3+ were trained without SEnSeI. First, a version
simply trained with Sentinel-2 data, using all bands (referred to as ‘DLv3 -
S2’ in the experiments). Second, a version trained on both the Sentinel-2
training data, and the Landsat 8 CCA dataset, using only those bands which are
found on both Sentinel-2 and Landsat 8, making it an example of the transfer
learning approach outlined in Section II-B. This model is referred to as ‘DLv3
- S2&L8 [common bands]’. The common bands—in the naming scheme of
Sentinel-2—were bands B1, B2, B3, B4, B8, B10, B11, and B12.

TABLE II: The five performance metrics for each model, across the 257 Sentinel-2 subscenes used as a test set. The highest performance for each metric is highlighted in bold. All variants of DeepLabv3+ outperform the other models in OA, BA, Recall and $F_{1}$, however s2cloudless does achieve the highest precision. Model | OA | BA | Prec. | Rec. | $\mathbf{F_{1}}$
---|---|---|---|---|---
DLv3 - S2 | 94.04 | 94.03 | 94.57 | 94.17 | 94.37
DLv3 - S2&L8 [common bands] | 95.19 | 95.18 | 95.61 | 95.31 | 95.46
SEnSeI+DLv3 - S2 [fixed bands] | 94.46 | 94.42 | 94.58 | 94.99 | 94.79
SEnSeI+DLv3 - S2 | 93.08 | 93.10 | 94.00 | 92.89 | 93.44
SEnSeI+DLv3 - S2&L8 | 93.73 | 93.73 | 94.42 | 93.73 | 94.07
SEnSeI+DLv3 - S2&L8 [no pre-training of SEnSeI] | 93.11 | 93.16 | 94.47 | 92.43 | 93.44
NN - S2 | 90.04 | 90.04 | 91.06 | 90.08 | 90.57
NN - S2&L8 [common bands] | 89.86 | 89.98 | 92.50 | 88.02 | 90.20
SEnSeI+NN - S2 | 88.90 | 88.75 | 88.19 | 91.31 | 89.72
SEnSeI+NN - S2&L8 | 89.04 | 89.16 | 91.70 | 87.23 | 89.41
CloudFCN - S2 [35] | 91.58 | 91.67 | 93.58 | 90.34 | 91.93
s2cloudless [64] | 92.42 | 92.61 | 95.90 | 89.54 | 92.61
Standard L1C product mask [65] | 83.11 | 83.29 | 86.88 | 80.27 | 83.45

Four different configurations of DeepLabv3+ were used for SEnSeI. All four of
these had the exact same architecture. Three began training with the same pre-
trained weights in both SEnSeI and DeepLabv3+. These three configurations are
increasingly sensor independent, adding to the ability of the model to
generalise to other sensors. Another configuration used a SEnSeI with randomly
initialized weights, to see what advantage pre-training offered:

  1. 1.

‘SEnSeI+DLv3 - S2 [fixed bands]’ was given all Sentinel-2 bands at every
training step. This means it was only ever given Sentinel-2 data with all
bands present, thus producing a model that could not be generalised to work on
any other satellite or subset of Sentinel-2 bands, but containing the SEnSeI
architecture.

  2. 2.

‘SEnSeI+DLv3 - S2’, was trained using random combinations of the available
Sentinel-2 bands, from a minimum of 3, up to a maximum of all 13. This
additional complexity in training produces a model which is theoretically
capable of predicting clouds from sensors with any combination of the existing
Sentinel-2 bands (this assumption of sensor independence will be tested
further in Section IV-C).

  3. 3.

‘SEnSeI+DLv3 - S2&L8’, was also given the Landsat 8 Cloud Cover Assessment
dataset as training data. It then received random combinations of the
available bands on each satellite, creating what is shown in later experiments
to be a truly sensor independent model.

  4. 4.

‘SEnSeI+DLv3 - S2&L8 [no pre-training of SEnSeI]’, was the same as the
previous but the weights in SEnSeI were randomly initialized.

As well as DeepLabv3+, we combine SEnSeI with a 2-layer neural network, which
has no convolutions. This neural network has only a few hundred parameters,
and represents an extremely simple machine learning solution. By combining
SEnSeI with both this small neural network, and the very complex DeepLabv3+,
we hope to show that SEnSeI works with a wide range of model architectures.
The naming conventions for the neural network model configurations are the
same as for DeepLabv3+, with “NN” replacing “DLv3”.

In addition to DeepLabv3+ and the neural network, we also attempted to train
CloudFCN [35] with SEnSeI to further show SEnSeI’s general applicability;
however the training became unstable when they were used together, and it was
not possible to get the models to reliably converge. The reason for this is
not clear, but it may be that, when used with SEnSeI, CloudFCN would need to
be trained with a larger batch size than we were able to process on the 8GB of
video memory used in training. This is an avenue for future research, as it
may be that SEnSeI adds some instability to certain model architectures
(perhaps those with an architecture similar to U-Net [34], like CloudFCN)
during training. We do however include results from CloudFCN without SEnSeI in
this experiment, trained on the Sentinel-2 dataset.

We also include results from s2cloudless [64], using a confidence threshold of
0.5. This algorithm is a pixel-based machine learning model, which uses the
XGBoost algorithm [66].

Lastly, the Standard L1C product mask is the cloud mask generated during L1C
processing [65]. We create a rasterized version of the polygonal information
in the GML file provided with each L1C product. All “OPAQUE” and “CIRRUS”
polygons were included. The full product masks were then cropped to the extent
of the subscene, and visually inspected to confirm they had been correctly
processed and were aligned with the image. Comparison with this mask allows us
to judge the relative performance of a deep learning-based model to a
thresholding model.

To assess each model’s performance in this experiment, we consider five
metrics. Overall Accuracy (OA), Balanced Accuracy (BA), Precision, Recall, and
$F_{1}$, as defined in Section III-F. These are calculated across the entire
Sentinel-2 test set. Table II provides numerical values for these metrics over
the dataset. Fig. 4 gives some visual examples of the masks produced by the
models. Appendix B gives a further breakdown of results for the models across
different image types.

Overall, DLv3 - S2&L8 [common bands] performs better than the other models.
This shows that transfer learning techniques (as used by others, e.g. [36, 44,
45]) can result in boosts to performance, despite the removal of the bands
which Landsat 8 does not also include. Besides this model, SEnSeI+DLv3 - S2
[fixed bands] also exceeds the performance of DLv3 - S2 in all metrics,
showing that prepending SEnSeI to a deep learning model does not produce a
significant drop in performance, supporting Hyp. (A). When a more challenging
training configuration was used, in which band combinations were sampled
randomly, performance does drop somewhat (SEnSeI+DLv3 - S2). Promisingly,
however, performance of SEnSeI+DLv3 - S2&L8 is higher than that of SEnSeI+DLv3
- S2 in every metric, making it clear that adding data from more than one
satellite boosts the model’s reliability. Whilst SEnSeI+DLv3 - S2&L8 falls
short of DLv3 - S2&L8 [common bands] in terms of numerical performance, it is
important to remember that the former is able to produce cloud masks over a
huge range of different spectral band combinations, whilst the latter requires
all bands shared by Sentinel-2 and Landsat 8 to be present. We also observe a
small decrease in performance of roughly 0.6% in OA, BA, and $F_{1}$ when
using a randomly initialised version of SEnSeI, rather than the pre-trained
weights. This suggests pre-training SEnSeI is worthwhile, and has a moderate
but noticeable impact on performance.

SEnSeI’s effect on DeepLabv3+ is mirrored closely by its effect on the 2-layer
neural network. The creation of a sensor independent model leads to a drop of
about 1% from NN - S2 to SEnSeI+NN - S2 in OA, BA and $F_{1}$. This 1% drop in
performance (on test data from Sentinel-2) seems to be the cost of making a
model which works across many sensors when using SEnSeI. Interestingly, the
addition of data from Landsat 8 did not boost the neural network’s
performance, like it did for DeepLabv3+, suggesting that such a small, simple
model as this cannot benefit from an expanded training set, whereas DeepLabv3+
can.

In order to determine whether the differences in performance seen in Table II
are statistically significant or not we carried out a cross-validation and
bootstrapping experiment to investigate the uncertainties in our results. Due
to computational limitations in training many models, we were only able to
analyse the uncertainty for just two of the model configurations (DLv3 - S2
and SEnSeI+DLv3 - S2&L8). Although not all models were included, we believe
that given their architectures, training data, and the test set are identical
to those not included, this analysis can be safely extrapolated to approximate
the uncertainty for all our model results. For each model configuration, we
use 5-fold cross-validation to train 5 different models, each with a different
40% of the data as training, and 10% as validation. The same 257 scenes are
always held as the test set for all models. We then bootstrap the test set
results, with 1 million iterations per model, to measure the data-driven
uncertainty in our performance. With this method, both models tested were
found to have a range of $\pm 0.5\%$ in OA, BA and $F_{1}$. We take this range
as a rough limit for statistical significance in our analysis of the results.

Figure 4: Visualisation of seven models’ predictions tested on five scenes
from the Sentinel-2 test set. Blue is correctly identified cloud, red is
missed cloud, and yellow is false cloud detection. These scenes were selected
as they contained interesting, diverse and challenging cloud structures.
Figure 5: Visualisation of six models’ predictions tested on five scenes from
the Landsat 8 SPARCS test set. Blue is correctly identified cloud, red is
missed cloud, and yellow is false cloud detection. DLv3 - L8 shows much higher
precision and lower recall than other models. The boundaries of the thin cloud
in the 2nd row are understandably error-prone, given the smooth transition
between cloudy and clear. Some models misclassify parts the snow-covered
mountains in the 4th row as cloud, but successfully reject the snow on the
left of the 3rd row’s scene.

###  IV-B Landsat 8

We now present results on the Landsat 8 SPARCS dataset (described in Section
III-D2). In testing these models, we demonstrate true sensor independence (a
model working on multiple satellite sensors at once), and show strong evidence
for Hyp. (B) and Hyp. (C). In addition, we also use this experiment as an
opportunity to investigate the effects of different band combinations, by
testing the same models with different spectral band combinations. The first
model, DLv3 - L8, is trained using the 96 scenes from the Landsat 8 Cloud
Cover Assessment dataset, but without the panchromatic band (to make the model
compatible with the SPARCS dataset). DLv3 - S2&L8 [common bands] is also used
in this experiment, and is the same model as that used in Section IV-A. Then,
in a similar fashion to Section IV-A, we include two configurations of
DeepLabv3+ with SEnSeI:

  1. 1.

‘SEnSeI+DLv3 - S2’ is the same model (with the exact same weights) as used in
Section IV-A with mixed band combinations as input in training. No retraining
takes place to prepare it for use with Landsat 8, however, bands that
Sentinel-2 does not have (thermal bands) are not used in testing.

  2. 2.

‘SEnSeI+DLv3 - S2&L8’ is also the identical model to that which is used in
Section IV-A, with no retraining.

For both of these SEnSeI configurations, we run tests on SPARCS using several
different band combinations. For 10 bands, there are 968 possible combinations
of 3 or more bands. We therefore only select a few of interest:

  1. 1.

‘ALL’: All Landsat 8 bands (excluding the panchromatic band, which is not
distributed with the SPARCS dataset).

  2. 2.

‘COMMON’: All bands shared by Sentinel-2 and Landsat 8. SPARCS already
excludes Landsat 8’s panchromatic band, so this band combination is just all
available bands except the two thermal bands.

  3. 3.

‘RGB’: Bands 2, 3, and 4.

  4. 4.

‘AeroRGB’: Bands 1, 2, 3, and 4.

  5. 5.

‘AeroRGBNIR’: Bands 1, 2, 3, 4 and 5.

  6. 6.

‘No-RGB’: All possible bands except 2, 3 and 4.

TABLE III: Results of models across the Landsat 8 SPARCS dataset. The versions of RS-Net which are highlighted were tested on SPARCS using 5-fold cross-validation, and thus have an advantage over other models in that they have been exposed to the style and distribution of SPARCS at training. Therefore, in metrics for which this version of RS-Net is best, we also highlight the best model not trained on SPARCS. A version of Cloud-Net+ was trained using 64 SPARCS images and tested on 16, but is omitted here as the results do not cover all 80 scenes. Model | Bands | OA | BA | Precision | Recall | $\mathbf{F_{1}}$
---|---|---|---|---|---|---
DLv3-L8 | ALL | 93.03 | 83.32 | 95.17 | 67.46 | 78.96
DLv3-S2&L8 |  COMMON | 93.70 | 87.04 | 89.82 | 76.14 | 82.42
| COMMON | 91.86 | 93.05 | 71.96 | 94.99 | 81.89
| RGB | 90.91 | 91.77 | 69.92 | 93.18 | 79.89
| RGBNIR | 91.11 | 92.23 | 70.18 | 94.06 | 80.38
| AeroRGBNIR | 91.11 | 92.45 | 70.02 | 94.64 | 80.49
SEnSeI+DLv3 S2 | No-RGB | 91.52 | 92.45 | 71.34 | 93.96 | 81.10
| ALL | 93.91 | 86.31 | 93.24 | 73.90 | 82.45
| COMMON | 94.24 | 90.98 | 84.77 | 85.65 | 85.21
| RGB | 92.45 | 92.34 | 74.73 | 92.18 | 82.54
| RGBNIR | 93.51 | 92.29 | 79.12 | 90.31 | 84.34
| AeroRGBNIR | 93.63 | 91.67 | 80.54 | 88.47 | 84.32
SEnSeI+DLv3 S2&L8 | No-RGB | 93.83 | 86.19 | 93.00 | 73.71 | 82.24
| ALL | 92.53 | – | 88.57 | 70.53 | 78.35
| COMMON | 93.26 | – | 91.04 | 72.34 | 80.62
| RGBNIR | 92.53 | – | 95.35 | 64.56 | 76.99
RS-Net trained on Biome [37] | RGB | 92.38 | – | 86.54 | 71.83 | 78.50
| ALL | 94.54 | – | 87.62 | 83.66 | 85.59
| COMMON | 95.60 | – | 89.47 | 87.58 | 85.59
| RGBNIR | 94.85 | – | 88.92 | 83.89 | 86.33
RS-Net trained on SPARCS (5-fold C.V.) [37] | RGB | 94.86 | – | 88.58 | 84.34 | 86.41
Cloud-Net+ trained on 38-Cloud [40] | RGBNIR | 91.30 | 86.47 | 76.94 | 78.60 | 77.76
Cloud-Net+ trained on 95-Cloud [40] | RGBNIR | 90.45 | 85.69 | 74.14 | 77.90 | 75.97
Fmask [37] | – | 92.47 | – | 77.47 | 86.21 | 81.61

As well as our own models’ results, we also include the results of Fmask, the
standard Landsat 8 cloud masking software, and RS-Net, another state-of-the-
art deep learning model discussed in Section II-B, which was trained on
several different band combinations of Landsat 8. We also compare the results
of Cloud-Net+ when trained on their 38-Cloud and 95-Cloud datasets and tested
on SPARCS. We did not compute predictions for these three models ourselves,
but simply recover the results from prior studies ([37] for Fmask and RS-Net,
[40] for Cloud-Net+). For Fmask, there are differing results on the SPARCS
dataset in different studies, possibly because different versions of the code
of were used, and so the best published result for the SPARCS dataset is taken
[37].

The results of this experiment are shown in Table III, and some of the models
can be compared visually in Fig. 5. Out of the models not trained on SPARCS
data, SEnSeI+DLv3 - S2&L8 (COMMON) achieves the highest OA and $F_{1}$ score,
although it is beaten by models trained using 5-fold cross-validation on
SPARCS. Perhaps a more important result for Hyp. (C), though, is the
performance of SEnSeI+DLv3 - S2 across the different band combinations, which
shows a model performing with $>90\%$ OA on data from a sensor it has not seen
in training. Falling just short of Fmask in OA, we believe this shows adequate
performance for a model working on a different sensor. Finally, the
significant increase in performance between DLv3 - L8 and SEnSeI+DLv3 - S2&L8,
shows that adding data from another satellite (Sentinel-2) increases
performance on Landsat 8, agreeing with Hyp. (B). Not only is SENSeI allowing
for models to become more generalised and work across different sensors, but,
in this instance, it is allowing models to enhance their predictive
capabilities on one sensor with data from another.

The change in performance of the same models with different band combinations
is relatively small. For example, OA for most models varies by around 1-2%,
whilst the $F_{1}$ score is slightly more sensitive, varying by around 2-3%.
Generally, models with limited band combinations, e.g. RGB and RGBNIR,
suffered in comparison to more extensive band combinations, which makes sense
given that they do not have access to as much spectral information from their
inputs. Surprisingly, for all three models which were tested both with and
without the thermal infrared bands (SEnSeI+DLv3 - S2&L8, and both versions of
RS-Net), performance actually decreased when they were included. This is
surprising, given that thermal bands provide quite different information to
the others, which would ordinarily be a good thing for a model to have.

###  IV-C Other Satellites

In this section we seek to test the limits of SEnSeI’s sensor independence, by
predicting cloud masks for satellites not included in the model’s training,
and different resolutions to the 20–30 m/pixel range of Sentinel-2 and Landsat
8. We include three sensors in the following experiments, two of which
(Landsat 7 and PerúSat-1) have datasets that allowed us to measure our models’
performance, and another (Sentinel-3 SLSTR) which we include for visual
inspection only, as no suitable dataset was available.

Landsat 7’s non-thermal bands all have close equivalents in Sentinel-2 and
Landsat 8. It’s thermal band, whilst not directly shared by Landsat 8, spans
roughly the equivalent of both of Landsat 8 TIR bands, and so can be
approximately simulated by averaging them together. Whilst the spectral
profiles and resolution of Landsat 7 are familiar to our model, the sensor
design and image quality differ between Landsat 7 and the other satellites. An
8-bit digitisation depth was used for Landsat 7, unlike the larger bit depths
on more recent missions, which meant that some of the higher reflectances are
saturated, in order to allow for more radiometric precision at lower
reflectances. This leads some high reflectance surfaces such as snow, ice and
clouds, to have a textureless appearance, as many pixels are at the maximum of
the range.

We use the large Landsat 7 CCA dataset (described further in Section III-D4)
to test SEnSeI’s performance. As before, we use versions of SEnSeI trained
either on Sentinel-2, or Sentinel-2 and Landsat 8, and compare the test set
results with different band combinations as input. We compare our results to
the those published in 2012 [26], which present accuracy scores for three
models over 103 of the Landsat 7 CCA images. As well as the numerical findings
in Table IV, we also visualise of some failures and successes, in Fig. 6.

Figure 6: Visualisation of five models’ predictions tested on 6 scenes from the Landsat 7 CCA test set. We take 2000 pixel squares from the scenes, as the full scenes are too detailed to display in this manner. Models using RGB bands are inconsistent, while those using other bands are more successful. The fourth row shows an image with very thin cloud which is picked up by some of the models, but is not annotated as cloud. TABLE IV: Results for models over the 197 scenes from the Landsat 7 CCA dataset. Models using only RGB made substantial mistakes, perhaps due to saturation in Landsat 7 images. The combination of all bands other than RGB (No-RGB) were generally the best, and the model trained on both Sentinel-2 and Landsat 8 data is generally better than the Sentinel-2-only model. Model | Bands | OA | BA | Precision | Recall | F1
---|---|---|---|---|---|---
SEnSeI+DLv3 S2 | RGB | 76.23 | 73.17 | 64.27 | 64.05 | 64.16
COMMON | 85.24 | 86.25 | 72.61 | 89.26 | 80.08
No-RGB | 85.25 | 86.63 | 72.09 | 90.73 | 80.34
SEnSeI+DLv3 S2&L8 | RGB | 74.62 | 71.72 | 61.52 | 63.06 | 62.28
ALL | 89.08 | 87.77 | 83.35 | 83.88 | 83.62
No-RGB | 89.90 | 88.90 | 84.03 | 85.92 | 84.96
C5 CCA [26] | - | 88.5 | - | - | - | -
AT-ACCA [26] | - | 76.6 | - | - | - | -
Expanded AT-ACCA [26] | - | 89.7 | - | - | - | -
TABLE V: Results of models across the CloudPeru2 dataset, from PerúSat-1. We test our two models at the original resolution (2.8 m) and 5.6 m. The masks at the coarser resolution are then bilinearly resampled to retrieve a mask at the full 2.8 m resolution. The models examined in [62] are included, but direct comparison to them is not strictly valid, given that they tested the models on 5% of the available data, using the rest for training. Model | Bands | OA | BA | Precision | Recall | F1
---|---|---|---|---|---|---
| RGB | 90.86 | 90.90 | 89.03 | 92.71 | 90.83
SEnSeI-DLv3 S2 (2.8 m) | RGBNIR | 90.95 | 91.00 | 89.04 | 92.92 | 90.94
| RGB | 91.63 | 91.68 | 89.78 | 93.52 | 91.61
SEnSeI-DLv3 S2&L8 (2.8 m) | RGBNIR | 91.52 | 91.56 | 89.97 | 93.01 | 91.47
| RGB | 92.22 | 92.20 | 92.41 | 91.59 | 92.00
SEnSeI-DLv3 S2 (5.6 m) | RGBNIR | 92.51 | 92.50 | 92.72 | 91.88 | 92.30
| RGB | 92.70 | 92.67 | 93.49 | 91.42 | 92.44
SEnSeI-DLv3 S2&L8 (5.6m) | RGBNIR | 92.74 | 92.69 | 94.36 | 90.54 | 92.41
CloudNet [39] (2.8 m) | RGBNIR | 94.01 | 93.91 | 97.82 | 89.78 | 93.63
[67]’s method (2.8 m) | RGBNIR | 91.34 | 91.97 | 86.52 | 92.36 | 89.34
[62]’s method (2.8 m) | RGBNIR | 97.50 | 97.52 | 96.45 | 98.46 | 97.44

Overall, the models are reasonably successful on Landsat 7, achieving
accuracies close to 90%, with the best version just out-performing the highest
reported accuracy from [26]. However, there is a noticeable increase in error
rate when compared to our results on Sentinel-2 and Landsat 8. In particular,
the model appeared to struggle in regions of the image which were uniformly
high reflectance, and when it was given only RGB bands. This may have been
caused by Landsat 7’s tendency to saturate pixels with high reflectances,
leading to a lack of texture which perhaps confused the model. It may be
possible to mitigate these errors by adding artificial saturation to images in
training.

Another likely reason for reduced accuracy on Landsat 7 is the discrepancy in
annotation styles between datasets, with the Landsat 7 CCA dataset classifying
as clear some areas that we would judge to be thin cloud, which was also noted
by [26]. This means that a model trained on the Sentinel-2 and Landsat 8
datasets is biased in relation to the Landsat 7 CCA labels. We still treat
this experiment as partially successful, given the challenges, but we suggest
more work is needed on understanding why the model failed where it did, and
why in this instance Hyp. (C) is not well supported.

PerúSat-1’s resolution is an order of magnitude finer than that of Sentinel-2
and Landsat 8, at 2.8 m/pixel. We use the CloudPeru2 dataset (Section III-D5)
to see how a sensor independent model copes with a significant change in
scale. The model results are given in Table V. In order to investigate how a
change in scale affects results, we produce cloud masks at both the original
resolution (2.8 m/pixel) and at 50% downsampling, with an effective resolution
of 5.6 m/pixel.

Direct comparison with the other models included in the table is difficult, as
those considered by [62] were all trained on 90% of the dataset, and tested
only on 5%, giving their models a significant advantage in having been exposed
to the same distribution of data as the test set. Given this difference in
approach, we believe our results, especially at 5.6 m, resolution are
excellent, out-performing some of the models trained specifically on the
dataset. The drop in performance seen at 2.8 m is perhaps a sign that we are
reaching the limit of our model’s resolution range, having only been trained
on a resolution of 20 m/pixel or more.

Finally, we present results over a scene from Sentinel-3’s SLSTR. We use the 6
bands S1–S6, as these have close equivalents in Sentinel-2 and Landsat 8 from
Green into the SWIR. They are all at 500 m resolution, and can be easily
converted into reflectance values using SNAP’s rad2refl preprocessing tool. We
predicted clouds on the nadir-view stripe “A” data, and present the results in
Fig. 7. As in our experiment using PerúSat-1, we predict clouds for different
effective resolutions. This time, however, we upsample the images,
artificially raising their resolutions from 500 m, to 250 m and 167 m. This
was done because the original 500 m data contains very high-frequency clouds
not seen in the 20–30 m data the model was trained on. What is more, because
DeepLabv3’s output is interpolated over an output stride of 8, it would not be
able to segment these very small clouds successfully.

Whilst the results for SLSTR are only visual, we believe they show great
promise. At higher spatial resolutions, especially, the models seem able to
accurately segment clouds, at a resolution roughly 5-10 times coarser than
they were trained on, up to 250 m/pixel. Whilst the development and the
testing of SEnSeI focused on independence with respect to spectral band
combinations, these results from SLSTR (and those from PerúSat-1) show SEnSeI-
based models can work well across large resolution ranges, which massively
increases the number of sensors they can be applied to without any retraining.

Figure 7: Predictions of SEnSeI+DLv3 - S2 and SEnSeI+DLv3 - S2&L8 for a
Sentinel-3 scene over the UK and Ireland from August 2020. The scene has been
cropped slightly for better visualisation. Here we display it in false colour
with S3 as Red, S2 as Green, and S1 as Blue. The green overlay used for
detections does not necessarily indicate correct detection, because we do not
have a ground-truth for this scene. Masks produced at the original resolution
of 500 m fail in areas with high frequency clouds, whereas at 250 m and 167 m,
these are segmented more successfully. Some visible improvements in
performance can be observed in the SEnSeI+DLv3 - S2&L8 predictions, e.g. the
false detections over Cardigan Bay, Wales made by the model trained only on
Sentinel-2 are avoided. This figure is best viewed digitally in colour.

##  V Discussion

###  V-A Findings

In this work we have introduced the concept of sensor independence with
respect to machine learning models in remote sensing, and applied it
successfully to cloud masking, with both a neural network and a large
convolutional model. The experimental results in Section IV not only showcase
state-of-the-art cloud masking performance in our models, but also the novel
property of sensor independence. Sensor independence is an opportunity for
deep learning’s strength (assimilation of information from massive amounts of
training data) to be combined with thresholding methods’ strength
(universality over many sensors, through their use of physical measurements).

In developing SEnSeI, we have produced a highly performant cloud masking
algorithm that achieved Overall Accuracy greater than 92% on 3 test sets from
3 different satellites, Sentinel-2, Landsat 8, and PerúSat-1, and with
visually promising results on Sentinel-3’s SLSTR instrument, all without any
retraining whatsoever. We believe this proven compatibility with multiple
sensors, beyond those used in training, is a novel result in the field of deep
learning for remote sensing.

The experimental results do not unambiguously show SEnSeI to always offer an
advantage to models in terms of statistical performance. Whilst the experiment
on Landsat 8 demonstrated that sensor independence can lead to improvements in
performance, the Sentinel-2 experiment also showed that sometimes a transfer
learning approach (as we define it in Section II-B) can be more successful. In
both cases, though, the use of aggregated training data from multiple
satellites was shown to be preferable to using data from a single satellite.
Based on this, we argue that further work into sensor independence—and
interoperability more generally—is vital for continuing to improve performance
and expand usability of deep learning models in remote sensing.

Throughout Section IV we have pointed to how the results correspond to our
three hypotheses. Here, we consider each hypothesis in turn and pull together
the outcomes of our experiments in relation to them:

Hyp. A: “If SEnSeI is added to a deep learning model, there is no reduction in
performance when trained and tested on data from a single sensor”.

Whilst there is some good evidence for this hypothesis in our results, it is
somewhat nuanced. The six DeepLabv3+ models tested on Sentinel-2—four with
SEnSeI and two without—show a spread of results which fall slightly outside
the $\pm 0.5\%$ range that was measured using cross-validation on one of the
models. Clearly, the increase in performance from DLv3 - S2 to SEnSeI+DLv3 -
S2 [fixed bands] shows that adding SEnSeI does not necessarily cause a drop in
performance, with the OA actually increasing by 0.42%, though this is just
below what we deem to be a statistically significant difference of 0.5%.

However, the drop in performance for SEnSeI+DLv3 - S2 is significant, at
around 1%—and is consistent with the 1% drop seen for between NN - S2 and
SEnSeI+NN - S2. Consider, though, that this model has been trained to produce
cloud masks on any combination of 3 or more Sentinel-2 bands, which is a much
more general problem than one in which all bands are always present. So, we
believe this experiment is evidence that SEnSeI has no deleterious effects on
performance when the training task is the same, but that there is a small cost
to performance (1%) when using SEnSeI to produce a more generalised, sensor
independent model.

Hyp. B: “If SEnSeI is added to a deep learning model, the model can be trained
on multiple sensors without a drop in performance relative to when it is
trained on an individual sensor.”

To fully appreciate whether this hypothesis holds, we must consider two cases.
First, what effect adding Landsat 8 data in training has on Sentinel-2 test
performance, and second, the reciprocal case, when Sentinel-2 data is added,
what affect is seen on Landsat 8 test performance. In the prior case, the
performance of SEnSeI+DLv3 - S2&L8 actually shows a moderate increase (0.65%
in OA, 0.63% in BA, and 0.63% in $F_{1}$ score) over the equivalent model
trained on Sentinel-2 alone, SEnSeI+DLv3 - S2. This gain in performance,
though, does not quite make up for the initial decrease of 1% seen when adding
SEnSeI with random band combinations.

In the second case, on Landsat 8 test data, the results are even more
positive. The performance of SEnSeI+DLv3 - S2&L8 (COMMON) is significantly
higher than the performance of the specialised, non-sensor independent Landsat
8 model, DLv3 - L8. OA is 1.21% higher, BA is 7.66% higher, and $F_{1}$ 6.25%
higher. This seems to have been driven by the models trained only on the
Landsat 8 CCA dataset (including those published by others) having highly
imbalanced Precision and Recall values. This may be because SPARCS does not
have the same sampling distribution as the Landsat 8 CCA dataset, as it
focuses on clouds which are not extended across the whole scene, and which
have complex and diverse structures. This difference in the data distribution
between training and test sets appears to lead to models which are too ‘picky’
and miss many of the SPARCS dataset’s clouds. The introduction of Sentinel-2
data at training has produced more balanced predictions on the SPARCS dataset,
with Precision and Recall much more equal. This is promising evidence for the
power of sensor independence; training on data from more sources, with
different labellers and sensors, produces models which are more robust to
changes in data sampling, and predictions which draw on information from a
wider range of situations. Therefore, we believe the Landsat 8 SPARCS results
strongly support the hypothesis, and indeed exceeds it, by showing in this
instance that performance on one sensor, Landsat 8, can be improved by
training data from another sensor, Sentinel-2.

Hyp. C: “SEnSeI can enable a model to perform adequately on a previously
unseen sensor, given training data that provide close equivalents to the
unseen sensor’s spectral bands.”

Several times in this work, models are shown to perform well on sensors not
included in training. First, let us consider the performance of SEnSeI+DLv3 -
S2 in the experiment on the Landsat 8 SPARCS dataset. Across several band
combinations, OA remained above 90%, with the highest being for the band
combination using all possible bands (except thermal bands, which were not
available during training and so cannot be included in inference). This model
achieved the highest BA of any model (93.05%), and had very high recall but
low precision—the inverse of the models trained solely on the Landsat 8 CCA
dataset.

Landsat 7 presented difficulties for our models regarding Hyp. (C), which
shows the often unpredictable nature of deep learning models when used in
unfamiliar settings. In the case of Landsat 7, we believe that the significant
saturation over high reflectance areas (often clouds) led to a flat texture
that the model was unfamiliar with, and thus struggled to correctly classify.
As we see in Fig. 6, high reflectance, saturated areas (e.g. the thick clouds
in the 2nd row in Fig. 6) are sometimes mis-classified. Another issue is a
difference in annotation style, with some thin clouds that the model detected
not being included in the ground-truth (e.g. the thin clouds not annotated as
such in the 4th row in Fig. 6).

Meanwhile, at resolution scales outside those that the model was trained on,
SEnSeI performed surprisingly well. For the finer resolution data in the
CloudPeru2 dataset, sensor independent models achieved OA consistently above
92% at 5.6 m, with a likely drop off in performance in the range of 3–5 m per
pixel, with predictions at 2.8 m showing a roughly 1% decrease in performance
relative to 5.6 m results.

At the coarser resolution range of Sentinel-3’s SLSTR, no suitable labelled
dataset could be found, however using our model on a product over the UK shows
that visually impressive predictions can be made, although only by
artificially upsampling the scene, which is somewhat inefficient. Predictions
seem to become less reliable beyond 250 m/pixel, with the model failing to
produce good results at 500 m/pixel. We think this limitation is related to
the structure of clouds at this resolution, with the high frequency changes in
cloud cover being too fine for DeepLabv3+’s output stride of 8 pixels to
segment properly, leading to an averaging over class predictions and thus
either all-cloud or all-clear predictions over areas with both clear and
cloudy pixels.

###  V-B Dataset Design

One factor which greatly affects the practicality of sensor independence, is
the homogeneity of labelling styles, and data sampling schemes, between
datasets. In some of our experiments, differences in annotation style may have
caused drops in performance, notably for Landsat 7. This does not mean either
style of annotation was “wrong” per se, but rather that they have chosen to
define what is and isn’t included in the category of ‘cloud’ differently. In
other experiments, such as those using Landsat 8 SPARCS as a test set, we
believe the imbalanced Precision and Recall of some models compared to others
is evidence that the data sampling scheme and class distribution of training
data can have significant impacts on performance on a fixed test set. This
issue was also noted by [37].

Where possible, we believe datasets should be designed with a focus on
compatibility with prior efforts, both in terms of their labelling styles, and
in their file formats and structure. Not only will this help to accelerate
sensor independent model design by reducing the time spent on preprocessing
work for each sensor and dataset, but will also allow studies with non-sensor
independent designs to rapidly train and test on multiple datasets.
Additionally, dataset creators should provide as much information as possible
on the sampling of the data (how it was selected, what the edge cases are in
class labels, etc.), so that users can understand what a model’s performance
in a given dataset is representative of.

##  VI Conclusions & Future Work

We propose SEnSeI as a possible route to building sensor independence into
existing models for remote sensing applications. By treating each spectral
band as a member of a set of all possible bands, and using a permutation
invariant network design, SEnSeI translates any satellite’s measured
reflectances and brightness temperatures into a common representation. The
framework is presented in the context of cloud masking, and is shown to work
well with both a simple neural network, and the large convolutional model,
DeepLabv3+. It enables DeepLabv3+ to simultaneously predict clouds over five
different satellites without retraining, and using all available bands.
Furthermore, the performance in Landsat 8 was significantly higher when
trained on data from Sentinel-2 as well as Landsat 8, showing that a better
model comes when using training data sampled from multiple sensors.

The sensor independent models appeared to struggle when given data that is
very unlike that used in training (e.g. Landsat 7 with its saturation, or
resolutions very different from those in training). Nevertheless, we
demonstrated our model—not trained on any Landsat 7 data—had a comparable
accuracy to purpose-built models [26] (Table IV). Future work could improve
matters further by adding information beyond spectra to the inputted
descriptors, e.g. saturation levels, Point Spread Function, noise, or
resolution. This could allow models to better handle images with different
statistics to those used in training, and help to expand training sets to
include even more sensors.

Training a model to be sensor independent appears to cause some penalty to
performance in some circumstances (e.g. in the Sentinel-2 experiment).
However, this is not always the case, as the sensor independent model
outperformed others on Landsat 8. The additional usability and versatility of
the sensor independent model, being shown to work on many other satellites for
which it was not directly trained, is the key advantage of sensor
independence. Robustness to unfamiliar sensors is not only essential when
training data does not exist for those satellites, but can also be incredibly
helpful if deep learning is to be used operationally on-board future missions,
where real training data cannot be created prior to launch, or in ground
segment processing, where algorithms are usually developed before downlinking
of data begins.

In this work, we have focused on the problem of cloud masking. Many of the
datasets we used also contain cloud shadow annotations, presenting an avenue
for future research. For many users of cloud masks, shadows are as important
to detect as the clouds themselves. As well as cloud masking, cloud removal is
also a topic of significant interest, e.g. [68, 69]. With data fusion
techniques, cloudy areas can be in-filled with synthetic multispectral imagery
generated from other data sources which are unaffected by cloud cover. In the
future, our work could be extended to this challenge.

More widely, we believe we have shown that sensor independence is a worthwhile
strategy for remote sensing tasks, not limited to cloud masking. We expect the
greatest benefit to come for sensors with little annotation data, which can
now benefit from the extensive annotations on other sensors. We plan to apply
SEnSeI to a greater variety of remote sensing computer vision tasks in the
future, including, for example, object detection and classification. SEnSeI’s
architecture is not specific to the problem of cloud masking, and therefore it
should be possible to use it without modification for other tasks and with
other models.

Finally, we wish to emphasise that the main thrust of this paper is not the
raw performance of our models. Whilst comparisons between competing methods
are an important part of machine learning research, we believe our findings
invite collaboration, not competition. SEnSeI is an accessory which can be
added to existing models, making them sensor independent (although future work
should seek to ascertain why it did not work well with CloudFCN,
specifically). For many computer vision tasks in remote sensing, we see
limited adoption of published techniques in real-world settings, even when
performance is high [70]. This is often because the models are highly
specialised to a specific sensor or dataset, and cannot be deployed beyond it.
We suggest that pursuing sensor independence—with SEnSeI or something like
it—can give models real-world utility, which may ultimately prove to be more
important than further increasing their performance.

##  Appendix A Computational Efficiency

SEnSeI is a lightweight model with a fairly small footprint in terms of both
memory and computation time. Table VI gives a breakdown of the tuneable
parameters in SEnSeI’s different modules. An experiment was also conducted to
measure the computation time for the models used in this paper, and to measure
how the addition of SEnSeI affected this computation. Each model was given a
257-by-257 pixel scene 1,600 times. For variants with SEnSeI, different
numbers of bands were given, but for models without SEnSeI, 13 bands was used,
as this is the number of bands which Sentinel-2 data contains. The number of
parameters for each model, and the time taken per megapixel, is given in Table
VII. SEnSeI adds a small amount of computation time to the models, although
when only using a few bands, SEnSeI has no noticeable affect.

TABLE VI: Number of parameters per module in SEnSeI. The names of the modules correspond to those in Fig. 2. The band multiplication block uses a fixed offset of 0.5, and so has no trainable parameters. Module | No. Parameters
---|---
DESCRIPTOR BLOCK | 24’208
PERMUTATION BLOCK | 62’848
COMBINED DESCRIPTOR BLOCK | 123’488
BAND MULTIPLICATION BLOCK | 0
TOTAL | 210’544
TABLE VII: Measured computational efficiency across both the 2-layer neural network and DeepLabv3+. SEnSeI adds roughly 30 ms/Megapixel to either model, when 13 spectral bands are inputted. Model | No. Parameters | Speed (ms/Megapixel)
---|---|---
NN | 1154 | 36.8
SEnSeI+NN (3 bands) | 2.27e5 | 33.2
SEnSeI+NN (8 bands) | 2.27e5 | 51.6
SEnSeI+NN (13 bands) | 2.27e5 | 71.4
DLv3 | 2.11e6 | 94.6
SEnSeI+DLv3 (3 bands) | 2.34e6 | 88.4
SEnSeI+DLv3 (8 bands) | 2.34e6 | 103
SEnSeI+DLv3 (13 bands) | 2.34e6 | 121

##  Appendix B Detailed Results on Sentinel-2

As well as measuring the metrics over the total Sentinel-2 test sets, we also
subdivided the test set based on the categorical tags assigned to each image
(see Table VIII). Relating the various models’ performance with the content of
the images gives a more nuanced view of how they can be expected to behave in
the real world.

TABLE VIII: DeepLabv3+ model results over the Sentinel-2 test set (257 scenes). We also sub-divide the total test by the presence of several categories of surface type and cloud properties, described in Section III-D1. Categories with fewer than 30 subscenes are omitted. The best result for each metric and image category is boldened. | OA
---
Total test set |  forest/jungle |  snow/ice |  agricultural |  desert/barren |  shrublands |  open_water |  low |  high |  thin |  thick |  isolated |  extended |  cumulus |  stratocumulus |  cirrus |  ice_clouds
DLv3 - S2 | 94.04 | 92.52 | 86.05 | 92.22 | 94.88 | 95.31 | 94.14 | 94.92 | 92.97 | 91.10 | 95.26 | 89.54 | 93.92 | 95.00 | 96.42 | 91.93 | 95.30
DLv3 - S2&L8 [common bands] | 95.19 | 93.28 | 89.71 | 95.07 | 95.85 | 95.92 | 93.83 | 94.82 | 94.52 | 92.28 | 95.93 | 90.48 | 94.83 | 94.93 | 96.63 | 93.84 | 96.85
SEnSeI+DLv3 - S2 [fixed bands] | 94.46 | 91.60 | 88.36 | 92.61 | 95.94 | 94.56 | 94.01 | 94.15 | 93.43 | 91.11 | 95.27 | 88.95 | 94.64 | 94.20 | 95.93 | 92.86 | 96.58
SEnSeI+DLv3 - S2 | 93.08 | 90.92 | 87.14 | 91.35 | 93.16 | 93.81 | 93.61 | 93.69 | 92.07 | 90.43 | 94.31 | 89.15 | 92.99 | 93.86 | 95.78 | 91.33 | 93.66
SEnSeI+DLv3 - S2&L8 | 93.73 | 90.70 | 84.87 | 90.20 | 93.77 | 93.78 | 94.31 | 94.66 | 92.49 | 89.97 | 95.68 | 88.90 | 94.23 | 94.74 | 96.86 | 91.79 | 97.45
| BA
---
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
DLv3 - S2 | 94.03 | 91.35 | 85.94 | 91.60 | 96.43 | 94.31 | 94.12 | 94.56 | 91.05 | 91.16 | 94.05 | 88.58 | 89.99 | 94.57 | 94.08 | 89.42 | 74.35
DLv3 - S2&L8 [common bands] | 95.18 | 91.94 | 89.41 | 93.61 | 97.21 | 94.75 | 93.84 | 94.09 | 93.33 | 92.37 | 93.82 | 89.60 | 89.27 | 94.12 | 93.67 | 92.46 | 73.15
SEnSeI+DLv3 - S2 [fixed bands] | 94.42 | 90.44 | 88.49 | 91.04 | 96.38 | 92.69 | 94.01 | 93.54 | 90.81 | 91.03 | 93.16 | 88.25 | 90.21 | 93.48 | 92.35 | 89.72 | 78.89
SEnSeI+DLv3 - S2 | 93.10 | 89.10 | 87.20 | 89.21 | 93.07 | 90.57 | 93.60 | 93.37 | 91.00 | 90.67 | 93.10 | 87.89 | 89.93 | 93.46 | 93.05 | 89.99 | 80.03
SEnSeI+DLv3 - S2&L8 | 93.73 | 88.43 | 85.04 | 87.75 | 92.31 | 89.87 | 94.27 | 94.57 | 91.30 | 90.26 | 94.64 | 87.23 | 91.27 | 94.51 | 94.46 | 90.06 | 82.88
| Precision
---
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
DLv3 - S2 | 94.57 | 91.64 | 83.03 | 85.03 | 62.24 | 90.33 | 94.67 | 96.79 | 96.71 | 93.81 | 97.69 | 85.09 | 98.80 | 96.96 | 98.45 | 96.48 | 98.63
DLv3 - S2&L8 [common bands] | 95.61 | 93.76 | 88.83 | 93.44 | 67.07 | 92.39 | 93.25 | 96.09 | 97.66 | 94.84 | 97.22 | 86.43 | 98.61 | 96.32 | 98.25 | 97.66 | 98.54
SEnSeI+DLv3 - S2 [fixed bands] | 94.58 | 90.00 | 84.55 | 88.23 | 68.00 | 90.61 | 93.78 | 95.95 | 96.37 | 93.20 | 97.02 | 83.33 | 98.79 | 96.10 | 97.88 | 96.36 | 98.87
SEnSeI+DLv3 - S2 | 94.00 | 91.36 | 83.47 | 86.98 | 55.35 | 92.28 | 94.14 | 96.24 | 97.03 | 94.09 | 97.40 | 85.37 | 98.87 | 96.45 | 98.19 | 97.01 | 98.99
SEnSeI+DLv3 - S2&L8 | 94.42 | 92.89 | 80.26 | 85.27 | 58.12 | 94.23 | 95.97 | 97.15 | 97.07 | 93.95 | 97.94 | 86.25 | 99.00 | 97.16 | 98.51 | 96.90 | 99.08
| Recall
---
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
DLv3 - S2 | 94.17 | 87.18 | 85.18 | 90.03 | 98.30 | 92.18 | 93.30 | 95.60 | 94.34 | 90.81 | 96.20 | 85.31 | 94.59 | 95.72 | 97.35 | 93.45 | 96.51
DLv3 - S2&L8 [common bands] | 95.31 | 87.15 | 87.19 | 89.95 | 98.83 | 92.24 | 94.24 | 96.20 | 95.38 | 91.82 | 97.57 | 86.62 | 95.77 | 96.30 | 97.81 | 94.67 | 98.22
SEnSeI+DLv3 - S2 [fixed bands] | 94.99 | 86.27 | 89.46 | 87.08 | 96.91 | 88.70 | 94.02 | 95.32 | 95.29 | 91.51 | 96.92 | 85.86 | 95.40 | 95.43 | 97.36 | 94.77 | 97.61
SEnSeI+DLv3 - S2 | 92.89 | 82.60 | 87.65 | 83.84 | 92.95 | 83.67 | 92.75 | 94.30 | 92.83 | 89.28 | 95.26 | 83.60 | 93.50 | 94.54 | 96.87 | 92.14 | 94.45
SEnSeI+DLv3 - S2&L8 | 93.73 | 80.32 | 86.27 | 81.58 | 90.56 | 81.54 | 92.28 | 94.84 | 93.34 | 88.59 | 96.49 | 81.56 | 94.74 | 95.13 | 97.82 | 92.84 | 98.29
| $\mathbf{F_{1}}$
---
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
DLv3 - S2 | 94.37 | 89.35 | 84.09 | 87.46 | 76.22 | 91.25 | 93.98 | 96.20 | 95.51 | 92.28 | 96.94 | 85.20 | 96.65 | 96.33 | 97.90 | 94.94 | 97.56
DLv3 - S2&L8 [common bands] | 95.46 | 90.33 | 88.00 | 91.66 | 79.91 | 92.32 | 93.74 | 96.15 | 96.50 | 93.30 | 97.40 | 86.53 | 97.17 | 96.31 | 98.03 | 96.14 | 98.38
SEnSeI+DLv3 - S2 [fixed bands] | 94.79 | 88.10 | 86.94 | 87.65 | 79.92 | 89.64 | 93.90 | 95.64 | 95.83 | 92.35 | 96.97 | 84.58 | 97.06 | 95.76 | 97.62 | 95.56 | 98.23
SEnSeI+DLv3 - S2 | 93.44 | 86.76 | 85.51 | 85.38 | 69.38 | 87.76 | 93.44 | 95.26 | 94.88 | 91.62 | 96.32 | 84.47 | 96.11 | 95.49 | 97.52 | 94.52 | 96.66
SEnSeI+DLv3 - S2&L8 | 94.07 | 86.15 | 83.16 | 83.38 | 70.80 | 87.43 | 94.09 | 95.98 | 95.17 | 91.19 | 97.21 | 83.84 | 96.82 | 96.13 | 98.16 | 94.83 | 98.68
No. of scenes | 257 | 42 | 41 | 41 | 34 | 40 | 66 | 129 | 126 | 110 | 156 | 90 | 137 | 140 | 92 | 104 | 51
Cloud % | 53.05 | 36.02 | 43.29 | 30.14 | 8.34 | 26.53 | 49.06 | 67.21 | 79.22 | 58.60 | 78.09 | 35.30 | 92.74 | 68.64 | 85.74 | 81.09 | 97.27

Of the metrics used, OA and Recall are relatively unaffected by the ratio of
pixels which are cloudy, whereas BA, Precision and $F_{1}$ are all highly
dependent on the class distribution. Therefore, when inspecting these results,
it is also important to consider the percentage of cloudy pixels that are
present in a given category of images, and only compare results across
categories in metrics where it is reasonable to do so.

## Acknowledgments

The authors would like to thank the many researchers whose datasets and models
were used in this work, including those at the USGS for the Landsat datasets,
and Giorgio Morales and CONIDA for the CloudPeru2 dataset. We would also like
to thank Pierre-Philippe Mathieu and colleagues at the $\Phi$-lab, ESRIN for
giving continued support throughout this project. This project was funded
through STFC Grant 1912521, and ESA Contract 4000130413/20/I-DT.

## References

  * [1] X. X. Zhu, D. Tuia, L. Mou, G.-S. Xia, L. Zhang, F. Xu, and F. Fraundorfer, “Deep learning in remote sensing: A comprehensive review and list of resources,” _IEEE Geoscience and Remote Sensing Magazine_ , vol. 5, no. 4, pp. 8–36, 2017.
  * [2] L. Ma, Y. Liu, X. Zhang, Y. Ye, G. Yin, and B. A. Johnson, “Deep learning in remote sensing applications: A meta-analysis and review,” _ISPRS Journal of Photogrammetry and Remote Sensing_ , vol. 152, pp. 166–177, 2019.
  * [3] A. Orynbaikyzy, U. Gessner, and C. Conrad, “Crop type classification using a combination of optical and radar remote sensing data: a review,” _International Journal of Remote Sensing_ , vol. 40, no. 17, pp. 6553–6595, 2019.
  * [4] N. Y. Zakhvatkina, V. Y. Alexandrov, O. M. Johannessen, S. Sandven, and I. Y. Frolov, “Classification of sea ice types in envisat synthetic aperture radar images,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 51, no. 5, pp. 2587–2600, 2012.
  * [5] M. A. Tanase, M. A. Belenguer-Plomer, E. Roteta, A. Bastarrika, J. Wheeler, Á. Fernández-Carrillo, K. Tansey, W. Wiedemann, P. Navratil, S. Lohberger _et al._ , “Burned area detection and mapping: Intercomparison of sentinel-1 and sentinel-2 based algorithms over tropical africa,” _Remote Sensing_ , vol. 12, no. 2, p. 334, 2020.
  * [6] M. Wurm, H. Taubenbock, A. Roth, and S. Dech, “Urban structuring using multisensoral remote sensing data: By the example of the german cities cologne and dresden,” in _2009 Joint Urban Remote Sensing Event_. IEEE, 2009, pp. 1–8.
  * [7] U. Kanjir, H. Greidanus, and K. Oštir, “Vessel detection and classification from spaceborne optical images: A literature survey,” _Remote Sensing of Environment_ , vol. 207, pp. 1–26, 2018.
  * [8] Y. Roh, G. Heo, and S. E. Whang, “A survey on data collection for machine learning: a big data-ai integration perspective,” _IEEE Transactions on Knowledge and Data Engineering_ , 2019.
  * [9] L. Gómez-Chova, D. Tuia, G. Moser, and G. Camps-Valls, “Multimodal classification of remote sensing images: A review and future directions,” _Proceedings of the IEEE_ , vol. 103, no. 9, pp. 1560–1584, 2015.
  * [10] D. Hong, L. Gao, N. Yokoya, J. Yao, J. Chanussot, Q. Du, and B. Zhang, “More diverse means better: Multimodal deep learning meets remote-sensing imagery classification,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 59, no. 5, pp. 4340–4354, 2020.
  * [11] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoder-decoder with atrous separable convolution for semantic image segmentation,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , September 2018.
  * [12] A. Francis and J. Mrziglod, “Sensei,” 2021. [Online]. Available: https://github.com/aliFrancis/SEnSeI
  * [13] D. Helder, B. Markham, R. Morfitt, J. Storey, J. Barsi, F. Gascon, S. Clerc, B. LaFrance, J. Masek, D. P. Roy _et al._ , “Observations and recommendations for the calibration of landsat 8 oli and sentinel 2 msi for improved data interoperability,” _Remote Sensing_ , vol. 10, no. 9, p. 1340, 2018.
  * [14] J. A. Barsi, B. Alhammoud, J. Czapla-Myers, F. Gascon, M. O. Haque, M. Kaewmanee, L. Leigh, and B. L. Markham, “Sentinel-2a msi and landsat-8 oli radiometric cross comparison over desert sites,” _European Journal of Remote Sensing_ , vol. 51, no. 1, pp. 822–837, 2018.
  * [15] M. Claverie, J. Ju, J. G. Masek, J. L. Dungan, E. F. Vermote, J.-C. Roger, S. V. Skakun, and C. Justice, “The harmonized landsat and sentinel-2 surface reflectance data set,” _Remote Sensing of Environment_ , vol. 219, pp. 145–161, 2018.
  * [16] S. Falanga Bolognesi, E. Pasolli, O. R. Belfiore, C. De Michele, and G. D’Urso, “Harmonized landsat 8 and sentinel-2 time series data to detect irrigated areas: An application in southern italy,” _Remote Sensing_ , vol. 12, no. 8, p. 1275, 2020.
  * [17] D. K. Bolton, J. M. Gray, E. K. Melaas, M. Moon, L. Eklundh, and M. A. Friedl, “Continental-scale land surface phenology from harmonized landsat 8 and sentinel-2 imagery,” _Remote Sensing of Environment_ , vol. 240, p. 111685, 2020.
  * [18] S. Skakun, B. Franch, E. Vermote, J.-C. Roger, C. Justice, J. Masek, and E. Murphy, “Winter wheat yield assessment using landsat 8 and sentinel-2 data,” in _IGARSS 2018-2018 IEEE International Geoscience and Remote Sensing Symposium_. IEEE, 2018, pp. 5964–5967.
  * [19] A. Banskota, N. Kayastha, M. J. Falkowski, M. A. Wulder, R. E. Froese, and J. C. White, “Forest monitoring using landsat time series data: A review,” _Canadian Journal of Remote Sensing_ , vol. 40, no. 5, pp. 362–384, 2014\.
  * [20] J. Rouse, R. H. Haas, J. A. Schell, D. W. Deering _et al._ , “Monitoring vegetation systems in the great plains with erts,” _NASA special publication_ , vol. 351, no. 1974, p. 309, 1974.
  * [21] S. Mahajan and B. Fataniya, “Cloud detection methodologies: Variants and development—a review,” _Complex & Intelligent Systems_, pp. 1–11, 2019\.
  * [22] N. Champion, “Automatic cloud detection from multi-temporal satellite images: towards the use of pléiades time series,” _International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences_ , vol. 39, p. B3, 2012.
  * [23] Q. Zhang, Q. Yuan, J. Li, Z. Li, H. Shen, and L. Zhang, “Thick cloud and cloud shadow removal in multitemporal imagery using progressively spatio-temporal patch group deep learning,” _ISPRS Journal of Photogrammetry and Remote Sensing_ , vol. 162, pp. 148–160, 2020.
  * [24] Q. Zhang, Q. Yuan, Z. Li, F. Sun, and L. Zhang, “Combined deep prior with low-rank tensor svd for thick cloud removal in multitemporal images,” _ISPRS Journal of Photogrammetry and Remote Sensing_ , vol. 177, pp. 161–173, 2021.
  * [25] R. R. Irish, J. L. Barker, S. N. Goward, and T. Arvidson, “Characterization of the landsat-7 etm+ automated cloud-cover assessment (acca) algorithm,” _Photogrammetric Engineering & Remote Sensing_, vol. 72, no. 10, pp. 1179–1188, 2006.
  * [26] P. L. Scaramuzza, M. A. Bouchard, and J. L. Dwyer, “Development of the landsat data continuity mission cloud-cover assessment algorithms,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 50, no. 4, pp. 1140–1154, 2011.
  * [27] S. Bley and H. Deneke, “A threshold-based cloud mask for the high-resolution visible channel of meteosat second generation seviri,” _Atmospheric Measurement Techniques_ , vol. 6, no. 10, pp. 2713–2723, 2013.
  * [28] J. Louis, V. Debaecker, B. Pflug, M. Main-Knorn, J. Bieniarz, U. Mueller-Wilm, E. Cadau, and F. Gascon, “Sentinel-2 sen2cor: L2a processor for users,” in _Proceedings Living Planet Symposium 2016_. Spacebooks Online, 2016, pp. 1–8.
  * [29] Z. Zhu and C. E. Woodcock, “Object-based cloud and cloud shadow detection in landsat imagery,” _Remote Sensing of Environment_ , vol. 118, pp. 83–94, 2012.
  * [30] Z. Zhu, S. Wang, and C. E. Woodcock, “Improvement and expansion of the fmask algorithm: Cloud, cloud shadow, and snow detection for landsats 4–7, 8, and sentinel 2 images,” _Remote Sensing of Environment_ , vol. 159, pp. 269–277, 2015.
  * [31] S. Foga, P. L. Scaramuzza, S. Guo, Z. Zhu, R. D. Dilley, T. Beckmann, G. L. Schmidt, J. L. Dwyer, M. Joseph Hughes, and B. Laue, “Cloud detection algorithm comparison and validation for operational Landsat data products,” _Remote Sensing of Environment_ , vol. 194, pp. 379–390, 2017. [Online]. Available: http://dx.doi.org/10.1016/j.rse.2017.03.026
  * [32] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” _Advances in Neural Information Processing Systems_ , vol. 25, pp. 1097–1105, 2012.
  * [33] M. Z. Alom, T. M. Taha, C. Yakopcic, S. Westberg, P. Sidike, M. S. Nasrin, B. C. Van Esesn, A. A. S. Awwal, and V. K. Asari, “The history began from alexnet: A comprehensive survey on deep learning approaches,” _arXiv preprint arXiv:1803.01164_ , 2018.
  * [34] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2015, pp. 234–241.
  * [35] A. Francis, P. Sidiropoulos, and J.-P. Muller, “Cloudfcn: Accurate and robust cloud detection for satellite imagery with deep learning,” _Remote Sensing_ , vol. 11, no. 19, p. 2312, 2019.
  * [36] M. Wieland, Y. Li, and S. Martinis, “Multi-sensor cloud and cloud shadow segmentation with a convolutional neural network,” _Remote Sensing of Environment_ , vol. 230, p. 111203, 2019.
  * [37] J. H. Jeppesen, R. H. Jacobsen, F. Inceoglu, and T. S. Toftegaard, “A cloud detection algorithm for satellite imagery based on deep learning,” _Remote Sensing of Environment_ , vol. 229, pp. 247–259, 2019.
  * [38] J. Zhang, X. Li, L. Li, P. Sun, X. Su, T. Hu, and F. Chen, “Lightweight u-net for cloud detection of visible and thermal infrared remote sensing images,” _Optical and Quantum Electronics_ , vol. 52, no. 9, pp. 1–14, 2020.
  * [39] S. Mohajerani and P. Saeedi, “Cloud-net: An end-to-end cloud detection algorithm for landsat 8 imagery,” in _IGARSS 2019-2019 IEEE International Geoscience and Remote Sensing Symposium_. IEEE, 2019, pp. 1029–1032.
  * [40] ——, “Cloud-net+: A cloud segmentation cnn for landsat 8 remote sensing imagery optimized with filtered jaccard loss function,” _arXiv preprint arXiv:2001.08768_ , 2020.
  * [41] M. Khoshboresh-Masouleh and R. Shah-Hosseini, “A deep learning method for near-real-time cloud and cloud shadow segmentation from gaofen-1 images,” _Computational Intelligence and Neuroscience_ , vol. 2020, 2020.
  * [42] M. Molinier, N. Reunanen, A. Lämsä, H. Astola, and T. Räty, “Deepcloud-a fully convolutionnal neural network for cloud and shadow masking in optical satellite images,” in _IGARSS 2018-2018 IEEE International Geoscience and Remote Sensing Symposium_. IEEE, 2018, pp. 2107–2110.
  * [43] Z. Li, H. Shen, Y. Wei, Q. Cheng, and Q. Yuan, “Cloud detection by fusing multi-scale convolutional features,” _Proceedings of the ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Science, Beijing, China_ , pp. 7–10, 2018.
  * [44] Y. Shendryk, Y. Rist, C. Ticehurst, and P. Thorburn, “Deep learning for multi-modal classification of cloud, shadow and land cover scenes in planetscope and sentinel-2 imagery,” _ISPRS Journal of Photogrammetry and Remote Sensing_ , vol. 157, pp. 124–136, 2019.
  * [45] Z. Li, H. Shen, Q. Cheng, Y. Liu, S. You, and Z. He, “Deep learning based cloud detection for medium and high resolution remote sensing images of different sensors,” _ISPRS Journal of Photogrammetry and Remote Sensing_ , vol. 150, pp. 197–212, 2019.
  * [46] C. R. Qi, H. Su, K. Mo, and L. J. Guibas, “Point-net: Deep learning on point sets for 3d classification and segmentation. corr abs/1612.00593 (2016),” _arXiv preprint arXiv:1612.00593_ , 2016.
  * [47] C. R. Qi, L. Yi, H. Su, and L. J. Guibas, “Pointnet++: Deep hierarchical feature learning on point sets in a metric space,” _arXiv preprint arXiv:1706.02413_ , 2017.
  * [48] N. Guttenberg, N. Virgo, O. Witkowski, H. Aoki, and R. Kanai, “Permutation-equivariant neural networks applied to dynamics prediction,” _arXiv preprint arXiv:1612.04530_ , 2016.
  * [49] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 40, no. 4, pp. 834–848, 2017.
  * [50] L.-C. Chen, G. Papandreou, F. Schroff, and H. Adam, “Rethinking atrous convolution for semantic image segmentation,” _arXiv preprint arXiv:1706.05587_ , 2017.
  * [51] S. Lazebnik, C. Schmid, and J. Ponce, “Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories,” in _2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06)_ , vol. 2. IEEE, 2006, pp. 2169–2178.
  * [52] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “Mobilenetv2: Inverted residuals and linear bottlenecks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 4510–4520.
  * [53] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin _et al._ , “Tensorflow: Large-scale machine learning on heterogeneous distributed systems,” _arXiv preprint arXiv:1603.04467_ , 2016.
  * [54] J. Golbstein, “Keras deeplab v3.1+,” 2019. [Online]. Available: https://github.com/Golbstein/Keras-segmentation-deeplab-v3.1/commit/e3f0daaa79a729c022da658fc86eef82a6c7ceeb
  * [55] D. Hoiem, S. K. Divvala, and J. H. Hays, “Pascal voc 2008 challenge,” _World Literature Today_ , 2009.
  * [56] A. Francis, J. Mrziglod, P. Sidiropoulos, and J.-P. Muller, “Sentinel-2 cloud mask catalogue,” _Zenodo_ , Nov. 2020. [Online]. Available: https://doi.org/10.5281/zenodo.4172871
  * [57] J. M. Hughes, “L8 sparcs cloud validation masks,” 2016. [Online]. Available: https://www.usgs.gov/core-science-systems/nli/landsat/spatial-procedures-automated-removal-cloud-and-shadow-sparcs
  * [58] “L8 biome cloud validation masks,” U.S. Geological Survey, 2016. [Online]. Available: https://landsat.usgs.gov/landsat-8-cloud-cover-assessment-validation-data
  * [59] “L7 irish cloud validation masks,” U.S. Geological Survey, 2016. [Online]. Available: https://landsat.usgs.gov/landsat-7-cloud-cover-assessment-validation-data
  * [60] G. Morales, A. Ramírez, and J. Telles, “Cloudperu2 dataset,” 2019. [Online]. Available: http://didt.inictel-uni.edu.pe/dataset/CloudPeru2.zip
  * [61] J. Mrziglod and A. Francis, “Iris,” 2020. [Online]. Available: https://github.com/ESA-PhiLab/iris
  * [62] G. Morales, A. Ramírez, and J. Telles, “End-to-end cloud segmentation in high-resolution multispectral satellite imagery using deep learning,” in _2019 IEEE XXVI International Conference on Electronics, Electrical Engineering and Computing (INTERCON)_. IEEE, 2019, pp. 1–4.
  * [63] D. A. Van Dyk and X.-L. Meng, “The art of data augmentation,” _Journal of Computational and Graphical Statistics_ , vol. 10, no. 1, pp. 1–50, 2001.
  * [64] A. Zupanc, “Improving cloud detection with machine learning,” _Availabe online: https://medium. com/sentinel-hub/improving-cloud-detection-with-machine-learning-c09dc5d7cf13 (accessed on 13 May 2020)_ , 2017.
  * [65] R. Coluzzi, V. Imbrenda, M. Lanfredi, and T. Simoniello, “A first assessment of the sentinel-2 level 1-c cloud mask product to support informed surface analyses,” _Remote Sensing of Environment_ , vol. 217, pp. 426–443, 2018\.
  * [66] T. Chen and C. Guestrin, “Xgboost: A scalable tree boosting system,” in _Proceedings of the 22nd SCM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , 2016, pp. 785–794.
  * [67] G. Morales, S. G. Huamán, and J. Telles, “Cloud detection for perusat-1 imagery using spectral and texture descriptors, ann, and panchromatic fusion,” in _Brazilian Technology Symposium_. Springer, 2017, pp. 1–7.
  * [68] A. Meraner, P. Ebel, X. X. Zhu, and M. Schmitt, “Cloud removal in sentinel-2 imagery using a deep residual neural network and sar-optical data fusion,” _ISPRS Journal of Photogrammetry and Remote Sensing_ , vol. 166, pp. 333–346, 2020.
  * [69] V. Sarukkai, A. Jain, B. Uzkent, and S. Ermon, “Cloud removal from satellite images using spatiotemporal generator networks,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2020, pp. 1796–1805.
  * [70] D. Thompson and P. Brodrick, “Realizing machine learning’s promise in geoscience remote sensing,” _Eos_ , vol. 102, Jul. 2021. [Online]. Available: https://doi.org/10.1029/2021eo160605

|  Alistair Francis was born in Maryport, UK, in 1994. He received a B.A. Hons
in Physics from the University of Oxford in 2016. He was then awarded an M.Sc.
in Space Science & Engineering from University College London in 2017, and
completed his Ph.D. at Mullard Space Science Laboratory, University College
London, in 2021. He has previously worked on projects with Createc Ltd.,
Cockermouth, UK, from 2015–2016, and Cortexica Ltd., London, UK, in 2018.
Since October 2021, he holds a postdoctoral research fellowship at ESRIN’s
$\Phi$-lab, Frascati, Italy. His research interests include machine learning
and computer vision applications in remote sensing data of Earth and Mars,
semi-automated data labelling, and problems involving data fusion across
multispectral, hyperspectral, and SAR instruments.
---|---
|  John Mrziglod received the B.Sc. and M.Sc. degrees in Meteorology from the
Universität Hamburg, Germany, in 2014 and 2018, respectively. He spent one and
a half years at the $\Phi$-Lab of the European Space Agency, Rome, Italy, as
Young Graduate Trainee for Digital Earth Observation Technologies, after which
he was an Earth Observation Data Analyst at the UN World Food Programme, Rome,
Italy. He is currently a Video Game Developer in Hamburg, Germany. His
research interests include the application of deep learning to remote sensing
and climate data.
---|---
|  Panagiotis Sidiropoulos received his Ph.D. degree in artificial
intelligence from the Centre for Vision, Speech, and Signal Processing,
University of Surrey, Surrey, UK, in 2012. He is currently an honorary
lecturer in the Mullard Space Science Laboratory, UCL, while holding a Head of
Data and Knowledge Science position in industry. His work is in the
intersection of academic excellence and commercial applicability, having co-
authored more than 100 papers while receiving $>$£5 million in commercial
funding. His research interests include the use of machine learning, computer
vision and data science techniques in remote sensing applications.
---|---
|  Jan-Peter Muller received a B.Sc. degree (Hons.) in Physics from the
University of Sheffield, U.K., in 1976; the M.Sc. and D.I.C. degrees in
atmospheric physics and dynamics from Imperial College London, U.K., in 1977,
and a Ph.D. degree in planetary meteorology from the University of London, UK,
in 1982. He is currently Professor of image understanding and remote sensing
with University College London (UCL), where he has been on the faculty since
1984. He is head of the Imaging Group at the Mullard Space Science Laboratory,
which is part of the Department ofSpace and Climate Physics at UCL. He was a
member of the MODIS science team from 1989 until 2016 and is an active member
of the MISR science Team since 1990. He is a member of the ESA Harmony mission
MAG responsible for developing automated methods for the retrieval of 3D
cloud-top heights, winds and cloud presence. His research interests include
the practical applications of machine vision and deep learning to remote
sensing of the earth and planets, with a special focus on 3-D imaging and
super-resolution restoration technologies for applications in climate
modelling, weather forecasting, and planetary exploration.
---|---