# Attention-augmented Spatio-Temporal Segmentation for Land Cover Mapping

Rahul Ghosh1 These authors contributed equally to this article Praveen
Ravirathinam∗1 Xiaowei Jia2 Chenxi Lin1 Zhenong Jin1 &Vipin Kumar1 1University
of Minnesota
2University of Pittsburgh {ghosh128, pravirat, lin00370, jinzn,
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

The availability of massive earth observing satellite data provide huge
opportunities for land use and land cover mapping. However, such mapping
effort is challenging due to the existence of various land cover classes,
noisy data, and the lack of proper labels. Also, each land cover class
typically has its own unique temporal pattern and can be identified only
during certain periods. In this article, we introduce a novel architecture
that incorporates the UNet structure with Bidirectional LSTM and Attention
mechanism to jointly exploit the spatial and temporal nature of satellite data
and to better identify the unique temporal patterns of each land cover. We
evaluate this method for mapping crops in multiple regions over the world. We
compare our method with other state-of-the-art methods both quantitatively and
qualitatively on two real-world datasets which involve multiple land cover
classes. We also visualise the attention weights to study its effectiveness in
mitigating noise and identifying discriminative time period.

##  1 Introduction

Growth in the world’s population and the acceleration of urbanization are
straining already scarce natural resources and food supplies, which must scale
up to keep pace with growing demand. The consequences of the resulting large-
scale changes include tremendous stresses on the environment, as well as
challenges to our ability to feed the world’s population, that could be
calamitous at the current rate of change if they are not managed sustainably.
Timely information on land use and land cover changes can provide critical
information at desired spatial and temporal scales to assist in decision
making for development investment and sustainable resource management. In
particular, mapping crops is a key step towards many applications, such as
forecasting yield, guiding sustainable management practices and evaluating
progress in conservation efforts.

Figure 1: Yearly Normalized Difference Vegetation Index (NDVI) time series for
some land cover classes.

Recent advances in storing and processing remote sensing data collected by
sensors on-board aircrafts or satellites provide tremendous potential for
mapping a variety of land covers. Deep learning on remote sensing has shown
promising results for mapping specific land covers, e.g., plantations Jia and
others (2017), agricultural facilities Handan-Nader and others (2019), roads
Xie et al. (2015), and buildings Nayak and others (2020), by extracting their
distinct spatial and temporal patterns. In particular, Convolutional Neural
Networks (CNN)-based models have been widely used in automated land cover
mapping given their capacity to capture the spatial correlation, i.e., land
covers are often contiguous over space Maggiori and others (2016); Postadjian
and others (2017). Extracting the spatial information also assists in
mitigating the noise of remote sensing data at individual locations (e.g.,
aerosols). While CNNs have mostly been used for studying land covers from a
single image, Recurrent Neural Networks (RNN) and Long-Short Term Memory
(LSTM) can take advantage of temporal patterns Jia and others (2014); Rußwurm
and Korner (2017) obtained from multi-temporal remote sensing data. Many land
covers, e.g., crops, are indistinguishable at a single time step but require
the modeling of their growing or seasonal patterns.

Researchers have also built spatio-temporal models for land cover mapping Ji
and others (2018); Jia and others (2017); Mazzia and others (2020) and these
approaches have shown encouraging results in isolated scenarios for studying a
specific land cover. However, there is a major challenge that prevents these
approaches from successfully mapping crops in large regions. In particular,
these approaches use CNN and RNN in a straightforward way without fully
exploiting the characteristics of land covers. Different crops show
discriminative signatures at different points of time Jia and others (2019);
Sakamoto et al. (2010). Moreover, different crops have different seeding time
and harvesting time, depending on the weather conditions. To highlight these
characteristics, we show the averaged yearly vegetation index for different
crops in Fig. 1. In addition, the images from some time steps may be affected
by natural disturbances (e.g. clouds, weather) or data acquisition errors
which can severely degrade the classification performance. Thus, identifying
these discriminative periods automatically for different locations and
different years and filtering out the noisy time steps is essential for
distinguishing between different types of crops accurately.

In this paper, we propose Spatio-Temporal segmentation networks with ATTention
(STATT) for automated land-cover detection. Specifically, the proposed model
uses an UNet Ronneberger and others (2015) encoder to embed the spatial
information from each image into a more representative feature space, and then
introduces a Bidirectional Long-Short Term Memory (Bi-LSTM) layer to capture
long-term temporal dependencies and generate spatio-temporal features. We
further use the attention networks to aggregate the obtained spatio-temporal
features at different time steps. The attention networks help identify the
most relevant time steps for classifying land covers and mitigate the impact
from noisy images by assigning lower weights to those time steps.

We further enhance the detection by fusing spatio-temopral features extracted
at multiple resolutions through the skip connections. Specifically, for each
resolution level (obtained through the UNet encoder) we use the same weights
obtained from the aforementioned attention network to aggregate the spatial
features extracted by the encoder at multiple time steps. These aggregated
features at different resolutions are transferred to the UNet decoder to
improve the segmentation.

We show the superiority of our method over existing spatio-temporal learning
methods in mapping crops in different regions, including agriculture-intensive
areas in US and another region that is dominated by tree plantations in
Africa. We also showcase the usability of attention weights in filtering out
noisy time steps and also identifying time steps in which the classes are
distinguishable. We make the code and a dataset used in our evaluation
publicly available
111https://drive.google.com/drive/folders/1qQHqhf90VbI_xWLjy5cUlreWqEk9f1Fy?usp=sharing.

##  2 Related Work

#### Land Use and Land Cover mapping

In recent literature, machine learning techniques, especially deep learning
(DL), have been heavily used for land use and land cover (LULC) mapping. In
particular CNNs Hu and others (2018); Stoian and others (2019) have been used
to extract representations for both spectral and spatial information, whereas
RNN and LSTM Jia and others (2017) make use of the temporal information in
modeling land cover transitions and have shown promising performance in
sequence labelling. Land Cover mapping can also be framed as a semantic
segmentation problem Ulmas and Liiv (2020); Su and Chen (2019); Saralioglu and
Gungor (2020), where each pixel in an aerial/satellite image must classified
into one of several land cover classes.

#### Spatio-temporal modeling for LULC changes

To capture both the spatial and temporal contextual, existing works commonly
follow the end-to-end learning paradigm using a combination of convolutional
and recurrent networks. Shi et al. Xingjian and others (2015) proposed a novel
ConvLSTM layer which replaced the standard feed-forward neural network in an
LSTM with a convolutional one. Rußwurm et al. Rußwurm and Körner (2018)
applied a single ConvLSTM layer to detect cloud obstruction. Another approach
Garnot and others (2019) used a shared CNN to embed the image time-series and
the resulting sequence was fed to an RNN for spatio-temporal image
classification. This approach is inspired by the LRCN model Donahue and others
(2015) which has been successfully used for activity recognition in videos.
All these methods have been focused on the image classification and do not
address the semantic segmentation problem. 3D convolutions is another method
that has been used in land cover classification Ji and others (2018) to
capture the temporal information of satellite images. The convolutional
approach suffers from the inability to handle sequences of different lengths.

Figure 2: A diagrammatic representation of STATT.

#### Semantic Segmentation

One of the most fundamental techniques used in semantic segmentation is the
Fully Convoluted Network (FCN) Long and others (2015) which supplements the
output of the deeper layers with that of the shallower layers to increase the
resolution of the prediction. Researchers have also developed several variants
of FCN such as SegNet Badrinarayanan and others (2017), DeconvNet Noh and
others (2015) and UNet Ronneberger and others (2015). Given the effectiveness
of UNet in a variety of segmentation tasks, we adopt UNet as the base of our
proposed method.

##  3 Problem definition and preliminaries

###  3.1 Problem Setting

In this paper, we consider the task of land cover mapping and frame it as a
semantic segmentation problem using the multi-spectral satellite/aerial image
time-series. In particular, we aim to predict the land cover class
$\boldsymbol{l}\in\\{1,...,\text{L}\\}$ for each pixel in an image. We denote
an input image time-series as $X=[X^{1},\dots,X^{T}]$, where each
$X^{t}\in\mathbb{R}^{\text{H}\times\text{W}\times\text{C}}$ is an
aerial/satellite image of size $(\text{H},\text{W})$ at time $t$ with C multi-
spectral channels. We are also provided with the corresponding labels (in one-
hot representation)
$Y\in\mathbb{\\{0,1\\}}^{\text{H}\times\text{W}\times\text{L}}$, where L is
the number of land-cover classes.

###  3.2 Segmentation network

A segmentation network $f(X_{i};\theta)$ aims to predict the label of each
pixel for an image $X_{i}$. The parameter $\theta$ is estimated through a
training process on a labeled dataset by minimizing an objective function of
empirical risk, such as the pixel-wise cross entropy, as follows:

$\small\mathcal{L}_{CE}(\boldsymbol{\theta}|\mathbf{X},\mathbf{Y})=-\frac{1}{\text{NHW}}\sum_{i}\sum_{(h,w)}\sum_{k}(Y_{i})_{h,w}^{k}\log
f(X_{i};\theta)_{h,w}^{k}$ (1)

where $f(X_{i};\theta)_{h,w}^{k}$ is the likelihood of the $(h,w)$’th pixel
belonging to the class $k$ as predicted by the fully-convolutional network and
$(Y_{i})_{h,w}^{k}=1$ if the $(h,w)$’th pixel of the image $i$ belongs to the
class $k$.

##  4 Method

In this section, we describe the proposed STATT method for land cover mapping
(see Fig. 2). Our method extends the UNet architecture Ronneberger and others
(2015) to capture the spatial and temporal land cover patterns, identify
discriminative land cover patterns and noisy time periods, and fuse multi-
resolution information through skip connections.

###  4.1 Spatio-Temporal modeling

The standard UNet encodes the semantic information of a single image into a
hidden representation, which is then decoded to produce the segmentation
results along with the help of skip connections to recover fine-level details.
Similarly, in our proposed architecture, each satellite image
$X^{t}\in\mathbb{R}^{\text{H}\times\text{W}\times\text{C}}$ of a sequence
$X=[X^{1},\dots,X^{T}]$ is passed through an convolutional encoder
$E(\,\cdot\,;\theta_{E})$ to encode the spatial information of each image into
a latent representation
$Z^{t}\in\mathbb{R}^{\text{H}^{\prime}\times\text{W}^{\prime}\times\text{C}^{\prime}}$.
This results in a sequence of $(\text{H}^{\prime},\text{W}^{\prime})$ spatial
features of dimension $\text{C}^{\prime}$ which is represented as
$Z=[Z^{1},\dots,Z^{T}]$ in dimension
$\text{T}\times\text{H}^{\prime}\times\text{W}^{\prime}\times\text{C}^{\prime}$.
This ordered set of spatial features is then passed into a recurrent sequence
learning module to capture the temporal dependencies in the time series of
spatial features. Specifically, given a sequence of input features
$Z_{ij}=[Z^{1}_{ij},\dots,Z^{T}_{ij}]$, we use an LSTM network to compute a
sequence of hidden states $H_{ij}=[H^{1}_{ij},\dots,H^{T}_{ij}]$. Each hidden
representation $H_{ij}^{t}$ is generated using the hidden state $H_{ij}^{t-1}$
and cell state $C_{ij}^{t-1}$ from the previous time step and the input
spatial-features using the following set of equations:

$\displaystyle\small\begin{rcases}F_{ij}^{t}&=\sigma(W_{H}^{F}H_{ij}^{t-1}+W_{Z}^{F}Z_{ij}^{t})\\\
I_{ij}^{t}&=\sigma(W_{H}^{I}H_{ij}^{t-1}+W_{Z}^{I}Z_{ij}^{t})\\\
O_{ij}^{t}&=\sigma(W_{H}^{O}H_{ij}^{t-1}+W_{Z}^{O}Z_{ij}^{t})\\\
G_{ij}^{t}&=\text{tanh}(W_{H}^{G}H_{ij}^{t-1}+W_{Z}^{G}Z_{ij}^{t})\\\
C_{ij}^{t}&=F_{ij}^{t}\odot C_{ij}^{t-1}+I_{ij}^{t}\odot G_{ij}^{t}\\\
H_{ij}^{t}&=O_{ij}^{t}\odot\text{tanh}(C_{ij}^{t})\end{rcases}{i,j}\in\left(\text{H}^{\prime},\text{W}^{\prime}\right)$
(2)

Table 1: Comparison with baselines in terms of Crop Wise as well as Mean F1
Score. The numbers in bold and with $*$ symbol correspond to the best and
second best method for each row respectively. We also show the time taken for
training(for 1 epoch) and testing of baselines and our method on D1.

| CLASS(Percentage) | Count | UNet | Bi-LSTM Attn | CALD | ConvLSTM | 3D-CNN | STMean | STATT
---|---|---|---|---|---|---|---|---|---
D1 | Barren Land (13.77) | 2139439 | 0.6913 | 0.4662 | 0.6571 | 0.7486 | 0.6860 | 0.7498* | 0.7761
Urban (11.97) | 1859910 | 0.8282 | 0.5869 | 0.6918 | 0.8110 | 0.8143 | 0.8705 | 0.8690*
Grapes (11.58) | 1799302 | 0.8013 | 0.7959 | 0.7103 | 0.7441 | 0.8359 | 0.8477* | 0.8493
Almonds (10.79) | 1676985 | 0.8357 | 0.7684 | 0.7640 | 0.8407* | 0.8389 | 0.8283 | 0.8426
Grass (10.22) | 1588288 | 0.7859 | 0.7315 | 0.7582 | 0.7862 | 0.7595 | 0.8124* | 0.8290
Tree crops (09.77) | 1517775 | 0.7770 | 0.7137 | 0.6443 | 0.7118 | 0.7952 | 0.8064* | 0.8249
Corn (07.73) | 1200614 | 0.8172 | 0.9223 | 0.9105 | 0.9332 | 0.9217 | 0.9220 | 0.9262*
Cotton (07.04) | 1093876 | 0.9335 | 0.9519 | 0.9575 | 0.9623 | 0.9696 | 0.9586 | 0.9676*
Pistachio (05.17) | 802644 | 0.6702 | 0.5821 | 0.5875 | 0.6808 | 0.8202* | 0.7921 | 0.8358
Alfa Alfa (03.87) | 600932 | 0.8438 | 0.7971 | 0.6148 | 0.8098 | 0.8706 | 0.8907 | 0.8798*
Tomatoes (03.07) | 477757 | 0.7046 | 0.8891 | 0.9069 | 0.9244* | 0.9149 | 0.8803 | 0.9406
Walnut (02.42) | 376654 | 0.8457 | 0.5984 | 0.4758 | 0.7511 | 0.8666 | 0.8456 | 0.8477*
Wheat (02.20) | 342596 | 0.3228 | 0.7522 | 0.6594 | 0.7369 | 0.7397 | 0.6887 | 0.7431*
Water (0.40) | 62738 | 0.6967 | 0.5566 | 0.5649 | 0.5700 | 0.5650 | 0.6533 | 0.6756*
MEAN | 15539510 | 0.7539 | 0.7223 | 0.7074 | 0.7865 | 0.8142 | 0.8247* | 0.8434
D2 | Tree crops (51.63) | 2071563 | 0.8264 | 0.8180 | 0.8197 | 0.8864 | 0.8751 | 0.8969* | 0.9103
Barren Land (30.75) | 1233691 | 0.7723* | 0.6742 | 0.6521 | 0.7362 | 0.7263 | 0.7337 | 0.7832
Cashew (17.10) | 685997 | 0.6782 | 0.7817 | 0.7572 | 0.7865 | 0.7929 | 0.8019* | 0.8410
Urban (0.53) | 21304 | 0.5513 | 0.4395 | 0.4349 | 0.4807 | 0.4555 | 0.5586* | 0.5857
MEAN | 4012555 | 0.7071 | 0.6716 | 0.6660 | 0.7225 | 0.7124 | 0.7478* | 0.7800
| TRAIN TIME(in sec/epoch) | 11040 | 517 | 4668 | 3833 | 1812 | 1141 | 1154
| TEST TIME(in sec) | 4008 | 176 | 1848 | 546 | 321 | 241 | 245

Different crops have specific growing and harvesting patterns, and thus,
capturing their specific growing and harvesting patterns are essential in
successful identification. Traditional LSTMs tend to be biased towards the
information provided by the previous time step. To avoid the previous time
step bias and learn spatio-temporal features at each time step using
sufficient context of growing and harvesting patterns we use a Bi-LSTM.
Specifically, we build two LSTM structures called the forward LSTM and
backward LSTM using eq 2. The two LSTM structures are the same except that the
time-series is reversed for the backward LSTM. The forward LSTM models the
growing patterns whereas the backward LSTM models the harvesting patterns of a
crop. The spatio-temporal features at each time step are obtained by
concatenating the hidden representation of both the forward and backward LSTMs
as shown in figure 2. The obtained hidden representations
$H=[H^{1},\dots,H^{T}]$ capture the spatial information as well as the
temporal information by modeling the change in the spatial features.

###  4.2 Attention based aggregation

The spatio-temporal hidden representations are aggregated using an attention
mechanism which calculates the relevance scores for each time-step based on
their contribution to the classification performance. Specifically, we use a
feed-forward neural network $f_{\theta_{A}}(\cdot)$ with parameters
$\theta_{A}$, followed by a spatial averaging and softmax normalization over
all the time steps, as follows:

$\displaystyle\small\alpha^{t}$
$\displaystyle=\text{softmax}(\frac{1}{\text{H}^{\prime}\times\text{W}^{\prime}}\sum_{\left(i,j\right)\in\left(0,0\right)}^{\left(\text{H}^{\prime},\text{W}^{\prime}\right)}f_{\theta_{A}}(H_{ij}^{t}))$
(3)

Using the obtained attention weights, we combine the spatio-temporal features
$H^{t}$ of all the time steps into an aggregated spatio-temporal features $C$,
as follows:

$\displaystyle\small C$ $\displaystyle=\sum_{t=1}^{T}\alpha^{t}H^{t}$ (4)

Here the higher value of $\alpha^{t}\in[0,1]$ indicates that the time step $t$
contains critical information for detecting target land covers. In contrast,
the lower value of $\alpha^{t}$ indicates less importance of the time step
$t$, either because this time step contains much noise or it is out of the
discrimiantive period.

###  4.3 Aggregating multi-resolution features through skip connections

Given the weighted spatio-temporal features $C$, we build a convolutional
decoder $D(\,\cdot\,,\theta_{D})$ to generate segmentation labels.
Additionally, we use skip connections as a direct pathway to fuse spatio-
temporal features extracted by the encoder at multiple resolutions.
Specifically, our architecture has multiple convolution blocks in the encoder
and multiple up-convolution blocks in the decoder. Here these blocks extract
spatial features at different spatial resolutions. For the $k^{th}$
convolution block in the encoder, it outputs multiple intermediate images
$(Z_{1}^{k},\dots,Z^{k}_{T})$ over multiple time steps. We use the same
attention weights calculated in Eq. 4 and aggregate the outputs $Z^{k}_{t}$
using Eq. 5, as follows:

$\small C_{k}=\sum_{t=1}^{T}\alpha^{t}Z^{k}_{t}$ (5)

The obtained spatio-temporal features at the specific resolution
(corresponding to $k^{th}$ block) are then merged to the corresponding layer
in the decoder with the same resolution. The output of the decoder
$D(C,\theta_{D})$ is passed through a linear classification layer
$f(,\theta_{c})$ followed by a softmax function to generate class
probabilities. This model can be trained using a pixel-wise cross entropy loss
as shown in Eq. 1.

Figure 3: Predicted segmentation maps for baselines and STATT along with
groundtruth labels from CDL and high resolution satellite image. Please zoom
for better view

##  5 Experimental Results

###  5.1 Datasets and Implementation details

#### D1:

Sentinel based crop mapping for Central Valley in California, USA This data
set contains multi-spectral images observed by the Sentinel-2 Constellation
for the year 2018 (see the data release link in Section 1). The Sentinel-2
data product has 13 spectral bands 222https://developers.google.com/earth-
engine/datasets/catalog/COPERNICUS_S2_SR##bands at three different spatial
resolutions of 10, 20 and 60 metres. We leave out the atmospheric bands (Band
1, 9 and 10) of 60 metres resolution and re-sample all the bands to 10 metres
using the nearest neighbour method. The AOI corresponds to the Sentinel tile
T11SKA and has a wide variety of crop types, resulting in a challenging task
for land cover mapping. We use images for every 15 days from January to
December (in total 24 time steps) with the size of (10980, 10980). This
dataset contains a mixture of clean and noisy (cloudy) images. For our
experiment, we aim to classify each pixel to a class label, where the labels
for this data set are taken from CDL as the ground truth. To reduce the effect
of noisy labels from CDL, we use a label cleaning strategy where for each
class label we do a 1 pixel erosion followed by removal of connected
components of size less than 10.

#### D2:

Planet based Cashew tree-crop mapping in Benin, Africa This data set contains
multi-spectral monthly composites from Planet Labs
333https://www.planet.com/products/basemap/ for a region in Benin, Africa,
where cashew tree crops are a major source of income for farmers (they
contribute nearly 10% of the countries export income). The images are of size
(2700, 2400) pixels, having 4 spectral bands namely red, green, blue and
NIR(near infrared) at a spatial resolution of 3 metres. For our experiment, we
aim to classify each pixel into a class label $l\in\\{$ Forest, Barren land,
Cashew, Urban $\\}$. The ground truth was created using manual annotation over
the entire study region using high resolution Airbus imagery provided by our
collaborators in Benin, Africa 444Given the proprietary nature of the Planet
Lab composite and the Airbus imagery, we do not have permission to make this
data publicly available..

In both datasets D1 and D2, we divide the whole images into $10\times 10$
grids and randomly select 60%, 20%, 20% grids for training, validation and
testing, respectively. For D1 we use an input patch size of $32\times 32$ to
output a patch of size $16\times 16$ while in D2, we use an input patch size
of $64\times 64$ to output a patch of size $60\times 60$. The same settings
were used for all baseline experiments and all the methods were trained using
the Adam optimizer with a batch-size of 32.

In our proposed architecture, we create convolutional blocks with the filter
size $3\times 3$. Each convolution block in the encoder is followed by a
maxpool layer($2\times 2$) to downsample the the output. The decoder has
upconvolutional layers of the same scales at the corresponding layers to
upsample the output. We also use a single-layer Bi-LSTM network with 256
hidden units. Since the dataset D2 has a higher spatial resolution, we use a
deeper architecture with 5 convolutional blocks in both encoder and decoder
while using 3 blocks for the dataset D1.

We compare our proposed method against the standard UNet (applied to each
image separately and then averaged to get outputs), Bi-LSTM with Attention
mechanism (Bi-LSTM Attn, applied to each pixel), CALD Jia and others (2019),
3D-CNN Ji and others (2018), and ConvLSTM Xingjian and others (2015). Also we
create a variant of our method, STMean, which directly takes the mean of
hidden representations from the LSTM layer and feed it to the decoder instead
of using the attention mechanism.

###  5.2 Predictive Performance

The mean F1 score along with the class-wise F1 score for all the algorithms
are reported in Table 1. We also report the number of pixels of each class
along with their percentage of the total. We observe that STATT achieves the
best or the second best performance for most land cover classes (and the best
mean F1-score). It can also be seen that the incorporation of both spatial and
temporal information leads to an increase in F1 score in both datasets. On one
hand, UNet has worse performance than the spatio-temporal models because it
uses only spatial data for prediction and lacks temporal modeling. On the
other hand, the Bi-LSTM Attn approach performs the classification at the pixel
level by focusing on the temporal information, while disregarding the spatial
correlation. Hence, the predicted maps can contain more errors at individual
pixels, resulting in degraded performance. 3D-CNN is one of such methods which
exploit both the spatial and temporal information. Since the 3D-CNN model uses
convolution in the time-dimension to model the temporal information, it does
not track long-term changes but only considers a few temporally adjacent
frames. ConvLSTM models the long-term temporal information using LSTMs but
lacks due to its inability to model the discriminative time-periods for
different land covers. Moreover, ConvLSTM has a much larger number of
parameters, which requires more training data and higher computational cost.
Compared with these approaches, STATT models the spatial correlation and the
temporal progression while identifying the discriminative time-periods.
Moreover, STATT has much fewer parameters and thus can perform well even in
the data scarcity scenario. The benefit of the attention mechanism can be
sensed by comparing the performance of STMean and STATT. In Table 1, we also
report the runtime of all the algorithms in the form of Train and Test time on
D1. We note that STATT has a lower runtime while having a higher F1 score
among the spatio-temporal methods. Note that the high runtime of UNet is due
to the fact that the model need to be applied to each time step independently.

Figure 3 shows predicted segmentation maps on randomly selected patches along
with ground-truth labels from CDL and high resolution google satellite images.
The first three row shows of example from the Sentinel dataset (D1). Bi-LSTM
Attn results in a predicted maps that is not spatially smooth due to the lack
of spatial contextual information. There are also errors in other methods such
as ConvLSTM and 3D-CNN as they suffer from false predictions. The segmentation
maps of STATT match closely with the CDL labels and are more consistent
spatially. Similarly, in the fourth row we show an example from the Planet
dataset (D2). As expected, the predicted map produced by CALD is not spatially
smooth, whereas 3D-CNN and ConvLSTM produced maps which shows confusion
between Tree Crops and Cashew. From the qualitative analysis of the predicted
maps, it can be seen that STATT not only detects challenging classes but also
has less false positives when compared to the baselines. Specific details of
the arrows are provided in the supplementary material of the paper.

Figure 4: Impact of noisy images on the predictive performance for multiple
land covers. The x-axis shows the percentage of noisy time-steps in the time-
series and y axis shows F1-Score

###  5.3 Robustness to Noisy Images

STATT can reduce the effect of noisy time-steps by assigning low attention
weights to the noisy time-steps. We demonstrate this empirically in Figure 4
by increasing the number of noisy images from 0% to 50%. In particular, we
manipulate image sequence in D1 by replacing the clean images with Sentinel
images containing noise (cloud and other atmospheric disturbances) from the
same time-period. We report the change in the mean F1 score as well as the F1
score of an example class from each type, CROP(Cotton), TREE-CROPS(Pistachio)
and COVER-CROP(Alfa-Alfa). From the results in Figure 4, we observe that the
mean and class-wise F1 score of both STATT and 3D-CNN decreases with the
increase in noise percentage. However, by using the attention based
aggregation, the performance of STATT is only slightly affected compared to
the drop in F1 score (both mean and class-wise) of 3D-CNN.

Figure 5: Visualisation of Attention Weights predicted for Cotton on Sentinel
data.

###  5.4 Visualization of Attention Weights

Here we visualize the obtained attention weights to verify the effectiveness
of the attention mechanism in identifying the discriminative period specific
to a land cover and thus helps distinguish between different land covers. In
Figure 5, we plot the attention weights assigned to each time-step for the
cotton class(in red). To verify the attention weights, we show the satellite
image corresponding to three time-steps(3, 12 and 22). In the first
image(02/06/2018), the crop is not yet planted and thus the farm is
indistinguishable from a barren patch of land nearby which verifies the low
attention weight. In the second satellite image(06/26/2018) it can be easily
observed that the cotton is fully grown and thus it can be easily
distinguished from a barren patch of land and thus the attention weight is the
highest for this time-step. Finally, in the last time-step, the crop is
harvested leaving the land barren which explains the low attention weight.

##  6 Conclusion

Our proposed STATT architecture highlights the advantages of bringing in both
spatial and temporal aspects of data for land cover mapping, especially in
capturing the spatial correlation and temporal progression for each land
cover. The experimental results on real world datasets have shown the
superiority of STATT over its counterparts. Attention based mechanism helps in
boosting the overall performance, especially in scenarios where the images are
affected by clouds and other atmospheric disturbances. It also identifies the
discriminative time-periods which is essential in identifying different land-
covers. The proposed scheme can be extended along many directions. The current
scheme models each class individually, but in reality, many of them are more
closely related than others (e.g., corn is more similar to soybean than to
almonds which is a tree crop). Being able to model these hierarchical
relationships in the learning framework can make the scheme more robust. We
believe that with these advances, such techniques have a huge potential in
applications such as yield estimation, crop insurance, and improving
management practices to maximize yield.

## References

  * Badrinarayanan and others [2017] Vijay Badrinarayanan et al.  Segnet: A deep convolutional encoder-decoder architecture for image segmentation.  TPAMI, 2017.
  * Donahue and others [2015] Jeffrey Donahue et al.  Long-term recurrent convolutional networks for visual recognition and description.  In CVPR, 2015.
  * Garnot and others [2019] V Sainte Fare Garnot et al.  Time-space tradeoff in deep learning models for crop classification on satellite multi-spectral image time series.  In IGARSS, 2019.
  * Handan-Nader and others [2019] Cassandra Handan-Nader et al.  Deep learning to map concentrated animal feeding operations.  Nature Sustainability, 2(4):298–306, 2019.
  * Hu and others [2018] Yunfeng Hu et al.  A deep convolution neural network method for land cover mapping: a case study of qinhuangdao, china.  Remote Sensing, 10(12), 2018.
  * Ji and others [2018] Shunping Ji et al.  3d convolutional neural networks for crop classification with multi-temporal remote sensing images.  Remote Sensing, 10(1):75, 2018.
  * Jia and others [2014] Kun Jia et al.  Land cover classification of landsat data with phenological features extracted from time series modis ndvi data.  Remote sensing, 6(11), 2014.
  * Jia and others [2017] Xiaowei Jia et al.  Incremental dual-memory lstm in land cover prediction.  In 23rd ACM SIGKDD, 2017.
  * Jia and others [2019] Xiaowei Jia et al.  Spatial context-aware networks for mining temporal discriminative period in land cover detection.  In SDM. SIAM, 2019.
  * Long and others [2015] Jonathan Long et al.  Fully convolutional networks for semantic segmentation.  In CVPR, 2015.
  * Maggiori and others [2016] Emmanuel Maggiori et al.  Convolutional neural networks for large-scale remote-sensing image classification.  TGRS, 2016.
  * Mazzia and others [2020] Vittorio Mazzia et al.  Improvement in land cover and crop classification based on temporal features learning from sentinel-2 data using recurrent-convolutional neural network (r-cnn).  Applied Sciences, 2020.
  * Nayak and others [2020] Guruprasad Nayak et al.  Semi-supervised classification using attention-based regularization on coarse-resolution data.  In SDM. SIAM, 2020.
  * Noh and others [2015] Hyeonwoo Noh et al.  Learning deconvolution network for semantic segmentation.  In ICCV, 2015.
  * Postadjian and others [2017] Tristan Postadjian et al.  Investigating the potential of deep neural networks for large-scale classification of very high resolution satellite images.  ISPRS, 2017.
  * Ronneberger and others [2015] Olaf Ronneberger et al.  U-net: Convolutional networks for biomedical image segmentation.  In MICCAI. Springer, 2015.
  * Rußwurm and Korner [2017] Marc Rußwurm and Marco Korner.  Temporal vegetation modelling using long short-term memory networks for crop identification from medium-resolution multi-spectral satellite images.  In CVPR Workshops, 2017.
  * Rußwurm and Körner [2018] Marc Rußwurm and Marco Körner.  Convolutional lstms for cloud-robust segmentation of remote sensing imagery.  arXiv preprint arXiv:1811.02471, 2018.
  * Sakamoto et al. [2010] Toshihiro Sakamoto, Michio Shibayama, Eiji Takada, Akihiro Inoue, Kazuhiro Morita, Wataru Takahashi, Shigenori Miura, and Akihiko Kimura.  Detecting seasonal changes in crop community structure using day and night digital images.  Photogrammetric Engineering & Remote Sensing, 76(6):713–726, 2010\.
  * Saralioglu and Gungor [2020] Ekrem Saralioglu and Oguz Gungor.  Semantic segmentation of land cover from high resolution multispectral satellite images by spectral-spatial convolutional neural network.  Geocarto International, pages 1–21, 2020.
  * Stoian and others [2019] Andrei Stoian et al.  Land cover maps production with high resolution satellite image time series and convolutional neural networks: Adaptations and limits for operational systems.  Remote Sensing, 11(17), 2019.
  * Su and Chen [2019] Renee Su and Rong Chen.  Land cover change detection via semantic segmentation.  arXiv preprint arXiv:1911.12903, 2019.
  * Ulmas and Liiv [2020] Priit Ulmas and Innar Liiv.  Segmentation of satellite imagery using u-net models for land cover classification.  arXiv preprint arXiv:2003.02899, 2020.
  * Xie et al. [2015] Michael Xie, Neal Jean, Marshall Burke, David Lobell, and Stefano Ermon.  Transfer learning from deep features for remote sensing and poverty mapping.  arXiv preprint arXiv:1510.00098, 2015.
  * Xingjian and others [2015] SHI Xingjian et al.  Convolutional lstm network: A machine learning approach for precipitation nowcasting.  In NeurIPS, 2015.