# Habitat classification from satellite observations with sparse annotations

Mikko Impiö111Programme for Environmental Information, Finnish Environment
Institute, Finland<EMAIL_ADDRESS>Pekka Härmä222Data and Information
Centre, Finnish Environment Institute, Finland Anna
Tammilehto333Metsähallitus, National Parks, Finland Saku Anttila444Data and
Information Centre, Finnish Environment Institute, Finland Jenni
Raitoharju555Programme for Environmental Information, Finnish Environment
Institute, Finland

###### Abstract

Remote sensing benefits habitat conservation by making monitoring of large
areas easier compared to field surveying especially if the remote sensed data
can be automatically analyzed. An important aspect of monitoring is
classifying and mapping habitat types present in the monitored area. Automatic
classification is a difficult task, as classes have fine-grained differences
and their distributions are long-tailed and unbalanced. Usually training data
used for automatic land cover classification relies on fully annotated
segmentation maps, annotated from remote sensed imagery to a fairly high-level
taxonomy, i.e., classes such as forest, farmland, or urban area. A challenge
with automatic habitat classification is that reliable data annotation
requires field-surveys. Therefore, full segmentation maps are expensive to
produce, and training data is often sparse, point-like, and limited to areas
accessible by foot. Methods for utilizing these limited data more efficiently
are needed.

We address these problems by proposing a method for habitat classification and
mapping, and apply this method to classify the entire northern Finnish Lapland
area into Natura2000 classes. The method is characterized by using finely-
grained, sparse, single-pixel annotations collected from the field, combined
with large amounts of unannotated data to produce segmentation maps.
Supervised, unsupervised and semi-supervised methods are compared, and the
benefits of transfer learning from a larger out-of-domain dataset are
demonstrated. We propose a convolutional neural network (CNN) biased towards
center pixel classification ensembled with a random forest classifier, that
produces higher quality classifications than the models themselves alone. We
show that cropping augmentations, test-time augmentation and semi-supervised
learning can help classification even further.

###### keywords:

habitat classification , machine learning , semi-supervised learning ,
transfer learning , random forests

We focus on the challenging problem of automatic fine-grained biotype
classification in remote sensed imagery using sparse pixel-wise ground-truth
annotations.

We propose a region-based approach, where CNNs are used to identify only the
center-pixel using a wider region around it.

We compare unsupervised pretraining, supervised pretraining using a large out-
of-domain dataset, and semi-supervised training as approaches to improve our
CNN model.

We propose an ensemble of a random forest (RF) and our CNN approach with test-
time augmentation and achieve the most promising results with this ensemble
approach.

We make all our Machine learning (ML) implementations along with our field
survey based annotations publicly available.

##  1 Introduction

Earth observation based land cover classification, where each pixel of a
satellite raster is classified as belonging to a discrete class, is a well-
studied problem for classes such as roads, farmlands, and urban areas [1, 2,
3]. As the number of classes increases, variability between classes decreases,
making classification harder. [4, 5]. Finely-grained categories are found in
commonly used habitat taxonomies, such as in the Natura2000 system defined in
the EU Habitats Directive [6]. Deep learning methods have become widely-used
in land cover classification lately [7, 8, 9, 10, 1, 11], but the size of
available annotations remains a challenge for habitat classification.

Global biodiversity is declining due to human actions [12, 13, 14, 15].
Habitat mapping is a fundamental component of biodiversity monitoring, making
it possible to better understand biodiversity loss and the impacts of climate
change [16]. In addition to the intrinsic value habitats have, they also
provide important ecological services, such as production of oxygen and food,
carbon capture and water filtration [17, 18]. Habitat conservation and mapping
is a high priority in the EU Biodiversity Strategy for 2030 [19], aiming to
enlarge existing Natura2000 conservation areas and provide protection for
endangered habitat types. The EU Habitats Directive [20] also requires the
classification and mapping of endangered habitats. However, production of up-
to-date data on distribution and state of these endangered habitats is a
difficult task, especially in remote areas.

On-site field surveys and aerial photographic remote sensing are the two main
methods for habitat mapping [21]. Field surveys provide high-quality, in-situ
information, but conducting them is expensive especially in remote areas [22].
Remote sensing can provide information on a much larger scale, but with
reduced information content, as some information only visible in the field is
always lost. However, the information gained via remote sensing can be close
to the explanatory power of field surveys, making remote sensing a cost-
effective approach [23]. Remote sensing is also the only approach that enables
large-scale ecosystem and community observation, global coverage and
continuous monitoring. [24].

The ability to discriminate different habitats from remote sensed imagery
depends highly on the chosen class mapping [25], which is referred to as class
taxonomy in our study. Some class taxonomies are based on differentiating
habitats on-site, while others are designed especially for remote sensing
purposes, focusing on large, spatially contiguous areas [25]. This study
focuses on the Natura2000 class system [6], which is an in-situ class taxonomy
and challenging for remote sensing purposes.

Remote sensing is a particularly effective mapping method in remote and
spatially extensive areas, such as in Northern Lapland, Finland, which is the
focus area of our study. A number of endangered fell habitat types are endemic
to Lapland [26, 27], and reliable land cover mapping helps in their
conservation. Field surveys have to be conducted during the short summer
period and is a demanding and laborious task, as surveyors have to spend long
periods of time in the field, far away from human settlements. Field surveys
are limited due to difficult and time-consuming access to many remote areas,
making automated remote sensing approaches necessary for covering the full
variance of habitat types efficiently. While field surveys remain
irreplaceable, as ground truth data is always needed for building models,
remote sensing may also help targeting the costly field surveys to the most
promising areas for improving models.

LULC classification of remote sensed grids has been studied extensively for
natural environments, e.g., in [28, 29, 30]. Traditional machine learning
based models, such as RFs [31, 32], support vector machines [33], multi-layer
perceptrons [34, 35], and maximum entropy models [36] have been applied to the
problem. RF have proven to be especially useful in _pixel-based_
classification, where each pixel is classified separately, and have been used
for classifying habitats [37, 28], croplands [38] and tree species [39].
However, pixel-based classification fails to leverage the useful information
that the local environment may contain and some classes, such wetlands, cannot
be distinguished from each other based on a single pixel [4]. _Object-based_
classification, where objects formed by pixels of similar characteristics are
classified based on features extracted from the whole object, is another
commonly used approach for land use and land cover (LULC) classification [40,
41, 42]. A critical preprocessing step for object-based classification is
segmenting the image into objects of similar characteristics and in particular
under-segmentation may lead to poor results [40]. Some works relying on
traditional machine learning methods combine pixel-based and object-based
classification [41, 42].

Recently, also CNNs have been used for different LULC tasks [39, 43]. CNNs
take as their input an image patch and they typically output either an image
of the same size defining a class for each pixel (semantic segmentation) or a
vector showing the probabilities of the patch to belong to each class. The
latter can be used either for single-label classification, where the patch is
assigned to the most probable class [39, 43, 8] or multi-label classification
where multiple classes can be assigned for a single patch [7]. The patch
classification has been used, for example, for 4-class tree classification of
airborne hyperspectral data [39], different land cover classification tasks on
airborne hyperspectral data [8], forest classification into deciduous and
evergreen types using drone data [7], and 10-class general land cover
classification of Very High Resolution satellite imagery [43]. Patch
classification approaches either require uniform patches or sacrifice the
pixel level accuracy and focus only on the majority class(es) within the
patches.

Semantic segmentation has been used for urban environments [9, 3], crop
mapping [1], and general LULC into distinct classes [44]. A bottleneck for
semantic segmentation is that models need to be trained with dense semantic
segmentation maps, where each pixel is annotated. Due to annotation
requirements, most works focus on easily accessible areas such as urban and
agricultural areas [3, 1], use very generic classes that can be annotated
without field surveys [44], or rely on airborne data with higher resolution
leading to lower areal coverage [39, 7]. Producing segmentation maps from
sparse annotations has been studied ([45, 2]), but the approaches still rely
on full segmentation maps in addition to sparse annotations. An alternative
approach was proposed in [44], where sparse annotations of satellite imagery
are used to create synthetic segmentation maps for training, overcoming the
need for real fully-annotated segmentation maps.

Remote sensing has been used for habitat mapping [37], tree species and forest
classification [39, 7], as well as wetland classification [4, 31]. Remote
sensing has vast possibilities in biodiversity assessment and bioindicator
extraction [46]. Niche classification taxonomies such as the Natura2000 have
been less used in lieu of more high-level and easier classes. Random forests
have been used for grassland [36] and wetland [47] Natura2000 classes, but we
could not find studies considering a large number of very different Natura2000
classes simulatenously like it is done in our study. Lack of studies can be
explained by the Natura2000 class taxonomy being highly difficult for remote
sensing purposes due to some classes being distinguishable only in-situ, or
being too small to be detected remotely.

Our study differs from earlier studies by considering a general classification
model - a single model that classifies land cover to all possible Natura2000
types, not focusing on a single class group such as grasslands or wetlands.
The large amounts of unannotated data for pretraining or semi-supervised
learning has not been used for habitat classification previously. Our proposed
methods solve problems in sparsely-annotated land cover classification, and is
the first study where CNN-based methods have been used for classification of
arctic habitats via remote sensing.

In this paper, we focus on the challenging problem of with fine-grained
habitat classification from satellite and airborne measurements using very
sparse pixel-wise annotations. We propose a region-based classification
scheme, where the surrounding area is exploited in classifying the central
pixel as illustrated in Fig. 1. This approach is different from the object-
based approach, because the aim is not to classify the full patch/object, but
simply extract useful information from the surrounding area of the pixel of
focus. Thus, there is no need for homogeneous objects/patches, while CNNs can
be applied. To improve the CNN classification with scarce training data we
explore different state-of-the-art training techniques including transfer
learning, unsupervised and semi-supervised training. Furthermore, we propose a
novel approach for center-crop augmentation, a method for producing center-
pixel biased CNNs. We also show the benefits of test-time augmentation (TTA)
during inference, an approach that makes the final predictions more reliable.
We compare the performance, including the advantages and disadvantages, of the
CNN-based methods with RFs trained for pixel-wise classification. Finally, we
achieve the best performance by combining the pixel-based RF classifier with
the region-based CNN.

Figure 1: How the proposed method compares to other classification tasks

The main contributions of this paper are as follows:

  * 1.

We propose a region-based approach for pixel-based habitat classification
where a CNN and RF ensemble is used to identify the center-pixel of a patch
using a wider region around it.

  * 2.

We compare different pretraining approaches with post-training semi-supervised
learning.

  * 3.

We show how beneficial it is to apply pretraining with out-of-domain data, for
example using the CORINE land cover datasets.

  * 4.

Our experiments show that the region-based ensemble model outperforms both RFs
and CNNs separately.

  * 5.

We also analyze the suitability of the methods to assist in locating certain
habitats in field-surveys and to guide future field data collection.

The rest of this article is organized as follows. Section 2 discusses the
field-collected data and remote sensing imagery used for our experiments.
Section 3 discusses the details of our proposed model and training approaches,
as well as implementation details. Section 5 displays the results and Sections
6 and 7 discuss and conclude the results.

##  2 Materials

###  2.1 Annotations

####  2.1.1 Field survey-based annotations

Figure 2: The study area in northern Lapland

The classification ground-truth is based on field surveyed annotations. A
total of 2558 georeferenced points representing the environment around a 10m
radius was collected by Metsähallitus from Northern Lapland during summer
2020. Field data collection was done as a part of a project aiming at updating
the habitat maps in nature conservation and wilderness areas in Northern
Lapland. The data was collected mostly by foot by professional nature
surveyors. The study area is shown in Figure 2.

The collected dataset is rich in information and contains several attributes,
for example habitat homogeneity and vegetation height. Two different class
labels based on two different classification taxonomies, namely Natura2000 [6]
and the general classification system for Finland’s biotopes [48], are
provided for most of the points.

Natura2000 is a commonly used classification taxonomy consisting of 233
different classes [6]. The Natura2000 classification taxonomy is designed for
nature conservation and focuses on endangered habitat types. The taxonomy is
hierarchical in four levels, having an unique identifier for each class
determining the hierarchy. For example the ’7310 Aapa mires’ class belongs to
the ’73XX Boreal mires’ group, which belongs to the highest ’7XXX Raised bogs
and mires and fens’ group. The dataset we are using has in total 25 unique
Natura2000 classes, presented with their names and number of points in Table
1.

The collected dataset has some limitations. Some of the collected points do
not have class information for both taxonomies, making the dataset size for
the two taxonomies different. The total number of Natura2000-annotated points
is 2036. The dataset is also highly imbalanced, as seen in Table 1, with the
largest class containing 472 samples and the smallest classes only a single
sample. Although the dataset is a large field survey dataset, it is fairly
small for machine learning purposes. To address the limitations of a small
dataset, we collected a larger out-of-domain dataset for pretraining.

Code |  English name [6] |  #
---|---|---
3110 |  Oligotrophic waters containing very few minerals of sandy plains (Littorelletalia uniflorae) |  28
3160 |  Natural dystrophic lakes and ponds |  1
3220 |  Alpine rivers and the herbaceous vegetation along their banks |  4
4060 |  Alpine and Boreal heaths |  472
4080 |  Sub-Arctic Salix spp. scrub |  9
6150 |  Siliceous alpine and boreal grasslands |  121
6270 |  Fennoscandian lowland species-rich dry to mesic grasslands |  1
6430 |  Hydrophilous tall herb fringe communities of plains and of the montane to alpine levels |  13
6450 |  Northern boreal alluvial meadows |  46
7140 |  Transition mires and quaking bogs |  165
7160 |  Fennoscandian mineral-rich springs and springfens |  104
7220 |  Petrifying springs with tufa formation (Cratoneurion) |  8
7230 |  Alkaline fens |  26
7240 |  Alpine pioneer formations of the Caricion bicoloris-atrofuscae |  2
7310 |  Aapa mires |  27
7320 |  Palsa mires |  17
8110 |  Siliceous scree of the montane to snow levels (Androsacetalia alpinae and Galeopsietalia ladani) |  7
8210 |  Calcareous rocky slopes with chasmophytic vegetation |  2
8220 |  Siliceous rocky slopes with chasmophytic vegetation |  64
9010 |  Western Taïga |  271
9040 |  Nordic subalpine/subarctic forests with Betula pubescens ssp. czerepanovii |  453
9050 |  Fennoscandian herb-rich forests with Picea abies |  58
9080 |  Fennoscandian deciduous swamp woods |  12
91D0 |  Bog woodland |  19
91E0 |  Alluvial forests with Alnus glutinosa and Fraxinus excelsior (Alno-Padion, Alnion incanae, Salicion albae) |  106
|  |  2036
Table 1: Natura2000 classes present in our dataset, accompanied with their
Finnish and English names according to [6], and the number of samples in the
full dataset

####  2.1.2 Supplementary annotations

The larger out-of-domain dataset consists of 35 600 points sampled from the
CORINE land cover raster from 2018 [49]. The sampled points were selected
randomly, with the requirement of a minimum of 500 meters between points to
avoid overlapping patches. CORINE is a general land cover class taxonomy,
containing 44 classes in five major groups. Unlike the Natura2000 classes, the
classes are general land cover categories, such as agricultural areas,
artificial surfaces, and forests. The classes are coarsely-grained and contain
also human-built environments. A more coarse taxonomy here means that a single
class, for example ’Natural grasslands’ in CORINE, may correspond to several
Natura2000 classes.

###  2.2 Remote sensing data

High quality source data is important in land cover classification. Remote
sensing data should cover a wide spectral and temporal range of observation
with high spatial resolution. We use three different data sources: laser
scanning, Sentinel-2 (S2) imagery, and Normalized Difference Vegetation Index
(NDVI) time-series aggregates derived from the S2 imagery.

The laser scanning data was collected from fly-over scannings performed in
2018-2020. Point clouds were processed into surface models with spatial
resolution of 8m, including information on vegetation height and density.
Sentinel imagery is mosaiced from imagery collected in the summer of 2020. A
subset of 9 bands is used, leaving out the 60m resolution bands and the 8A
band correlating highly with band 8. Sentinel 2 Near-infrared (NIR) and red
channels were used to calculate aggregated NDVI rasters for the year 2020.
NDVI is a good indication of vegetation, and time-series of NDVI data can be
processed to produce rasters describing characteristics of the growth season.
We use three rasters: NDVI amplitude, NDVI sum, and maximum NDVI. Thus, our
data consists of a total of 14 different channels.

The original data was produced by Finnish Environment Institute for other
purposes. For this study, we processed the raw rasters further, resampling the
rasters to uniform 10m resolution and calculating the mean and standard
deviations across the full raster for standardization.

We used both pixel-based classification methods and methods taking the local
neighborhood into account. Therefore, in addition to collecting pixel-wise
training data for each of our annotated points, we collected a separate
dataset for 49x49 pixel patches surrounding the annotated points.

###  2.3 Train-test splits

Preventing data leakage is challenging with georeferenced data. For CNNs
training, we use patches collected around the ground-truth annotation points.
A traditional random train-test split can not be used, as the patches that are
geographically very close to each other, resulting in overlapping patches. The
CNN might learn some features present in the training set and validation on
patches that contain the same features would not reflect the predictive power
of the model.

Figure 3: The field-collected data point locations

To address this problem, we chose a random test set of approximately 20% of
the original dataset and removed all the remaining patches that overlap with
this set. The resulting non-overlapping patches then became the training set.
The train-test splitting for the first fold of total 5 cross-validation folds
is visualized in Figure 3.

This approach reduces data leakage, but has still some problems, as train and
test patches might be collected near a common geographical feature, such as a
river. A better approach would be using totally different regions for training
and testing. However, as some habitat classes are very local, regional
splitting would lead to problems in class balances, as all classes would not
be present in all regions.

##  3 Methods

##  4 Overview of the applied methods

To solve the problem of sparsely-annotated habitat classification from
remotely-sensed data we compared many different approaches, including
traditional pixel-based training, state-of-the-art deep learning algorithms,
and some novel techniques proposed below. As our classifiers, we used commonly
used RF that classifies each pixel separately. We also formulated a CNN-
classifier which exploits also the surrounding of the pixel in its decision-
making, but still only classifies the center pixel as illustrated in 1. Our
final best-performing model is an ensemble of these pixel-based and region-
based classifiers. The details of the classifier are described in Section 4.1.

The models were trained in three (optional) phases, as shown in Figure 4. In
the first phase, we pretrained the CNNs either in an unsupervised manner or
using out-of-domain annotations. In the second phase, we applied transfer
learning and trained the CNNs using our sparse annotations. In the third
phase, we tested whether any performance improvement is gained using semi-
supervised learning using a larger, unlabeled dataset. The details of the
training approaches are given in Section 4.2.

CORINE-pretraining Transfer learning Semi-supervised learning Figure 4:
Training phases

Data augmentation [50] is another approach commonly used to improve CNN
training. While there are many standard tricks for data augmentation, many of
them cannot be directly applied in our case due to the input channels
representing different modalities and due to center-pixel focused
classification approach. Therefore, we present a novel data augmentation
technique based on random cropping of the surrounding area, along with test
time data augmentation. Data augmentation details are discussed in Section
4.4.

###  4.1 Classifiers

####  4.1.1 RF

RFs have been frequently used in classifying remotely-sensed data [32, 31].
They take as an input a vector containing information about a single
pixel/polygon, which typically means different channels of the satellite data
or some derived data products for that pixel/polygon, and they output a class
label for that pixel (illustrated as single-pixel classification in Fig. 1).

The RFs [51] is an ensemble of decision trees that are trained with different
subsets of the original dataset, commonly referred to as bagging, as well as
using different subsets of the original features. The main benefit of RFs are
that they need significantly less data than CNNs or other deep learning models
and need a lot less fine-tuning [52]. RF also excel with unbalanced data [53,
54]. A drawback of RFs is that classification is pixel-based, which can be
problematic because the semantic content of a pixel can change depending on
the surroundings. For example, a palsa mire and an aapa mire are
differentiated by occurence of permafrost mounds - features that are larger
than a single pixel. Wetland areas between permafrost mounds are highly
similar, and without contextual information, the in-between areas of palsa
mires are classified as aapa mires.

####  4.1.2 Region-based center-pixel focused convolutional neural network

CNN classifiers differ from RFs and other traditional classifiers so that they
take the whole image as their input. The most typical CNN classifiers [55, 56]
output a class label for the whole image (illustrated as image classification
in 1). Another typical approach is to use fully convolutional networks [57]
that output an image that has equal size to the input images and contains
class labels for each pixel (illustrated as image segmentation in 1). As we
only know ground-truth for single pixels, but we assume that the surrounding
pixels may contain valuable information for the decision-making, we formulate
an approach, where the input is still an image, put the output is a class-
label for the center pixel (illustrated as the proposed classification method
in 1).

While CNNs have outperformed traditional classifiers in several computer
vision tasks [58, 55], their main drawback is the demand for very large
amounts of training samples, which may be a bottleneck for applications such
as ours, where annotating samples is slow and laborious. Nevertheless, also
several techniques to overcome this drawback have been proposed in the
literature [59, 60], some of which are also considered in this work and
discussed in the following sections.

####  4.1.3 Ensemble classifier

Figure 5: Classifying an image with the ensemble classifier

We introduce a novel approach combining the best sides of both RFs and CNNs.
The model is based on training a CNN and a RF separately and using them as a
stacked ensemble during inference. The ensemble addresses the drawbacks of
both models: context-blindness of RFs and the large data requirements of CNNs.

Figure 5 illustrates the classification process with the ensemble model. A
center pixel $x$ and its surrounding patch $I$ are fed to the RF and CNN
models, respectively, to produce outputs $y_{RF}=RF(x)$ and $y_{CNN}=CNN(I)$.
When using TTA, the image $I$ can be augmented before feeding it to the CNN.
Ensemble model combines the outputs of both the RF and the CNN to a final
output vector $y$ by averaging the output separately for each class:

$y^{i}=\alpha y_{RF}^{i}+(1-\alpha)y_{CNN}^{i},$ (1)

where $i$ is the class index and $\alpha$ is the weighting coefficient for
classifier importance. We use an equal weight of $\alpha=0.5$ for all
experiments.

###  4.2 Training approaches

We apply several different training approaches for center-pixel focused CNN
training. Pretraining and transfer learning are widely-used approaches and
their benefits compared to random initialization of neural networks are well-
known [61, 62]. A large out-of-domain dataset can be used in pretraining,
which has been shown to yield good results in several applications [63]. If a
large annotated dataset is not available, unsupervised pretraining can also
help initializing the network weights for further fine-tuning with a smaller
domain dataset [64, 65].

We apply the following three methods for CNN training: unsupervised
pretraining using invariant information clustering (IIC) approach by Ji et al.
[60], out-of-domain pretraining with the CORINE dataset, and semi-supervised
training using the Noisy student-method proposed by Xie et al. [59].

####  4.2.1 Out-of-domain pretraining

Pretraining classification models with out-of-domain datasets is a standard
practice in deep learning, and has been shown to improve model performance
[64, 63]. A large out-of-domain dataset with the same number of input channels
as the smaller target dataset is first used for initial training of the
selected model and the model is later fine-tuned via transfer learning by
training the final classification layer with the target dataset. The main
benefit of this approach is that it provides the model with enough data to
learn to extract some useful generic features, which are often useful also for
the target data even if the classes are quite different. Even in earth
observation, pretraining with a general purpose dataset containing all sorts
of natural images, can improve transfer learning to the domain of LULC
classification [11]. We show that sampling points from the CORINE land cover
inventory and using them to pretrain the final fine-grained model improves
classification performance.

Arguably, the most common out-of-domain dataset for many specialized
applications is the ImageNet dataset [66], as ImageNet-pretrained weights are
readily available for many model architectures. ImageNet pretraining can be
used to train intermediate layers in the network, but the first convolutional
layer typically needs to be trained as remote sensed imagery usually contains
more channels than the three present in RGB images. In our experiments, we
show that a custom dataset, i.e., the CORINE land cover data [67] described in
Section 2 can be successfully used in pretraining for biotype classification.

####  4.2.2 Unsupervised pretraining using invariant information clustering

IIC is a image clustering method proposed by Ji et al. [60]. It is based on
applying a transformation $\Phi$ to image $x$ and finding a network
configuration $Phi$ that maximizes the mutual information $I(\cdot,\cdot)$
between the output vectors of a neural network for both the original ($x$) and
a transformed image ($x^{\prime}$):

$\underset{\Phi}{\text{max}}\ I(\Phi(x),\Phi(x^{\prime})).$ (2)

In addition to minimizing the entropy between an original image and its
transformation, this approach maximizes the intra-class entropy, addressing
the problem of class-collapse, where all samples are clustered to only a few
classes. The only hyperparameter that needs to be chosen is the number of
clusters $C$. As the model is a CNN, complex textures can be learned.

###  4.3 Transfer learning

Transfer learning refers to a practice where a model is initially trained with
a different, often larger dataset than the final in-domain dataset. This often
yields better results than starting the in-domain training with a random
initialization of network weights [62]. Initial training is often called
pretraining, and datasets like the ImageNet are often used for the task. In
this paper, we test both ImageNet, and a custom dataset collected from the
data domain as the pretraining dataset.

The pretrained model is used as a starting point for the actual model training
with data from the final classification domain. The final fully connected
layer of the model is changed to match the number of classes in the final
taxonomy and the model is fine-tuned using the final field-collected, 3000
patch dataset with the desired taxonomy. In the transfer learning phase,
either only the classification head or also the convolutional layers are
trained, depending on the regularization methods and augmentations.

The practice where convolutional layer weights are not modified during
training is referred to as freezing. Only the fully connected classification
head weights are learned during the transfer learning phase. The frozen
weights are used straight from the pre-trained model.

####  4.3.1 Semi-supervised Noisy Student training

Training with a small dataset can lead to overfitting the model to the
dataset. Generality of the model suffers and it may not work on areas not
present in the original training patch set. To combat this, we attempt to
produce a model with better generalization by continuing the training using a
large unlabeled dataset. The dataset patches are the same as the CORINE-
annotated dataset, but it is now used without the CORINE class labels. We show
that using semi-supervised learning with an unannotated dataset generally
improves classifier performance in habitat classification.

Fully unsupervised and semi-supervised training, where a large pool of
unannotated images are used during training has leaped forward in recent
years, with several studies showing improvements in general purpose image
classification [68, 69, 56]. OUr semi-supervised training approach is based on
Noisy Student training proposed by Xie et al. in [59]. The method is based on
knowledge distillation, pseudo-labeling and augmentation invariance. The
finetuned model produced in the transfer learning phase is used as an teacher
model to a new model that is trained from scratch. The teacher model produces
pseudo-labels for the unlabeled patches in the large dataset, which are then
used to train the student model. The student model is fed images from the
small labeled dataset, with additional noise for regularization. Cross-entropy
between the student model output and the soft pseudo-labels is minimized to
find the parameters for the final model.

###  4.4 Augmentations

####  4.4.1 Training-time augmentation

Augmentations plays a large role in successfully training a classifier. Image
data augmentation by different geometrical and color transformations has been
shown to improve generalization of a classifier and avoid overfitting [50].
Although several augmentation methods for RGB images exist, the multispectral
data presented as an N-channel image can not be augmented using the same
techniques. The channels can contain data from different modalities and value
ranges, where value changes can change the represented scene drastically. In
our case, the annotated pixel is located in the center of an image and,
therefore, geometric augmentations, such as cropping, need to take this into
account.

Due to the data and annotation constraints, we select only basic augmentation
methods. Horizontal and vertical flipping does not affect the represented
scene and the annotated pixel is unmoved. We experimented with value
transforming augmentations and included random Gaussian blur as an additional
augmentation. Gaussian blur is a suitable augmentation for remote sensing data
because the value ranges can be arbitrary and the blurred scene is still a
valid representation of the underlying true natural scene.

The center pixel annotation gives a possibility of using this information
during augmentation. We propose using random center cropping as an additional
augmentation method for images where the center pixel is annotated. Because
the ResNet implementation uses an adaptive average pooling layer the input
size can be changed between batches. During training, a batch is randomly
cropped to a square with an odd side length between $3$ pixels and the maximum
size of a training square, in this case $19$.

####  4.4.2 Test-time augmentation

Test-time augmentation can improve the robustness of a prediction by
classifying a input several times with different augmentations applied. For
the test-time-augmented results, we augmented and ran inference on each input
region separately five times, and produced a final prediction based on the
average of the softmax outputs of the network.

##  5 Results

###  5.1 Evaluation metrics

The results are evaluated using commonly used classification metrics: Top 1,3,
and 5 accuracies, f1-score, precision and recall. In addition, threshold
metrics average precision (AP) and return-on-characteristics area under curve
(ROC-AUC) are also calculated.

With imbalanced data, the choice of the averaging method across classes can
have a large effect on the interpretation of the results [70]. We use the
weighted mean across classes for each metric, calculating a metric for each
class separately and weighing it depending on the amount of samples in the
class. Other approaches, such as micro- or macro averaging are less useful in
single-score metrics, as micro-averaging produces the same score for all
metrics and macro-averaging (i.e., uniform weighted average) is biased towards
the smallest classes. However, non-weighted micro- and macro averages are
displayed for precision-recall curves in Figure 7. Similar averaging needs to
be done across cross-validation folds, where we use the common macro-averaging
approach for calculating the final classifier performance metric and its
standard deviation.

###  5.2 Experimental setup

All the models trained in the three training phases seen in Figure 6. The
three phases of pretraining, transfer learning and semi-supervised learning
include 11 models trained with different approaches discussed in detail in
Section 3. We trained different versions of models in the same phase, with
different attributes applied. For example, the transfer learned models were
trained with convolutional layers both frozen and trainable, as well as with
random center cropping augmentation applied and not applied.

The transfer learned models were further trained with semi-supervised
learning, resulting in different models with convolutional freezing and center
cropping both applied and not.

Figure 6: Relationships between different trained models. The Base models are
trained only on the small target dataset. PT models are transfer learned from
a model trained on a large CORINE-labeled dataset. NS models are a
continuation of the PT models, using the PT models as a teacher model during
training. The UPT model is pretrained unsupervised using the CORINE dataset
without labels, and fine-tuned using the target dataset

We used a ResNet18 architecture [55] as a backbone to all the CNN models. The
first convolutional layer was changed to match the amount of channels in the
image, and the final fully connected layer was changed to match the number of
classes. All of the models were trained with same hyperparameters and
optimizers to ensure comparability. Optimization was done using the Adam
algorithm [71] with a learning rate of 1e-4. Batch size of 128 was chosen for
all tests. Series of test were conducted to choose training time of 500 epochs
for the transfer learning and distillation tasks, with pretraining of 50
epochs for the CORINE- and IIC-pretrained models.

For the RF classifier that we used both independently and as a part of the
ensemble classifier, we used a decision forest consisting of 100 decision
trees trained with Gini impurity splitting. Hyperparameter optimization with
grid and random search was done for different forest sizes, maximum tree
depths and node impurity measures. Different parameters or increasing the
amount of trees had little to no effect on the results, thus the most simple
configuration of 100 trees and Gini impurity was chosen as the final model.

Table 2: Trained models and their attributes | Attribute
---|---
| Pretraining |  |  |
Model |  Unsupervised |  CORINE |  Conv. layer freeze |  Random crop augmentation |  Semi-supervised learning
Base |  – |  – |  – |  – |  –
Base crop |  – |  – |  – |  + |  –
NS |  – |  + |  + |  – |  +
NS crop |  – |  + |  + |  + |  +
NS crop no freeze |  – |  + |  – |  + |  +
NS no freeze |  – |  + |  – |  – |  +
PT |  – |  + |  + |  – |  –
PT crop |  – |  + |  + |  + |  –
PT crop no freeze |  – |  + |  – |  + |  –
PT no freeze |  – |  + |  – |  – |  –
UPT |  + |  – |  + |  – |  –

###  5.3 Evaluation results

Table 3 shows the overall performance of different ResNet-models, the RF
baseline and the combined ensemble models. Model was evaluated with five-fold
cross-validation. All evaluations were performed with TTA with five
augmentation rounds.

Table 3: Ensemble results for Natura2000 classes: The CNN models ensembled with the random forest model. Trained models include a baseline model (Base) trained only with the final small dataset, a CORINE-pretrained model (PT), model pretrained in an unsupervised manner (UPT), and the PT model continued with semi-supervised Noisy Student learning (NS). Some models have alternative models with cropping or convolutional layer freezing applied. | Metric
---|---
Model |  | F1 weighted
---

| Prec. weighted
---

| Rec. weighted/Acc
---

| Top 3 Acc
---

Base | 0.494 | 0.478 | 0.527 | 0.805
Base crop | 0.504 | 0.494 | 0.553 | 0.823
NS | 0.519 | 0.512 | 0.565 | 0.820
NS crop | 0.534 | 0.525 | 0.586 | 0.822
NS crop no freeze | 0.522 | 0.521 | 0.585 | 0.812
NS no freeze | 0.517 | 0.504 | 0.555 | 0.816
PT | 0.543 | 0.550 | 0.590 | 0.820
PT crop | 0.532 | 0.545 | 0.589 | 0.831
PT crop no freeze | 0.512 | 0.511 | 0.551 | 0.823
PT no freeze | 0.499 | 0.485 | 0.526 | 0.823
UPT | 0.484 | 0.484 | 0.563 | 0.794
Random forest | 0.539 | 0.533 | 0.579 | 0.813

Best performance, measured by the weighted F1-score, was achieved using a
plain transfer learning model, pretrained with the CORINE land cover labels.
No cropping or semi-supervised learning was used for this model, however
further studies presented later show that cropping and semi-supervised
learning can help classification in the general case. The overall performance
of the baseline random forest model is better than any of the ResNet models
alone, but ensembling these models with the random forest model boosts
performance significantly.

The performance across different classes can be seen in the precision-recall
curves in Figure 7. The performance difference between classes is substantial,
with more populous classes being more reliably classified. This can be seen
also in the difference between the macro- and micro-averages of the
classifiers over classes. Due to the larger classes containing most examples,
and them being classified mostly correctly, the micro-average performance is
higher. Because the dataset contains several classes with only few examples
and the classifier performing poorly on these, the macro-averaged
classification performance is considerably lower.

(a) Random forest

(b) ResNet

(c) Ensemble model

Figure 7: Class-wise precision-recall curves for Natura2000 classifications
using the CORINE-pretrained model and test-time augmentation

Figure 8 illustrates the differences between different models and the effect
of test-time augmentation. As figures 8(a) and 8(b) show, the ensemble model
performance gain is higher for the macro-averaged models, indicating better
performance in smaller classes. The ensemble model gain is smaller in the
micro-averaged models, due to better random forest performance in the larger
classes.

(a) Macro average precision-recall curves

(b) Macro average ROC curves

Figure 8: Natura2000 classes precision-recall and ROC curve comparisons
between CORINE-pretrained ResNet, random forest, and ensemble models with
test-time augmetation applied five times (TTA5)

The confusion matrices in Figure 9 are very similar to each other, with
concentrating classifications to a few classes. Most different wetland types
are classified to the class ”7140 - Transition mires and quaking bogs”, and
smaller classes are often falsely classified to the largest class ”9040 -
Nordic subalpine/subarctic forests”.

(a) Random forest

(b) ResNet

Figure 9: Normalized confusion matrices for Natura2000 classes using CORINE-
pretrained models with test-time augmentation applied

###  5.4 Sensitivity studies

Different training methods can have a large effect on the final outcome of the
classifier. We tested four different hypotheses:

  1. 1.

Pretraining using a large dataset with a coarse class taxonomy improves
classification with fine-grained labels.

  2. 2.

Semi-supervised learning with the larger dataset (without labels) improves
performance from the pre-trained classifier

  3. 3.

Training only the classification head with the fine-grained labels leads to a
more generalized model

  4. 4.

Augmentation by random center cropping improves performance in this specific
scenario.

Figure 10 illustrates the effect of different model ablations on the best
performing model, which is the plain transfer-learned model with CORINE
pretraining, without crop augmentations and with convolutional layers freezed
after pretraining. The figure shows the weighted and macro F1 scores of each
cross-validation fold, with the mean and single standard deviation range
highlighted in bold. The effect of adding a certain attribute to the best
performing model is shown under the best model performance. Test-time
augmentation can be applied for each model at test time, and is plotted
separately for each attribute.

(a) F1 weighted

(b) F1 macro

Figure 10: Sensitivity study for attributes affecting the best Natura2000
CORINE-pretrained model, with test-time augmentation effect plotted
separately. Each scatterplot point is a result of a cross-validation fold.
Large points are mean of all cross-validation folds, with the standard
deviation as a bold line.

The effect of pretraining can be also seen in Figure 11, where the effect of
each attribute is plotted separately against all of the other models. Mean and
standard deviation over the cross-validation folds of all models where an
attribute is used, are plotted as well as the differences between having the
attribute and not having it.

(a) F1 weighted

(b) F1 macro

Figure 11: Natura2000 training attribute comparison for each attribute
separately, with the improvement between attribute being applied or not. Each
scatterplot point is a result of a cross-validation fold. Large points are
mean of all cross-validation folds, with the standard deviation as bold line.

If all models with a certain attribute are taken into account, all four
methods (pretraining, convolutional layer freezing, semi-supervised learning
and cropping) increase the weighted F1 score slightly. For the macro-averaged
F1 score the difference is smaller, accompanied with a high overlap in
standard deviation. Also, cropping has a negative effect on macro-averaged F1
score.

###  5.5 Classification maps

We produced land cover maps for the entire Northern Lapland study area (extent
seen in Figure 2), using the models presented in this paper. Examples of
classification maps for different models are displayed in Figures 12, 13, and
14. The differences between the models are quite large, the main difference
being the more uniform classification map of the ResNet model compared to the
fragmentation in the Random Forest’s map. Ensembling these models combines
these features with some tradeoff.

The models can output not just classification class, but a heuristic
probability distribution over all classes in the taxonomy. Using this
probability distribution, it is possible to map a single class’ classification
confidence over the whole study area. This information can be valuable when
researchers are trying to find rare nature types from new areas. Even lower
confidences of a class might indicate presence in the area.

The class with the maximum confidence is chosen as the final classification
for each pixel. In pixels that are harder to classify, the confidence is more
distributed among classes, and the maximum class gets a lower confidence
score. Mapping the maximum confidence for each pixel, like in Figure 15, it is
possible to compare areas where the model is more confident with areas where
confidence is low.

(a) Random forest Natura2000

(b) ResNet Natura2000

(c) Ensemble model Natura2000

Figure 12: Classification maps of the Saana fell area for all model types and
class taxonomies.

(a) Random forest

(b) ResNet

(c) Ensemble model

Figure 13: Classification maps of an area near Lätäseno river for all model
types.

(a) Random forest

(b) Random forest

(c) Ensemble model

Figure 14: Close-up classification maps of an wetland area near Lätäseno river
for all model types.

(a) Random forest

(b) ResNet

Figure 15: Maximum class heuristic confidence maps of the Saana fell area

(a) Random forest

(b) ResNet

(c) Ensemble model

(d) Reference classification

Figure 16: Classification heatmaps for the Natura2000 class ”7310 Aapa mires”
at wetland area east of Lätäseno river, with the ensemble classification map
as reference.

##  6 Discussion

###  6.1 Quantitative evaluation

Overall best performance for both taxonomies were achieved with a plain
transfer learning approach from the CORINE-pretrained model, without cropping
augmentations or further semi-supervised learning. The overall performance of
the baseline random forest model is better than any of the ResNet models
alone, but ensembling these models with the random forest model boosts
performance significantly. Altough the CORINE-classes are quite different from
the final classes, pretraining a CNN with a large CORINE-dataset improves
classification performance by a lot.

It can be seen that both crop augmentation and semi-supervised learning lead
to a worse performing model. The differences in macro-averaged F1 score are
larger due to the nature of macro-averaging, where the score is biased towards
smaller classes with few examples. Contrary to the original hypothesis, that
semi-supervised learning and cropping would lead to the best performance, the
best performing model applies neither of these. Layer freezing and pretraining
however improve the model, as the model does not overfit the convolutional
layers to the small dataset, and learns better representations from the
CORINE-dataset.

###  6.2 Qualitative evaluation

Quantitative evaluation and performance metrics provide a numeric way for
model performance assessment, but they fail to illustrate some differences
between models. Qualitative evaluation, or the analysis of final
classification maps and their differences provides an alternative approach to
model evaluation. Some differences, such as classification uniformity, noise
and granularity can only be seen from these maps and visual assessment.

Figure 16 shows the classification confidence heatmaps for the Natura2000
class ”7310 - Aapa mires”. Typical to CNN-models, the ResNet model displays
highly confident classifications for certain areas, while the Random Forest
model is less confident and more fragmented. As the CNN performs better with
more mosaiced classes, such as wetlands, it is able to classify areas as aapa
mires more confidently. The CNN classifications are also more uniform,
compared to the RF predictions.

In general the ResNet detects larger and more mosaicked areas, such as
wetlands, better, while losing some accuracy in finer-grained areas with rapid
land-cover changes. Some rarer classes are found only in small areas and may
not be detected by the ResNet model. The windowed nature of the ResNet
classifier leads to misclassifications in transitional areas, like in rivers
and surrounding wetlands. Often, when the center pixel to be classified hits a
river, it is classified into a class surrounding the river, as seen around in
Figure 13.

##  7 Conclusions

We have discussed the challenging task of fine-grained habitat classification
from remote sensed imagery, and proposed approaches for utilizing sparse
pixelwise annotations with large amounts of unannotated data. We showed that
pretraining a model with out-of-domain data (such as CORINE land cover labels)
improves performance with fine-grained habitat classes. We also proposed a new
model for single-pixel classification, using a ensemble of random forest
classifier and a convolutional neural network, showing that the approach
performs better than either model by themselves.

## References

  * [1] M. O. Turkoglu, S. D’Aronco, G. Perich, F. Liebisch, C. Streit, K. Schindler, J. D. Wegner, Crop mapping from image time series: Deep learning with multi-scale label hierarchies, Remote Sensing of Environment 264 (2021) 112603\.
  * [2] Y. Wei, S. Ji, Scribble-based weakly supervised deep learning for road surface extraction from remote sensing images, IEEE Transactions on Geoscience and Remote Sensing (2021).
  * [3] B. Huang, B. Zhao, Y. Song, Urban land-use mapping using a deep convolutional neural network with high spatial resolution multispectral remote sensing imagery, Remote Sensing of Environment 214 (2018) 73–86.
  * [4] S. Mahdavi, B. Salehi, J. Granger, M. Amani, B. Brisco, W. Huang, Remote sensing for wetland classification: a comprehensive review, GIScience & Remote Sensing 55 (5) (2018) 623–658.
  * [5] G. Van Horn, O. Mac Aodha, Y. Song, Y. Cui, C. Sun, A. Shepard, H. Adam, P. Perona, S. Belongie, The INaturalist Species Classification and Detection Dataset, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 8769–8778.
  * [6] European Commission DG Environment, Interpretation manual of European Union habitats (2013).
  * [7] S. Kentsch, M. L. Lopez Caceres, D. Serrano, F. Roure, Y. Diez, Computer Vision and Deep Learning Techniques for the Analysis of Drone-Acquired Forest Images, a Transfer Learning Study, Remote Sensing 12 (8) (2020) 1287, number: 8 Publisher: Multidisciplinary Digital Publishing Institute.
  * [8] L. Mou, X. X. Zhu, Learning to pay attention on spectral domain: A spectral attention module-based convolutional network for hyperspectral image classification, IEEE Transactions on Geoscience and Remote Sensing 58 (1) (2019) 110–122.
  * [9] J. Sherrah, Fully Convolutional Networks for Dense Semantic Labelling of High-Resolution Aerial Imagery, arXiv:1606.02585 [cs]ArXiv: 1606.02585 (Jun. 2016).
  * [10] J. Nalepa, M. Myller, Y. Imai, K.-I. Honda, T. Takeda, M. Antoniak, Unsupervised segmentation of hyperspectral images using 3-d convolutional autoencoders, IEEE Geoscience and Remote Sensing Letters 17 (11) (2020) 1948–1952.
  * [11] D. Marmanis, M. Datcu, T. Esch, U. Stilla, Deep learning earth observation classification using imagenet pretrained networks, IEEE Geoscience and Remote Sensing Letters 13 (1) (2015) 105–109.
  * [12] E. S. Brondizio, J. Settele, S. Díaz, H. T. Ngo, Global assessment report on biodiversity and ecosystem services of the intergovernmental science-policy platform on biodiversity and ecosystem services (2019).
  * [13] J. M. Hoekstra, T. M. Boucher, T. H. Ricketts, C. Roberts, Confronting a biome crisis: global disparities of habitat loss and protection, Ecology letters 8 (1) (2005) 23–29.
  * [14] J. A. Foley, R. DeFries, G. P. Asner, C. Barford, G. Bonan, S. R. Carpenter, F. S. Chapin, M. T. Coe, G. C. Daily, H. K. Gibbs, et al., Global consequences of land use, science 309 (5734) (2005) 570–574.
  * [15] C. B. Anderson, Biodiversity monitoring, earth observations and the ecology of scale, Ecology letters 21 (10) (2018) 1572–1585.
  * [16] W. Turner, S. Spector, N. Gardiner, M. Fladeland, E. Sterling, M. Steininger, Remote sensing for biodiversity science and conservation, Trends in ecology & evolution 18 (6) (2003) 306–314.
  * [17] B. J. Cardinale, J. E. Duffy, A. Gonzalez, D. U. Hooper, C. Perrings, P. Venail, A. Narwani, G. M. Mace, D. Tilman, D. A. Wardle, et al., Biodiversity loss and its impact on humanity, Nature 486 (7401) (2012) 59–67.
  * [18] P. Dasgupta, The Economics of Biodiversity: the Dasgupta Review., HM Treasury, 2021\.
  * [19] European commission, EU Biodiversity strategy for 2030 (2020).
  * [20] Council directive 92/43/eec of 21 may 1992 on the conservation of natural habitats and of wild fauna and flora (1992).
  * [21] S. Lengyel, E. Déri, Z. Varga, R. Horváth, B. Tóthmérész, P.-Y. Henry, A. Kobler, L. Kutnar, V. Babij, A. Seliškar, et al., Habitat monitoring in europe: a description of current practices, Biodiversity and Conservation 17 (14) (2008) 3327–3339.
  * [22] P. Mumby, E. Green, A. Edwards, C. Clark, The cost-effectiveness of remote sensing for tropical coastal resources assessment and management, Journal of Environmental Management 55 (3) (1999) 157–166.
  * [23] C. J. Rhodes, P. Henrys, G. M. Siriwardena, M. J. Whittingham, L. R. Norton, The relative value of field survey and remote sensing for biodiversity assessment, Methods in Ecology and Evolution 6 (7) (2015) 772–781.
  * [24] C. S. Reddy, Remote sensing of biodiversity: what to measure and monitor from space to species?, Biodiversity and Conservation 30 (10) (2021) 2617–2631.
  * [25] C. Corbane, S. Lang, K. Pipkins, S. Alleaume, M. Deshayes, V. E. G. Millán, T. Strasser, J. V. Borre, S. Toon, F. Michael, Remote sensing for mapping natural habitats and their conservation status–new opportunities and challenges, International Journal of Applied Earth Observation and Geoinformation 37 (2015) 7–16.
  * [26] T. Kontula, A. Raunio, Threatened habitat types in finland 2018. red list of habitats – results and basis for assessmen, The Finnish Environment 2 (2019) 254.
  * [27] J. Janssen, J. S. Rodwell, M. G. Criado, G. Arts, R. Bijlsma, J. Schaminee, European red list of habitats: Part 2. Terrestrial and freshwater habitats, European Union, 2016.
  * [28] S. Van Beijma, A. Comber, A. Lamb, Random forest classification of salt marsh vegetation habitats using quad-polarimetric airborne sar, elevation and optical rs data, Remote Sensing of Environment 149 (2014) 118–129.
  * [29] K. J. Salovaara, S. Thessler, R. N. Malik, H. Tuomisto, Classification of amazonian primary rain forest vegetation using landsat etm+ satellite imagery, Remote Sensing of Environment 97 (1) (2005) 39–51.
  * [30] Q. Sun, P. Zhang, H. Wei, A. Liu, S. You, D. Sun, Improved mapping and understanding of desert vegetation-habitat complexes from intraannual series of spectral endmember space using cross-wavelet transform and logistic regression, Remote Sensing of Environment 236 (2020) 111516.
  * [31] R. Í. Magnússon, J. Limpens, D. Kleijn, K. van Huissteden, T. C. Maximov, S. Lobry, M. M. Heijmans, Shrub decline and expansion of wetland vegetation revealed by very high resolution land cover change detection in the siberian lowland tundra, Science of the Total Environment 782 (2021) 146877\.
  * [32] P. O. Gislason, J. A. Benediktsson, J. R. Sveinsson, Random Forests for land cover classification, Pattern Recognition Letters 27 (4) (2006) 294–300.
  * [33] Y. Liu, B. Zhang, L.-m. Wang, N. Wang, A self-trained semisupervised SVM approach to the remote sensing land cover classification, Computers & Geosciences 59 (2013) 98–107.
  * [34] V. N. Mishra, P. K. Rai, A remote sensing aided multi-layer perceptron-markov chain analysis for land use and land cover change prediction in patna district (bihar), india, Arabian Journal of Geosciences 9 (4) (2016) 1–18.
  * [35] Q. Yuan, H. Shen, T. Li, Z. Li, S. Li, Y. Jiang, H. Xu, W. Tan, Q. Yang, J. Wang, et al., Deep learning in environmental remote sensing: Achievements and challenges, Remote Sensing of Environment 241 (2020) 111716.
  * [36] S. Stenzel, H. Feilhauer, B. Mack, A. Metz, S. Schmidtlein, Remote sensing of scattered Natura 2000 habitats using a one-class classifier, International Journal of Applied Earth Observation and Geoinformation 33 (2014) 211–217.
  * [37] G. J. McDermid, S. E. Franklin, E. F. LeDrew, Remote sensing for large-area habitat mapping, Progress in Physical Geography 29 (4) (2005) 449–474.
  * [38] P. Teluguntla, P. S. Thenkabail, A. Oliphant, J. Xiong, M. K. Gumma, R. G. Congalton, K. Yadav, A. Huete, A 30-m landsat-derived cropland extent product of australia and china using random forest machine learning algorithm on google earth engine cloud computing platform, ISPRS Journal of Photogrammetry and Remote Sensing 144 (2018) 325–340.
  * [39] J. Mäyrä, S. Keski-Saari, S. Kivinen, T. Tanhuanpää, P. Hurskainen, P. Kullberg, L. Poikolainen, A. Viinikka, S. Tuominen, T. Kumpula, P. Vihervaara, Tree species classification from airborne hyperspectral and LiDAR data using 3D convolutional neural networks, Remote Sensing of Environment 256 (2021) 112322.
  * [40] D. Liu, F. Xia, Assessing object-based classification: advantages and limitations, Remote Sensing Letters 1 (4) (2010) 187–194.  doi:10.1080/01431161003743173.
  * [41] Y. Chen, Y. Zhou, Y. Ge, R. An, Y. Chen, Enhancing land cover mapping through integration of pixel-based and object-based classifications from remotely sensed imagery, Remote Sensing 10 (1) (2018).  doi:10.3390/rs10010077.
  * [42] B. Zou, X. Xu, L. Zhang, Object-based classification of polsar images based on spatial and semantic features, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 13 (2020) 609–619.  doi:10.1109/JSTARS.2020.2968966.
  * [43] L. Dong, H. Du, F. Mao, N. Han, X. Li, G. Zhou, D. Zhu, J. Zheng, M. Zhang, L. Xing, T. Liu, Very High Resolution Remote Sensing Imagery Classification Using a Fusion of Random Forest and Deep Learning Technique—Subtropical Area for Example, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 13 (2020) 113–128, conference Name: IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.
  * [44] N. Laban, B. Abdellatif, H. M. Ebeid, H. A. Shedeed, M. F. Tolba, Sparse pixel training of convolutional neural networks for land cover classification, IEEE Access 9 (2021) 52067–52078.
  * [45] Y. Hua, D. Marcos, L. Mou, X. X. Zhu, D. Tuia, Semantic segmentation of remote sensing images with sparse annotations, IEEE Geoscience and Remote Sensing Letters (2021).
  * [46] Z. Petrou, M. Petrou, A review of remote sensing methods for biodiversity assessment and bioindicator extraction, in: 2011 2nd International Conference on Space Technology, 2011, pp. 1–5.
  * [47] D. Kopel, D. Michalska-Hejduk, T. Berezowski, M. Borowski, S. Rosadzifski, J. Chormafski, et al., Application of multisensoral remote sensing data in the mapping of alkaline fens natura 2000 habitat, Ecological Indicators 70 (2016) 196–208.
  * [48] S. Tuominen, H. Eeronheimo, H. Toivonen, Yleispiirteinen biotooppiluokitus (2001).
  * [49] Corine land cover 2018 dataset finland, ”https://ckan.ymparisto.fi/dataset/corine-maanpeite-2018” (2018).
  * [50] C. Shorten, T. M. Khoshgoftaar, A survey on image data augmentation for deep learning, Journal of Big Data 6 (1) (2019) 1–48.
  * [51] L. Breiman, Random forests, Machine learning 45 (1) (2001) 5–32.
  * [52] K. P. Murphy, Machine learning: a probabilistic perspective, MIT press, 2012.
  * [53] M. Khalilia, S. Chakraborty, M. Popescu, Predicting disease risks from highly imbalanced data using random forest, BMC medical informatics and decision making 11 (1) (2011) 1–13.
  * [54] L. Zhou, H. Wang, Loan default prediction on large imbalanced data using random forests, TELKOMNIKA Indonesian Journal of Electrical Engineering 10 (6) (2012) 1519–1525.
  * [55] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  * [56] P. Khosla, P. Teterwak, C. Wang, A. Sarna, Y. Tian, P. Isola, A. Maschinot, C. Liu, D. Krishnan, Supervised Contrastive Learning, arXiv:2004.11362 [cs, stat] (Apr. 2020).
  * [57] E. Maggiori, Y. Tarabalka, G. Charpiat, P. Alliez, Fully convolutional neural networks for remote sensing image classification, in: IEEE International Geoscience and Remote Sensing Symposium (IGARSS), 2016, pp. 5071–5074.  doi:10.1109/IGARSS.2016.7730322.
  * [58] C. Yoo, D. Han, J. Im, B. Bechtel, Comparison between convolutional neural networks and random forest for local climate zone classification in mega urban areas using landsat images, ISPRS Journal of Photogrammetry and Remote Sensing 157 (2019) 155–170.
  * [59] Q. Xie, M.-T. Luong, E. Hovy, Q. V. Le, Self-training with noisy student improves imagenet classification, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 10687–10698.
  * [60] X. Ji, J. F. Henriques, A. Vedaldi, Invariant information clustering for unsupervised image classification and segmentation, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 9865–9874.
  * [61] S. J. Pan, Q. Yang, A survey on transfer learning, IEEE Transactions on knowledge and data engineering 22 (10) (2009) 1345–1359.
  * [62] K. Weiss, T. M. Khoshgoftaar, D. Wang, A survey of transfer learning, Journal of Big data 3 (1) (2016) 1–40.
  * [63] F. Zhuang, Z. Qi, K. Duan, D. Xi, Y. Zhu, H. Zhu, H. Xiong, Q. He, A comprehensive survey on transfer learning, Proceedings of the IEEE 109 (1) (2020) 43–76.
  * [64] D. Erhan, A. Courville, Y. Bengio, P. Vincent, Why does unsupervised pre-training help deep learning?, in: Proceedings of the thirteenth international conference on artificial intelligence and statistics, JMLR Workshop and Conference Proceedings, 2010, pp. 201–208.
  * [65] O. Manas, A. Lacoste, X. Giró-i Nieto, D. Vazquez, P. Rodriguez, Seasonal contrast: Unsupervised pre-training from uncurated remote sensing data, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 9414–9423.
  * [66] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, L. Fei-Fei, Imagenet: A large-scale hierarchical image database, in: 2009 IEEE conference on computer vision and pattern recognition, Ieee, 2009, pp. 248–255.
  * [67] Corine land cover, https://land.copernicus.eu/pan-european/corine-land-cover (2018).
  * [68] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. A. Pires, Z. D. Guo, M. G. Azar, B. Piot, K. Kavukcuoglu, R. Munos, M. Valko, Bootstrap your own latent: A new approach to self-supervised Learning, arXiv:2006.07733 [cs, stat] (Sep. 2020).
  * [69] T. Chen, S. Kornblith, M. Norouzi, G. Hinton, A Simple Framework for Contrastive Learning of Visual Representations, arXiv:2002.05709 [cs, stat] (Jun. 2020).
  * [70] G. Forman, M. Scholz, Apples-to-apples in cross-validation studies: pitfalls in classifier performance measurement, Acm Sigkdd Explorations Newsletter 12 (1) (2010) 49–57.
  * [71] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980 (2014).

  *[CNN]: convolutional neural network
  *[RF]: random forest
  *[ML]: Machine learning
  *[RFs]: random forest
  *[LULC]: land use and land cover
  *[CNNs]: convolutional neural network
  *[TTA]: test-time augmentation
  *[NDVI]: Normalized Difference Vegetation Index
  *[NIR]: Near-infrared
  *[IIC]: invariant information clustering
  *[AP]: average precision
  *[ROC-AUC]: return-on-characteristics area under curve