An eXtended Finite Element Method Implementation in COMSOL Multiphysics: Solid Mechanics

civilUNSW]Ahmad Jafari
civilShiraz]Pooyan Broumand
civilUNSW]Mohammad Vahab*
[cor]Corresponding author:
civilUNSW]Nasser Khalili

[civilUNSW]School of Civil and Environmental Engineering, The University of New South Wales, Sydney 2052, Australia
[civilShiraz]Department of Civil and Environmental Engineering,  Shiraz University, Shiraz, Iran

This paper presents the first time implementation of the eXtended Finite Element Method (XFEM) in the general purpose commercial software COMSOL Multiphysics. An enrichment strategy is proposed, consistent with the structure of the software. To this end, for each set of enrichment functions, an additional Solid Mechanics module is incorporated into the numerical framework, coupled with compatible modifications to the internal variables. The Linear Elastic Fracture Mechanics (LEFM) is exclusively adopted for the crack analysis. The model pre-processing, level set update, stress intensity factor calculation and crack propagation analysis are conducted by employing COMSOL's built-in features in conjunction with external MATLAB functions through COMSOL LiveLink. All implementational aspects and suggested remedies for the treatment of enriched  elements,  framework  setup,  evaluation  of stress intensity factors, and numerical integration are described in detail. The accuracy and robustness of the proposed method are examined by several numerical examples for stationary and propagating crack problems in 2D and 3D settings. The results represent excellent agreement with available analytical, numerical and experimental observations in the literature.

XFEM; COMSOL Multiphysics; Crack analysis; Fracture propagation

* XFEM implementation in COMSOL Multiphysics is proposed for the first time.

* A straight-forward enrichment procedure is proposed with enormous potential for extension to multi-field problems.

* Benchmark problems regarding single/multiple stationary/propagating cracks  in 2D/3D settings are explored and validated.


#  DiverseNet: Decision Diversified Semi-supervised Semantic Segmentation
Networks for Remote Sensing Imagery

Wanli Ma, Oktay Karakus, Paul L. Rosin

###### Abstract

Semi-supervised learning is designed to help reduce the cost of the manual
labelling process by exploiting the use of useful features from a large
quantity of unlabelled data during training. Since pixel-level manual
labelling in large-scale remote sensing imagery is expensive, semi-supervised
learning becomes an appropriate solution to this. However, most of the
existing semi-supervised learning methods still lack efficient perturbation
methods to promote diversity of features and the precision of pseudo labels
during training. In order to fill this gap, we propose DiverseNet
architectures which explore multi-head and multi-model semi-supervised
learning algorithms by simultaneously promoting precision and diversity during
training. The two proposed methods of DiverseNet, namely the DiverseHead and
DiverseModel, achieve the highest semantic segmentation performance in four
widely utilised remote sensing imagery data sets compared to state-of-the-art
semi-supervised learning methods. Meanwhile, the proposed DiverseHead
architecture is relatively lightweight in terms of parameter space compared to
the state-of-the-art methods whilst reaching high-performance results for all
the tested data sets.

###### keywords:

Semi-supervised Learning, Consistency Learning, Land Cover Classification

organization=School of Computer Science and Informatics, Cardiff
University,city=Cardiff, postcode=CF24 4HJ, country=United Kingdom

##  1 Introduction

Supervised deep learning has gradually become the dominant technique in
computer vision, especially during the last decade. Specifically, with its
great success in semantic segmentation in computer vision, automated land
cover classification approaches have been significantly improved (Vali et al.,
2020). Nevertheless, supervised learning necessitates a substantial,
meticulously labelled dataset. In the case of extensive remote sensing data,
such as satellite imagery and drone-captured images in complex terrains,
acquiring pixel-wise expert annotations proves to be a time-consuming, labour-
intensive, and costly endeavour. While the field of computer vision
accommodates numerous well-annotated datasets, transferring deep learning
models trained on these datasets to the remote sensing domain is a formidable
challenge. This is primarily because of the significant differences between
images generally used in computer vision and remote sensing data, including
hyperspectral and synthetic aperture radar (SAR) imagery, in which images
often possess unconventional and non-intuitive characteristics. Fortunately,
thanks to open-access remote sensing data sources, a large amount of
unlabelled imagery can be accessed easily and freely. Thus, only using a
limited amount of labelled data and exploiting the benefits of a large amount
of unlabelled data could be a feasible solution for solving the problem of the
lack of labelled remote sensing datasets (Wang et al., 2022a).

In computer vision, semi-supervised learning has become a trending research
direction to eliminate the labour-intensive and expensive annotation stages
mentioned above. It leads to competitive performance compared to supervised
learning in applications such as image classification (Zhang et al., 2021;
Huang et al., 2021; Wang et al., 2021b) and segmentation (Hu et al., 2021;
Wang et al., 2022b, 2021a), which just use a part of labelled data and a large
amount of unlabelled data. Specifically, mainstream semi-supervised learning
methods exploit “ _pseudo_ ” labels that are fake and/or auto-generated label
information obtained as a result of an unsupervised step. Since pseudo labels
are involved in the model training stage, the accuracy of pseudo labels is
regarded as a key factor affecting the performance of semi-supervised learning
approaches. However, the accuracy of pseudo labels is not always satisfactory
especially when labelled training data is not complete and sufficient enough.
In this case, augmenting the pseudo-labels to increase diversity has emerged
as another key research direction within the semi-supervised learning domain
(French et al., 2019). A greater diversity aids in constructing more
comprehensive pseudo-labels, which, in turn, indirectly enhances the quality
of these labels and the training effectiveness of the model.

(a) DiverseHead

(b) DiverseModel

Figure 1: Two kinds of pseudo label generation methods based on multiple heads
(a) DiverseHead and multiple models (b) DiverseModel.

In order to empower networks to harness the unexplored potentials within
unlabeled data, a notable technique known as consistency regularization has
emerged as a widely adopted method for semi-supervised learning. (French et
al., 2019; Filipiak et al., 2021). Specifically, consistency regularization
methods rely on the assumption of smoothing, which posits that if two points
reside in a high-density region of feature space and are close to each other,
their corresponding labels should be the same or consistent (Chen and Wang,
2010). In practice, these methods for consistency regularization ensure
adherence to this assumption by compelling networks to generate predictions
that remain consistent across modified iterations of unlabeled inputs.

The strategies for the above-mentioned perturbations can be categorised into
three groups, namely input, feature, and network perturbations. Input
perturbation is a method that involves making adjustments or alterations to
the input data as part of the training process. In this way, the semi-
supervised learning approach promotes the consistency of predictions made in
response to these modified inputs. Widely used input perturbation methods
employed in semi-supervised learning include Cutout (DeVries and Taylor,
2017), MixUp (Zhang et al., 2017), and CutMix (Yun et al., 2019). Among them,
CutMix stands out by combining aspects of MixUp and CutOut techniques,
demonstrating superior performance compared to them (French et al., 2019).
However, input perturbation introduces artificial noise to the data, which
might lead to incorrect or noisy labels when generating pseudo-labels for
unlabeled examples. This might negatively impact the training process by
providing incorrect guidance to the model when using pseudo labels. Cross-
consistency training (CCT) (Ouali et al., 2020) presents a feature
perturbation method that introduces noise to both low- and high-level
features. These perturbed features are then fed into multi-decoders to
generate multiple outputs, followed by the enforcement of consistency among
the outputs obtained from different decoders. However, feature perturbation
alters the intrinsic characteristics and relationships between features, which
also might lead to the creation of artificial feature representations that do
not accurately reflect the underlying patterns in the original data.
Consequently, the perturbed features could result in incorrect or noisy labels
when generating pseudo-labels for unlabeled examples, which negatively impacts
the learning process. Lastly, network perturbation uses different networks or
the same network with different initialization weights to promote diversity of
predictions, such as cross pseudo supervision (CPS) (Chen et al., 2021),
confidence-guided semi-supervised learning (CGSSL) (Ma et al., 2023), mean
teacher (MT), (Tarvainen and Valpola, 2017), CCT (Ouali et al., 2020), guided
collaborative training (GCT) (Ke et al., 2020) and cross teaching between CNN
and transformer (Luo et al., 2022). Compared to the previous two perturbation
methods, although network perturbation methods introduce perturbations created
by the model itself instead of artificial noise, they generally require
rapidly increased computational resources due to the increased number of
networks (whole or their inner stages) that are used.

###  1.1 Contributions

The preceding overview underscores that, although perturbation techniques have
their advantages, they also bring inherent drawbacks. Therefore, the creation
of methods based on perturbations is a complex task, necessitating the
exploration of more sophisticated approaches to augment the potential of semi-
supervised learning frameworks by incorporating advanced perturbation methods.
This paper proposes perturbation-based semi-supervised network architectures,
coined as DiverseNet, which consist of two novel feature and decision
diversifying network architectures for the purposes of semi-supervised
semantic segmentation of remote sensing imagery. First, we develop a
lightweight semi-supervised semantic segmentation network called DiverseHead
based on multiple decision heads to promote precision and diversity of pseudo
labels whilst using two novel perturbation methods of dropout (DT) and dynamic
freezing (DF). In addition, we also propose a high-performance multi-model
structure called DiverseModel for scenarios equipped with high-memory
computational resources. Specifically, the contributions of this work are as
follows:

  1. 1.

We promote the utilisation of multiple decision heads to provide a semi-
supervised learning structure, called DiverseHead. This structure is inspired
by bagging (also called bootstrap aggregating) and provides diversity for
parameters and features. The main source of perturbation is from the dynamic
choosing heads to be updated for each iteration. This architecture does not
distinguish between the main or auxiliary decoder. Also, all decoders are
trained by both supervised and unsupervised losses, which comes with the
advantages of high-quality perturbation and diversity of semi-supervised
segmentation tasks for remote sensing imagery, whilst providing a relatively
lightweight architecture.

  2. 2.

For the purposes of creating perturbation in diversifying decisions, we used
two important techniques, namely ‘dynamic freezing’ and ‘dropout’. We make
these techniques fit with the DiverseHead semi-supervised learning
architecture for the purpose of diversifying the parameters of the network.
The two perturbation methods differ from input/feature perturbation methods,
in that the source of feature diversity is from parameter diversity instead of
introducing artificial noise to data or features.

  3. 3.

A previously proposed architecture DiverseModel (Ma et al., 2023) based on
consistency learning among multiple neural networks with distinct weights has
been extended for detailed evaluation and we provide a detailed comparison
study for various semantic segmentation datasets. Also, we use Grad-CAM to
verify the observation that different networks exhibit varied attention to the
same input, culminating in the combined pseudo label derived from the multiple
model predictions showcasing superior quality.

The rest of the paper is organised as follows: Section 2 discusses the related
work on semi-supervised semantic segmentation in remote sensing and some basic
knowledge on ensemble machine learning whilst in Section 3 and 4, the proposed
algorithms DiverseHead and DiverseModel are presented. Section 5 describes the
utilised segmentation dataset of remote sensing imagery. The experimental
setting and the qualitative and quantitative analyses of the experimental
results are performed in Section 6. Section 7 concludes the paper with a
summary.

##  2 Related Work

Semantic segmentation is a well-studied and wide-spread research topic in
remote sensing, incorporated within applications such as land cover
classification/mapping (Dong et al., 2019), building change detection (Zheng
et al., 2022), road extraction (Ghandorh et al., 2022), and marine debris
detection (Kikaki et al., 2022; Booth et al., 2023). The volume of research on
semantic segmentation in remote sensing in the literature is rapidly
increasing with the success of deep learning in computer vision, due to the
strong task-similarity between two areas. Fully convolutional networks (FCNs)
(Long et al., 2015) is one group of the most widespread networks for
segmentation tasks. They have made a considerable contribution to various
segmentation tasks either in remote sensing or computer vision. Following the
FCNs’ success, SegNet (Badrinarayanan et al., 2017) and UNet (Ronneberger et
al., 2015) adopt a symmetrical encoder-decoder structure with skip
connections, leveraging multi-stage features within the encoder.
Alternatively, PSPNet (Zhao et al., 2017) introduces a pyramid pooling
structure that helps provide a global contextual understanding for pixel-level
scene parsing. DeepLab architecture (Chen et al., 2017) proposes a new type of
convolution called atrous convolution and an atrous spatial pyramid pooling
(ASPP) operation to enable the network to have the ability to control the
spatial perception range of convolution kernels by setting different dilation
rates. DeepLab has been extended to DeepLabv3+ (Chen et al., 2018) with an
improved hybrid approach that combines an encoder-decoder structure and the
ASPP. Within DeepLabv3+, the ASPP module is incorporated following a ResNet
backbone, enabling the network to delve into deeper layers and extract high-
level features, which is helpful for accurately segmenting pixels around
boundaries (Xia et al., 2021; Zhang et al., 2019). DeepLabv3+ is one of the
most used networks in the literature for semi-supervised learning segmentation
in the computer vision area.

Semi-supervised learning aims to alleviate the need for expensive annotation
work and improve the models’ performances by making use of unlabelled data.
Self-training (Lee et al., 2013) (also known as pseudo labelling) represents
one of the primitive semi-supervised learning strategies for both
classification (Kim et al., 2020) and segmentation (Zhu et al., 2021), and
involves the generation of pseudo labels through model predictions. These
pseudo-labels are subsequently used to retrain the model. Various variations
of self-training have been proposed focusing on determining the appropriate
pseudo labels. Another widely developed semi-supervised learning approach
called consistency regularization (French et al., 2019) is to force networks
to give consistent predictions for unlabeled inputs that undergo diverse
perturbations. CCT (Ouali et al., 2020) employs an encoder-decoder
architecture with multiple auxiliary decoders. These decoders introduce
diversity in the output by feature perturbations specific to each auxiliary
decoder. They calculate MSE loss between predictions of the main decoder and
each auxiliary decoder without creating pseudo labels. It is worth noting that
the unsupervised loss is not used to supervise the main decoder. Following
CCT, subsequent methods like GCT (Ke et al., 2020) and CPS (Chen et al., 2021)
have been proposed to introduce network perturbation for consistency
regularization. Specifically, they use two same network structures but with
different weight initialization. CPS differs from GCT by employing pseudo
labels to enforce consistency, whereas GCT achieves consistency regularization
by utilizing predictions of networks cooperating with a flow detector.
Although the network perturbation achieved through different weight
initialization in the previous two works shows a considerable ability to
generate diversity, we believe that employing different networks could yield
even greater diversity.

Ensemble machine learning is a concept that employs multiple learners and
combines their predictions (Sewell, 2008). Bagging, a form of ensemble
learning, is a technique aimed at reducing prediction variance by creating
multiple iterations of a predictor and then utilizing them to form an
aggregated predictor (Breiman, 1996). Specifically, bagging creates sample
subsets by randomly selecting from the training data set and subsequently
utilises these acquired subsets to train the foundational models for
integration. When forecasting a numerical result, the aggregation involves
taking an average of the versions, while predicting a class, it relies on a
majority vote. Bagging is a commonly employed approach for enhancing the
robustness and precision of machine learning algorithms for classification and
regression (Ren et al., 2016).

##  3 Cross-head Supervision (DiverseHead)

Figure 2: DiverseHead: an online semi-supervised learning approach. The slide
windows of the freezer randomly select a certain number of heads to freeze the
parameter update of heads. Additionally, during every iteration, all heads
undergo supervision through a supervised loss, yet only a single head is
randomly chosen to be updated by unsupervised loss.

This section introduces the details of the proposed DiverseHead Semi-
Supervised Learning method which is proposed to create various predictions by
multiple heads, each with the same structure. The source of perturbation is
from the dynamic choosing heads to be updated for each iteration, called
dynamic freezing, which is equivalent to using different training data for
different heads in each iteration. The proposed DiverseHead differs from CCT,
which does not distinguish the main or auxiliary decoder. Also, all decoders
are trained by both supervised and unsupervised losses. In addition, the
proposed method combines self-training and consistency regularization, which
calculates unsupervised loss by using optimized pseudo labels. Specially,
given both a labelled dataset
$\mathcal{B}^{l}=\left\\{\left(x_{i},y_{i}\right)\right\\}_{i=1}^{M}$
containing $M$ images and an unlabelled data set
$\mathcal{B}^{u}=\left\\{u_{i}\right\\}_{i=1}^{N}$ with $N$ images, the
network $Q$ is constructed with multiple heads denoted as
$\left\\{head^{i}\right\\}_{i=1}^{length(heads)}$, where length(heads) denotes
the number of heads. Each of these heads is initialized differently. The
expectation is to gain the trained network $Q$ leveraging these labelled and
unlabelled data based on the proposed semi-supervised learning methods.

The output of each head refers to a probability-based prediction for all
classifications corresponding to its input. When working with labelled data,
the supervised loss $\mathcal{L}_{sup}^{j}$ between the $k^{\prime}th$ ground
truth $y_{k}$ and its corresponding prediction $p_{k}^{j}$ for $j^{\prime}th$
head is defined by using the standard cross-entropy loss function $\ell_{ce}$:

$\mathcal{L}_{sup}^{j}=\frac{1}{W\times H}\sum_{k=1}^{W\times
H}\left(\ell_{ce}\left(p_{k}^{j},y_{k}\right)\right),$ (1)

where $W$ and $H$ refer to the weight and height of input images. The final
supervised loss is determined by the mean of the losses across all heads

$\mathcal{L}_{sup}=\frac{1}{length(heads)}\sum_{j=1}^{length(heads)}\mathcal{L}_{sup}^{j}.$
(2)

In each iteration, we also get prediction $r_{k}^{j}$ corresponding to
unlabelled data $u_{k}^{j}$ from the $j^{\prime}th$ head. In order to get
higher precision pseudo labels, inspired by ensemble learning, the predictions
from all heads are aggregated to create a combined prediction
$\hat{r}_{k}^{sum}$. Subsequently, this combined prediction is used to
calculate pseudo labels $\hat{y}_{k}^{sum}$ through an $argmax$ operation,
which returns the indices of the maximum values of the prediction along the
class dimension. Next, there is a random selector to pick only one head as the
main branch to be supervised by minimizing the unsupervised loss

$\mathcal{L}_{unsup}=\frac{1}{W\times H}\sum_{k=1}^{W\times
H}\left(\ell_{ce}\left(r_{k}^{main},\hat{y}_{k}^{sum}\right)\right),$ (3)

where $r_{k}^{main}$ is the prediction from the selected head.

Finally, the whole loss is written as

$\mathcal{L}=\mathcal{L}_{sup}+\lambda\mathcal{L}_{unsup},$ (4)

where $\lambda$ is the trade-off weight between the supervised and
unsupervised losses.

Two perturbation methods. Based on the framework of DiverseHead, we propose a
parameter perturbation method called dynamic freezing and also apply dropout
as another type of perturbation to increase the diversity of features during
training. Specifically, the first step of dynamic freezing refers to a process
where, during each iteration, half of the heads are randomly selected to be
frozen. This implies that the parameters within these selected heads will
remain unchanged and not undergo updates. Every head has an equal probability
of being chosen. Before going to the next iteration, those frozen heads are
unfrozen. The algorithm of the DiverseHead semi-supervised learning approach
with the perturbation of dynamic freezing is expressed in Algorithm 1.

Another form of perturbation involves the use of dropout layers. We introduce
a dropout layer before the last convolutional layer for each head. The
pseudocode of this method is expressed in Algorithm 2. In the training phase,
specific components of the weights in the network are randomly assigned a
value of zero with a dropout rate (determining the probability), using samples
derived from a Bernoulli distribution. This dropout operation is employed to
enhance the variability of the output predictions. In the conducted
experiments, the dropout rate is set as a hyperparameter with a value of 0.3.
Similar to the training procedure of the semi-supervised learning approach
based on dynamic freezing, the dropout-based approach adheres to the same
training pipeline outlined above, with the distinction that all heads remain
unfrozen. To evaluate the efficacy of individual perturbation methods in
conjunction with the proposed DiverseHead techniques, each method is applied
independently within the proposed framework. The performance of each
combination is assessed in Section 6.

Algorithm 1 DiverseHead Semi-Supervised Learning with Dynamic Freezing
Pseudocode

Initialization:

Initialise backbone using ResNet-50

Randomly initialise parameters for each head

INPUT: Labelled Training Dataset
$\mathcal{B}^{l}=\left\\{\left(x_{i},y_{i}\right)\right\\}_{i=1}^{M}$

Unlabelled Training Dataset $\mathcal{B}^{u}=\left\\{u_{i}\right\\}_{i=1}^{N}$

for
$\left\\{\left(x_{k},y_{k}\right),u_{k}\right\\}_{k=1}^{P}\in\left\\{\textit{cycle}\left(\mathcal{B}^{l}\right),\mathcal{B}^{u}\right\\}$
do

$R=Randint(0,length(heads),\frac{1}{2}length(heads))$

$\left\\{head^{i}\right\\}_{i=R}\leftarrow
sample\left(\left\\{head^{i}\right\\}_{i=1}^{length(heads)}\right)$

$Freeze(\left\\{head^{i}\right\\}_{i=R})$

for $j\in\left\\{1,...,length(heads)\right\\}$ do

$p_{k}^{j}=Q^{head_{j}}(x_{k}^{j})$

$\mathcal{L}_{sup}^{j}=loss(p_{k}^{j},y_{k})$ based on (1)

$r_{k}^{j}=Q^{head_{j}}(u_{k}^{j})$

$r_{k}^{sum}=sum\left(\left\\{r_{k}\right\\}_{k=1}^{length(heads)}\right)$

$\hat{y}_{k}^{sum}\leftarrow argmax(r_{k}^{sum})$

$r_{k}^{main}\leftarrow
sample\left(\left\\{r_{k}^{1},r_{k}^{2},...,r_{k}^{length(heads)}\right\\}\right)$

$\mathcal{L}_{unsup}\leftarrow loss(r_{k}^{main},\hat{y}_{k}^{sum})$ based on
(3)

$\mathcal{L}\leftarrow\frac{1}{length(heads)}\sum_{j=1}^{length(heads)}\mathcal{L}_{sup}^{j}+\lambda\mathcal{L}_{unsup}$

Update model to minimize $\mathcal{L}$

$\mbox{\it Unfreeze}(\left\\{head^{i}\right\\}_{i=R})$

OUTPUT: Trained Model Q

Algorithm 2 DiverseHead Semi-Supervised Learning with Dropout Pseudocode

Initialization:

Initialise backbone using ResNet-50

Randomly initialise parameters for each head

Add dropout layer before the last convolutional layer

INPUT: Labelled Training Dataset
$\mathcal{B}^{l}=\left\\{\left(x_{i},y_{i}\right)\right\\}_{i=1}^{M}$

Unlabelled Training Dataset $\mathcal{B}^{u}=\left\\{u_{i}\right\\}_{i=1}^{N}$

for
$\left\\{\left(x_{k},y_{k}\right),u_{k}\right\\}_{k=1}^{P}\in\left\\{\textit{cycle}\left(\mathcal{B}^{l}\right),\mathcal{B}^{u}\right\\}$
do

for $j\in\left\\{1,...,length(heads)\right\\}$ do

$p_{k}^{j}=Q^{head_{j}}(x_{k}^{j})$

$\mathcal{L}_{sup}^{j}=loss(p_{k}^{j},y_{k})$ based on (1)

$r_{k}^{j}=Q^{head_{j}}(u_{k}^{j})$

$r_{k}^{sum}=sum\left(\left\\{r_{k}\right\\}_{k=1}^{length(heads)}\right)$

$\hat{y}_{k}^{sum}\leftarrow argmax(r_{k}^{sum})$

$r_{k}^{main}\leftarrow
sample\left(\left\\{r_{k}^{1},r_{k}^{2},...,r_{k}^{length(heads)}\right\\}\right)$

$\mathcal{L}_{unsup}\leftarrow loss(r_{k}^{main},\hat{y}_{k}^{sum})$ based on
(3)

$\mathcal{L}\leftarrow\frac{1}{length(heads)}\sum_{j=1}^{length(heads)}\mathcal{L}_{sup}^{j}+\lambda\mathcal{L}_{unsup}$

Update model to minimize $\mathcal{L}$

OUTPUT: Trained Model Q

##  4 Cross-model Supervision (DiverseModel)

Figure 3: DiverseModel: an online semi-supervised learning approach.

The proposed DiverseModel differs from CPS, exploring different networks in
parallel to generate comprehensive pseudo labels. Specially, the DiverseModel
structure includes three distinct networks which are PSPNet (Zhao et al.,
2017), UNet (Ronneberger et al., 2015), SegNet (Badrinarayanan et al., 2017),
respectively. Algorithm 3 presents the pseudocode of the DiverseModel method.
Since different networks pay different and complementary attention to the same
input, this offers the basis that they are able to benefit from each other. In
order to provide evidence for this claim, we executed the Gradient-weighted
Class Activation Mapping (Grad-CAM) (Selvaraju et al., 2020) technique for
every network employed within the framework of the DiverseModel architecture.
Grad-CAM visualizes the areas of an image that are important to the model
predictions from each network. Figure 4 depicts an example analysis for this
grad-CAM analysis for BUILDING class in Potsdam data set. Examining Figure 4,
we can see that different networks pay different and complementary attention
to the same input, and the pseudo labels from the DiverseModel prediction show
the highest quality.

Figure 4: The upper section displays Grad-CAM outputs from individual networks
within the DiverseModel architecture using the Potsdam dataset. The lower
section showcases both the ground truth and pseudo labels generated through
predictions from each network. In the lower right corner, the pseudo label
from DiverseModel is presented.

The labelled data is used in a regular supervised learning manner to train
these models by using the standard cross-entropy loss function $\ell_{ce}$.
The supervised loss is expressed as:

$\mathcal{L}_{sup}=\sum_{n=1}^{3}\frac{1}{W\times H}\sum_{k=1}^{W\times
H}\ell_{ce}\left(p_{k}^{n},y_{k}\right)),$ (5)

where $p_{k}^{n}$ represents the prediction of $k\leavevmode\nobreak\ th$
pixel from $n\leavevmode\nobreak\ th$ network.

In addition, unlabelled data is used to generate pseudo labels, which are then
exploited for cross-supervision to inform each network. Different from the
version presented in (Ma et al., 2023), all loss calculations in this work
solely focus on cross-entropy loss to avoid the effects on performance
resulting from different types of loss functions. The predictions obtained by
each network are denoted as $\\{p^{1},p^{2},p^{3}\\}$, which are used for
generating pseudo labels $\\{\hat{r}^{1},\hat{r}^{2},\hat{r}^{3}\\}$ through
the $argmax$ operation. For instance, the cross pseudo supervision loss
$\mathcal{L}_{unsup}^{12}$ between the prediction $p^{1}$ from the first
network and the pseudo label $r^{2}$ generated by the second network is
defined as:

$\mathcal{L}_{unsup}^{12}=\frac{1}{W\times H}\sum_{k=1}^{W\times
H}\left(\ell_{ce}\left(p_{k}^{1},\hat{r}_{k}^{2}\right)\right).$ (6)

Algorithm 3 DiverseModel Semi-supervised Learning Pseudocode

Initialization:

Randomly initialise 3 models, PSPNet $Q^{1}$, UNet $Q^{2}$, SegNet $Q^{3}$

INPUT: Labelled Training Dataset
$\mathcal{B}^{l}=\left\\{\left(x_{i},y_{i}\right)\right\\}_{i=1}^{M}$

Unlabelled Training Dataset $\mathcal{B}^{u}=\left\\{u_{i}\right\\}_{i=1}^{N}$

for
$\left\\{\left(x_{k},y_{k}\right),u_{k}\right\\}_{k=1}^{P}\in\left\\{\textit{cycle}\left(\mathcal{B}^{l}\right),\mathcal{B}^{u}\right\\}$
do

$\mathcal{L}_{sup}=loss(Q^{1}(x_{k}),y_{k})$ \+ $loss(Q^{2}(x_{k}),y_{k})$

\+ $loss(Q^{3}(x_{k}),y_{k})$ based on (1)

$\hat{r}_{k}^{1}\leftarrow argmax(Q^{3}(u_{k}))$

$\hat{r}_{k}^{2}\leftarrow argmax(Q^{3}(u_{k}))$

$\hat{r}_{k}^{3}\leftarrow argmax(Q^{3}(u_{k}))$

$\mathcal{L}_{unsup}=loss(Q^{1}(u_{k}),\hat{r}_{k}^{2})$ \+
$loss(Q^{1}(u_{k}),\hat{r}_{k}^{3})$

\+ $loss(Q^{2}(u_{k}),\hat{r}_{k}^{1})$ \+
$loss(Q^{2}(u_{k}),\hat{r}_{k}^{3})$

\+ $loss(Q^{3}(u_{k}),\hat{r}_{k}^{1})$ \+
$loss(Q^{3}(u_{k}),\hat{r}_{k}^{2})$

$\mathcal{L}\leftarrow\mathcal{L}_{sup}+\lambda\mathcal{L}_{unsup}$

Update model to minimize $\mathcal{L}$

OUTPUT: Trained Model $Q^{1}$, $Q^{2}$, $Q^{3}$

The cross-pseudo supervision among three networks creates six losses in the
same way. The unsupervised loss $\mathcal{L}_{unsup}$ is the result of
linearly adding the six individual losses, as shown below

$\begin{array}[]{r}\mathcal{L}_{unsup}=\mathcal{L}_{unsup}^{12}+\mathcal{L}_{unsup}^{13}+\mathcal{L}_{unsup}^{21}+\mathcal{L}_{unsup}^{23}+\mathcal{L}_{unsup}^{31}+\mathcal{L}_{unsup}^{32}.\end{array}$
(7)

The total loss $\mathcal{L}$ is the linear addition of $\mathcal{L}_{sup}$ and
$\mathcal{L}_{unsup}$, which is previously given in equation (4)

##  5 Dataset Description

We employed diverse datasets to assess both the proposed techniques and state-
of-the-art methods, specifically including (1) the ISPRS Potsdam dataset
(Rottensteiner et al., 2012), (2) the DFC2020 dataset (Robinson et al., 2021),
(3) the RoadNet dataset (Liu et al., 2018), and (4) the Massachusetts
Buildings dataset (Mnih, 2013). In the sequel, we share the details of each
datasets utilised in this paper.

1) ISPRS Potsdam Semantic Labeling dataset is an open-access benchmark dataset
provided by the International Society for Photogrammetry and Remote Sensing
(ISPRS). The ground sampling distance of the two modalities of the true
orthophoto (TOP) and the DSM is 5 cm. This dataset was annotated manually into
six land cover classes, which are impervious surfaces, buildings, low
vegetation, trees, cars, and clutter/background. This dataset provides 38
patches (all with the size of $6000\times 6000$ pixels), which contain
infrared (IR), red, green and blue bands orthorectified optical images with
corresponding digital surface models (DSM). For computation purposes, we
partitioned all these data tiles into $512\times 512$ patches, resulting in
3456, 201, and 1815 samples for training, validation, and test sets,
respectively. We randomly select a quarter of the training data as labelled
data and use the remaining three quarters as unlabelled data for semi-
supervised learning.

2) DFC2020 is the 2022 IEEE GRSS Data Fusion Contest dataset which is based on
the SEN12MS dataset (Schmitt et al., 2019). It provides Sentinel-1 SAR
imagery, Sentinel-2 multispectral imagery, and corresponding land cover maps
across seven areas namely Mexico City, Kippa-Ring, Khabarovsk, Black Forest,
Mumbai, Cape Town, Bandar Anzali between 2016 and 2017. The size of all
patches is $256\times 256$ pixels. There are 6112, 986 and 5127 images for
training, validation and test sets, respectively. In this paper, we utilize
only one-fifth of the labelled data for training, while the remaining four-
fifths are employed as unlabeled data. The fine-grained International
Geosphere-Biosphere Program (IGBP) classification scheme in SEN12MS was
aggregated to ten coarser-grained classes, which are forest (11.3%), shrubland
(6.9%), savanna (23.6%), grassland (16.8%), wetlands (1.1%), croplands
(17.9%), urban/built-up (10.6%), snow/ice (0.0%), barren (5.2%), and water
(6.5%).

3) RoadNet is a benchmark dataset for road detection with 0.21-m spatial
resolution per pixel covering 21 typical urban regions about 8 $km^{2}$ in
Ottawa, Canada. The annotation data includes road surfaces, road edges, and
road centerlines for each image. we only used road surfaces for the
segmentation task. the number of samples for training, validation and testing
is 410, 45 and 387, respectively. In the semi-supervised learning approaches,
only a quarter of the annotated training data is used, while the remaining
three quarters of the training data are employed as unlabeled data for
training purposes.

4) Massachusetts Buildings Dataset predominantly encompasses urban and
suburban regions, encompassing structures of varying scales. This includes
diverse buildings such as individual houses and garages, all of which are
included in the labels. The dataset consists of 151 aerial images capturing
the Boston area covering roughly 340 square kilometres, with a resolution of
$1\leavevmode\nobreak\ m^{2}$ per pixel. The dataset was divided into subsets
as follows: 137 images for training, 10 images for testing, and 4 images for
validation. Each subset consists of images sized $1500\times 1500$ pixels. A
quarter of the annotated training images and the remaining three quarters of
the unlabelled training data are used in semi-supervised learning.

##  6 Experimental analysis

###  6.1 Implementation Details

We implemented our methods by using the PyTorch framework. Following (Chen et
al., 2018), we used a mini-batch SGD optimizer and adopted a poly learning
rate policy, where the current learning rate equals the initial learning rate
multiplied by $\left(1-\frac{\text{ iter }}{\text{ max-iter
}}\right)^{\text{power }}$. The initial learning rate and power are set to
0.01 and 0.9, respectively. All experiments were conducted on the GW4 Isambard
with an NVIDIA A100-sxm GPU and an AMD EPYC 7543P CPU (McIntosh-Smith, 2014).

We implemented various classic semi-supervised learning methods (MT, CCT, GCT
and CPS) on remote sensing datasets. All these methods use DeepLabv3+ with
ResNet-50 pre-trained on ImageNet as the semantic segmentation network. To be
fair the proposed DiverseHead is also based on DeepLabv3+ with the pretrained
ResNet-50.

We comprehensively analyzed all methods by quantifying performance via class-
related measures, including overall accuracy (OA), user’s accuracy (UA),
producer’s accuracy (PA), mean intersection over union (mIoU), and
$F_{1}$-score. Expressions of all five performance metrics are given as
follows: $\mathrm{OA}=\frac{TP+TN}{TP+TN+FP+FN}$,
$\mathrm{UA}=\frac{TP}{TP+FP}$, $\mathrm{PA}=\frac{TP}{TP+FN}$,
$\mathrm{mIoU}=\frac{|\mathrm{TP}|}{|\mathrm{TP}+\mathrm{FN}+\mathrm{FP}|}$,
$F_{1}=\frac{2\cdot PA\cdot UA}{PA+UA}$, where TP, TN, FP, and FN refer to the
numbers of pixels that are true positives, true negatives, false positives,
and false negatives for each class, respectively.

###  6.2 Quantitative Results and Analysis

We evaluated the performance of the proposed approach and reference networks
using the four previously mentioned datasets, and the average results are
presented in Table 1. The reason we first share average results across
datasets is to provide a global performance for all utilised techniques, and
the performance comparison for all methods on each dataset will be discussed
later. Evaluating the performance results in Table 1, we conclude that the
proposed DiverseModel achieves the best performance among all metrics. In
particular, for metrics like UA and mIoU, the proposed DiverseModel exhibits
an improvement of over 5% compared to another cross-model-based approach, CPS.
Although the performance of DiverseHead is lower than that of DiverseModel, it
outperforms other reference methods and reaches the second best. With these
used remote sensing datasets, MT demonstrates notably inferior average
performance across various metrics, with the exception of UA. Both CCT and GCT
exhibit comparable performance, yet the performance superiority of CCT is
notably evident in its PA metric. CPS consistently yields superior outcomes
across the majority of metrics when juxtaposed against the other state-of-the-
art methodologies.

Meanwhile, it is worth noting that the DiverseHead consists of a notably
lighter architecture during training compared to all other models where Table
2 presents the required number of parameters during training for each semi-
supervised learning architecture. It is evident that the architectures using
multiple segmentation networks during training typically need higher memory
demands. In contrast, the proposed DiverseHead is remarkably lightweight as it
only employs a single model with multiple heads which consists of only two
convolution layers, eliminating the need for multiple models during training.
Thus, the parameter size of the DiverseHead (DT&DF) is only 16% bigger than
that of the single network (Base in Table 2), whereas the parameter size of
other reference semi-supervised architectures is at least twice that of the
single network. Although just a smaller number of parameters are required, the
DiverseHead proposed method outperforms all state-of-the-art with at least 1%
accuracy whilst obtaining similar performance to the companion method of
DiverseModel.

Table 1: Average performance comparison with the state of the art methods on four datasets. DT and DF present dropout and dynamic freezing, respectively. Models |  OA |  UA |  PA |  mIoU |  $F_{1}$
---|---|---|---|---|---
MT (Tarvainen and Valpola, 2017) |  85.63% |  73.14% |  80.10% |  63.93% |  76.39%
CCT (Ouali et al., 2020) |  86.60% |  72.17% |  83.29% |  64.07% |  77.22%
GCT (Ke et al., 2020) |  86.53% |  73.33% |  80.50% |  64.59% |  76.72%
CPS (Chen et al., 2021) |  87.17% |  72.89% |  84.62% |  65.02% |  78.18%
DiverseHead (DT) |  88.42% |  77.80% |  84.95% |  69.70% |  81.05%
DiverseHead (DF) |  88.29% |  76.98% |  84.87% |  68.85% |  80.58%
DiverseModel |  88.86% |  78.24% |  86.06% |  70.53% |  81.79%
Table 2: The required parameter size in each semi-supervised learning approach. Base means supervised segmentation network DeepLabv3+. DT and DF present dropout and dynamic freezing, respectively. |  CPS (Chen et al., 2021) |  MT (Tarvainen and Valpola, 2017) |  CCT (Ouali et al., 2020) |  GCT (Ke et al., 2020)
---|---|---|---|---
Size (MB) | 303.378 | 303.378 | 337.655 | 335.049
| Base | DiverseHead(DT) | DiverseHead(DF) | DiverseModel
Size (MB) | 151.689 | 175.806 | 175.806 | 936.262

Specifically, we present the results of each semi-supervised learning method
for the 4 utilised datasets in Table 3. DiverseNet including DiverseHead and
DiverseModel architectures reach the best performance for each of the datasets
compared to state-of-the-art methods. Specifically, the proposed DiverseModel
achieves the best performance on both RoadNet and Massachusetts building
datasets for all metrics, and the DiverseHead (DT) shows the second best for
most of the metrics on these two datasets. For the other two datasets, Potsdam
and DFC2020, The DiverseHead (DF) method shows competitive and better
performance compared to the companion DiverseModel for most of the metrics.

Table 3: Performance comparison with the state of the arts on four datasets. DT and DF present Dropout and dynamic freezing, respectively. |  Model |  OA |  UA |  PA |  mIoU |  $F_{1}$
---|---|---|---|---|---|---
Potsdam |  MT(Tarvainen and Valpola, 2017) |  81.98% |  73.66% |  78.39% |  63.07% |  75.95%
CCT(Ouali et al., 2020) |  82.66% |  74.64% |  77.62% |  64.16% |  76.10%
GCT(Ke et al., 2020) |  83.99% |  75.65% |  80.81% |  65.80% |  78.14%
CPS(Chen et al., 2021) |  85.00% |  75.76% |  82.94% |  66.69% |  79.19%
DiverseHead (DT) |  84.60% |  77.42% |  80.49% |  67.18% |  78.92%
DiverseHead (DF) |  85.37% |  78.43% |  81.86% |  68.49% |  80.11%
DiverseModel |  85.76% |  76.75% |  83.45% |  67.85% |  79.96%
DFC2020 |  MT(Tarvainen and Valpola, 2017) |  78.64% |  59.59% |  73.57% |  50.44% |  65.85%
CCT(Ouali et al., 2020) |  79.71% |  59.87% |  76.40% |  51.04% |  67.13%
GCT(Ke et al., 2020) |  80.84% |  61.47% |  71.43% |  52.17% |  66.07%
CPS(Chen et al., 2021) |  81.50% |  61.75% |  79.47% |  53.21% |  69.50%
DiverseHead (DT) |  82.11% |  62.31% |  80.38% |  53.93% |  70.20%
DiverseHead (DF) |  82.19% |  62.29% |  80.73% |  54.02% |  70.32%
|  DiverseModel |  81.87% |  62.20% |  80.69% |  53.71% |  70.25%
RoadNet |  MT(Tarvainen and Valpola, 2017) |  93.00% |  80.74% |  85.54% |  72.96% |  83.07%
CCT(Ouali et al., 2020) |  93.90% |  83.85% |  87.26% |  76.23% |  85.52%
GCT(Ke et al., 2020) |  93.57% |  79.26% |  89.25% |  73.51% |  83.96%
CPS(Chen et al., 2021) |  94.36% |  84.21% |  88.85% |  77.44% |  86.47%
DiverseHead (DT) |  96.28% |  90.89% |  91.95% |  84.84% |  91.42%
DiverseHead (DF) |  95.36% |  87.99% |  90.31% |  81.35% |  89.13%
|  DiverseModel |  96.79% |  92.82% |  92.57% |  86.87% |  92.70%
Massachusetts |  MT(Tarvainen and Valpola, 2017) |  88.92% |  78.56% |  82.90% |  69.25% |  80.67%
CCT(Ouali et al., 2020) |  89.65% |  79.91% |  84.23% |  70.95% |  82.01%
GCT(Ke et al., 2020) |  87.73% |  76.95% |  80.51% |  66.88% |  78.69%
CPS(Chen et al., 2021) |  87.85% |  69.85% |  87.24% |  62.76% |  77.58%
DiverseHead (DT) |  90.69% |  80.56% |  86.97% |  72.86% |  83.64%
DiverseHead (DF) |  90.25% |  79.22% |  86.59% |  71.53% |  82.74%
|  DiverseModel |  91.01% |  81.19% |  87.53% |  73.67% |  84.24%

We also considered a downgraded version of the current DiverseHead approach,
called single-head supervision (SHS). The segmentation network consists of
only a single head. During each iteration, the current model’s predictions for
unlabeled data are used to generate pseudo labels through an $\arg\max$
operation. These generated pseudo labels are then used to calculate the
unsupervised loss. The baseline is the segmentation network only trained by
labelled data. The performance of the baseline and SHS, along with the
proposed DiverseHead, are presented in Table 4 in order to provide
experimental evidence of the novelty of use of multiple heads in the
DiverseHead architecture. While the single-head supervision benefits from the
pseudo labels and exhibits better performance compared to the baseline model,
especially improving the metric of PA, both versions of DiverseHead approaches
demonstrate a further significant improvement by taking advantage of multiple
heads. Specifically, the mIoU of DiverseHead (DF) is 5.7% higher than that of
Baseline and approximately 4% higher than that of SHS for Potsdam. Also, the
mIoU value of DiverseHead (DT) is about 10% higher than that of SHS.

Table 4: Performance comparison with single-head supervision (SHS) and baseline model which is only supervised by labelled data. DT and DF present dropout and dynamic freezing, respectively. |  Model |  OA |  UA |  PA |  mIoU |  $F_{1}$
---|---|---|---|---|---|---
Potsdam |  Baseline |  81.64% |  73.86% |  76.05% |  62.72% |  74.94%
SHS |  83.43% |  74.57% |  79.93% |  64.51% |  77.16%
DiverseHead (DT) |  84.60% |  77.42% |  80.49% |  67.18% |  78.92%
|  DiverseHead (DF) |  85.37% |  78.43% |  81.86% |  68.49% |  80.11%
RoadNet |  Baseline |  93.56% |  82.85% |  86.53% |  75.06% |  84.65%
SHS |  93.79% |  81.19% |  88.62% |  74.87% |  84.74%
DiverseHead (DT) |  96.28% |  90.89% |  91.95% |  84.84% |  91.42%
|  DiverseHead (DF) |  95.36% |  87.99% |  90.31% |  81.35% |  89.13%
Table 5: Ablation study on the number of heads for DiverseHead with dynamic freezing |  # of heads |  OA |  UA |  PA |  mIoU |  $F_{1}$
---|---|---|---|---|---|---
Potsdam |  8 |  85.70% |  78.47% |  82.09% |  68.82% |  80.24%
10 |  85.37% |  78.43% |  81.86% |  68.49% |  80.11%
|  12 |  85.51% |  77.23% |  83.03% |  68.14% |  80.02%
RoadNet |  8 |  94.28% |  80.20% |  92.23% |  75.59% |  85.79%
10 |  95.36% |  87.99% |  90.31% |  81.35% |  89.13%
|  12 |  95.35% |  85.57% |  92.22% |  80.52% |  88.77%
Average |  8 |  89.99% |  79.33% |  87.16% |  72.21% |  83.02%
10 |  90.36% |  83.21% |  86.08% |  74.92% |  84.62%
|  12 |  90.43% |  81.40% |  87.62% |  74.33% |  84.40%

An ablation study was conducted on the Potsdam and RoadNet datasets to explore
the impact of the number of heads for the proposed DiverseHead method with
dynamic freezing perturbation. The results in Table 5 indicate that using 10
heads yields the optimal choice across most metrics in terms of average
performance. The performance difference between the versions with different
numbers of heads is very small on Potsdam, but the version of eight heads
shows a performance drop. Considering the stability, We chose the 10-head
version of DiverseHead (DF) for all experiments in this paper.

Table 6: Performance comparison of the DiverseModel with its constituent networks  Model |  OA |  UA |  PA |  mIoU |  $F_{1}$
---|---|---|---|---|---
UNet |  83.28% |  74.18% |  78.88% |  64.51% |  76.46%
SegNet |  82.37% |  73.53% |  77.46% |  63.24% |  75.45%
PSPNet |  81.96% |  74.23% |  76.99% |  63.08% |  75.58%
DiverseModel |  85.76% |  76.75% |  83.45% |  67.85% |  79.96%

Since the DiverseModel technique uses three different segmentation networks,
we also evaluated the performance of each individual member network within the
DiverseModel in Table 6. The results indicate that the cross-supervision of
all networks shows a significant improvement in the segmentation task
performance compared to each separate network.

Figure 5: Visual results of each method on Potsdam Dataset. Values in sub-
captions refer to the IoU scores for each method of this case. Figure 6:
Visual results of each method on the RoadNet Dataset. Values in sub-captions
refer to the IoU scores for each method of this case. Figure 7: Visual results
of each method on the DFC2020 Dataset. Values in sub-captions refer to the IoU
scores for each method of this case. Figure 8: Visual results of each method
on Massachusetts Dataset. Values in sub-captions refer to the IoU scores for
each method of this case.

###  6.3 Qualitative Results and Analysis

The visible results are presented in Figures 5 to 8. For each dataset, we
chose one case and presented the RGB image, ground truth, and predictions of
all the methods. Compared to the ground truth, the proposed three methods
(right-top 3 sub-figures) achieve better similarity to the ground truth
compared to the reference models. We also calculated the IoU scores for all
predictions with the ground truth for this case, which are shown in the sub-
caption of each prediction. The IoU scores also demonstrate that the
performance of the proposed DiverseNet methods is more competitive than that
of the other approaches.

##  7 Conclusion

In this paper, we proposed a lightweight semi-supervised learning based on a
multi-head structure called DiverseHead, which used a dynamic freezing and
dropout perturbation approaches. The multi-head idea benefits from bagging and
shows competitive performance for the segmentation of remote sensing imagery.
Also, another high-efficiency multiple network based semi-supervised learning
method called DiverseModel is investigated for further analysis. DiverseModel
shows the strongest performance on various datasets compared to the state of
the arts but requires more computational resources, whilst the performance of
DiverseHead is slightly lower than DiverseModel but dramatically lighter than
all reference models.

## References

  * Badrinarayanan et al. (2017) Badrinarayanan, V., Kendall, A., Cipolla, R., 2017.  SegNet: A deep convolutional encoder-decoder architecture for image segmentation.  IEEE Transactions on Pattern Analysis and Machine Intelligence 39, 2481–2495.
  * Booth et al. (2023) Booth, H., Ma, W., Karakuş, O., 2023.  High-precision density mapping of marine debris and floating plastics via satellite imagery.  Scientific Reports 13, 6822\.
  * Breiman (1996) Breiman, L., 1996.  Bagging predictors.  Machine learning 24, 123–140.
  * Chen and Wang (2010) Chen, K., Wang, S., 2010.  Semi-supervised learning via regularized boosting working on multiple semi-supervised assumptions.  IEEE Transactions on Pattern Analysis and Machine Intelligence 33, 129–143.
  * Chen et al. (2017) Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L., 2017.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  IEEE transactions on pattern analysis and machine intelligence 40, 834–848.
  * Chen et al. (2018) Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H., 2018.  Encoder-decoder with atrous separable convolution for semantic image segmentation, in: Proceedings of the European Conference on Computer Vision (ECCV), pp. 801–818.
  * Chen et al. (2021) Chen, X., Yuan, Y., Zeng, G., Wang, J., 2021.  Semi-supervised semantic segmentation with cross pseudo supervision, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2613–2622.
  * DeVries and Taylor (2017) DeVries, T., Taylor, G.W., 2017\.  Improved regularization of convolutional neural networks with cutout.  arXiv preprint arXiv:1708.04552 .
  * Dong et al. (2019) Dong, S., Zhuang, Y., Yang, Z., Pang, L., Chen, H., Long, T., 2019.  Land cover classification from vhr optical remote sensing images by feature ensemble deep learning network.  IEEE Geoscience and Remote Sensing Letters 17, 1396–1400.
  * Filipiak et al. (2021) Filipiak, D., Tempczyk, P., Cygan, M., 2021.  $n$-CPS: Generalising cross pseudo supervision to $n$ networks for semi-supervised semantic segmentation.  arXiv preprint arXiv:2112.07528 .
  * French et al. (2019) French, G., Laine, S., Aila, T., Mackiewicz, M., Finlayson, G., 2019.  Semi-supervised semantic segmentation needs strong, varied perturbations.  arXiv preprint arXiv:1906.01916 .
  * Ghandorh et al. (2022) Ghandorh, H., Boulila, W., Masood, S., Koubaa, A., Ahmed, F., Ahmad, J., 2022\.  Semantic segmentation and edge detection—approach to road detection in very high resolution satellite images.  Remote Sensing 14, 613\.
  * Hu et al. (2021) Hu, H., Wei, F., Hu, H., Ye, Q., Cui, J., Wang, L., 2021.  Semi-supervised semantic segmentation via adaptive equalization learning.  Advances in Neural Information Processing Systems 34, 22106–22118.
  * Huang et al. (2021) Huang, A., Wang, Z., Zheng, Y., Zhao, T., Lin, C.W., 2021.  Embedding regularizer learning for multi-view semi-supervised classification.  IEEE Transactions on Image Processing 30, 6997–7011.
  * Ke et al. (2020) Ke, Z., Qiu, D., Li, K., Yan, Q., Lau, R.W., 2020.  Guided collaborative training for pixel-wise semi-supervised learning, in: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIII 16, Springer. pp. 429–445.
  * Kikaki et al. (2022) Kikaki, K., Kakogeorgiou, I., Mikeli, P., Raitsos, D.E., Karantzalos, K., 2022.  MARIDA: A benchmark for Marine Debris detection from Sentinel-2 remote sensing data.  PloS one 17, e0262247.
  * Kim et al. (2020) Kim, J., Hur, Y., Park, S., Yang, E., Hwang, S.J., Shin, J., 2020.  Distribution aligning refinery of pseudo-label for imbalanced semi-supervised learning.  Advances in neural information processing systems 33, 14567–14579.
  * Lee et al. (2013) Lee, D.H., et al., 2013.  Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks, in: Workshop on challenges in representation learning, ICML, Atlanta. p. 896.
  * Liu et al. (2018) Liu, Y., Yao, J., Lu, X., Xia, M., Wang, X., Liu, Y., 2018.  Roadnet: Learning to comprehensively analyze road networks in complex urban scenes from high-resolution remotely sensed images.  IEEE Transactions on Geoscience and Remote Sensing 57, 2043–2056.
  * Long et al. (2015) Long, J., Shelhamer, E., Darrell, T., 2015.  Fully convolutional networks for semantic segmentation, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431–3440.
  * Luo et al. (2022) Luo, X., Hu, M., Song, T., Wang, G., Zhang, S., 2022\.  Semi-supervised medical image segmentation via cross teaching between cnn and transformer, in: International Conference on Medical Imaging with Deep Learning, PMLR. pp. 820–833.
  * Ma et al. (2023) Ma, W., Karakus, O., Rosin, P.L., 2023.  Confidence-guided semi-supervised learning in land cover classification.  arXiv preprint arXiv:2305.10344 .
  * McIntosh-Smith (2014) McIntosh-Smith, S., 2014.  GW4 Isambard.  https://gw4.ac.uk/isambard/.  Accessed: 2023-10-1.
  * Mnih (2013) Mnih, V., 2013.  Machine Learning for Aerial Image Labeling.  Ph.D. thesis. University of Toronto.
  * Ouali et al. (2020) Ouali, Y., Hudelot, C., Tami, M., 2020.  Semi-supervised semantic segmentation with cross-consistency training, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12674–12684.
  * Ren et al. (2016) Ren, Y., Zhang, L., Suganthan, P.N., 2016.  Ensemble classification and regression-recent developments, applications and future directions.  IEEE Computational intelligence magazine 11, 41–53.
  * Robinson et al. (2021) Robinson, C., Malkin, K., Jojic, N., Chen, H., Qin, R., Xiao, C., Schmitt, M., Ghamisi, P., Hänsch, R., Yokoya, N., 2021\.  Global land-cover mapping with weak supervision: Outcome of the 2020 ieee grss data fusion contest.  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 14, 3185–3199.
  * Ronneberger et al. (2015) Ronneberger, O., Fischer, P., Brox, T., 2015.  U-Net: Convolutional networks for biomedical image segmentation, in: Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, Springer. pp. 234–241.
  * Rottensteiner et al. (2012) Rottensteiner, F., Sohn, G., Jung, J., Gerke, M., Baillard, C., Benitez, S., Breitkopf, U., 2012.  The ISPRS benchmark on urban object classification and 3D building reconstruction.  ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences; I-3 1, 293–298.
  * Schmitt et al. (2019) Schmitt, M., Hughes, L.H., Qiu, C., Zhu, X.X., 2019.  Sen12ms–a curated dataset of georeferenced multi-spectral sentinel-1/2 imagery for deep learning and data fusion.  arXiv preprint arXiv:1906.07789 .
  * Selvaraju et al. (2020) Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D., 2020\.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  International Journal of Computer Vision 128, 336–359.
  * Sewell (2008) Sewell, M., 2008.  Ensemble learning.  RN 11, 1–34.
  * Tarvainen and Valpola (2017) Tarvainen, A., Valpola, H., 2017\.  Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.  Advances in Neural Information Processing Systems 30\.
  * Vali et al. (2020) Vali, A., Comai, S., Matteucci, M., 2020.  Deep learning for land use and land cover classification based on hyperspectral and multispectral earth observation data: A review.  Remote Sensing 12, 2495\.
  * Wang et al. (2021a) Wang, C., Zhao, S., Zhu, L., Luo, K., Guo, Y., Wang, J., Liu, S., 2021a.  Semi-supervised pixel-level scene text segmentation by mutually guided network.  IEEE Transactions on Image Processing 30, 8212–8221.
  * Wang et al. (2022a) Wang, J.X., Chen, S.B., Ding, C.H., Tang, J., Luo, B., 2022a.  Semi-supervised semantic segmentation of remote sensing images with iterative contrastive network.  IEEE Geoscience and Remote Sensing Letters 19, 1–5.
  * Wang et al. (2021b) Wang, L., Liu, Y., Di, H., Qin, C., Sun, G., Fu, Y., 2021b.  Semi-supervised dual relation learning for multi-label classification.  IEEE Transactions on Image Processing 30, 9125–9135.
  * Wang et al. (2022b) Wang, Y., Wang, H., Shen, Y., Fei, J., Li, W., Jin, G., Wu, L., Zhao, R., Le, X., 2022b.  Semi-supervised semantic segmentation using unreliable pseudo-labels, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4248–4257.
  * Xia et al. (2021) Xia, M., Wang, T., Zhang, Y., Liu, J., Xu, Y., 2021\.  Cloud/shadow segmentation based on global attention feature fusion residual network for remote sensing imagery.  International Journal of Remote Sensing 42, 2022–2045.
  * Yun et al. (2019) Yun, S., Han, D., Oh, S.J., Chun, S., Choe, J., Yoo, Y., 2019.  Cutmix: Regularization strategy to train strong classifiers with localizable features, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6023–6032.
  * Zhang et al. (2021) Zhang, B., Wang, Y., Hou, W., Wu, H., Wang, J., Okumura, M., Shinozaki, T., 2021\.  FlexMatch: Boosting semi-supervised learning with curriculum pseudo labeling.  Advances in Neural Information Processing Systems 34, 18408–18419.
  * Zhang et al. (2019) Zhang, C., Li, G., Du, S., 2019.  Multi-scale dense networks for hyperspectral remote sensing image classification.  IEEE Transactions on Geoscience and Remote Sensing 57, 9201–9222.
  * Zhang et al. (2017) Zhang, H., Cisse, M., Dauphin, Y.N., Lopez-Paz, D., 2017\.  mixup: Beyond empirical risk minimization.  arXiv preprint arXiv:1710.09412 .
  * Zhao et al. (2017) Zhao, H., Shi, J., Qi, X., Wang, X., Jia, J., 2017\.  Pyramid scene parsing network, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2881–2890.
  * Zheng et al. (2022) Zheng, H., Gong, M., Liu, T., Jiang, F., Zhan, T., Lu, D., Zhang, M., 2022.  Hfa-net: High frequency attention siamese network for building change detection in vhr remote sensing images.  Pattern Recognition 129, 108717\.
  * Zhu et al. (2021) Zhu, Y., Zhang, Z., Wu, C., Zhang, Z., He, T., Zhang, H., Manmatha, R., Li, M., Smola, A.J., 2021\.  Improving semantic segmentation via efficient self-training.  IEEE transactions on pattern analysis and machine intelligence .