# Active Learning for Improved Semi-Supervised Semantic Segmentation in
Satellite Images

Shasvat Desai
Orbital Insight
<EMAIL_ADDRESS>Authors contributed equally Debasmita Ghose
††footnotemark:
Yale University
<EMAIL_ADDRESS>
###### Abstract

Remote sensing data is crucial for applications ranging from monitoring forest
fires and deforestation to tracking urbanization. Most of these tasks require
dense pixel-level annotations for the model to parse visual information from
limited labeled data available for these satellite images. Due to the dearth
of high-quality labeled training data in this domain, there is a need to focus
on semi-supervised techniques. These techniques generate pseudo-labels from a
small set of labeled examples which are used to augment the labeled training
set. This makes it necessary to have a highly representative and diverse
labeled training set. Therefore, we propose to use an active learning-based
sampling strategy to select a highly representative set of labeled training
data. We demonstrate our proposed method’s effectiveness on two existing
semantic segmentation datasets containing satellite images: UC Merced Land Use
Classification Dataset and DeepGlobe Land Cover Classification Dataset. We
report a 27% improvement in mIoU with as little as 2% labeled data using
active learning sampling strategies over randomly sampling the small set of
labeled training data.

a) b)

c) d)

Figure 1: a) An image from the UC Merced Land Use Classification Dataset [63],
b) Ground Truth of the same image provided by the DLSRD dataset [48], c)
Baseline semi-supervised semantic segmentation model trained with 2% labeled
data, d) Output of our active learning based semi-supervised semantic
segmentation model trained with 2% labeled data.

##  1 Introduction

Semantic segmentation has found vast applications in the domain of remote
sensing, including but not limited to environmental monitoring [64, 66], land
use classification, and change detection [11, 28, 4, 12, 18, 55]. The largest
barrier to applying these segmentation techniques is the availability of
representative labeled data across different geographies and terrains. Each
pixel in a satellite image can represent a large area on the ground, thus
requiring domain knowledge and experience to annotate pixel-level labels. This
makes it significantly expensive in terms of cost and time to collect a large
set of pixel-wise labels [23]. To alleviate this problem, recent work in the
computer vision community has explored using fewer pixel-wise labels along
with information from unlabeled images in a semi-supervised fashion [17, 32,
52]. However, these small sets of images that are labeled pixel-wise are
chosen randomly from a dataset [17, 32]. This might bias the semi-supervised
network towards a particular set of classes, degrading its performance.
Therefore, we propose to use active learning to select a representative set of
labeled examples for semi-supervised semantic segmentation for land cover
classification.

This work is the first to explore a semi-supervised approach to semantic
segmentation in satellite images to the best of our knowledge. We use a
conditional GAN [31] based on Mittal et al. [32] which takes in a small number
of labeled examples and a large unlabeled pool of data. This conditional GAN
generates pseudo-labels based on limited labeled examples to augment the
labeled pool. This makes it essential to have a diverse set of labeled
training data. Thus, we propose to use active learning to select a highly
representative set of labeled training samples.

Active learning aims to select the most informative and representative data
instances for labeling from an unlabeled data pool based on some information
measure. We sample a subset of the images and their corresponding labels at
random from a dataset, which serves as our labeled training set for the
conditional GAN. We do the sampling again using an active learning-based
sampling strategy which would provide a more diverse set of training data and
show a performance improvement even when only very few training samples are
available. With as little as 2% labeled data, we report an improvement of up
to 27% in mIoU over random sampling. We demonstrate our proposed method’s
efficacy on two existing semantic segmentation datasets containing satellite
images: UC Merced Land Use Classification Dataset [48, 63], and DeepGlobe Land
Cover Classification Dataset [9].

Active learning for semantic segmentation [29, 60] yields patches of the given
input image that are most informative. However, in this work, we require an
active learning-based sampling strategy that gives us the set of most
informative images from the given dataset. To achieve this, we propose using
active learning for image classification to select entire images from the
given dataset, which are the most informative. We then query and obtain dense-
pixel level annotations only for the actively selected samples, giving us our
diverse labeled training data for semi-supervised semantic segmentation.

Finally, we propose this method for sample selection to act as a guiding
process for large-scale dataset creation, requiring the collection of dense
pixel-level annotations. It would require significantly less cost and effort
to obtain coarse image-level labels for the images and then use our proposed
methodology to sample informative images labeled at pixel-level using image-
classification-based active learning. The code has been made publicly
available***https://github.com/immuno121/ALS4GAN.

Our key contributions are summarized as follows:

  * •

We use pool-based active learning sampling strategies to intelligently select
labeled examples and improve performance for a GAN-based semi-supervised
semantic segmentation network for satellite images.

  * •

We demonstrate the applicability of the proposed method for selecting an
optimal subset of data instances for which pixel-level annotations should be
obtained.

##  2 Background and Related Work

###  2.1 Active Learning

Active Learning is a technique that uses a learning algorithm which learns to
select samples from an unlabeled pool of data for which the labels should be
queried.

Scenarios for Active Learning: Active Learning is typically employed in the
following settings: [44]. Membership Query Synthesis [45] is a setting where
the learner generates an instance from an underlying distribution. Stream-
based Selective Sampling [1] queries each unlabeled instance one at a time
based on some information measure. The last scenario Pool-based sampling, used
in this paper, assumes a large pool of unlabeled data and draws instances from
the pool according to some information measure.

Query Strategies: There are several query strategies in the pool-based
sampling strategies to select samples for which we need to query labels.

The margin sampling strategy [2, 40] selects the instance that has the
smallest difference between the first and second most probable labels.

$\displaystyle x_{M}$
$\displaystyle=\underset{x}{\arg\max}[P_{\theta}({\hat{y_{2}}}|x)-P_{\theta}({\hat{y_{1}}}|x)]$
(1)

Intuitively, instances with small margins are more ambiguous and knowing the
true label should help the model discriminate more effectively.

The third and the most common strategy is Entropy-based sampling [24, 25].

$\displaystyle x_{H}^{*}$
$\displaystyle=\underset{x}{\arg\max}-\sum{{}_{y}}P_{\theta}(y|x)\log(P_{\theta}(y|x))$
(2)

where $y$ ranges over all possible labels of $x$. Entropy is a measure of a
variable’s average information content. So intuitively, this method selects
samples by ranking them based on their information content.

Applications:  Active learning techniques have found numerous applications in
medical imaging [16, 19, 30, 47, 51] and remote sensing [20, 27, 39, 54, 58,
59] communities because obtaining labeled data in those domains have been
particularly challenging [36, 43]. Some recent work have used deep learning
techniques for active learning-based image classification [38, 47], semantic
segmentation [29, 60, 62] and object detection [41]. However, to the best of
our knowledge, this is the first work that uses a deep active learning-based
image classifier to select labeled examples for a semi-supervised semantic
segmentation network.

###  2.2 Semi-Supervised Semantic Segmentation

Most of the existing semi-supervised semantic segmentation techniques either
use consistency regularization strategies [7, 13, 33, 34, 35] or generative
model to augment pseudo-labels to existing labeled data pool [17, 52] or some
combination of the two [32].

Consistency Regularization-based strategies: The core idea in consistency
regularization is that predictions for unlabeled data should be invariant to
perturbations. Some recent work has pointed out the difficulty in performing
consistency regularization for semi-supervised semantic segmentation because
it violates the cluster assumption [13, 35]. Some other work [6, 13, 21] use
data augmentation techniques like CutMix [65] and ClassMix [34], which
composite new images by mixing two original images. They hypothesize that this
would enforce consistency over highly varied mixed samples while respecting
the original images’ semantic boundaries.

Figure 2: Proposed Framework: 1) The active learning module expects an
unlabeled pool of data as its input. It is an image classification network
that returns samples $X_{L}$ selected based on some information measure
determined by the sampling strategy used for the active learner. 2) A
get_label operation is performed to obtain pixel-level labels corresponding to
images returned by the active learning module. 3) A conditional GAN is then
trained where the generator module is a semantic segmentation network and
expects the labeled images returned by the active learning module, $X_{L}$,
the corresponding pixel-wise labels for these samples $Y_{L}^{P}$, along with
the remaining unlabeled images $X_{U}$. It outputs a segmentation mask. 4) The
discriminator expects the predicted segmentation masks from the generator
along with the pixel-wise ground truth labels $Y_{L}^{P}$, and outputs a
prediction confidence score. 5) prediction masks with a score greater than the
predefined confidence threshold $\tau$ are selected and treated as pseudo-
labels to train the GAN and are augmented to the labeled pool as shown by the
”+” sign on the bottom left corner.

GAN-based strategies: Souly et al. [52] was the first work to perform semi-
supervised semantic segmentation using a GAN. They employ the generator to
generate realistic visual data that forced the discriminator to learn better
features for more accurate pixel classification. However, these generated
images were not sufficiently close to the real images since it is challenging
to generate realistic-looking images from pixel-wise maps.

To overcome the drawback of poorly generated images, Hung et al. [17] propose
a conditional GAN. The generator is a standard semantic segmentation network
that takes in images and their ground truth maps. The discriminator is a fully
convolutional network (FCN) that takes the ground truth, and the segmentation
map predicted by the generator and aims to distinguish between the two. Thus,
it is difficult for the discriminator to determine if the pixels belong to the
real or the fake distribution by looking at one pixel at a time without
context.

Mittal et al. [32] propose to replace the FCN-based discriminator with an
image-wise discriminator that determines if the image belongs to the real or
the fake distribution, which is a relatively easy task. Additionally, they
propose to use a supervised multi-label classification branch [56] which
decides on the classes present in the image and thus aids the segmentation
network to make globally consistent decisions. During evaluation, they fuse
the two branches to alleviate both low-level and high-level artifacts that
often occur when working in a low-data regime. In this work, we use the s4GAN
branch of the network presented by Mittal et al. [32] and propose to select a
more optimal set of labeled examples to improve the performance of the network
over a random selection of labeled examples.

##  3 Our Method

We propose to use active learning techniques to select a small informative
subset of labeled data that would help the semi-supervised semantic
segmentation model learn more effectively with a representative pool of
labeled data. The proposed framework is demonstrated in Figure 2. It should be
noted that for any given image, our method assumes the ability to gain access
to its corresponding image and pixel-level annotations.

Input:

1 Labeled ratio $R$ and unlabeled pool of data $X_{N}$

Output:

2 Informative samples and their image-level labels ($X_{L}$,$Y_{L}^{I}$)

Define: $learner\leftarrow\text{Neural network based image classifier}$

3 $oracle\leftarrow\text{source of labels}$

4 Number of data points to sample: $X_{N_{L}}\leftarrow R*X_{N}$

5 Size of initial labeled pool for learner:
$init\\_size\leftarrow\lceil\alpha_{init}*X_{N_{L}}\rceil$

6 Initial labeled pool for active learner:($X_{init}$, $Y_{init}$)

7 Train the active learner: $learner(X_{init},Y_{init})$

8 Unlabeled pool: $X_{pool}\leftarrow X_{N}-X_{init}$

9 $X_{L}\leftarrow\\{\\}$, $Y_{L}^{I}\leftarrow\\{\\}$

10 Number of samples to query in each iteration :
$N_{Q}=\beta_{Q}*init\\_size$

while  _$n(X_{L})\leq X_{N_{L}}$_ do

11 Query Step:

Inference on unlabeled pool using $learner$:

12 $prediction\\_scores=learner(X_{pool})$

Select top-Q most informative instances ($N_{Q}$):

13 $X_{Q}$ = $sampling\\_strategy(prediction\\_scores)$

14 $Y_{Q}=oracle(X_{Q})$

15 $X_{pool}=X_{pool}-X_{Q}$

16 $X_{L}=X_{L}\bigcup X_{Q}$

17 $Y_{L}^{I}=Y_{L}^{I}\bigcup Y_{Q}$

Teach Step:

Retrain the $learner$ with updated labeled pool:

$learner(X_{L},Y_{L}^{I})$

18 end while

return $X_{L},Y_{L}^{I}$

Algorithm 1 Active Learning for Labeled Sample Selection

###  3.1 Active Learning for Image Classification

Algorithm 1 describes how active learning was used to select the most
informative and diverse set of labeled samples for semi-supervised semantic
segmentation. We use active image classification and sample images using pool-
based sampling strategies [44] as described in Section 2.1. Algorithm 1 is
used as an offline process to sample informative samples and it accepts two
inputs: labeled ratio $R$, and an unlabeled pool of data, $X_{N}$. The labeled
ratio $R$, determines the number of labeled samples used to train the semi-
supervised model. The labeled ratio $R$, used in this paper for each dataset,
can be found in Table 1.

Initialization: The active learner is initialized with $init\\_size$ number of
image-level labels (line 1), which is a function of the number of labeled
samples to be returned. We define the parameter $\alpha_{init}\in(0,1]$ (line
1) to control the size of the initial labeled pool of the active learner.
$\alpha_{init}$ helps in determining the optimal size of the labeled pool that
the active learner should be initialized with for every labeled ratio, $R$. A
low value of $\alpha_{init}$ would result in the active learner being
initialized with a tiny pool of data not providing sufficient information
about the data distribution. In contrast, a large value of $\alpha_{init}$
might bias the active learner toward a particular set of initial samples,
which might lead to under-sampling of a particular class. Intuitively, setting
$\alpha_{init}$ to 1 would make to outcome close to being equivalent to random
sampling. We found this approach to perform better than having a fixed initial
labeled pool size, irrespective of the labeled ratio, $R$. The $learner$ is
then trained with this initial labeled pool (line 1). Once the initial labeled
samples are selected for the active learner, they are removed from the
unlabeled pool (line 1). We sample data instances and their labels by
performing the query and teach steps in an interleaved fashion.

Query Step: We run inference using the trained $learner$ on the entire
unlabeled pool (line 1) and obtain the model’s confidence scores for each
sample in that pool. Then using some uncertainty measure based on the active
learning strategy used, the oracle queries image-level labels for top-Q
uncertain samples, $N_{Q}$. For instance, if entropy-based sampling [24, 25]
is used, then the oracle will return labels for samples with the highest
entropy measure. The optimal number of data instances $N_{Q}$, queried from
the $oracle$ in every iteration is a function of the initial labeled pool size
$init\\_size$ and another parameter, $\beta_{Q}\in(0,1]$ (line 1). We define
$\beta_{Q}$ to determine the number of iterations for which the active learner
will be trained. It is crucial for the active learner’s performance because a
small value of $\beta_{Q}$ will add only a small number of labels in each
iteration, resulting in a negligible weight update of the active learner. In
contrast, a large value of $\beta_{Q}$ will cause a massive update in the
learner’s weights at every step. It will also reduce the total number of steps
the learning algorithm will take to reach its target ${X_{N_{L}}}$. This will
leave little room for the learner to learn from its mistakes in each
iteration, directly impacting the quality labels produced. Once the labels for
$N_{Q}$ number of samples are queried from $oracle$, the images and their
corresponding labels are added to the result set, $X_{L}$ and $Y_{L}^{I}$
(lines 1, 1).

Teach Step: In this step, the $learner$ is trained with the updated labeled
pool of samples obtained from the query step (line 1). The image
classification network’s capacity for the $learner$ is also crucial in
determining the quality of the selected samples. Any network with low capacity
tends to underfit, while any network with a higher capacity than required
could overfit and detrimentally affect the downstream task’s performance.

Labeled Ratio(R) |  2% |  5% |  12.5% |  100%
---|---|---|---|---
UC Merced[63] |  34 |  85 |  211 |  1680
DeepGlobe[9] |  12 |  32 |  80 |  642
Table 1: Number of Labeled Examples per Labeled Ratio in the UC Merced and
DeepGlobe datasets

###  3.2 Semi-Supervised Semantic Segmentation

We use the s4GAN network proposed by Mittal et al. [32] for performing semi-
supervised semantic segmentation using a small number of pixel-wise labeled
examples along with a pool of unlabeled examples. This is a conditional GAN-
based technique where the generator $G$ is a segmentation network. The
generator takes in all the labeled and unlabeled images, along with the ground
truth masks. The discriminator $D$ takes the predicted segmentation map and
the available ground truth masks concatenated with their respective images.
The network attempts to match the real and the predicted segmentation maps’
distribution through adversarial training.

Notation:
$x_{L}^{P},y_{L}^{P}$: image with their pixel-wise labels
$x_{U}^{P}$: image with no pixel-wise ground-truth labels

####  3.2.1 Segmentation Network (Generator)

The segmentation network S is trained with loss $L_{S}$, which is a
combination of three losses: the standard cross-entropy loss, the feature
matching loss, and the self-training loss. Cross Entropy Loss: Standard
supervised pixel-wise cross entropy loss term evaluated only for the labeled
samples $x_{L}^{p}$ is shown is Equation 3.

$L_{ce}=-\sum y_{L}^{P}\cdot log(S(x_{L}^{P}))$ (3)

Feature-Matching Loss:  The feature matching loss $L_{fm}$ [42] aims to
minimize the mean discrepancy between the feature statistics of the predicted,
$S(x_{U}^{P})$ and the ground truth segmentation maps, $y_{L}^{P}$ as shown in
Equation 4. This loss uses both labeled and unlabeled training examples.

$L_{fm}=||\mathbb{E}_{(y_{L}^{P},x_{L}^{P})\sim D_{l}}D(x_{L}^{P}\oplus
y_{L}^{P})-\\\ \mathbb{E}_{(x_{U}^{P})\sim D_{u}}D(x_{U}^{P}\oplus
S(x_{U}^{P}))||$ (4)

Self-Training Loss:  This loss is used for only unlabeled data. This loss aims
to pick the best outputs of the segmentation network (i.e., those outputs that
could fool the discriminator) that do not have a corresponding ground truth
mask and reuse them for supervised training. Intuitively, it pushes the
segmentation network to produce predictions that the discriminator cannot
distinguish from real. The discriminator’s output is a score between 0 and 1,
denoting the discriminator’s confidence that the predicted segmentation mask
is real. The predicted segmentation mask with a score greater than the
predefined confidence threshold $\tau$, is selected and treated as a pseudo-
label to train the GAN.

Equation 5 describes the self-training loss.

$L_{st}=\begin{cases}-\sum{\textbf{y*}\cdot log(S(x_{U}^{P}))}&\text{if
}D(x_{U}^{P})\geq\tau\\\ 0&\text{otherwise}\end{cases}$ (5)

y* = pseudo pixel-wise labels which are the predictions of the segmentation
network
Finally, the objective function for the generator is given by Equation 6.

$L_{S}=L_{ce}+\lambda_{fm}L_{fm}+\lambda_{st}L_{st}$ (6)

where, $\lambda_{fm}>0$ and $\lambda_{st}>0$ are the weighting parameters for
the feature matching and the self-training losses.

####  3.2.2 Discriminator

The discriminator is trained to distinguish between the real labeled examples
and the fake segmentation masks generated by the network concatenated with the
corresponding input images. It is trained using the original GAN loss proposed
by Goodfellow et al. [14] as shown in Equation 7.

$L_{D}=\mathbb{E}_{(y_{L}^{P},x_{L}^{P})\sim D_{l}}logD(x_{L}^{P}\oplus
y_{L}^{P})+\\\ \mathbb{E}_{(x_{U}^{P})\sim D_{u}}log(1-D(x_{U}^{P}\oplus
S(x_{U}^{P}))$ (7)

###  3.3 Labeled Example Selection for Semi-Supervised Semantic Segmentation
using Active Learning

To obtain labeled samples for a semi-supervised semantic segmentation network,
the proposed framework uses the active learning module from Algorithm 1
defined as $Active\\_Sampler$ in Algorithm 2 and shown in Figure 2. The active
learning module expects an unlabeled pool of data ($X_{N}$) and the labeled
ratio $R$ as its input. It returns samples $X_{N_{L}}$ (line 2), which are
selected based on some information measure determined by the sampling strategy
used for the active learner. The active learning module is called only once to
select informative data instances for the semi-supervised model. In the
get_label stage, we obtain pixel-level labels corresponding to only those
images returned by the $Active\\_Sampler$ to serve as the initial labeled
training data for the semi-supervised segmentation module. This enables the
semi-supervised semantic segmentation network to learn with a representative
labeled set instead of a random subset of the data.

The conditional GAN described in Section 3.2 is then trained, where the
generator module is a semantic segmentation network and expects the labeled
images returned by the active learning module $X_{N_{L}}$, the corresponding
pixel-wise labels for these samples $Y_{N_{L}}^{P}$, along with the remaining
unlabeled images, $X_{N_{U}}$. The output of the generator network is a
segmentation mask. The discriminator expects the predicted masks along with
the pixel-wise ground truth labels $Y_{N_{L}}^{P}$, and outputs a probability
score between 0 and 1, denoting its confidence in the predicted mask being
real and belonging to the ground truth. If this confidence score is greater
than the predefined confidence threshold $\tau$, then it implies the generator
has successfully predicted a mask that appears real to the discriminator.
Hence, this prediction is augmented to the ground-truth, $Y_{N_{L}}^{P}$ (line
2) and used as a pseudo-label, and the GAN is trained with this updated
dataset. These pseudo-labels contribute to the self-training loss detailed in
Section 3.2.1. The generator and the discriminator are trained adversarially
until a stopping criterion is satisfied.

2%
5%
12.5%

a) Original Image b) Ground Truth c) Baseline d) Our Results

Figure 3: Qualitative Results from the UC Merced Land Use Classification
Dataset for different labeled ratios

2%
5%
12.5%

a) Original Image b) Ground Truth c) Baseline d) Our Results

Figure 4: Qualitative Results from the DeepGlobe Land Cover Classification
Dataset for different labeled ratios

Input:

1 Labeled Ratio, R and unlabeled pool, $X_{N}$

Define:

2 $Active\\_Sampler\leftarrow\text{Active Learning module}$

3 $\tau\leftarrow\text{Confidence Threshold}$

4 $G\leftarrow\text{Generator network of conditional GAN}$

5 $D\leftarrow\text{Discriminator network of conditional GAN}$

6 Active sampler from Algorithm 1: $X_{N_{L}},\\_=Active\\_Sampler(R,X_{N})$

7 Get_Label: Obtain pixel-wise labels for the images returned by the active
sampler: $Y_{N_{L}}^{P}=get\\_label(X_{N_{L}})$

8 Images without pixel-wise labels: $X_{N_{U}}\leftarrow X_{N}-X_{N_{L}}$

Semi-Supervised Semantic Segmentation:

9 while  _$i <iterations$_ do

10 $pseudo\\_label=\\{\\}$

11 for  _$(x_{N_{L}},x_{N_{U}},y_{N_{L}}^{P})$ in
$(X_{N_{L}},X_{N_{U}},Y_{N_{L}}^{P})$_ do

12 $mask=G(x_{N_{L}},x_{N_{U}},y_{N_{L}}^{P})$

13 $confidence=D(mask,y_{N_{L}}^{P})$

14

15 if  _$confidence >\tau$_ then

16 $pseudo\\_label=pseudo\\_label\bigcup mask$

17

18 end for

19 Augment pseudo-labels to ground-truth and train the Generator model via
self-training loss: $Y_{N_{L}}^{P}=Y_{N_{L}}^{P}\bigcup pseudo\\_label$

20 end while

Algorithm 2 Semi-Supervised semantic segmentation using samples obtained from
the Active Learner

##  4 Experiments and Results

###  4.1 Datasets and Evaluation Metric

UC Merced Land Use Classification Dataset: The UC Merced Land Use
Classification dataset [63] has 2100 RGB images of size 256x256 pixels and
0.3m spatial resolution, with image-level annotations for each of the 21
classes. We use the pixel-level annotations for the UC Merced dataset made
publicly available by Shao et al. [48] which has 17 classes as proposed in
[3]. The dataset was randomly split into training and validation sets with
1680 training images (80%) 420 validation images (20%).
DeepGlobe Land Cover Classification Dataset: The DeepGlobe land cover
classification dataset is comprised of DigitalGlobe Vivid+ images of
dimensions 2448x2448 pixels and spatial resolution of 0.5 m. There are 803
pixel-wise annotated training images, each with pixel-wise label covering
seven land cover classes. Since there are no image-level annotations available
for the DeepGlobe dataset, to generate image-level annotations, we calculate
which class contains the highest number of pixels for every image and assign
that particular coarse class to the image. The dataset was randomly split into
training and validation sets such that 642 training images (80%) and 161
validation images (20%).
Evaluation Metric:  We use mean Intersection-over-Union (mIoU) as our
evaluation metric.

###  4.2 Implementation Details

Active Learning for Image Classification We used ResNet 101 and ResNet 50 [15]
as our image classification networks for the UC Merced and the DeepGlobe
Datasets respectively which were trained using the Cross-Entropy loss. The
network was trained using the SGD optimizer with a base learning rate of 0.001
and momentum of 0.9. We used a step learning rate scheduler where the learning
rate is dropped by a factor of 0.1 every 7 epochs. We used a batch size of 4
and trained for 50 epochs after each query. Through cross-validation, we found
the optimal value of $\alpha_{init}$ = 0.1 and $\beta_{Q}$ = 0.5. We
implemented the network using the skorch [57] framework. The different active
learning query strategies were implemented using the modAL toolbox [8] and
trained on a NVIDIA GTX-2080ti GPU.

Semi-Supervised Semantic Segmentation We use a GAN-based semi-supervised
semantic segmentation technique proposed by Mittal et al. [32]. The generator
is comprised of a segmentation network which in our case is DeepLabv2 [5]
trained with a ResNet-101 [15] backbone pretrained on the ImageNet dataset
[10]. The discriminator is a binary classifier with four convolutional layers
with 4x4 kernels with 64, 128, 256, 512 channels each followed by a Leaky ReLU
activation [61] with negative slope of 0.2 and a dropout layer [53] with
dropout probability of 0.5. The segmentation network in the generator is
trained with SGD optimizer base learning rate of 2.5e-4, momentum of 0.9, and
a weight decay of 5e-4 as described in [17, 32]. The image classification
network in the discriminator is trained using the Adam optimizer [22] with a
base learning rate of 1e-4. Through cross-validation, we found the optimal
loss weights to be $\lambda_{fm}=0.1$ and $\lambda_{st}=1.0$ and the optimal
value of $\tau$ to be 0.6. For the DeepGlobe dataset, we resize each image to
320x320 pixels to reduce the training time. We implemented the network using
PyTorch [37] on NVIDIA Tesla V100 GPUs.

Labeled Ratio(R) |  2% |  5% |  12.5%
---|---|---|---
s4GAN [32] (Baseline) |  0.347 $\pm$ 0.0077 |  0.408 $\pm$ 0.0035 |  0.473 $\pm$ 0.0080
s4GAN + Entropy (Ours) |  0.398 |  0.456 |  0.494
s4GAN + Margin (Ours) |  0.360 |  0.480 |  0.546
Table 2: mIoU Scores for the UC Merced Land Use Classification Dataset [63]

###  4.3 Results and Analysis

Our baseline is a vanilla s4GAN [32] network, where the labeled data is
selected randomly from a given dataset. We report the mean and standard
deviation of mIoUs across three experiments with different random seeds for
robust evaluation for our baseline method. We compare our approach of using
active learning to select representative labeled examples with this baseline.
We experiment with two pool-based query strategies, entropy and margin
sampling and demonstrate qualitative and quantitative performance improvements
on two datasets, DeepGlobe Land Cover Classification Dataset [9], and UC
Merced Land Use Classification Dataset [48, 63] over the stated baseline. We
evaluated our approach with labeled ratios of 2%, 5%, and 12.5%. The
qualitative results in Figures 3 and 4 are shown for the best out of the two
sampling strategies for each labeled ratio. Table 1 shows the number of
labeled images in each dataset for different labeled ratios.

UC Merced Land Use Classification Dataset: Table 2 shows a quantitative
comparison of our method with the baseline for the UC Merced Land Use
Classification Dataset [48, 63]. We compare the performance of entropy and
margin sample selection strategies with the baseline and show significant and
consistent performance improvements. Both the active learning strategies out-
perform the baseline by a significant margin. We report a maximum mIoU
improvement of close to 15% with as little as 2% labeled data, a maximum
improvement of about 18% over the baseline when training with 5% and 12.5%
labeled data across the two active learning strategies. Figure 3 shows how our
proposed method qualitatively improves over the UC Merced Dataset baseline for
different labeled ratios. Our method predicts a finer coastline with no false
positives, even with only as few as 34 labeled images which are 2% of labeled
data (Row 1 of Figure 3). Similarly, we demonstrate that even when using only
5% (85 images) of labeled data (Row 2 of Figure 3). our method predicts the
green river that is camouflaging into the background while the baseline method
completely misses it (Row 2 of Figure 3). This shows the importance of having
a representative pool of labeled data, especially in a low data regime, as is
our case. With 12.5% (211 images) of labeled data (Row 3 of Figure 3), our
method accurately predicts the complex shape of the airplane (Column d), as
opposed to the baseline (Column c), which was confused between multiple
unrelated classes.

DeepGlobe Land Use Classification Dataset: Table 5 shows a quantitative
comparison of our method with the baseline for the DeepGlobe Land Cover
Classification Dataset [9]. We report significant performance improvements
over the baseline using both entropy and margin sampling strategies. We report
a maximum mIoU improvement of close to 27% with as little as 2% labeled data,
a maximum improvement of about 6% over the baseline when training with 5%
labeled data, and an improvement of approximately 8% with 12.5% labeled data
across the two active learning strategies. Figure 4 shows some visualizations
from the DeepGlobe Dataset where it is seen that our method results in fewer
false positives than the baseline.

Labeled Ratio(R) |  2% |  5% |  12.5%
---|---|---|---
s4GAN [32] (Baseline) |  0.403 $\pm$ 0.035 |  0.486 $\pm$ 0.018 |  0.511 $\pm$ 0.007
s4GAN + Entropy (Ours) |  0.469 |  0.513 |  0.554
s4GAN + Margin (Ours) |  0.511 |  0.513 |  0.529
Table 3: mIoU Scores for the DeepGlobe Land Cover Classification Dataset [9]

##  5 Conclusion

This work proposes a method to leverage active learning-based sampling
techniques to improve performance on the downstream task of semi-supervised
semantic segmentation for land cover classification in satellite images. We do
this by intelligently selecting samples for which pixel-wise labels should be
obtained using coarse image classification-based active-learning strategies.
Our method helps the semi-supervised semantic segmentation network start with
an optimal set of labeled examples to help it get the right amount of initial
information to learn the suitable representation. We prototype this method for
a GAN-based semi-supervised semantic segmentation network, where the labeled
images were selected using pool-based active learning strategies. We
demonstrate the efficacy of our method for two satellite image datasets, both
quantitatively and qualitatively, and report sizable performance gains.

## Acknowledgements

We would like to thank our peers who helped us improve our paper with their
feedback, in no particular order - Joseph Weber, Wencheng Wu, Gowdhaman
Sadhasivam, Surya Teja, Rheeya Uppal, Julius Simonelli, Jing Tian, Kaushik
Patnaik.

## References

  * [1] Les E Atlas, David A Cohn, and Richard E Ladner.  Training connectionist networks with queries and selective sampling.  In Advances in neural information processing systems, pages 566–573. Citeseer, 1990.
  * [2] Maria-Florina Balcan, Andrei Broder, and Tong Zhang.  Margin based active learning.  In International Conference on Computational Learning Theory, pages 35–50. Springer, 2007.
  * [3] Bindita Chaudhuri, Begüm Demir, Subhasis Chaudhuri, and Lorenzo Bruzzone.  Multilabel remote sensing image retrieval using a semisupervised graph-theoretic method.  IEEE Transactions on Geoscience and Remote Sensing, 56(2):1144–1158, 2017.
  * [4] Hao Chen and Zhenwei Shi.  A spatial-temporal attention-based method and a new dataset for remote sensing image change detection.  Remote Sensing, 12(10):1662, 2020.
  * [5] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  IEEE transactions on pattern analysis and machine intelligence, 40(4):834–848, 2017.
  * [6] Xiaokang Chen, Yuhui Yuan, Gang Zeng, and Jingdong Wang.  Semi-supervised semantic segmentation with cross pseudo supervision, 2021\.
  * [7] Ying Chen, Xu Ouyang, Kaiyue Zhu, and Gady Agam.  Mask-based data augmentation for semi-supervised semantic segmentation.  arXiv preprint arXiv:2101.10156, 2021.
  * [8] Tivadar Danka and Peter Horvath.  modAL: A modular active learning framework for Python.  available on arXiv at https://arxiv.org/abs/1805.00979.
  * [9] Ilke Demir, Krzysztof Koperski, David Lindenbaum, Guan Pang, Jing Huang, Saikat Basu, Forest Hughes, Devis Tuia, and Ramesh Raskar.  Deepglobe 2018: A challenge to parse the earth through satellite images.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 172–181, 2018.
  * [10] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [11] M Dharani and G Sreenivasulu.  Land use and land cover change detection by using principal component analysis and morphological operations in remote sensing applications.  International Journal of Computers and Applications, pages 1–10, 2019.
  * [12] Huihui Dong, Wenping Ma, Yue Wu, Jun Zhang, and Licheng Jiao.  Self-supervised representation learning for remote sensing image change detection based on temporal prediction.  Remote Sensing, 12(11):1868, 2020.
  * [13] Geoffrey French, Samuli Laine, Timo Aila, Michal Mackiewicz, and Graham Finlayson.  Semi-supervised semantic segmentation needs strong, varied perturbations.  In British Machine Vision Conference, number 31, 2020.
  * [14] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  Advances in neural information processing systems, 27:2672–2680, 2014.
  * [15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Identity mappings in deep residual networks.  In European conference on computer vision, pages 630–645. Springer, 2016.
  * [16] Steven CH Hoi, Rong Jin, Jianke Zhu, and Michael R Lyu.  Batch mode active learning and its application to medical image classification.  In Proceedings of the 23rd international conference on Machine learning, pages 417–424, 2006.
  * [17] Wei-Chih Hung, Yi-Hsuan Tsai, Yan-Ting Liou, Yen-Yu Lin, and Ming-Hsuan Yang.  Adversarial learning for semi-supervised semantic segmentation.  arXiv preprint arXiv:1802.07934, 2018.
  * [18] Gong Jianya, Sui Haigang, Ma Guorui, and Zhou Qiming.  A review of multi-temporal remote sensing data change detection algorithms.  The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 37(B7):757–762, 2008.
  * [19] Ajay J Joshi, Fatih Porikli, and Nikolaos P Papanikolopoulos.  Scalable active learning for multiclass image classification.  IEEE transactions on pattern analysis and machine intelligence, 34(11):2259–2273, 2012.
  * [20] Benjamin Kellenberger, Diego Marcos, Sylvain Lobry, and Devis Tuia.  Half a percent of labels is enough: Efficient animal detection in uav imagery using deep cnns and active learning.  IEEE Transactions on Geoscience and Remote Sensing, 57(12):9524–9533, 2019.
  * [21] Jongmok Kim, Jooyoung Jang, and Hyunwoo Park.  Structured consistency loss for semi-supervised semantic segmentation.  arXiv preprint arXiv:2001.04647, 2020.
  * [22] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [23] Adriana Kovashka, Olga Russakovsky, Li Fei-Fei, and Kristen Grauman.  Crowdsourcing in computer vision.  arXiv preprint arXiv:1611.02145, 2016.
  * [24] David D Lewis and Jason Catlett.  Heterogeneous uncertainty sampling for supervised learning.  In Machine learning proceedings 1994, pages 148–156. Elsevier, 1994\.
  * [25] David D Lewis and William A Gale.  A sequential algorithm for training text classifiers.  In SIGIR’94, pages 3–12. Springer, 1994.
  * [26] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [27] Yaping Lin, George Vosselman, Yanpeng Cao, and Michael Ying Yang.  Active and incremental learning for semantic als point cloud segmentation.  ISPRS Journal of Photogrammetry and Remote Sensing, 169:73–92, 2020\.
  * [28] Xiaoping Liu, Jialv He, Yao Yao, Jinbao Zhang, Haolin Liang, Huan Wang, and Ye Hong.  Classifying urban land use by integrating remote sensing and social media data.  International Journal of Geographical Information Science, 31(8):1675–1696, 2017.
  * [29] Radek Mackowiak, Philip Lenz, Omair Ghori, Ferran Diego, Oliver Lange, and Carsten Rother.  Cereals-cost-effective region-based active learning for semantic segmentation.  arXiv preprint arXiv:1810.09726, 2018.
  * [30] Dwarikanath Mahapatra, Behzad Bozorgtabar, Jean-Philippe Thiran, and Mauricio Reyes.  Efficient active learning for image classification and segmentation using a sample selection and conditional generative adversarial network.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 580–588. Springer, 2018.
  * [31] Mehdi Mirza and Simon Osindero.  Conditional generative adversarial nets.  CoRR, abs/1411.1784, 2014.
  * [32] Sudhanshu Mittal, Maxim Tatarchenko, and Thomas Brox.  Semi-supervised semantic segmentation with high-and low-level consistency.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019\.
  * [33] Viktor Olsson and Wilhelm Tranheden.  Consistency regularization for semantic segmentation.  2020\.
  * [34] Viktor Olsson, Wilhelm Tranheden, Juliano Pinto, and Lennart Svensson.  Classmix: Segmentation-based data augmentation for semi-supervised learning.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1369–1378, 2020.
  * [35] Yassine Ouali, Céline Hudelot, and Myriam Tami.  Semi-supervised semantic segmentation with cross-consistency training.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12674–12684, 2020.
  * [36] Sakrapee Paisitkriangkrai, Jamie Sherrah, Pranam Janney, and Anton Van Den Hengel.  Semantic labeling of aerial and satellite imagery.  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 9(7):2868–2881, 2016.
  * [37] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer.  Automatic differentiation in pytorch.  2017\.
  * [38] Hiranmayi Ranganathan, Hemanth Venkateswara, Shayok Chakraborty, and Sethuraman Panchanathan.  Deep active learning for image classification.  In 2017 IEEE International Conference on Image Processing (ICIP), pages 3934–3938. IEEE, 2017.
  * [39] Andrés C Rodríguez, Stefano D’Aronco, Konrad Schindler, and Jan D Wegner.  Mapping oil palm density at country scale: An active learning approach.  Remote Sensing of Environment, 261:112479, 2021.
  * [40] Dan Roth and Kevin Small.  Margin-based active learning for structured output spaces.  In European Conference on Machine Learning, pages 413–424. Springer, 2006.
  * [41] Soumya Roy, Asim Unmesh, and Vinay P Namboodiri.  Deep active learning for object detection.  In BMVC, page 91, 2018.
  * [42] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen.  Improved techniques for training gans.  arXiv preprint arXiv:1606.03498, 2016.
  * [43] Jo Schlemper, Ozan Oktay, Michiel Schaap, Mattias Heinrich, Bernhard Kainz, Ben Glocker, and Daniel Rueckert.  Attention gated networks: Learning to leverage salient regions in medical images.  Medical Image Analysis, 53:197–207, 2019.
  * [44] Burr Settles.  Active learning literature survey.  Computer Sciences Technical Report 1648, University of Wisconsin–Madison, 2009.
  * [45] H Sebastian Seung, Manfred Opper, and Haim Sompolinsky.  Query by committee proceedings of 5th annual workshop on computational learning theory, 287–294.  New York, ACM Press, 10:130385–130417, 1992.
  * [46] Claude E Shannon.  A mathematical theory of communication.  The Bell system technical journal, 27(3):379–423, 1948.
  * [47] Wei Shao, Liang Sun, and Daoqiang Zhang.  Deep active learning for nucleus classification in pathology images.  In 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pages 199–202. IEEE, 2018.
  * [48] Zhenfeng Shao, Weixun Zhou, Xueqing Deng, Maoding Zhang, and Qimin Cheng.  Multilabel remote sensing image retrieval based on fully convolutional network.  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 13:318–328, 2020.
  * [49] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition, 2015\.
  * [50] E Simpson.  Medición de la diversidad.  Nature, 163(688):1, 1949.
  * [51] Asim Smailagic, Pedro Costa, Hae Young Noh, Devesh Walawalkar, Kartik Khandelwal, Adrian Galdran, Mostafa Mirshekari, Jonathon Fagert, Susu Xu, Pei Zhang, et al.  Medal: Accurate and robust deep active learning for medical image analysis.  In 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), pages 481–488. IEEE, 2018.
  * [52] Nasim Souly, Concetto Spampinato, and Mubarak Shah.  Semi supervised semantic segmentation using generative adversarial network.  In Proceedings of the IEEE International Conference on Computer Vision, pages 5688–5696, 2017.
  * [53] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: a simple way to prevent neural networks from overfitting.  The journal of machine learning research, 15(1):1929–1958, 2014\.
  * [54] André Stumpf, Nicolas Lachiche, Jean-Philippe Malet, Norman Kerle, and Anne Puissant.  Active learning in the spatial domain for remote sensing image classification.  IEEE transactions on geoscience and remote sensing, 52(5):2492–2507, 2013.
  * [55] Shuting Sun, Lin Mu, Lizhe Wang, and Peng Liu.  L-unet: An lstm network for remote sensing image change detection.  IEEE Geoscience and Remote Sensing Letters, 2020.
  * [56] Antti Tarvainen and Harri Valpola.  Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.  arXiv preprint arXiv:1703.01780, 2017.
  * [57] Marian Tietz, Thomas J. Fan, Daniel Nouri, Benjamin Bossan, and skorch Developers.  skorch: A scikit-learn compatible neural network library that wraps PyTorch, July 2017.
  * [58] Devis Tuia, Frédéric Ratle, Fabio Pacifici, Mikhail F Kanevski, and William J Emery.  Active learning methods for remote sensing image classification.  IEEE Transactions on Geoscience and Remote Sensing, 47(7):2218–2232, 2009.
  * [59] Devis Tuia, Michele Volpi, Loris Copa, Mikhail Kanevski, and Jordi Munoz-Mari.  A survey of active learning algorithms for supervised remote sensing image classification.  IEEE Journal of Selected Topics in Signal Processing, 5(3):606–617, 2011.
  * [60] Shuai Xie, Zunlei Feng, Ying Chen, Songtao Sun, Chao Ma, and Mingli Song.  Deal: Difficulty-aware active learning for semantic segmentation.  In Proceedings of the Asian Conference on Computer Vision, 2020\.
  * [61] Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li.  Empirical evaluation of rectified activations in convolutional network.  arXiv preprint arXiv:1505.00853, 2015.
  * [62] Lin Yang, Yizhe Zhang, Jianxu Chen, Siyuan Zhang, and Danny Z Chen.  Suggestive annotation: A deep active learning framework for biomedical image segmentation.  In International conference on medical image computing and computer-assisted intervention, pages 399–407. Springer, 2017.
  * [63] Yi Yang and Shawn Newsam.  Bag-of-visual-words and spatial extensions for land-use classification.  In Proceedings of the 18th SIGSPATIAL international conference on advances in geographic information systems, pages 270–279, 2010.
  * [64] Qiangqiang Yuan, Huanfeng Shen, Tongwen Li, Zhiwei Li, Shuwen Li, Yun Jiang, Hongzhang Xu, Weiwei Tan, Qianqian Yang, Jiwen Wang, et al.  Deep learning in environmental remote sensing: Achievements and challenges.  Remote Sensing of Environment, 241:111716, 2020.
  * [65] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.  Cutmix: Regularization strategy to train strong classifiers with localizable features.  In Proceedings of the IEEE International Conference on Computer Vision, pages 6023–6032, 2019.
  * [66] Bing Zhang, Di Wu, Li Zhang, Quanjun Jiao, and Qingting Li.  Application of hyperspectral remote sensing for environment monitoring in mining areas.  Environmental Earth Sciences, 65(3):649–658, 2012.

## Appendices

###  A Ablation Study

####  A.1 Active Learning Parameters

Tables 4 and 5 show the results of our experiments with different combinations
of $\alpha_{init}$ and $\beta_{Q}$ on UC Merced Land Use Classification [63]
and the DeepGlobe Land Cover Classification [9] datasets respectively. We vary
both the parameters between 0.1 and 0.9 for both entropy and margin-based
sampling strategies for three different labeled ratios. We found the best
performing $\alpha_{init}$ and $\beta_{Q}$ values to be 0.1 and 0.5
respectively. Overall we noticed out method to be sensitive to changes in
$\alpha_{init}$ and $\beta_{Q}$ as the average difference in the worst
performing and best performing model across all labeled ratios and sampling
techniques is 4 mIoU points for the UC Merced Land Use Classification Dataset
and 2.4 mIoU points for the DeepGlobe Land Cover Classification Dataset.

Active Learning Parameters | 2% | 5% | 12.5%
---|---|---|---
$\alpha_{init}$ | $\beta_{Q}$ | Entropy | Margin | Entropy | Margin | Entropy | Margin
0.1 | 0.1 | 0.381 | 0.358 | 0.423 | 0.450 | 0.484 | 0.497
0.1 | 0.5 | 0.398 | 0.36 | 0.456 | 0.48 | 0.494 | 0.546
0.9 | 0.9 | 0.352 | 0.353 | 0.411 | 0.421 | 0.478 | 0.478
Table 4: Ablation Study for the different Active Learning parameters on the UC Merced Land Use Classification Dataset [63] Active Learning Parameters | 2% | 5% | 12.5%
---|---|---|---
$\alpha_{init}$ | $\beta_{Q}$ | Entropy | Margin | Entropy | Margin | Entropy | Margin
0.1 | 0.1 | 0.464 | 0.497 | 0.507 | 0.502 | 0.549 | 0.513
0.1 | 0.5 | 0.469 | 0.511 | 0.513 | 0.513 | 0.554 | 0.529
0.9 | 0.9 | 0.449 | 0.462 | 0.495 | 0.498 | 0.527 | 0.512
Table 5: Ablation Study for the different Active Learning parameters on the DeepGlobe Land Cover Classification Dataset [9] | 2% | 5% | 12.5%
---|---|---|---
Backbone | Entropy | Margin | Entropy | Margin | Entropy | Margin
VGG-16 | 0.355 | 0.351 | 0.421 | 0.426 | 0.481 | 0.498
Resnet-50 | 0.371 | 0.354 | 0.434 | 0.452 | 0.489 | 0.524
Resnet-101 | 0.398 | 0.36 | 0.456 | 0.48 | 0.494 | 0.546
Table 6: Impact of different network architectures for the active learner in UC Merced Land Use Classification Dataset [63] on mIoU values | 2% | 5% | 12.5%
---|---|---|---
Backbone | Entropy | Margin | Entropy | Margin | Entropy | Margin
VGG-16 | 0.421 | 0.445 | 0.492 | 0.499 | 0.523 | 0.52
Resnet-50 | 0.469 | 0.511 | 0.513 | 0.513 | 0.554 | 0.529
Resnet-101 | 0.443 | 0.482 | 0.505 | 0.492 | 0.534 | 0.518
Table 7: Impact of different network architectures for the active learner in
the DeepGlobe Land Cover Classification Dataset [9] on mIoU values

a)

b)

Figure 5: Visualization of quantitative results for different labeled ratios
for the a) UC Merced Land Use Classification Dataset [63] and, b) DeepGlobe
Land Cover Classification Dataset [9]

####  A.2 Network Capacity of Active Learner

Tables 6 and 7 show the results of our experiments with different backbone
networks on UC Merced Land Use Classification [63] and the DeepGlobe Land
Cover Classification [9] datasets respectively. We experiment with VGG-16
[49], ResNet-50 [15] and ResNet-101 [15] which have different network
capacities. We found the best performing backbone network to be ResNet-101 for
the UC Merced Land Use Classification dataset and ResNet-50 for the DeepGlobe
Land Cover Classification dataset. As shown by the results, the image
classification network’s capacity for the $learner$ is crucial in determining
the quality of the selected samples. Any network with low capacity with
respect to the size of the dataset and the number of classes tends to
underfit, while any network with a higher capacity than required could overfit
and detrimentally affect the downstream task’s performance. We noticed our
method to be sensitive to networks with different capacities as the average
difference in the worst performing and best performing model across all
labeled ratios and sampling techniques is 3.3 mIoU points for the UC Merced
Land Use Classification Dataset and 2.9 mIoU points for the DeepGlobe Land
Cover Classification Dataset. Notably, we see that in most cases, VGG-16
performed significantly performed poorly across all labeled ratios in both the
datasets as compared to the ResNet-50 and ResNet-101 models reinforcing the
hypothesis that models with insufficient network capacity underperform at the
downstream task.

###  B Quantitative Evaluation of Diversity

In this paper, we proposed a method which aims to select the most diverse and
representative set of samples to serve as an initial labeled set of data for
the semi-supervised network. We empirically showed the success of the proposed
method on different datasets. In this section, we evaluate the robustness of
our method using statistical indices which measure the diversity of the
selected samples. To achieve this, we choose two diversity indices which are
frequently used in ecological studies that measure species diversity, but the
same analysis can also be applied to measure diversity of any set of random
samples.

####  B.1 Shannon’s Diversity Index

The Shannon index [46] was developed from information theory and is based on
measuring uncertainty. Shannon’s index accounts for both abundance and
evenness of the samples present. Shannon index is defined in Equation 8:

$\displaystyle H(x)=-\sum_{i=1}^{N}p_{i}\log p_{i}$ (8)

In our case, each sample is a pixel. Hence, $p_{i}$ indicates the probability
that a given pixel belongs to class $i$. N indicates the total number of
classes that a given pixel can belong to. Thus, we are measuring how diverse
are the samples selected by the active learning method as compared to samples
selected randomly. Therefore, samples with a large number of pixels from
different classes that are evenly distributed are the most diverse. On the
other hand, samples that are dominated by pixels from one class are the least
diverse. We report the value of Shannon diversity index for our baseline
method averaged across our three experiments with different random seeds and
for samples selected by both the active learning techniques. Intuitively,
Shannon’s index quantifies the uncertainty in predicting the class to which a
given pixel belongs and hence a higher value of Shannon diversity index
indicates a more diverse set of samples.

Our results for Shannon’s diversity index are shown in Tables 8 and 9 for the
UC Merced Land Use Classification [63] and DeepGlobe Land Cover Classification
[9] datasets respectively. We notice a strong correlation between the mIoU
values reported in the paper for the baseline and active learning strategies
and the values of the Shannon’s diversity index obtained for the respective
experiments.

Labeled Ratio(R) |  2% |  5% |  12.5%
---|---|---|---
s4GAN [32] (Baseline) |  1.96 $\pm$ 0.08 |  2.16 $\pm$ 0.02 |  2.14 $\pm$ 0.03
s4GAN + Entropy (Ours) |  2.10 |  2.20 |  2.22
s4GAN + Margin (Ours) |  2.08 |  2.22 |  2.25
Table 8: Shannon’s Diversity Index for the UC Merced Land Use Classification Dataset [63] (Higher the better) Labeled Ratio(R) |  2% |  5% |  12.5%
---|---|---|---
s4GAN [32] (Baseline) |  1.01 $\pm$ 0.04 |  1.16 $\pm$ 0.05 |  1.19 $\pm$ 0.14
s4GAN + Entropy (Ours) |  1.06 |  1.25 |  1.38
s4GAN + Margin (Ours) |  1.09 |  1.24 |  1.36
Table 9: Shannon’s Diversity Index for the DeepGlobe Land Cover Classification
Dataset [9] (Higher the better)

####  B.2 Simpson’s Diversity Index

Traditionally, Simpson’s Diversity Index [50] measures the probability that
two individuals randomly selected from a sample will belong to the same
species (or some category other than species). We extend it to our use case to
measure the diversity of the selected samples. To make it easier and intuitive
to understand the relevance of this index, we use the inverse Simpson index.
Thus, greater the value, the greater the sample diversity. In this case, the
index represents the probability that two individuals randomly selected from a
sample will belong to different species. Thus, the inverse Simpson index is
defined in 9:

$\displaystyle D=1-\frac{\sum(n(n-1))}{N(N-1)}$ (9)

where,
$n$ = the number of pixels belonging to class $i$,
$N$ = total number of classes that exist in the dataset.

Similar to Shannon’s index in Section B.1, we report results on the UC Merced
Land Use Classification [63] and the DeepGlobe Land Cover Classification [9]
datasets in Tables 10 and 11 We show that both the active learning sampling
strategies used in this paper yield more diverse set of samples and show
strong correlation with the mIoU values reported on these datasets in the
paper.

Labeled Ratio(R) |  2% |  5% |  12.5%
---|---|---|---
s4GAN [32] (Baseline) |  0.79 $\pm$ 0.03 |  0.83 $\pm$ 0.009 |  0.83 $\pm$ 0.008
s4GAN + Entropy (Ours) |  0.85 |  0.84 |  0.85
s4GAN + Margin (Ours) |  0.82 |  0.86 |  0.87
Table 10: Simpson’s Diversity Index for the UC Merced Land Use Classification Dataset [63] (Higher the better) Labeled Ratio(R) |  2% |  5% |  12.5%
---|---|---|---
s4GAN [32] (Baseline) |  0.55 $\pm$ 0.04 |  0.64 $\pm$ 0.01 |  0.65 $\pm$ 0.02
s4GAN + Entropy (Ours) |  0.58 |  0.73 |  0.71
s4GAN + Margin (Ours) |  0.62 |  0.71 |  0.68
Table 11: Simpson’s Diversity Index for the DeepGlobe Land Cover
Classification Dataset [9] (Higher the better)

a)

b)

c)

d)

Figure 6: (a) Average number of semantic categories per image-level category
for UC Merced Land Use Classification Dataset. (b) Percentage of images
containing vs. number of pixel-level categories per image for UC Merced Land
Use Classification Dataset. (c) Average number of semantic categories per
image-level category for DeepGlobe Land Cover Classification Dataset. (d)
Percentage of images containing vs. number of pixel-level categories per image
for DeepGlobe Land Cover Classification Dataset

###  C Discussion

####  C.1 Applicability of Our Method to Land Use Classification

The average number of semantic categories per scene in the UC Merced and
DeepGlobe Landuse Classification datasets used in this paper is 3.39 and 2.51
respectively as depicted by figure 6. This implies that a given scene from the
UCM dataset with a given image-level label will have 3 or more different
pixel-level labels(semantic categories). Similarly, for the DeepGlobe dataset,
we have about 2 or more semantic categories per scene on an average. UCM
dataset has a total of 18 semantic categories and DeepGlobe has 6 semantic
categories. Thus, each satellite scene in the UCM dataset has about 18% of all
pixel level labels and similarly each satellite scene in the DeepGlobe dataset
has about 42% of all pixel-level labels on an average. Figure 6 also shows us
that about 90% of scenes in the UCM dataset have more than 1 semantic category
and similarly about 80% of scenes in the DeepGlobe dataset have more than 1
semantic category. This number if quite high when we compare this statistics
with that in some generic standard dataset. For instance, consider the COCO
dataset [26]. Less that 30% of the images in the COCO dataset have more than 1
semantic category. This tells us that the landuse scenes in the domain of
satellite imagery are inherently more diverse and hence our method is highly
applicable specifically for land use classification in satellite images. We
will get a more diverse set of samples for satellite domain as compared to
using our method on generic datasets like COCO.

####  C.2 Suitability of s4GAN as our baseline

[32] propose to fuse the output of the s4GAN network with another image
classification-based network called MLMT [56] during inference to reduce false
positives. This MLMT branch uses an image classification network to output a
confidence score for every category in the dataset. This output is combined
with the pixel level output of the s4GAN network to reduce the number of false
positives in the segmentation network. Therefore, one major constraint for
using MLMT is that there should be a one-to-one correspondence between the
image-level and the pixel-level labels. This would mean that the number of
image-level categories should equal the number of pixel-level categories in a
dataset. However, this does not always hold in the case of land use
classification. An image-level label for land use classification in a
satellite scene indicates predominant usage of land. However, the same scene
can have multiple semantic categories. This prevents us from using MLMT as
done by [32] as our baseline for the task of land use classification.

a) Original Image b) Ground Truth c) Baseline d) Our Results

Figure 7: Qualitative Results from the UC Merced Land Use Classification
Dataset for 2% labeled data

###  D More Qualitative Evaluation

In this section, we provide more qualitative results from our best performing
active learning strategies and compare them to our baseline for the UC Merced
Land Use Classification Dataset [63].

Figure 7 compares the performance of our method with the baseline when trained
with 2% labeled data. Row 1 shows how our method predicts the row of boats
parked on the harbor better than the baseline method. Rows 2, 3, and 4 show
that the baseline method gets confused between multiple unrelated classes,
whereas our method reasonably predicts the correct classes.

Similarly, Figure 8 qualitatively compares the performance of our method with
the baseline when trained with 5% labeled data. Rows 1 and 4 show an example
of our method predicting the complex shape of airplanes better than the
baseline method. Row 2 shows the baseline method being confused between cars
in a parking lot and boats parked along a harbor, whereas our method predicts
cars parked close together correctly. Row 3 shows how the baseline method
completely misses the river and gets confused between multiple classes, while
our method predicts the river reasonably well.

Finally, Figure 9 shows some qualitative examples of how our method
outperforms the baseline when trained with 12.5% labeled data. Row 1 shows the
baseline being confused between buildings and mobile homes, while our method
predicts buildings in a dense residential setting more accurately. Rows 2 and
4 show our method predicting the baseball diamond structures accurately
without being confused between other classes. Similarly, as shown by Row 3,
our method predicts the contours of the airplane better than the baseline.

a) Original Image b) Ground Truth c) Baseline d) Our Results

Figure 8: Qualitative Results from the UC Merced Land Use Classification
Dataset for 5% labeled data

a) Original Image b) Ground Truth c) Baseline d) Our Results

Figure 9: Qualitative Results from the UC Merced Land Use Classification
Dataset for 12.5% labeled data