11institutetext: Xiaochen Zheng
11email<EMAIL_ADDRESS>
ETH Zürich, Switzerland

# Rare Wildlife Recognition with Self-Supervised Representation Learning

Master Thesis

###### Abstract

Automated animal censuses with aerial imagery are a vital ingredient towards
wildlife conservation. Recent models are generally based on supervised
learning and thus require vast amounts of training data. Due to their scarcity
and minuscule size, annotating animals in aerial imagery is a highly tedious
process. In this project, we present a methodology to reduce the amount of
required training data by resorting to self-supervised pretraining. In detail,
we examine a combination of recent contrastive learning methodologies like
Momentum Contrast (MoCo) and Cross-Level Instance-Group Discrimination (CLD)
to condition our model on the aerial images without the requirement for
labels. We show that a combination of MoCo, CLD, and geometric augmentations
outperforms conventional models pretrained on ImageNet by a large margin.
Meanwhile, strategies for smoothing label or prediction distribution in
supervised learning have been proven useful in preventing the model from
overfitting. We combine the self-supervised contrastive models with image
mixup strategies and find that it is useful for learning more robust visual
representations. Crucially, our methods still yield favorable results even if
we reduce the number of training animals to just 10%, at which point our best
model scores double the recall of the baseline at similar precision. This
effectively allows reducing the number of required annotations to a fraction
while still being able to train high-accuracy models in such highly
challenging settings. Dr. Benjamin Kellenberger, ECEO, EPFL, Switzerland
Prof. Dr. Devis Tuia, ECEO, EPFL, Switzerland
Prof. Dr. Irena Hajnsek, EO, ETH Zürich, Switzerland August
2021111https://eo.ifu.ethz.ch/studium/master-thesis.html

##  1 Introduction

Background Biodiversity and ecosystem services are of fundamental importance
to the earth and to human society. However, human activities are causing rapid
habitat destruction and environmental degradation, making biodiversity
disappear at an unprecedented rate, which has reached over 1,000 times the
rate of natural disappearance of species [2]. This requires adaptive
technologies for effective environmental monitoring, which includes wildlife
censuses [24].

Wildlife censuses plays an increasingly significant role in protecting
endangered species. The censuses help determine the exact number and the
spatial-temporal distribution of wild animals, which is vital to assess living
conditions and potential survival risks of wildlife species [20, 2, 32].
Meanwhile wildlife censuses help monitor usual changes in wildlife communities
[24], which are essential to ecological balance and biodiversity.

Manual surveys are traditional approaches of wildlife censuses in remote
areas. However, serious deficiencies, such as health and life risk on human
operators, low accuracies and small area coverage, are caused by using manned
aircrafts, camera traps, or road driving for field work. One of the effective
and feasible ways to address these issues is to replace manual surveys of
wildlife reserves with counts derived automatically from images acquired by
Unmanned Aerial Vehicles (UAVs), paired with deep learning models to automate
the wildlife recognition task [20, 19, 33, 21, 10].

Supervised deep learning models have been shown to perform exceptionally well
for natural images. These models are typically pretrained on large-scale
curated datasets such as ImageNet [7], MS-COCO [23] with supervision and then
fine-tuned on the target natural imagery. Irrespective of the type of fine-
tuning, this last step requires thousands of animals to be annotated, and
hence expert knowledge. Meanwhile, large UAV campaigns can generate images in
high numbers [20], with many containing either large numbers of animals, or
none [21], both of which cases are cumbersome for manual annotators.
Furthermore, the vastness of wildlife reserves mean that wildlife is a rare
sight and background dominates the majority of images. This causes two
problems: (1) the datasets are strongly imbalanced toward the absence class;
(2) the objects of interest are extremely small compared with the original
aerial image size, illustrated in Figure. These two issues significantly
downgrade the capacity of a supervised model to solve the recognition task
[20]. Hence, methods to reduce the labelling requirement are urgently needed.

Figure 1: Wildlife in raw image is tiny and is a rare sight.

A promising direction to this end is to use self-supervised learning (SSL),
where models are first trained in a _pretext_ task on the target images
without the need for manually provided labels, and then fine-tuned on the
actual objective (_downstream task_) with manual labels [26]. Earlier
_pretext_ tasks required models to reconstruct transformations between
different _views_ of the same image. Recently, focus has shifted to
contrastive learning. Here, augmentation as a method of image transformation
is still employed, but with a different objective: while traditional SSL
methodologies forced the model to learn representations within one data point
to _e.g._ , in-paint cut-out regions [30, 46], contrastive learning employs
transformations in a comparison scheme and encourages the model to learn
representations by maximizing the similarity between two randomly augmented
_views_ of the same data point, resp. image (positive pairs) and dissimilarity
between different data points (negative pairs) [42, 13, 5, 38, 4, 12, 27]. The
different _views_ of same instance are randomly generated from a stochastic
data augmentation module. Recent works have identified a stronger augmentation
strategy to be vital for improved learning [5]. However, augmentation
functions need to be carefully selected with respect to the problem and data
at hand [43]. Choosing inadequate functions may result in removal of important
information (_e.g._ , random resized cropping may remove animals at the border
of UAV images). In contrast, some functions benefit certain scenarios more
than others (_e.g._ , random vertical flips and rotations may be of limited
use with natural images, but may provide strong learning signals for view-
independent aerial images).

However, applying self-supervised learning techniques, such as contrastive
learning, to UAV images on wildlife is challenging. One such problem is the
requirement of contrastive learning methods to receive dissimilar imagery. In
UAV acquisitions, as illustrated in Figure 3, high image sampling frequencies
will generate strong autocorrelations between acquisitions in short time
intervals (similar to adjacent frames in a video). Also, the vastness of many
wildlife areas result in consistent characteristics, and hence in repeated or
similar patches in the dataset. Many instance discrimination based methods,
such as NPID [42], MoCo [13, 5], and SimCLR [4], are based on the assumption
that each instance is significantly different from others and that each
instance can be treated as a separate category. The large similarity between
training images mean that the negative pairs used in the contrastive learning
process is likely to be composed of highly similar instances, which in turn
compromises feature representations due to incorrect repulsion between similar
images. Instead of exploring the effect of hard negative sampling [17, 34], in
this paper we propose to solve these problems with Cross-Level Instance-Group
Discrimination (CLD) [41], which aims to deal with highly correlated datasets.
Furthermore, we hypothesize that top view UAV imagery should be _invariant_ to
geometric transformations, _e.g._ , rotation. From this perspective, we
propose to apply extra geometric transformation to contrastive model, which
captures invariant information of UAV images introduced by different
augmentations.

Scope and Objective. The thesis is mainly focus on four aspects:

(1) Processing and analyzing the raw data from UAVs. The raw images are large
in size and have very high resolution. Due to the limited computation
capacity, the deep models can only handle with patches in appropriate sizes
cropped from raw images. The processing aims to find a cropping method without
using annotation information that adapts to self-supervised pretraining.

(2) Defining the downstream task. We aim to exploit how SSL pretraining will
perform on object recognition task. Thus for downstream task, all patches
should be image-level annotated and the cropping methods should be different
from the one of SSL pretraining.

(3) Designing domain-specific algorithm. We aims to design algorithms based on
the characteristics of our natural data and to select the best fitted
algorithm by applying certain criteria, _e.g._ , linear protocol and
classification accuracy. The features of remote sensing images in latent space
should be rotation invariant. Hence, our model should learn rotation-invariant
features to obtain good performance

(4) In-depth analysis of performance on downstream task.

The quantification of the training uncertainty associated with the random seed
selection is not necessary as part of the work. The study of model
generalization to other datasets as well as the influence of spatial and
temporal variability of data acquisition is not included in the project. The
numerical computer vision algorithms are not included.

Contributions of This Thesis. We propose an SSL model to pretrain on our
wildlife dataset by contrastive learning. We apply the pretraining process to
downstream wildlife recognition task. Our work build on the work of MoCo [13,
5] and Cross-Level Instance-Group Discrimination (CLD) [41]. The contributions
of this thesis can be summarized as follows:

(1) We propose a methodology for image-level wildlife recognition with a
reduced number of annotations by self-supervised pretraining.

(2) We show that using self-supervised pretraining outperforms supervised
ImageNet pretraining on downstream recognition task.

(3) We design a domain-specific geometric augmentation and apply it to state-
of-the-art models, _e.g._ , MoCo v2 [5] and MoCo+CLD [41].

(4) We apply image mixup strategies to self-supervised pretraining (MoCo),
which are initially used in supervised learning models. We further find that
MoCo with image mixup strategies outperforms ImageNet pretraining fine-tuned
with all available training labels. Namely, we improve the preformance of MoCo
v2 on our dataset without increasing the model parameters. Our self-supervised
pretraining learns representations of natural wildlife scenes more efficiently
than supervised pretraining.

##  2 Related Work

Self-Supervised Representation Learning. Self-Supervised representation
learning in computer vision aims to learn effective visual representations of
images without any supervision. Namely, Self-Supervised deep models learn
high-level representations from raw image inputs. [25] summarizes the self-
supervised mechanisms into three categories: generative model, contrastive
model, and adversarial model. In this thesis, we only focus on the contrastive
model. The basic framework is illustrated in Figure 2.

Figure 2: A basic framework of self-supervised representation learning. It
consists of data loader, encoder, pretext task.

Unlike in the field of classic computer vision, self-supervised learning of
aerial images has not yet been fully studied. [36] apply Contrastive Multiview
Coding [39] to learn aerial image representations on both RGB and
multispectral remote sensing images. [18] proposes a method based on
contrastive learning with different image augmentations. [37] analyze
different pretext tasks, _e.g._ , inpainting [30], context prediction [9], and
contrastive learning with different image augmentations on remote sensing
dataset. Besides, [1] use geo-location classification as the pretext task. The
encoder is trained by predicting the global geo-location of input image. [40]
generate pseudo-labels of unlabeled UAV imagry data to improve the downstream
classification accuracy. Most of the downstream tasks in aerial imaery domain
are scene classification, _e.g._ , land-cover or land-use [16] classification.
[1] apply Self-Supervised learning to transferred downstream tasks, _e.g._ ,
object detection and image segmentation. Different from those tasks, our task
is more domain specific, which is tiny and rare wildlife recognition in the
wild.

Pretext tasks. Self-Supervised representation learning is designed to solve
certain pretext tasks. [11] uses the rotation angle as pseudo label and learn
underlined structure of the objects by predicting rotation angle. [9, 26, 8]
perform region-level relative location prediction. Other missions like color
in-painting [30, 46], and solving jigsaw puzzles [28] are also applied as
pretext tasks. In contrastive learning, learning between different augmented
views are used as pretext tasks. In this work, we apply a instance
discrimination task [42] in addition to geometric invariant mapping to
contrastive self-supervised models.

Contrastive Representation Learning. Recently, the most competitive
representation learning method without labels is Self-Supervised contrastive
learning. Contrastive methods [4, 13, 5, 12, 3, 29, 26, 31, 27] train a visual
representation encoder by attracting positive pairs from the same instance in
latent space while repulsing negative pairs from different instances. One of
the most important parts in contrastive learning is the selection of positive
and negative pairs [39] for instance discrimination. To create positive pairs
without prior label information, one common way is to generate multiple views
of input images. [5, 13, 4] apply a stochastic augmentation module to randomly
augment an input image twice. [43] proposes a new model with multi-
augmentation and multi-head. It constructs multiple embeddings and captures
varying and invariant information introduced by different augmentations.
Methods combined contrastive learning with online clustering [3, 41, 22] are
proposed to boost the performance of self-supervised learning which explore
the data manifold to learn image representations by capturing invariant
information.

Image Mixture Strategies. Image mixture is well defined in supervised
learning. Image mixture mixes pixel values of two images. Images in mixup [45]
are generated by mixing pixel values of two randomly selected images. The
proportion of pixel strength during data mixing is decided by the mixing
factor $\lambda\in[0,1]$. The values for $\lambda$ are drawn from the beta
distribution. Cutmix [44] is the way to generate a new training sample by
locally combining two samples of their region and whole image. Different from
mixup [45], this operation will replace pixels within particular locations of
a region. Augmix [15] performs data mixing using the input image itself. It
mixes itself with different views of its original image. The views are
generated from different transforms, which are a series of randomly selected
augmentation operations. Those operations are applied with three parallel
augmentation chains. The output of these three chains are three mixed images.
This new image is further mixed with the original image to generate final
augmented output image. [35] explore applying both mixup and cutmix to
contrastive learning framework.

Technical Challenges. Most contrastive models are trained on curated dataset
with unique characteristics, _e.g._ , ImageNet [7]. In these datasets, images
contain only a single object which is located in the center of the image
(_object-centric_). And objects have _discriminative_ visual features. The
datasets also have uniformly distributed classes. In contrast, domain-specific
datasets (_e.g._ , our KWD) contain less discriminative visual feature, making
it hard to distinguish between similar objects _e.g._ , trees, wildlife from
the top view. As illustrated in Figure 3 (2), similar patches containing
wildlife should be distinguished (repulsed) from the ones without wildlife.
Whereas in Figure 3 (3) similar patches from the same category should be
attracted. Inside-image similarity and between-image similarity are difficult
for supervised learning and contrastive learning, respectively. Meanwhile, the
tiny objects contained in the drone image increase the difficulty of
recognition. Models should be sensitive to tiny objects. The patches
containing the target object account for a minimal number, which also brings
difficulty to model training.

##  3 Materials

###  3.1 Study Area and Data.

In this work, we use the data from [20], consisting of RGB aerial images
acquired with a SenseFly eBee222https://www.sensefly.com UAV over the Kuzikus
Wildlife Reserve in Namibia333https://kuzikus-namibia.de by the SAVMAP
consortium444http://lasig.epfl.ch/savmap. The UAV’s flight height varied
between 120 and 160m, resulting in a resolution of 4 to 8 cm with the given
camera (Canon PowerShot S110). The images were annotated with bounding boxes
for animals in a crowdsourcing campaign led by
MicroMappers555https://micromappers.wordpress.com; these annotations were then
refined in several iterations by the authors. This resulted in a total of 1183
animals. We derived the Kuzikus Wildlife Dataset Pre-training (KWD-Pre) and
Kuzikus Wildlife Dataset Long-Tail distributed (KWD-LT) for pre-traning and
fine-tuning/downstream task.

Figure 3: Overview of Kuzikus dataset. In (1), the region of interest (RoI)
(red bounding box) only occupies a small part of original UAV image. Wildlife
(yellow bounding boxes) is tiny which is hard to be recognized and labeled.
That makes supervised learning difficult. Foreground (wildlife)/background
crops are marked with red/green bounding boxes. Patches $a,c,d,e$ are cropped
from different locations of the same image. Patches
$a,a^{\prime},b,b^{\prime}$ are cropped from same locations of different
images (sampling time interval is $\Delta t$).

###  3.2 Datasets

KWD-Pre. We apply the same patches creating procedure as described in [20]. We
randomly crop 4 patches for every original $4000\times 3000$ image. The size
of each patch is $256\times 256$ pixels to save memory and have a larger batch
size. We perform overlapping cropping if one image contains animals. Cropping
this way increases the chances of extracting patches containing animals for
training, but we do not retain any labeled information nor bounding box
location. As this can be seen as a form of weak supervision in the patch
extraction process, we do not know whether each patch contains animal(s) or
not while applying random cropping. So the prior knowledge of classes and
locations is not exploited by self-supervised learning.

KWD-LT. The original images are taken by UAVs on different dates and times
[20]. We first split the original data into train, test, and validation set
with a ratio of 8:1:1. Then, for the background class, we apply a random
cropping procedure ($512\times 512$ pixels) to the original images and verify
each patch to make sure it contains no animal. For the foreground (wildlife)
class, we apply a random cropping procedure ($224\times 224$ pixels) around
the ground truth bounding boxes to make sure each patch contains whole
animal(s) body. We choose three different random seed to random cropping
procedures of train, test, and validation set to make sure the cropping
position is different. The train set is class imbalanced and long-tail
distributed with a foreground-to-background ratio of $\frac{1}{18}$. The test
and validation set are class balanced. In the experiments below, we evaluate
fine-tuning on KWD-LT with different percentages of annotated animals to
investigate the benefit of SSL for reduced annotation efforts.

Datasets Overview. After analyzing the content of the dataset, we can
intuitively divide all elements into the following categories: ground, animal,
tree, grass, and tire mark, as illustrated in Figure 4. Especially the animal
elements in the right are animals beneath the tree and the tree elements in
the right are dead tree trunks. This is extremely hard to be recognized [20].
It is also difficult to distinguish between dead tree trunks and animal. All
patches are random combination of those five elements.

Figure 4: Overview of KWD-LT dataset. (a) denotes all possible elements in the
dataset. Intuitively, all examples in the KWD-LT dataset consist of randomly
combined elements in (a). (b) is examples of KWD-LT dataset. Instances are
image-level annotated. Foreground represent images containing wildlife.

##  4 Methods

###  4.1 Cropping Methods

Random Cropping. This method randomly selects X and Y coordinates of each
patch. And based on predefined image length and width, it crops the raw images
into several patches.

Overlapping Cropping. Overlapping means some part of the previous patch is
repeated in the current patch. Overlapping cropping is to crop raw images with
stride equal to ((image length - overlapping length), (image width -
overlapping width)) from left to right, top to bottom. For example, 50%
overlap means half of the previous patch is repeated in the current patch,
both horizontally and vertically666For all cropping methods, we fix the random
seed to improve repeatability..

###  4.2 Contrastive Learning Framework

Figure 5: Overview of the employed SSL framework, consisting of MoCo [13]
(upper part) and CLD [41] (lower part). Firstly, the two different _views_
$I_{1}$ and $I^{+}$ of the same input $I$ are encoded by $f_{q}$ and $f_{k}$
respectively. Then, the two representations $z_{1}$ and $z^{+}$ are projected
into an embedding space. $q$ and $k$ are representations of query and key in
the hypersphere. Query $q_{1}$ and its positive key $k^{+}$ are from the
different augmented _views_ of the same input and negative keys $k^{-}$ are
encoded from different inputs (dashed bounding box). CLD first encodes two
different _views_ $I_{1},I_{2}$ of the same instance, then applies a different
projection head and projects the representations from the same query encoders
to a different embedding space from (1). Finally, a local K-Means clustering
is used to find the $k$ centroids of a batch of inputs. The centroid of
assigned cluster $g_{1}$ can be served as positive key of view $g_{2}$, and
vice versa.

Contrastive learning is a learning algorithm that describes similarity and
dissimilarity for deep learning models. Contrastive methods train deep models
to distinguish between similar and dissimilar input images in computer vision.
Namely, contrastive learning uses the contrastive loss of the latent space to
maximize the consistency between different augmented views of the same data
point to learn representation. It constructs representations by encoding
similar or different inputs.

More formally, for any data point $x$, contrastive methods aim to learn an
encoder $f$ such that:

$\textbf{score}\left(f(x),f(x^{+})\right)\gg\textbf{score}\left(f(x),f(x^{-})\right)$
(1)

where $x$ is commonly referred to as an “anchor” data point, $x^{+}$ is data
point similar to $x$, referred to as a positive sample, $x^{-}$ is a data
point dissimilar to $x$, referred to as a negative sample, the score function
is a metric that measures the similarity between two features.

Contrastive methods use data augmentation to generate different _views_.
Positive samples are different views of the same image, while negative samples
are views of different images. We choose cosine similarity as the score
function. Given that “query”, “positive key”, and “negative keys”, which are
representations of anchor, positive sample, and negative samples encoded by
encoder networks, denoted as $\boldsymbol{q}$, $\boldsymbol{k}^{+}$, and
$\boldsymbol{k}^{-}$, a popular choice of contrastive loss for positive pairs
$(\boldsymbol{q},\boldsymbol{k}^{+})$ and negative pairs
$(\boldsymbol{q},\boldsymbol{k}^{-})$ is InfoNCE [29], denoted as
$\mathcal{L}_{q}(\boldsymbol{q},\boldsymbol{k}^{+})$:

$\mathcal{L}_{q}(\boldsymbol{q},\boldsymbol{k}^{+})=-\log\frac{\exp\left(\boldsymbol{q}\\!\cdot\\!\boldsymbol{k}^{+}\\!/\tau\right)}{\exp\left(\boldsymbol{q}\\!\cdot\\!\boldsymbol{k}^{+}\\!/\tau\right)+\sum_{K}\exp\left(\boldsymbol{q}\\!\cdot\\!\boldsymbol{k}^{-}\\!/\tau\right)}$
(2)

where the dictionary contains $K$ negative samples and $\tau$ denotes the
temperature parameter, which is the hyper-parameter scaling the distribution
of distances [29, 13].

There are two methods for minimizing the empirical loss in Equation 2. The
first method is to increase the numerator
$(\boldsymbol{q}\\!\cdot\\!\boldsymbol{k}^{+})$, which means making the angle
between the feature vectors of positive pairs smaller. The second method is to
reduce the denominator
$\sum_{K}\exp\left(\boldsymbol{q}\\!\cdot\\!\boldsymbol{k}^{-}\\!/\tau\right)$,
which means increasing the dissimilarity of negative pairs. Thus, the number
of samples has a great influence on the quality of the learned representations
[26]. The purpose of minimizing Equation 2 is achieved by updating the
parameters of query encoder and key encoder through gradient descent.

###  4.3 Momentum Contrast (MoCo)

We apply MoCo [13] as our unsupervised learning method. MoCo is developed
based on two ideas:

  * •

Dynamic Dictionaries. Words in natural language processing (NLP) tasks have
discrete signal spaces. Thus tokenized dictionaries, a data structure that
contains all essential information units, can be constructed, which contain
representations of all the words. The data in the computer vision task are
distributed in high-dimensional continuous spaces, which makes it challenging
to build an image dictionary. In computer vision, the dictionary should
contain representations of all data points, which is difficult to implement.
Thus, MoCo propose a dynamic dictionary mechanism. The dynamic dictionary is
to randomly sample several images from all images during the training process
and to use it as the dictionary for the current training. Each batch
eliminates the earliest data encoded in the dynamic dictionary and adds the
latest encoded data. This aims to keep the dictionary as consistent as
possible.

  * •

Momentum Updating. With a large dictionary, it is hard to calculate the
gradient through all samples in the dictionary and to update the key encoder
by back-propagation. Thus a momentum updating mechanism is proposed:

$\theta_{k}\leftarrow m\theta_{k}+(1-m)\theta_{q}$ (3)

where $m\in[0,1)$ and $\theta_{k},\theta_{q}$ are parameters of key and query
encoders, respectively. Only query encoder is updated by back-propagation.

Given a batch of inputs, query and positive key are from two different
augmentation _views_ of the same input whereas query and negative keys are
from _views_ of different inputs. The dynamic dictionary stores both positive
and negative keys. With MoCo, self-supervised learning can be regarded as a
training process to perform dictionary look-up. MoCo learns image
representations by matching an encoded query $q$ to a dictionary of encoded
keys $k$ using a contrastive loss. Query $q$ should be similar to its matching
key, positive key $k^{+}$, and dissimilar to negative keys $k^{-}$. As
illustrated in Figure 5, the MoCo model consists of five parts:

(1) A _stochastic data augmentation_ module $\mathcal{T}$ [4] transforms one
given input image $I$ into different augmented _views_ with randomly applied
augmentations, denoted $I_{i}$ and $I^{+}$ for query $q_{i}$ and positive key
$k^{+}$ [13]. We sequentially apply five augmentations (Base and Color Aug. in
Table 1) similar as MoCo v2 [5].

(2) A _base encoder_ $f_{q}$ for query $q$ maps the augmented _views_ into
feature space: $f_{q}:I_{i},\rightarrow\boldsymbol{z}_{i},\ \text{where}\
I_{i}\in\mathbb{R}^{C\times W\times H},\boldsymbol{z}_{i}\in\mathbb{R}^{s}$,
where $z$ denotes the $s$-D encoded _representation_. The parameters are
updated by back-propagation. This takes the form of a CNN in our case.

(3) A _momentum-updated encoder_ $f_{k}$ for keys $k$ shares the same
structure with _base encoder_ $f_{q}$ and is initialized with the same
parameters. However, they are not learned through backpropagation during
training; instead, the parameters of $f_{k}(\cdot)$ are updated with a
momentum mechanism [13].

(4) A _projection head_ projects the _representations_ $\boldsymbol{z}_{i}$
into a unit hypersphere: $h:\boldsymbol{z}_{i}\rightarrow\boldsymbol{q}_{i},\
\text{where}\ \boldsymbol{q}_{i}\in\mathbb{R}^{m}\ \text{and}\
\left|\left|\boldsymbol{q}_{i}\right|\right|=1$, the same for
$\boldsymbol{z}^{+}$. The similarity is measured by a dot product.

(5) A _dynamic dictionary_ holds the prototypical feature for all instances
[42, 4, 41]. It is implemented as a queue of fixed size, fed with the stream
of mini-batches that are used for training: in current mini-batch, the encoded
representations are enqueued, and the oldest are dequeued [13].

# f_q, f_k: encoder networks for query and key
# queue: dictionary as a queue of K keys (CxK)
# m: momentum
# t: temperature

f_k.params = f_q.params # initialize

for x in loader: # load a minibatch x with N samples
x_q = aug(x) # a randomly augmented version
x_k = aug(x) # another randomly augmented version

q = f_q.forward(x_q) # queries: NxC
k = f_k.forward(x_k) # keys: NxC
k = k.detach() # no gradient to keys

# positive logits: Nx1
l_pos = bmm(q.view(N,1,C), k.view(N,C,1))

# negative logits: NxK
l_neg = mm(q.view(N,C), queue.view(C,K))

# logits: Nx(1+K)
logits = cat([l_pos, l_neg], dim=1)

# contrastive loss
labels = np.zeros(N) # positives are the 0-th
loss = CrossEntropyLoss(logits/t, labels)

# update gradients
optimizer.zero_grad()
loss.backward()
optimizer.step()

# momentum update: key network
f_k.params = m * f_k.params + (1 - m) * f_q.params

# update dictionary
enqueue(queue, k) # enqueue the current minibatch
dequeue(queue) # dequeue the earliest minibatch

# bmm: batch matrix multiplication
# mm: matrix multiplication
# cat: concatenation

Algorithm 1 PyTorch-like Pseudo-code of MoCo [13]

###  4.4 Clustering Based Contrastive Learning (CLD)

Applying MoCo to highly correlated dataset will cause the problem of false
repulsion. As shown in Figure 6, three entries in negative samples are from
the same category as the positive sample and anchor sample. Thus, those
samples are repelled by MoCo. Obviously, representation learning demands
encoder to generate similar features from different images in the same
category. True negative samples can have a better performance than ”mixing”
ones in supervised learning [17, 34]. Without access to any similarity or
dissimilarity information, we aims to find a method of ”sorting” positive and
negative samples.

Figure 6: Illustration of positive and negative samples in MoCo, MoCo+CLD, and
MoCo+CLD with geometric augmentation strategy scenarioes. Anchor $I$ is the
original image patch on KWD dataset. $I^{\prime}$ is the augmented view of
$I$. The red/green bounding boxes represent the foreground/background classes.
In MoCo, negative samples might contain images from the same category of the
anchor (positive samples). That causes the false repulsion. In MoCo+CLD, after
applying local $k$-means clustering to all samples, positive and negative
samples can be grouped into different categories. With CLD, there is less
repulsion between samples in the same category.

The key idea of CLD [41] is to cluster instances locally, and perform
contrastive loss to centroids and image representaions. Therefore, similar
instances are clustered into the same group and the false rejection of
instances with high similarity is alleviated, as illustrated in Figure 6. CLD
uses two different _views_ of the same instance as input. As such, the CLD
branch shares the same query encoder $f_{q}$ with MoCo but uses a different
projection head $h_{G}$, as illustrated in the lower part of Figure 5.

To perform CLD, the unit-length features $\boldsymbol{g}_{i}$ of all instances
in a mini-batch are first extracted from $f_{q}$ and $h_{G}$. Then, CLD
implements local $k$-means clustering to $\boldsymbol{g}_{i}$ for a mini-batch
of instances and finds $k$ local cluster centroids
$\\{\boldsymbol{c}_{i}^{1},\dots,\boldsymbol{c}_{i}^{k}\\}$ with
$\boldsymbol{g}_{i}$ assigned to $\boldsymbol{C}(\boldsymbol{g}_{i})$. The
same operation is performed to the other branch $I_{j}$ from all instances in
a mini-batch, denoted as $\boldsymbol{g}_{j}$,
$\\{\boldsymbol{c}_{j}^{1},\dots,\boldsymbol{c}_{j}^{k}\\}$, and
$\boldsymbol{C}(\boldsymbol{g}_{j})$. CLD applies the contrastive loss between
$\boldsymbol{g}_{i}$ and clustering of the other branch
$\\{\boldsymbol{c}_{j}^{1},\dots,\boldsymbol{c}_{j}^{k}\\}$. Each cluster
contains highly similar instances, and the assigned centroids together with
representations from the other branch can be regarded as positive pairs.
Namely, the centroids of the _other_ clusters act as negative samples [41].
Thus, feature vector $\boldsymbol{g}_{i}$ and its counterparts
$\boldsymbol{g}_{j}$ assigned centroid $\boldsymbol{C}(\boldsymbol{g}_{j})$
comprise positive pairs and all _other_ centroids comprise negative pairs. The
local contrastive loss for CLD is

$\mathcal{L}_{g}(\boldsymbol{g}_{i},\boldsymbol{C}(\boldsymbol{g}_{j}))=-\log\frac{\exp\left(\boldsymbol{g}_{i}\\!\cdot\\!\boldsymbol{C}(\boldsymbol{g}_{j})\\!/\tau\right)}{\sum_{\left\\{\boldsymbol{c}_{j}^{k}\right\\}}\exp\left(\boldsymbol{g}_{i}\\!\cdot\\!\boldsymbol{c}_{j}^{k}\\!/\tau\right)}$
(4)

where $\left\\{\boldsymbol{c}_{j}^{k}\right\\}$ denotes the set of $k$
centroids from the other branch. Thus, the loss of a dual-branch CLD in Figure
5 is:

$\mathcal{L}_{g}(\boldsymbol{g}_{1},\boldsymbol{C}(\boldsymbol{g}_{2}))+\mathcal{L}_{g}(\boldsymbol{g}_{2},\boldsymbol{C}(\boldsymbol{g}_{1}))$
(5)

Algorithm 2 provides the pseudo-code of CLD which can be combined with classic
MoCo model, described in Agorithm 1.

We combine CLD [41] with MoCo v2 [5] and construct a total CLD contrastive
loss over _views_ $I_{1}$, $I_{2}$, and $I^{+}$ with CLD weight $\lambda$ in a
mini-batch. We apply different temperatures $\tau_{q}$ and $\tau_{g}$ for
instance and group branches, respectively. The total CLD contrastive loss is
[41]:

$\displaystyle\mathcal{L}_{CLD}=$
$\displaystyle\frac{1}{2}\left[\mathcal{L}_{q}(\boldsymbol{q}_{1},\boldsymbol{k}^{+})+\mathcal{L}_{q}(\boldsymbol{q}_{2},\boldsymbol{k}^{+})\right]$
(6)
$\displaystyle+\lambda\times\frac{1}{2}\left[\mathcal{L}_{g}(\boldsymbol{g}_{1},\boldsymbol{C}(\boldsymbol{g}_{2}))+\mathcal{L}_{g}(\boldsymbol{g}_{2},\boldsymbol{C}(\boldsymbol{g}_{1})\right]$

where $\boldsymbol{q}_{\\{1,2\\}}$ and $\boldsymbol{g}_{\\{1,2\\}}$ are
feature representations issued from augmented versions of the original
samples, generated following the procedure described in Section 4.5.

# f: backbone encoder + projection mlp
# k: k means algorithm function, outputs are the label of assigned cluster and
centroids of G clusters

for x in loader: # load a minibatch x with N samples
x_1, x_2 = aug(x), aug(x) # two randomly augmented versions

g_1, g_2 = f(x_1), f(x_2) # query-like features: NxC

# label, cent: N, GxC
label1, cent1 = k(g_1)
label2, cent2 = k(g_2)

# logits: NxG
logits1 = mm(g_1, cent2.T).div(T)
logits2 = mm(g_2, cent1.T).div(T)

# contrastive loss
loss1 = CrossEntropyLoss(logits1/t, label2)
loss2 = CrossEntropyLoss(logits2/t, label1)
loss = 0.5 * loss1 + 0.5 * loss2

# update gradients
optimizer.zero_grad()
loss.backward()
optimizer.step()

# mm: matrix multiplication
# div(T): tensor division with rounding the results towards zero

Algorithm 2 PyTorch-like Pseudo-code of CLD [41]

###  4.5 Geometric Augmentation Strategies

Module | PyTorch-like Augmentation
---|---
Base Aug. | RandomCrop(224)∗
| RandomHorizontalFlip(p=0.5)
| GaussianBlur([0.1, 2.0])
Color Aug. | ColorJitter(0.4, 0.4, 0.4, 0.1)
| RandomGrayscale(p=0.2)
Rot. Aug. | RandomRotation()∗∗
Table 1: Overview of the employed random augmentation strategies. ∗ To avoid
information loss on tiny animals, we apply random crops without resizing. ∗∗
We randomly rotate the images by $\\{90^{\circ}$, $180^{\circ}$,
$270^{\circ}\\}$.

State-of-the-art contrastive learning [4, 6, 5] between multiple _views_ of
the data employs stronger augmentation strategies to improve performance. The
choices of different _views_ have a marked impact on the performance of self-
supervised pretraining [5, 47, 43, 39]. For different branches in CLD, _e.g._
, $I_{1}$ and $I_{2}$ in Figure 5, we apply multiple augmentations to the same
input image [43]. We keep each branch invariant to one specific augmentation
transformation. For example, $I_{1}$ and $I^{+}$ are always augmented by the
same color but different rotation augmentation while $I_{2}$ and $I^{+}$ are
always augmented by same rotation but different color augmentation.
Augmentation parameters are sampled randomly and independently from the
stochastic augmentation module $\mathcal{T}$ as outlined in Table 1. We
project the queries and key into one embedding space, keeping the embedding
space invariant to all augmentations. We aim to add extra geometric
transformations on top of the CLD framework. Given the queries $q_{1}$,
$q_{2}$ of $I_{1}$, $I_{2}$, and positive key $k^{+}$ of $I^{+}$, the loss of
MoCo with our proposed geometric augmentation strategies is:

$\mathcal{L}_{Geo}=\gamma\mathcal{L}_{q}(q_{1},k^{+})+(1-\gamma)\mathcal{L}_{q}(q_{2},k^{+})$
(7)

where $I_{1}$ and $I_{2}$ are two different views and $\gamma$ is the weight
for the combination. $\gamma$ is used to balance the color augmentation branch
and geometric augmentation branch.

###  4.6 Image Mixture Strategies

(a) Views generation and image combination.

(b) Illustration of mixup strategies.

Figure 7: Batch-wise mixing up. $I_{1},I_{2}$ are from different augmented
views of the same input.

Mixup [45] is a pixel-wise image-level mixture method. It combines two global
images with certain weight (or mixing ratio) $\lambda$. The mixed image stores
all information from two original images:

$I_{M}\leftarrow\lambda I_{1}+(1-\lambda)I_{2}$ (8)

where${I_{M},I_{1},I_{2}}$ denote the output mixed image and two original
images, respectively.

In our method, instead of mixing two different images, we aim to combine two
different views of the same input on image level. Intuitively, mixing two
versions with different geometric transformation from the same foreground
sample will increase the amount of target objects, as illustrated in Figure 7.
The mixing ratio of the two augmented images and the probability of applying
mixture strategy are the core information in our self-supervised model. Here
we introduce two simple strategies to retain this information for loss
calculation.

(1) Given $I$, $I_{1}$, and $I_{2}$ which are three different views of the
same input augmented by base, color, and rotation transformations in Table 1,
we mix two images with a weight $\lambda$, as illustrated in Figure 7. The
weight $\lambda$ is sampled from a Beta distribution
$\text{Beta}(\alpha,\beta)$ [35]:

$\displaystyle I_{M}=\lambda I_{2}+(1-\lambda)I$ (9) $\displaystyle
I_{M}^{\prime}=\lambda I_{2}+(1-\lambda)I_{1}$

Given that the feature vectors of $I_{M}$ and $I_{M}^{\prime}$ are $q_{M}$ and
$q_{M}^{\prime}$, we defined a _mixture_ loss over mixtures $I_{M}$ and
$I_{M}^{\prime}$ as:

$\mathcal{L}_{M}=\lambda\mathcal{L}_{q}(q_{M},k^{+})+(1-\lambda)\mathcal{L}_{q}(q_{M}^{\prime},k^{+})$
(10)

(2) We replace the MoCo loss in Equation 7 by _mixture_ loss in Equation 10
with a probablity of $p$. $p$ is the hyperparameter.

Obviously, for $\lambda=1$, $\mathcal{L}_{M}=\mathcal{L}_{q}(q_{1},k^{+})$,
while for $\lambda=0$, $\mathcal{L}_{M}=\mathcal{L}_{q}(q_{2},k^{+})$, which
are _original_ loss of MoCo v2 and MoCo with geometric augmentation,
respectively.

# p: probability of applying mixtures
# g: ratio of MoCo v2 loss
# beta: hyperparameter for Beta distribution
# lam: mixture ratio in geo-geo mixture or geo-color mixture
# base_aug: basic augmentation
# geo_aug: geometric augmentation
# color_aug: color augmentation

args.beta = 1.0

for x in loader: # load a minibatch x with N samples
x = base_aug(x)
x_1 = color_aug(x)
x_2 = geo_aug(x)

prob = np.random.rand(1)
lam = np.random.beta(args.beta, args.beta)

if prob < args.p: # Probability of applying mixtures
ggm = lam * x_2 + (1 - lam) * x # geo-geo mixtures
gcm = lam * x_2 + (1 - lam) * x_1 # geo-color mixtures
# if choosing mixture strategy, we only compute the
’mixture’ loss instead of the origin loss.
loss = lam * model(ggm, x) + (1 - lam) * model(gcm, x)
else:
# compute origin loss
loss = g * model(x_1, x) + (1 - g) * model(x_2, x)

# update gradients
optimizer.zero_grad()
loss.backward()
optimizer.step()

Algorithm 3 PyTorch-like Pseudo-code of Our Mixture Strategy.

###  4.7 Models Design and Comparison

Name | Backbone | CLD | Geometric Strategies | Mixture Strategies
---|---|---|---|---
MoCo v2 [13, 5] | MoCo v2 |  |  |
MoCo+CLD [41] | MoCo v2 | ✓ |  |
MoCo Geo | MoCo v2 |  | ✓ |
GeoCLD | MoCo v2 | ✓ | ✓ |
MixCo | MoCo v2 |  | ✓ | ✓
Table 2: MoCo based models.

In this work, we apply MoCo v2 [5] as our contrastive baseline model. We
explore the performance of MoCo v2 [5] and MoCo+CLD [41] on KWD datasets with
self-supervised pretraining. MoCo Geo replace the augmentation method of MoCo
v2 with geometric augmentation in Table 1. Based on the methods we proposed,
we construct two models: MoCo+CLD with geometric augmentation strategies
(GeoCLD) and MoCo with mixture augmentation strategies (MixCo). We compare
those MoCo based models with supervised models. For supervised models, we use
a ResNet-50 [14], pretrained on ImageNet [7]. Sup1 freezes the output of
ResNet-50 average pooling layer. Sup2 fine-tunes the pretrained ResNet-50 on
KWD-LT with full labels, as shown in Figure 2. denoted as MCC0. Instead of
using a RandomResizeCrop, we apply a PyTorch RondomCrop to input images to
keep more information. We set the $\lambda$, number of clusters to 0.25 and 32
respectively. For MixCo we set $\gamma=0.9$ and $p=0.3$. Detailed information
of different models are outlined in Table 2.

Here is the conceptual comparison of four model mechanisms, as illustrated in
Figure 8. (a) MoCo has two branches: query branch and key branch. MoCo encodes
the latest keys by a momentum-updated key encoder and maintains a dictionary
of keys. (b) CLD head has two symmetric branches and two views of inputs. Two
symmetric branches are used to compute centroids of a _mini_ -batch of
representations. CLD incorporates local clustering into an instance
discrimination _pretext_ task. The centroid of the assigned cluster serves as
a positive key and the other centroids as negative keys. The CLD head is
trained without a memory bank in a contrastive way like SimCLR [4]. (c) GeoCLD
is a combination of MoCo and CLD. The MoCo and CLD share the same inputs and
encoder but use a different projection head. We apply geometric augmentation
to one group of inputs. (d) MixCo has the same structure as MoCo. Instead of
MoCo v2 augmentation methods, we apply our mixup strategies to the inputs of
MixCo and then add a symmetric branch to original MoCo model. Parameters of
all models are updated by minimizing the total loss:

$\displaystyle\textbf{GeoCLD:}\quad\mathcal{L}_{CLD}=\frac{1}{2}\left[\mathcal{L}_{q}(\boldsymbol{q}_{1},\boldsymbol{k}^{+})+\mathcal{L}_{q}(\boldsymbol{q}_{2},\boldsymbol{k}^{+})\right]+\lambda\times\frac{1}{2}\left[\mathcal{L}_{g}(\boldsymbol{g}_{1},\boldsymbol{C}(\boldsymbol{g}_{2}))+\mathcal{L}_{g}(\boldsymbol{g}_{2},\boldsymbol{C}(\boldsymbol{g}_{1})\right]$
(11) $\displaystyle\textbf{MixCo (with
mixup):}\quad\mathcal{L}_{M}=\lambda\mathcal{L}_{q}(q_{M},k^{+})+(1-\lambda)\mathcal{L}_{q}(q_{M}^{\prime},k^{+})$
$\displaystyle\textbf{MixCo (without
mixup):}\quad\mathcal{L}_{Geo}=\gamma\mathcal{L}_{q}(q_{1},k^{+})+(1-\gamma)\mathcal{L}_{q}(q_{2},k^{+})$

Our idea is to keep the original model framework and apply our domain-specific
strategies to improve the downstream task performance.

(a) Original MoCo [13].

(b) Original CLD [41]

(c) GeoCLD.

(d) MixCo.

Figure 8: Overview and comparison of MoCo, CLD, GeoCLD, and MixCo

##  5 Experiments

Optimizer. We use stochastic gradient descent for self-supervised pretraining
and downstream task fine-tuning. For the self-supervised pretraining, we apply
the same cosine decay scheduler as proposed in [41]. For the semi-supervised
fine-tuning, the initial learning rate is 0.01 and we likewise apply a cosine
decay schedule [4]. For the downstream task, we set the initial learning rate
as 30 and we apply same strategy proposed in [13].

Training details. All models are trained on a single node with a single GPU.
We pretrain our SSL model on one Nvidia GeForce RTX 3090 (with 24 GB memory)
for 200 epochs with a batch size of 64. We train an ImageNet pretrained
ResNet-50 end-to-end with labels on Kuzikus-Patch dataset using EPFL cluster.
The linear classifier is trained on one Nvidia TITAN V (with 12 GB memory) for
100 epochs with a batch size of 256. We fine-tune the model _end-to-end_ on
one Nvidia GeForce RTX 3090 (with 24 GB memory) for 200 epochs with a batch
size of 256. One SSL pretraining costs 14-24 hours.

Base Encoder and Projection Head. We apply a ResNet-50 without pretrained on
ImageNet as our base encoder. We simply remove the last fully-connected layer
and use the output of average pooling as feature vector $\boldsymbol{z}_{i}$.
We adopt a Multi-Layer Perceptron (MLP) head following [4, 5], which is a
2-layer MLP (2048-dimensional hidden layer, with ReLU). We share the hidden
layer and apply a different final layer of the MLP head to MoCo branch and CLD
branch. The dimension of the unit-length feature representation $q_{i}$ and
$k^{\\{+,-\\}}$ is 128.

Hyperparameter Choice. For fair comparison and avoiding hyper-parameters
tuning redundancy, we select the CLD weight $\lambda$ and number of cluster by
linear classification on frozen features with labeled data. According to our
prior knowledge, the images in the KWD dataset are primarily composed of
artefacts as follows: animal, tree, grass, tire mark (road), animal beneath
tree, and dead tree trunk, as illustrated in Figure 4. Ideally, the number of
all possible clusters is therefore
$C_{6}^{0}+C_{6}^{1}+C_{6}^{2}+C_{6}^{3}+C_{6}^{4}+C_{6}^{5}+C_{6}^{6}=2^{6}=64$.
Among all elements, animals beneath tree are hard to recognize; also dead tree
trunks are easy confused with animals. Meanwhile, with a batch size of 64, it
might not yield all 64 combinations, but we still use 64 clusters to give the
model enough freedom. We train the MoCo+CLD model with different hyper-
parameters for 200 epochs and select the best hyper-parameter combination
based on accuracy on the validation set of the downstream recognition task.

Downstream Recognition Task. We verify different models by applying linear
classification on encoded image representations. We follow the same common
linear classification protocol as [13]. We first perform self-supervised
pretraining on KWD-Pre dataset. Then we perform two kinds of experiments: (1)
_frozen_ features: we freeze the output features of the global average pooling
layer of a ResNet and train a linear classifier (a fully-connected layer
followed by softmax) [13] in a supervised way on our KWD-LT downstream task
dataset; (2) _end-to-end_ : we fine-tune the base encoder and linear
classifier by softmax loss instead of contrastive loss. And we report the
linear classification top-1 accuracy on the KWD-LT validation set, as well as
recall and precision for foreground class.

##  6 Results and Discussion

###  6.1 Self-Supervised Pretraining

Model | Acc | Prec | Rec
---|---|---|---
ImageNet Pretraining (Sup1) | 86.7 | 96.5 | 76.1
End-to-End Finetuning (Sup2) | 88.6 | 99.3 | 77.7
MoCo v2 | 83.9 | 95.3 | 71.1
MoCo+CLD | 92.8 | 98.5 | 86.8
MoCo Geo (ours) | 88.9 | 96.6 | 80.4
GeoCLD (ours) | 93.6 | 99.5 | 87.4
MixCo (ours) | 94.2 | 99.3 | 89.0
Table 3: Linear classifier top-1 accuracy (%), foreground class precision and
recall (%) on _frozen_ features with 10% labels, comparison of self-supervised
learning on KWD-Pre (MoCo v2, MoCo+CLD, MoCo Geo, GeoCLD, MixCo) and
supervised pretraining on ImageNet (Sup1, Sup2) which is trained with full
labels. Figure 9: Comparison of five contrastive loss mechanisms under the
KWD-LT linear classification protocol. We only vary the models in Table 2.

(a) The loss of MoCo v2 and MixCo.

(b) The KNN accuracy of MoCo v2 and MixCo.

Figure 10: t-SNE feature visualization of MoCo v2, MoCo+CLD, GeoCLD, and MixCo
on KWD-LT. All our proposed methods have earlier and better separation between
foreground and background classes (indicated by color) than MoCo v2.

We verify different pretraining models by linear classification accuracies. We
follow the same common linear classification protocol as [13]. We first
perform self-supervised pre-training on KWD-Pre dataset. We freeze the output
features of the global average pooling layer of a ResNet. Then we train a
linear classifier in a supervised way on our KWD-LT dataset and report the
top-1 classification accuracy on the KWD-LT validation set. For the hyper-
parameters and training process of the linear classifier, we follow the same
settings as [13].

The possible repeated and highly correlated patches slow the training process
and lower the performance of SSL pretraining as it breaks the instance
discrimination presumption described in Section 1. For a fair comparison, we
evaluate the performance of self-supervised pretraining by linear
classification of _frozen_ features with 10% labels. The linear classifier
accuracy in Table 3 shows that SSL pretraining on the target dataset is an
intense competition of supervised pretraining on ImageNet. MoCo with CLD
(MoCo+CLD) performs much better than supervised fine-tuning (Sup2). With
controlled and designed augmentation for our scenario, GeoCLD outperforms
MoCo+CLD by 0.8%, and MixCo outperforms MoCo+CLD by 1.4%. We can tell that SSL
pretraining with controlled augmentation can improve the performance of rare
wildlife recognition. Our strategies help learn geometric invariant
information in the latent space as well as the less discriminative visual
features in the raw images. However, in Figure 9, the accuracy of GeoCLD in
epoch 200 is slightly lower than that in epoch 150. That might imply that
naïvely adding geometric transformation might cause the problem of
overfitting. The training process of MoCo fluctuates and applying geometric or
image mixup strategies can stable the training and increase the accuracy
steadily. Furthermore, feature visualization in Figure 10 shows that CLD with
geometric augmentation (GeoCLD) and MixCo converge faster and better towards a
more distinctive feature representation than MoCo v2 and MoCo+CLD.

###  6.2 Fine-Tuning with Few Labels

Self-supervised pretraining on our dataset can utilize annotations far more
efficiently than supervised pretraining on ImageNet. As shown in Table 4,
fine-tuning the encoder _end-to-end_ will completely destroy the capacity of
ImageNet pretrained model. Even though we freeze the feature representations,
the model perform poorly with small fraction of labeled instances. However,
fine-tuning only linear classifier on _frozen_ features with 1% and 10%
annotations outperforms fine-tuning the encoder _end-to-end_. When we only
train the linear classifier with 1% of labels, GeoCLD outperforms MoCo v2 by
9.4% and MoCo+CLD by 11.5%. For MoCo+CLD, fine-tuning encoder with 10%
annotations outperforms the recognition accuracy with full annotations. For
GeoCLD, we need 20% annotations to get a better result. Meanwhile, for
MoCo+CLD, fine-tuning the encoder with 20% of labeled instances will have the
same performance with fine-tuning ImageNet pretrained model (Sup2) with full
labeled instances. Whereas for GeoCLD, only 10% labeled data is required to
outperform the Sup2 supervised model. The results show that SSL model can
learn more information through geometric invariant mapping and capturing
geometric invariant information can benefit UAV top view imagery task.

The results of Sup1, Sup2, MoCo v2, and MoCo+CLD in Table 3 and of these
_frozen_ features in Table 4 show that increasing the fraction of labels used
can improve the performance of linear classification. But it has a bottleneck
performance of 88.6%. This is different in training linear classifier on
GeoCLD. Although adding extra geometric augmentation can boost performance.
Feeding all annotated data can decrease the performance of _frozen_ features
in GeoCLD, as shown in Table 3 and 4. That could be caused by imbalanced
classes: the model overfits to the background class and is over-confident.

Model | Fine-tuning | 1% labels | 10% labels | 20% labels
---|---|---|---|---
strategies | (on KWD-LT) | Acc | Prec | Rec | Acc | Prec | Rec | Acc | Prec | Rec
Sup2 | _end-to-end_ | 50.4 | 0 | 0 | 69.8 | 98.8 | 40.2 | 80.5 | 97.3 | 62.8
Sup1 |  _frozen_ features | 54.4 | 100.0 | 9.0 | 69.6 | 99.3 | 39.3 | 76.2 | 99.5 | 52.7
MoCo v2 | _end-to-end_ | 68.0 | 98.5 | 35.9 | 76.8 | 99.7 | 53.4 | 77.9 | 99.7 | 55.4
_frozen_ features | 74.0 | 97.1 | 49.1 | 76.9 | 96.8 | 55.1 | 77.4 | 97.3 | 56.1
MoCo+CLD | _end-to-end_ | 70.5 | 98.5 | 40.9 | 76.0 | 99.7 | 51.6 | 88.5 | 99.3 | 77.4
_frozen_ features | 71.9 | 94.8 | 46.0 | 82.5 | 98.2 | 66.0 | 83.8 | 98.2 | 68.8
GeoCLD | _end-to-end_ | 78.9 | 98.0 | 58.7 | 90.7 | 99.8 | 81.3 | 91.9 | 100 | 83.7
_frozen_ features | 83.4 | 97.0 | 68.6 | 91.7 | 98.8 | 84.5 | 90.1 | 99.1 | 81.4
Table 4: Animal recognition accuracy when using a portion of the available
labeled samples in the final classifier (_frozen_ features) or in the base
encoder (_end-to-end_) and when varying the type of pretraining, the self-
supervised strategy and the type of fine tuning. Prec = Precision, Rec =
Recall. ∗ The results are not updated and are different from Table 3 because
we improve the pretraining process and the linear classification of the models
in Table 3. For the pretraining process of these old result in this table,
please check our preprint paper https://arxiv.org/abs/2108.07582.

###  6.3 Why geometric and mixup performs better on UAV images?

For kNN monitor, we perform a grid search and find the optimal $k$, $t$ are 20
and 0.02 for our KWD-LT dataset. Comparing the kNN accuracies of MoCo v2 with
MoCo Geo in Figure 11(b), MoCo+CLD with GeoCLD in Figure 11(d), we can find
that adding geometric augmentation can prevent overfitting.

Mixing the image with its different views will increase the amount of objects
in the final image. Thus mixup strategies can reduce the imbalance between
positive and negative samples. Rotation is one kind of stronger augmentations.
The augmentation methods in MoCo v2 [5] are mainly about texture
transformation (color augmentation in Table 1), which will capture content
information, _e.g._ , color change. However, our data have very similar
pattern and the objects of interest have very similar colors and are very tiny
in the raw images. Thus, rotation augmentation will help models ”perceive” the
orientation change of tiny objects further capture the invariant information.

(a) Loss of MoCo v2, MoCo Geo, and MixCo

(b) kNN accuracy of MoCo v2, Geo, and MixCo

(c) Loss of MoCo+CLD and GeoCLD

(d) kNN accuracy of MoCo+CLD and GeoCLD

Figure 11: Monitoring loss and KNN accuracy of all models during the
pretraining.

###  6.4 Why CLD performs better on class imbalanced dataset?

CLD [41] groups similar samples and then perform the contrastive mechanism.
Clustering instead of directly comparing instances reduces the imbalance
between positive and negative samples. For instance, in MoCo, the dictionary
size is 4,096, which means each input instance is compared against 4,096
negative samples. After performing local clustering, there are only $k$
clusters, which means each input instance only needs comparing with $k$
negative samples (centroids of $k$ clusters). The importance of positives
increases from 1/4096 to 1/k [41]. CLD thus has better performance regarding
invariant mapping for both head and tail classes.

##  7 Longer Training

We train MoCo v2 and MixCo for 800 epoch. The results show that both models
benefit from a longer training compared to its supervised counterpart. In
supervised learning, longer training might cause the problem of overfitting.
The reason is that training longer provides more negative examples, which can
futher improve the results [4]. Furthermore, MixCo converges faster than MoCo
and longer training might cause overfitting. The results in Table 5 are
slightly different from the results in Table 3. This is caused by same initial
learning rate and different training epochs in the learning rate cosine decay
strategy.

Model | Epochs | 100 | 200 | 300 | 600 | 800
---|---|---|---|---|---|---
MoCo v2 | Linear Acc. | 83.1 | 79.0 | 84.6 | 87.0 | 90.0
MixCo (ours.) | Linear Acc. | 89.6 | 92.0 | 92.9 | 94.0 | 93.8
Table 5: Linear accuracy (10% labels) for long time training.

(a) The loss of MoCo v2 and MixCo.

(b) The KNN accuracy of MoCo v2 and MixCo.

Figure 12: Monitoring loss and KNN accuracy for a longer training.

###  7.1 Ablation Study

The results for the ablation studies on the number of clusters for CLD and
$\lambda$ are shown in Tables 6 and 7. We can clearly see that the model
performs almost equally when $\lambda=0.25$ or $\lambda=1$. Hence, we chose
the default $\lambda=0.25$ value as in [41]. However for the number of
clusters, there does not seem to be an obvious trend. we speculate that (1)
grouping projected _representations_ by $k$-means clustering is hard to
perform well, (2) the model is not able to recognize animals beneath the tree
or/and to distinguish between dead tree trunks and animals (number of clusters
= 16 or 32). When number of clusters = 16, 32, or 64, the accuracy is equally
good. Therefore, we chose the best recall with number of cluster = 32.

For MixCo, we exploit the effect of different ratio of original MoCo v2 branch
and mixup probability. The results have shown that with Geo:Color = 1:9 and
30% mixup probability, the model has the best performance on KWD dataset with
only 10% labels.

Model | $\lambda$ | Acc | Prec | Rec
---|---|---|---|---
MoCo+CLD | 0.1 | 86.2 | 98.6 | 73.4
MoCo+CLD | 0.25 | 88.4 | 98.9 | 77.4
MoCo+CLD | 0.5 | 86.4 | 98.9 | 73.5
MoCo+CLD | 1 | 88.1 | 98.5 | 77.2
CLD | 1 | 60.9 | 98.9 | 22.1
Table 6: Hyper-parameter $\lambda$ selection. All models are pretrained on KWD-Pre using the MoCo+CLD strategy. Model | Clusters | Acc | Prec | Rec
---|---|---|---|---
MoCo+CLD | 16 | 88.3 | 100.0 | 76.5
MoCo+CLD | 30 | 87.3 | 99.7 | 75.0
MoCo+CLD | 32 | 88.4 | 98.9 | 77.4
MoCo+CLD | 48 | 87.2 | 99.7 | 74.9
MoCo+CLD | 64 | 88.4 | 100.0 | 76.9
Table 7: Number of clusters selection. All models are pretrained on KWD-Pre
using the MoCo+CLD strategy with $\lambda=0.25$.

Model | $p$ | Acc | Prec | Rec
---|---|---|---|---
MixCo | 0 | 92.172 | 98.121 | 85.894
MixCo | 0.1 | 90.488 | 94.366 | 85.867
MixCo | 0.3 | 92.761 | 98.859 | 86.448
MixCo | 0.5 | 91.498 | 96.390 | 86.100
MixCo | 0.7 | 90.572 | 99.424 | 81.469
MixCo | 0.9 | 90.320 | 97.000 | 83.045
MixCo | 1 | 91.582 | 98.199 | 84.625
(a) $\gamma=0.5$, different mixup-p.

Model | $\gamma$ | Acc | Prec | Rec
---|---|---|---|---
MixCo | 0 | 88.889 | 94.719 | 82.264
MixCo | 0.3 | 90.152 | 91.166 | 88.718
MixCo | 0.5 | 92.761 | 98.859 | 86.448
MixCo | 0.6 | 92.172 | 99.046 | 84.945
MixCo | 0.9 | 94.192 | 99.263 | 89.040
MixCo | 1 | 91.667 | 99.613 | 83.592
(b) mixup-p=0.3, different $\gamma$.

Table 8: Ablation study of the MixCo model.

##  8 Conclusion

In this paper, our proposed strategy reduces the label requirements in the
wildlife recognition tasks. The problem is introduced by applying supervised
learning to automated animal censuses in largely remote areas with aerial
imagery, in which scenario the annotations are expensive to be obtained. The
contrastive self-supervised pretraining with domain-specific geometric
transformation and mixup strategies outperforms the performance of fine-tuning
ImageNet pretrained model with full labels. Results show that the geometric
invariant mapping method can capture information more efficiently of wildlife
in UAV images than method without geometric augmentation. Extensive
experiments further prove the effectiveness of recognizing rare wildlife with
reduced labels.

##  9 Acknowledgement

This project would not have been possible without the support of many people.
Many thanks to my adviser, Dr. Benjamin Alexander Kellenberger, who read my
numerous revisions and helped make some sense of the confusion. His unassuming
approach to research and science is a source of inspiration. This approach is
reflected by his simple but clear writing style, which is something I hope to
carry forward throughout my career. Thanks to my supervisor, Prof. Dr. Devis
Tuia. Devis has been an ideal teacher, mentor, and thesis supervisor, offering
advice and encouragement with a perfect blend of insight and humor. Also
thanks to my supervisor, Prof. Dr. Irena Hajnsek who offered invaluable
guidance and support. I’m proud of, and grateful for, my time working with
Beni, Devis, and Irena.

Thanks to my dear colleague, Rui Gong. Rui is a good friend and a good
teacher. Without his inspired idea, I could not get to my destination. Long
live our friendship and wish you all the success with your PhD journey.

Thanks to ETH Zürich for providing me with the financial support to complete
this project. And finally, thanks to parents, and numerous friends who endured
this long process with me, always offering support and love. I will remember
this experience for a life long time.

## References

  * [1] Ayush, K., Uzkent, B., Meng, C., Tanmay, K., Burke, M., Lobell, D., Ermon, S.: Geography-aware self-supervised learning. arXiv preprint arXiv:2011.09980 (2020)
  * [2] Balmford, A., Bennun, L., Ten Brink, B., Cooper, D., Côté, I.M., Crane, P., Dobson, A., Dudley, N., Dutton, I., Green, R.E., et al.: The convention on biological diversity’s 2010 target. Science 307(5707), 212–213 (2005)
  * [3] Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., Joulin, A.: Unsupervised learning of visual features by contrasting cluster assignments. In: Advances in Neural Information Processing Systems (NeurIPS) (2020)
  * [4] Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: Proceedings of the 37th International Conference on Machine Learning (ICML) (2020)
  * [5] Chen, X., Fan, H., Girshick, R., He, K.: Improved baselines with momentum contrastive learning. arXiv preprint arXiv: 2003.04297 (2020)
  * [6] Chen, X., He, K.: Exploring simple siamese representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021)
  * [7] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: A Large-Scale Hierarchical Image Database. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2009)
  * [8] Ding, J., Xie, E., Xu, H., Jiang, C., Li, Z., Luo, P., Xia, G.S.: Unsupervised pretraining for object detection by patch reidentification. arXiv preprint arXiv:2103.04814 (2021)
  * [9] Doersch, C., Gupta, A., Efros, A.A.: Unsupervised visual representation learning by context prediction. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2015)
  * [10] Eikelboom, J.A., Wind, J., van de Ven, E., Kenana, L.M., Schroder, B., de Knegt, H.J., van Langevelde, F., Prins, H.H.: Improving the precision and accuracy of animal population estimates with aerial image object detection. Methods in Ecology and Evolution 10(11), 1875–1887 (2019)
  * [11] Gidaris, S., Singh, P., Komodakis, N.: Unsupervised representation learning by predicting image rotations. In: International Conference on Learning Representations (ICLR) (2018)
  * [12] Grill, J.B., Strub, F., Altché, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., Piot, B., kavukcuoglu, k., Munos, R., Valko, M.: Bootstrap your own latent - a new approach to self-supervised learning. In: Advances in Neural Information Processing Systems (NeurIPS) (2020)
  * [13] He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)
  * [14] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2016)
  * [15] Hendrycks, D., Mu, N., Cubuk, E.D., Zoph, B., Gilmer, J., Lakshminarayanan, B.: Augmix: A simple method to improve robustness and uncertainty under data shift. In: International Conference on Learning Representations (ICLR) (2020)
  * [16] Jean, N., Wang, S., Samar, A., Azzari, G., Lobell, D., Ermon, S.: Tile2vec: Unsupervised representation learning for spatially distributed data. In: Proceedings of the AAAI Conference on Artificial Intelligence (2019)
  * [17] Kalantidis, Y., Sariyildiz, M.B., Pion, N., Weinzaepfel, P., Larlus, D.: Hard negative mixing for contrastive learning. In: Advances in Neural Information Processing Systems (NeurIPS) (2020)
  * [18] Kang, J., Fernandez-Beltran, R., Duan, P., Liu, S., Plaza, A.J.: Deep unsupervised embedding for remotely sensed images based on spatially augmented momentum contrast. IEEE Transactions on Geoscience and Remote Sensing 59(3), 2598–2610 (2020)
  * [19] Kellenberger, B., Marcos, D., Lobry, S., Tuia, D.: Half a percent of labels is enough: Efficient animal detection in UAV imagery using deep cnns and active learning. IEEE Transactions on Geoscience and Remote Sensing 57(12), 9524–9533 (2019). https://doi.org/10.1109/TGRS.2019.2927393
  * [20] Kellenberger, B., Marcos, D., Tuia, D.: Detecting mammals in UAV images: Best practices to address a substantially imbalanced dataset with deep learning. Remote Sensing of Environment 216, 139–153 (2018). https://doi.org/https://doi.org/10.1016/j.rse.2018.06.028, https://www.sciencedirect.com/science/article/pii/S0034425718303067
  * [21] Kellenberger, B., Marcos, D., Tuia, D.: When a few clicks make all the difference: Improving weakly-supervised wildlife detection in UAV images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2019)
  * [22] Li, J., Zhou, P., Xiong, C., Hoi, S.: Prototypical contrastive learning of unsupervised representations. In: International Conference on Learning Representations (ICLR) (2021)
  * [23] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft COCO: Common objects in context. In: Proceedings of the European Conference on Computer Vision (ECCV) (2014)
  * [24] Linchant, J., Lisein, J., Semeki, J., Lejeune, P., Vermeulen, C.: Are unmanned aircraft systems (uas s) the future of wildlife monitoring? a review of accomplishments and challenges. Mammal Review 45(4), 239–252 (2015)
  * [25] Liu, X., Zhang, F., Hou, Z., Mian, L., Wang, Z., Zhang, J., Tang, J.: Self-supervised learning: Generative or contrastive. IEEE Transactions on Knowledge and Data Engineering (2021)
  * [26] Misra, I., Maaten, L.v.d.: Self-supervised learning of pretext-invariant representations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)
  * [27] Misra, I., Maaten, L.v.d.: Self-supervised learning of pretext-invariant representations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)
  * [28] Noroozi, M., Favaro, P.: Unsupervised learning of visual representations by solving jigsaw puzzles. In: Proceedings of the European Conference on Computer Vision (ECCV) (2016)
  * [29] Oord, A.v.d., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018)
  * [30] Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: Feature learning by inpainting. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2016)
  * [31] Raia, H., Sumit, C., Yann, L.: Dimensionality reduction by learning an invariant mapping. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2006)
  * [32] Randin, C.F., Ashcroft, M.B., Bolliger, J., Cavender-Bares, J., Coops, N.C., Dullinger, S., Dirnböck, T., Eckert, S., Ellis, E., Fernández, N., et al.: Monitoring biodiversity in the anthropocene using remote sensing in species distribution models. Remote Sensing of Environment 239, 111626 (2020)
  * [33] Rey, N., Volpi, M., Joost, S., Tuia, D.: Detecting animals in african savanna with UAVs and the crowds. Remote Sensing of Environment 200, 341–351 (2017)
  * [34] Robinson, J.D., Chuang, C.Y., Sra, S., Jegelka, S.: Contrastive learning with hard negative samples. In: International Conference on Learning Representations (ICLR) (2021)
  * [35] Shen, Z., Liu, Z., Liu, Z., Savvides, M., Darrell, T., Xing, E.: Un-mix: Rethinking image mixtures for unsupervised visual representation learning. arXiv preprint arXiv:2003.05438 (2020)
  * [36] Stojnic, V., Risojevic, V.: Self-supervised learning of remote sensing scene representations using contrastive multiview coding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2021)
  * [37] Tao, C., Qi, J., Lu, W., Wang, H., Li, H.: Remote sensing image scene classification with self-supervised paradigm under limited labeled samples. IEEE Geoscience and Remote Sensing Letters (2020)
  * [38] Tian, Y., Krishnan, D., Isola, P.: Contrastive multiview coding. In: Proceedings of the European Conference on Computer Vision (ECCV) (2020)
  * [39] Tian, Y., Sun, C., Poole, B., Krishnan, D., Schmid, C., Isola, P.: What makes for good views for contrastive representation learning? In: Advances in Neural Information Processing Systems (NeurIPS) (2020)
  * [40] Tong, X.Y., Xia, G.S., Lu, Q., Shen, H., Li, S., You, S., Zhang, L.: Land-cover classification with high-resolution remote sensing images using transferable deep models. Remote Sensing of Environment 237, 111322 (2020). https://doi.org/https://doi.org/10.1016/j.rse.2019.111322, https://www.sciencedirect.com/science/article/pii/S0034425719303414
  * [41] Wang, X., Liu, Z., Yu, S.X.: Unsupervised feature learning by cross-level instance-group discrimination. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021)
  * [42] Wu, Z., Xiong, Y., Yu, S.X., Lin, D.: Unsupervised feature learning via non-parametric instance discrimination. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2018)
  * [43] Xiao, T., Wang, X., Efros, A.A., Darrell, T.: What should not be contrastive in contrastive learning. In: International Conference on Learning Representations (ICLR) (2021)
  * [44] Yun, S., Han, D., Oh, S.J., Chun, S., Choe, J., Yoo, Y.: Cutmix: Regularization strategy to train strong classifiers with localizable features. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (October 2019)
  * [45] Zhang, H., Cisse, M., Dauphin, Y.N., Lopez-Paz, D.: mixup: Beyond empirical risk minimization. In: International Conference on Learning Representations (ICLR) (2018)
  * [46] Zhang, R., Isola, P., Efros, A.A.: Colorful image colorization. In: Proceedings of the European Conference on Computer Vision (ECCV) (2016)
  * [47] Zhao, N., Wu, Z., Lau, R.W.H., Lin, S.: What makes instance discrimination good for transfer learning? In: International Conference on Learning Representations (ICLR) (2021)