††footnotetext: Copyright © 2020 for this paper by its authors. Use permitted
under Creative Commons License Attribution 4.0 International (CC BY 4.0). CLEF
2020, 22-25 September 2020, Thessaloniki, Greece.11institutetext:  German
Research Center for Artificial Intelligence (DFKI)
Saarland Informatics Campus, 66123 Saarbrücken, Germany
<EMAIL_ADDRESS>
# A Competitive Deep Neural Network Approach for the ImageCLEFmed Caption 2020
Task

Marimuthu Kalimuthu  Fabrizio Nunnari  Daniel Sonntag

###### Abstract

The aim of ImageCLEFmed Caption task is to develop a system that automatically
labels radiology images with relevant medical concepts. We describe our Deep
Neural Network (DNN) based approach for tackling this problem. On the
challenge test set of 3,534 radiology images, our system achieves an F1 score
of 0.375 and ranks high, 12th among all systems that were successfully
submitted to the challenge, whereby we only rely on the provided data sources
and do not use any external medical knowledge or ontologies, or pretrained
models from other medical image repositories or application domains.

Keywords: Medical Imaging, Concept Detection, Image Labeling, Multi-Label
Classification, Deep Convolutional Neural Networks.

##  1 Introduction

ImageCLEF organises 4 main tasks for the 2020 edition with a global objective
of promoting the evaluation of technologies for annotation, indexing, and
retrieval of visual data with the aim of providing information access to large
collections of images in various usage scenarios and application domains,
including medicine [4].

Interpreting and summarizing the insights gained from medical images is a
time-consuming task that involves highly trained experts and often represents
a bottleneck in clinical diagnosis pipelines. Consequently, there is a
considerable need for automatic methods that can approximate this mapping from
visual information to condensed textual descriptions. The more image
characteristics are known, the more structured are the radiology scans and
hence, the more efficient are the radiologists regarding interpretation, see
https://www.imageclef.org/2020/medical/caption/ and [11].

Recent years have witnessed tremendous advances in deep neural networks in
terms of architectures, optimization algorithms, tooling and techniques for
training large networks, and handling multiple modalities (e.g., text, images,
videos, speech, etc.). In particular, deep convolutional neural networks have
proved to be extremely successful image encoders and have thus become the de
facto standard for visual recognition [2, 3, 13]. We have been working on
machine learning problems in several medical application domains [14, 15, 16,
17] in our projects, see https://ai-in-medicine.dfki.de/. In this paper, we
describe how we built a competitive deep neural network approach based on
these projects.

The rest of the paper is organized as follows. In Section 2, we formally
describe the challenge and its goal. In Section 3, we present some statistics
on the dataset and explain the approach we adopted to tackle the challenge. In
Section 4, we describe the experiments that we conducted with different
architectures and introduce a new loss function that addresses the sparsity
problem in ground truth labels. In Section 5, results are presented followed
by a short discussion. Finally, we summarize our work in Section 6 and discuss
some future directions.

##  2 The Challenge

The overarching goal of ImageCLEFmed Caption challenge is to assist medical
experts such as radiologists in interpreting and summarising information
contained in medical images. As a first step towards this goal, a simpler task
would be to detect as many key concepts as possible, with the goal that these
concepts can then be composed into comprehensible sentences, and eventually
into medical reports. For full details about the challenge, we refer the
reader to Pelka et al. [11].

The challenge has evolved over the years, since its first edition in 2017, to
focus only on radiology images in this year’s version, and incorporating the
lessons learned from previous years. The aim of 2020 ImageCLEFmed Caption
challenge is to develop a system that would automatically assign medical
concepts to radiology images that were sorted into 7 different categories (see
Table 1). More concretely, given an image ($\mathcal{I}$), the objective is to
learn a function $\mathcal{F}$ that maps $\mathcal{I}$ to a set of concepts
($\mathcal{C}_{1}$, $\mathcal{C}_{2}$, …, $\mathcal{C}_{v_{\mathcal{I}}}$)
where $v_{\mathcal{I}}$ is the number of concepts associated with
$\mathcal{I}$. A peculiarity of this challenge is that $v\ll k$, where $k$ is
the total number of unique labels.

$\displaystyle\mathcal{F}:\mathcal{I}\rightarrow(\mathcal{C}_{1},\mathcal{C}_{2},...,\mathcal{C}_{v_{\mathcal{I}}})$
(1)

On the challenge data, the ground truth concepts are not known for images in
the test set. The only known constraint is that predictions of the test set
must be submitted with a maximum of 100 non-repeating concepts per image.

The performance of submissions to the challenge are evaluated on a withheld
test set of 3,534 radiology images using the $F1$ score evaluation metric,
which is defined as the harmonic mean of precision and recall values.

$\displaystyle{F}1\;$
$\displaystyle=\;2\>*\>\frac{precision\,*\,recall}{precision\,+\,recall}$ (2)

Firstly, the instance level F1 scores are computed using the predicted
concepts for images in the test set. Later, an average F1 score is computed
over all images in the test set using scikit-learn111https://scikit-
learn.org/stable/modules/generated/sklearn.metrics.f1_score.html library’s
default binary averaging method. This yields the final F1 score for an
accepted submission.

All registered teams are allowed for a maximum of 10 submissions. Successful
submissions and teams are then ranked based on the achieved F1 scores and
results are made publicly available.

##  3 Method

We provide information about the dataset and some analysis that we performed
on it during the exploratory phase. Then, we discuss our learning approach and
a suitable data preparation strategy for training our models.

###  3.1 Dataset

All participants are provided with ImageCLEFmed Caption dataset which is a
subset of the ROCO dataset [12]. It is a multi-modal, medical images dataset
containing radiology images that are each labelled with a set of medical
concepts, called as Concept Unique Identifiers (CUIs) in the literature. As
common in many challenges, the provided images are already split into _train_
, _validation_ , and _test_ sets. Images of the first two sets come with
ground truth labels, while the test set contains only images.

Table 1:  Terminology used in the ROCO dataset. Abbreviation | Full form
---|---
DRAN | DR Angiography
DRCO | DR Combined modalities in One image
DRCT | DR Computerized Tomography
DRMR | DR Magenetic Resonance
DRPE | DR Positron Emission Tomography
DRUS | DR Ultrasound
DRXR | DR X-Ray, 2D Tomography

Moreover, images in each of the splits are sorted into one of the 7 categories
as described in tables 1 and 2; such information can be inferred from the name
of the sub-directory containing the images. As we can observe from Table 2,
the images are not equally distributed across categories, indicating an
imbalance in the dataset. The DRCT category, which contains images captured
using Computerized Tomography (CT), has the highest representation, followed
by X-ray images (DRXR), while the least number of images (approximately
$\frac{1}{40}$th of the images in DRCT) are seen in DRCO category.

Table 2:  Splits and statistics of the ImageCLEFmed Caption dataset. |  Total Number of Images in the Category of  |
---|---|---
Split | DRAN | DRCO | DRCT | DRMR | DRPE | DRUS | DRXR | Total
Train | 4,713 | 487 | 20,031 | 11,447 | 502 | 8,629 | 18,944 | 64,753
Val | 1,132 | 73 | 4,992 | 2,848 | 74 | 2,134 | 4,717 | 15,970
Test | 325 | 49 | 1,140 | 562 | 38 | 502 | 918 | 3,534
Total | 6,170 | 609 | 26,163 | 14,857 | 614 | 11,265 | 24,579 | 84,257

Despite having these category labels as a meta-information, we could not
leverage them during model training due to time constraints. Thus, the
relationship between CUIs and image category labels is, for us, still
uninvestigated. Possibly, in a follow-up work we will investigate on how to
exploit such meta-information to improve classification performance of
predictive models, for example by using one of the metadata fusion strategies
tested by the authors in [9].

CUI |  UMLS Term (Concept)
---|---
C2951888 | set of bones of skull
C0037303 | set of bones of cranium
C0032743 | tomogr positron emission
C0342952 | increased basal metabolic rate

CUI | UMLS Term
---|---
C0022742 | knees
C0043299 | x-ray procedure
C1260920 | kneel
C0030647 | bone, patella

Figure 1: Sample images & their CUIs from the train split.

###  3.2 Data Analysis

Here, we outline some insights that we gained after performing analysis on the
CUIs and the category labels meta-information. Furthermore, this section sheds
light on the imbalance in the dataset, which is a common problem in many
research domains.

Figure 1 provides a conceptual representation of the input, viz. images paired
with relevant concepts. In this case, both images are labelled with four CUIs,
the descriptions of which are provided by Unified Medical Language System
(UMLS)222https://www.nlm.nih.gov/research/umls/ terms. These terms are
depicted in Figure 1 merely for the purpose of understanding since the mapping
of CUIs to UMLS terms is not part of the provided dataset.

Figure 2: Histogram of top 30 CUIs on the training split. Figure 3: Histogram
of top 30 CUIs on the validation split.

Figure 2 shows the frequencies of top 30 CUIs on the training split. Two CUIs
(C0040398, C0040405), which both occur around 20k times in the training set,
dominate this list and represent the concept “computer assisted tomography”.
Similar behavior is observed on the validation set (see Figure 3). However,
the frequencies of top 2 CUIs in this case are only around 4.6k.

Figure 4: Histogram of CUI counts vs. Images Count on the training set after
combining all of the 7 categories.

Figure 4 depicts a histogram representation of the number of images and the
CUI counts on the training set. For instance, there are around 5,200 images
that have exactly two CUIs as ground truth labels. On the contrary, there are
only around 100 images that have exactly 50 CUIs as ground truth labels in the
provided training set. This histogram is truncated at CUI count 50 (x-axis)
for clarity and uncluttered representation.

Figure 5: Histogram of CUI counts vs. Images Count on the validation set after
combining all of the 7 categories.

In a similar manner, Figure 5 shows a histogram representation of the number
of images and the CUI counts on the validation set. For instance, there are
around 1,300 images that have exactly two CUIs as ground truth labels. On the
contrary, there are only around 30 images that have exactly 50 CUIs as ground
truth labels in the validation set. This histogram is truncated at CUI count
50 (x-axis) for clarity and uncluttered representation.

On the combined training and validation set, there are 80,723 images and
907,718 non-unique CUIs. Among them, we counted 3,047 unique CUIs, which were
used to build the label space in our training objective (see Table 3).

Following Tsoumakas et al. [18], we compute the Label Cardinality (LC) on the
combined training and validation set, denoted as $\mathcal{D}$, using the
formula:

$\displaystyle LC(\mathcal{D})$
$\displaystyle=\frac{1}{|\mathcal{D}|}\sum_{i=1}^{|\mathcal{D}|}|{Y}_{i}|$ (3)

In a similar manner, we compute the Label Density (LD) using the following
formula:

$\displaystyle LD(\mathcal{D})$
$\displaystyle=\frac{1}{|\mathcal{D}|}\sum_{i=1}^{|\mathcal{D}|}\frac{|{Y}_{i}|}{|L|}$
(4)

where $|L|$ is the number of unique labels in our multi-label classification
objective. The LC and LD scores on the combined training and validation sets
are 11.24 and 0.0037 respectively.

###  3.3 Data preparation

As a first step in formatting the labels, we convert CUIs associated with
images to a format that is suitable as input for neural network learning.
Since our objective here is multi-label classification, we cannot use simple
one-hot encoding, as usually done in classification tasks, hence we apply a
multi one-hot encoding. An illustration of this representation can be found in
Table 3. Specifically, we sort the list of CUIs from the unique label set
($k$) in alphabetical order and use the positions of CUIs in the sorted list
to mark as $1$ if a specific CUI is associated with the image in question,
else as $0$. After this conversion step, the label set for each image
($\mathcal{I}$) is represented as a single multi one-hot vector of fixed size
$k$, which is equal to 3,047.

Table 3:  Encoding CUIs using multi one-hot representation. | Multi One-Hot Encoding
---|---
Image | CUI01, CUI02, CUI03, …, CUI3047
$ROCO2\\_CLEF\\_76012.jpg$ | …,0,1,0,0,1,0,0,0,1…
$ROCO2\\_CLEF\\_05856.jpg$ | …,0,0,0,0,0,0,0,0,1…
$ROCO2\\_CLEF\\_45763.jpg$ | …,0,1,0,0,1,0,1,0,0…
⋮ | ⋮

We used only the images from ImageCLEFmed Caption dataset and did not use pre-
training on external datasets, or utilize other modalities such as text during
model training.

For our experiments, we divided the validation set via random sampling into
two equally sized subsets, namely _val1_ and _val2_. We conducted our internal
evaluations by always training pairs of models, first using _val1_ for
validation and _val2_ for testing, and then vice-versa. When results were
promising, we then submitted our predictions on the test set images by using
the model trained with first configuration. Training a third model, based on
the validation on the full development set would have been the ideal solution.
However, this could not be applied in our case because of time constraints.

###  3.4 Learning Approach

The prediction problem for this challenge lays in the category of _multi-label
classification_ [18]. It differs from most common classification problems in
the fact that each sample of the dataset is simultaneously associated with
more than one class from the ground truth label pool.

Technically, when addressing such a problem with deep neural networks, it
means that the final classification layer relies on multiple sigmoidal units
rather than a single softmax probability distribution. The last layer of the
network contains one sigmoidal unit for each of the target classes, and the
association with a true/false result is performed by thresholding the final
sigmoid activation value (usually at 0.5).

To address the challenge, we followed a classical transfer learning approach
starting from a Convolutional Neural Network (CNN) model pre-trained on an
image classification problem, namely ImageNet, because the pre-trained network
already offers the ability to detect basic image features, viz. edges,
borders, and corners. Then, the final classification stage of the network
(i.e., all the layers after the last convolutional layer) is substituted with
randomly initialized fully connected layers. Finally, the network is fitted
for the new target training set.

In detail, we used VGG16 [13], ResNet50 [2], and DenseNet169 [3], all of them
pre-trained on ImageNet data used for ILSVRC [6]. An example configuration
based on VGG16 is shown in listing 1. Layers from block1_conv1 to block5_pool
are pre-trained on ImageNet, and unlocked for further training. Remaining
layers, from flatten to predictions, are newly instantiated and initialized
randomly (where applicable). The final predictions layer is a dense layer with
3,047 sigmoidal activation units (one per target class).

⬇

1_________________________________________________________________

2Layer (type) Output Shape Param #

3=================================================================

4input_1 (InputLayer) (None, 227, 227, 3) 0

5_________________________________________________________________

6block1_conv1 (Conv2D) (None, 227, 227, 64) 1792

7_________________________________________________________________

8block1_conv2 (Conv2D) (None, 227, 227, 64) 36928

9_________________________________________________________________

10block1_pool (MaxPooling2D) (None, 113, 113, 64) 0

11_________________________________________________________________

12[... 13 more layers ...]

13_________________________________________________________________

14block5_conv3 (Conv2D) (None, 14, 14, 512) 2359808

15_________________________________________________________________

16block5_pool (MaxPooling2D) (None, 7, 7, 512) 0

17_________________________________________________________________

18flatten (Flatten) (None, 25088) 0

19_________________________________________________________________

20fc1 (Dense) (None, 4096) 102764544

21_________________________________________________________________

22dropout_1 (Dropout) (None, 4096) 0

23_________________________________________________________________

24fc2 (Dense) (None, 4096) 16781312

25_________________________________________________________________

26dropout_2 (Dropout) (None, 4096) 0

27_________________________________________________________________

28predictions (Dense) (None, 3047) 12483559

29=================================================================

30Total params: 146,744,103

31Trainable params: 146,744,103

32Non-trainable params: 0

33_________________________________________________________________

Listing 1: An excerpt of the VGG16 architecture used for the multi-label
classification task.

The system was developed in a Python environment using Keras deep learning
framework [1] with TensorFlow [7] as the backend. For our experiments we used
a desktop machine equipped with an 8-core $9^{th}$-gen i7 CPU, 64GB RAM, and
NVIDIA RTX TITAN 24GB GPU memory.

##  4 Experiments

In this section, we describe our experimental procedure, model configuration,
and a variety of deep CNN architectures that we tried for achieving the multi-
label classification task.

Table 4 reports the results of the experiment we conducted throughout the
challenge. Because of time constraints, rather than running a grid search for
the best hyper-parameter values, we started from a reference configuration,
already successfully used in other past works [9, 10].

Table 4: The list of experiments conducted for the challenge. # | CNN arch | res | aug. | fc | do | bs | loss | lr-red. | test1 | test2 | score
---|---|---|---|---|---|---|---|---|---|---|---
|  | (px) |  | layers |  |  | func |  | ep | F1 | ep | F1 | F1
1 | VGG16 | 227 | none | 2x2k | 0.5 | 32 | bce | 0.2/3/loss | 12 | 0.333 | 22 | 0.335 |
2 | VGG16 | 227 | none | 2x4k | 0.5 | 32 | bce | 0.2/3/loss | 25 | 0.346 | 26 | 0.336 |
3 | VGG16 | 227 | hflip | 2x4k | 0.5 | 32 | bce | 0.2/5/f1 | 9 | 0.3475 | 9 | 0.3455 | 0.363
4 | VGG16 | 450 | hflip | 2x4k | 0.5 | 24 | bce | 0.2/5/f1 | n/a | crash | 6 | 0.3417 |
5 | ResNet50 | 224 | hflip | 2x4k | 0.5 | 16 | bce | nothing | 3 | 0.3484 | 3 | 0.3487 | 0.365
6 | DenseNet169 | 224 | none | 2x4k | 0.5 | 32 | bce | nothing | 8 | 0.3495 | 10 | 0.3450 |
7 | DenseNet169 | 224 | hflip | 2x4k | 0.5 | 32 | bce | nothing | 5 | 0.3500 | 4 | 0.3463 | 0.360
8 | VGG16 | 227 | hflip | 3x4k | 0.5 | 32 | bce | 0.2/5/f1 | 13 | 0.3465 | 14 | 0.3433 |
9 | VGG16 | 227 | hflip | 2x4k | 0.5 | 48 | $\overline{F1}*bce$ | 0.2/5/f1 | 11 | 0.3604 | 7 | 0.3606 | 0.374
10 | VGG16 | 227 | hflip | 2x4k | 0.5 | 48 | $\overline{F1}+bce$ | 0.2/5/f1 | 14 | 0.3636 | 12 | 0.3632 |

Together with the base architecture used for convolution (CNN arch) we report:
(res) the resolution in pixels of the input images of equal height and width,
(aug) the data augmentation strategy, (fc layers) the configuration of final
fully-connected stage of the CNN architectures, (do) the dropout value after
each fully connected layer, (bs) the batch size used for training, (loss func)
the loss function used for optimization, (lr-red) the learning rate reduction
strategy (reduction factor/patience/monitored metric). Additionally, we report
the best training epoch, based on an early stopping criteria by monitoring the
F1 score on the validation set. The last three columns report the F1 scores
achieved on the two internal cross-validation sets and finally on the
AIcrowd333https://www.aicrowd.com/challenges/imageclef-2020-caption-concept-
detection online submission platform.

Other training parameters, common to all configurations are: NAdam optimizer,
learning rate = 1e-5, and schedule decay 0.9. All images were scaled to the
input resolution of the CNN using _nearest_ filtering, without any cropping.

In the following, we report on the evolution of our tests and obtained
results.

#### Experiments 1-4: VGG16 baseline

We started with (1) a VGG16 architecture, pretrained on ImageNet, with the
last two fully connected (FC) layers configured with n=2048 nodes, each
followed by a dropout layer with the dropout probability $p$ set to 0.5.

(2) We observed an increase in performance by increasing the size of the FC
layers to 4096 nodes.

From the first two experiments, it was evident how the loss value (based on
binary cross-entropy) could not be effectively used to monitor the validation.
The ground truth of each sample, a vector of size 3,047, contains on average
about 11 concepts per image (see Section 3.2), and only a few images contain
more than 50 concepts. Hence, the ground truth matrix is very sparse. As a
consequence, the loss function quickly stabilizes into a plateau, as does the
accuracy, which saturates to values above 0.9966 after the first epoch. Hence,
to better handle early stopping, we implemented a training-time computation of
the F1 score.

(3) A further improvement was observed by applying a 2X data augmentation of
the input dataset. Each image is provided to the training procedure both as-
it-is and flipped horizontally. At the same time, learning rate reduction was
applied by monitoring F1 scores on the validation set, rather than the loss
values. However, increasing the learning rate happens only after an
overfitting occurs. This configuration led to an online evaluation score of
0.363. Figure 6 shows the evolution of loss values and F1 scores over epochs
during model training.

Loss (Binary cross-entropy) F1 score

Figure 6: Plots of the metrics computed on the validation set during training:
(left) binary cross-entropy and (right) F1 score. Here, epochs are counted
from 0. The best validation results are achieved at epoch 9, then the model
starts overfitting.

(4) We tried to improve the performance by increasing the size of input images
to 450x450 pixels, which forced a reduction of the batch size to 24. We could
not observe any significant improvement in the accuracy, suggesting that
higher image resolutions do not provide useful details for label selection in
our case.

#### Experiments 5-7: more powerful CNN architectures

By using deeper CNN architectures, we could observe a slight improvement in
the test accuracy. Indeed, the ResNet50 architecture (5) led to an F1 score of
0.365 in the online evaluation. The DenseNet169 architecture (6-7) led to
higher test values, but the online evaluation was slightly lower (0.360) than
the ResNet50 version.

#### Experiment 8: more layers

In order to increase the overall performance, we tried to increase the number
of FC layers to 3x4k (8). However, taking as reference the performance of
configuration (3), we could not observe a significant improvement by
introducing an additional 4k FC layer to the classification stage.

#### Experiments 9-10: a new loss function

To further improve performance, we decided to directly optimize for the F1
score evaluation metric. Notice that the F1 score used in the ImageCLEF
challenge is computed as an average F1 over the samples (and not over the
labels, as more often found in online code
repositories444https://towardsdatascience.com/the-unknown-benefits-of-using-a-
soft-f1-loss-in-classification-
systems-753902c0105d,555https://www.kaggle.com/rejpalcz/best-loss-function-
for-f1-score-metric).

We implemented a loss function $\overline{F1}=1-sF1$, where $sF1$ is called
the “soft F1 score”. The $sF1$ is a differentiable version of the F1 function
that computes true positives, false positives, and false negatives as
continuous sum of likelihood values, without applying any thresholding to
round the probabilities to 0 or 1. The implementation of $\overline{F1}$ is
shown in listing 2.

⬇

1def loss_1_minus_f1(y_true, y_pred):

2

3 import keras.backend as K

4 import tensorflow as tf

5

6 # The following is not differentiable.

7 # Round the prediction to 0 or 1 (0.5 threshold)

8 # y_pred = K.round(y_pred)

9 # By commenting, we implement what is called soft-F1.

10

11 # Compute precision and recall.

12 tp = K.sum(K.cast(y_true * y_pred, ’float’), axis=-1)

13 # tn = K.sum(K.cast((1 - y_true) * (1 - y_pred), ’float’), axis=-1)

14 fp = K.sum(K.cast((1 - y_true) * y_pred, ’float’), axis=-1)

15 fn = K.sum(K.cast(y_true * (1 - y_pred), ’float’), axis=-1)

16

17 p = tp / (tp + fp + K.epsilon())

18 r = tp / (tp + fn + K.epsilon())

19

20 # Compute F1 and return the loss.

21 f1 = 2 * p * r / (p + r + K.epsilon())

22 f1 = tf.where(tf.is_nan(f1), tf.zeros_like(f1), f1)

23 return 1 - K.mean(f1)

Listing 2: Python implementation of the soft-F1 based loss function for the
Keras environment with TensorFlow backend.

Experiments using the $\overline{F1}$ loss function could not converge.
Likely, the problem is due to the fact that the $\overline{F1}$ loss lies in
the range $[0,1]$. As such, the gradient search space can be abstractly seen
as a huge plateau, just below 1.0, with a solitary hole in the middle that
quickly converges to the global minimum. At the same level of abstraction, we
can visualize the binary cross-entropy $bce$ search space as a wide bag, with
a large flat surface, just above 0. It is easy to reach the bottom of the bag,
i.e., reach a very high binary accuracy due to the sparsity of the labeling,
but then we see a very mild slope towards the global minimum at its center.

Our intuition is that by combining (multiplying or adding) $\overline{F1}$ and
$bce$ results in a search space where $\overline{F1}$ does not affect the
identification of inside of the bag, and at the same time helps with the
identification of the $\overline{F1}$’s and $bce$’s common global minimum. The
implementation of the $\overline{F1}*bce$ loss function is straightforward and
is presented in listing 3.

⬇

1def loss_1mf1_by_bce(y_true, y_pred):

2 import keras.backend as K

3

4 loss_f1 = loss_1_minus_f1(y_true, y_pred)

5 bce = K.binary_crossentropy(target=y_true, output=y_pred, from_logits=False)

6

7 return loss_f1 * bce

Listing 3: Python implementation of the loss function combining soft-F1 score
with binary cross-entropy.

(9) An experiment using VGG16 confirms that the loss function
$\overline{F1}*bce$ leads to better results with 0.3604/0.3606 on our tests.
This is the configuration that performed best in the online submission
(0.374).

Further experiments, e.g., using ResNet50, could not be submitted to the
challenge due to time constraints. However, (10) an internal test using VGG16
in combination with the $\overline{F1}+bce$ loss function, led to the best
performance in our internal evaluation (0.3636/0.3632).

##  5 Results and Discussion

The results of our experiments for the ImageCLEFmedical 2020 challenge can be
summarized as follows.

The task of concept detection can be modeled as a multi-labeling problem and
solved by a transfer learning approach where deep CNNs pretrained on real-
world images can be fine-tuned on the target dataset. The multi-labeling is
technically addressed by using a sigmoid activation function on the output
layer and a label selection by thresholding. A good configuration consists of
a VGG16 deep CNN architecture followed by two fully connected layers of 4096
nodes, each followed by a dropout layer with probability $p$ set to 0.5.
Augmenting the training set with horizontally flipped images increases
accuracy and also reduces the number of epochs needed for training. Increasing
the resolution of input images does not prove to be useful, while better
results are achieved by substituting the convolution stage with a deeper CNN
architecture (ResNet50). We noticed that the learning rate reduction has never
helped in improving the results.

Using the standard binary cross-entropy loss function leads to competitive
results, which significantly increases when it is combined with a soft-F1
score computation. It is worth noticing that when using solely the soft-F1
score as a loss function, the network could not converge and this problem
needs further investigation.

In total, we made five online submissions to the challenge. Table 5 presents
the F1 scores achieved on the withheld test set and the overall ranking of our
team (iml) out of 47 successful submissions, as reported by the challenge
organizers. In addition, it is worth mentioning that the difference in F1
scores between our best submission and the system that achieved the highest
score in the challenge is 0.0195. What percentage of test set images on which
our model still needs to achieve correct labels to bridge this gap needs
further investigation.

Table 5:  F1 scores for submissions by our team ‘iml’. AIcrowd Submission Run | F1 Score | Rank
---|---|---
imageclefmed2020-test-vgg16-f1-bce-nomissing-iml.txt | 0.374525478882926 | 12
imageclefmed2020-test-vgg16-f1-bce-iml.txt | 0.374402134956526 | 13
imageclefmed2020-test-resnet50-iml.txt | 0.365168555515581 | 17
imageclefmed2020-test-vgg16-iml.txt | 0.363067945861981 | 18
imageclefmed2020-test-densenet169-iml.txt | 0.360156086299303 | 19

##  6 Conclusion and Future Directions

In this work we have proposed a deep convolutional neural network based
approach for concept detection in radiology images. Our best performance
($12^{th}$ position) is achieved by implementing a new loss function whereby
we combined the widely used binary cross-entropy loss together with a
differentiable version of the F1 score evaluation metric.

Still several aspects could be investigated to improve the achieved results.
For instance, as we can observe from the CUI distribution plots, there is an
imbalance in the dataset. Consequently the model is biased towards predicting
the concepts associated with over-represented samples. Our future work will
focus on the approaches to combat such type of biases. A straightforward
approach would be to undersample the over-represented samples using a query
strategy that maximizes informativeness of chosen samples. Such a strategy
showed promising results for incremental domain adaptation task in neural
machine translation [5].

Furthermore, we did not make use of the existing categorization of the images
in 7 sub-sets. A straightforward idea would be to train 7 different models,
one per category, and rely on their ensemble for a final global classification
result. Alternatively, the class identifier might be used as additional
metadata information, concatenated to the images’ internal features
representation in the CNN, and fed to a further shallow neural network for
improved classification (see [9]). Another promising direction to look would
be to consider further trends in the integration of vision and language
research [8].

## References

  * [1] Keras Special Interest Group.  Keras.  https://keras.io/.
  * [2] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770–778. IEEE Computer Society, 2016.
  * [3] Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger.  Densely connected convolutional networks.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 2261–2269. IEEE Computer Society, 2017.
  * [4] Bogdan Ionescu, Henning Müller, Renaud Péteri, Asma Ben Abacha, Vivek Datla, Sadid A. Hasan, Dina Demner-Fushman, Serge Kozlovski, Vitali Liauchuk, Yashin Dicente Cid, Vassili Kovalev, Obioma Pelka, Christoph M. Friedrich, Alba García Seco de Herrera, Van-Tu Ninh, Tu-Khiem Le, Liting Zhou, Luca Piras, Michael Riegler, Pål Halvorsen, Minh-Triet Tran, Mathias Lux, Cathal Gurrin, Duc-Tien Dang-Nguyen, Jon Chamberlain, Adrian Clark, Antonio Campello, Dimitri Fichou, Raul Berari, Paul Brie, Mihai Dogariu, Liviu Daniel Ştefan, and Mihai Gabriel Constantin.  Overview of the ImageCLEF 2020: Multimedia retrieval in medical, lifelogging, nature, and internet applications.  In Experimental IR Meets Multilinguality, Multimodality, and Interaction, volume 12260 of Proceedings of the 11th International Conference of the CLEF Association (CLEF 2020), Thessaloniki, Greece, September 22-25 2020. LNCS Lecture Notes in Computer Science, Springer.
  * [5] Marimuthu Kalimuthu, Michael Barz, and Daniel Sonntag.  Incremental domain adaptation for neural machine translation in low-resource settings.  In Wassim El-Hajj, Lamia Hadrich Belguith, Fethi Bougares, Walid Magdy, and Imed Zitouni, editors, Proceedings of the Fourth Arabic Natural Language Processing Workshop, WANLP@ACL 2019, Florence, Italy, August 1, 2019, pages 1–10. Association for Computational Linguistics, 2019.
  * [6] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  ImageNet Classification with Deep Convolutional Neural Networks.  In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 25, pages 1097–1105. Curran Associates, Inc., 2012.
  * [7] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng.  TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems.  2015\.
  * [8] Aditya Mogadala, Marimuthu Kalimuthu, and Dietrich Klakow.  Trends in integration of vision and language research: A survey of tasks, datasets, and methods.  CoRR, abs/1907.09358, 2019.
  * [9] Fabrizio Nunnari, Chirag Bhuvaneshwara, Abraham Obinwanne Ezema, and Daniel Sonntag.  A study on the fusion of pixels and patient metadata in CNN-based classification of skin lesion images.  In International IFIP Cross Domain Conference for Machine Learning and Knowledge Extraction (CD-MAKE). Springer, August 2020.
  * [10] Fabrizio Nunnari and Daniel Sonntag.  A CNN toolbox for skin cancer classification.  arXiv:1908.08187 [cs, eess], August 2019.  arXiv: 1908.08187.
  * [11] Obioma Pelka, Christoph M Friedrich, Alba García Seco de Herrera, and Henning Müller.  Overview of the ImageCLEFmed 2020 concept prediction task: Medical image understanding.  In CLEF2020 Working Notes, CEUR Workshop Proceedings, Thessaloniki, Greece, September 22-25 2020. CEUR-WS.org.
  * [12] Obioma Pelka, Sven Koitka, Johannes Rückert, Felix Nensa, and Christoph M Friedrich.  Radiology objects in context (roco): A multimodal image dataset.  In Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis, pages 180–189. Springer, 2018.
  * [13] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
  * [14] Daniel Sonntag.  AI in Medicine, Covid-19 and Springer Nature’s Open Access Agreement.  KI, 34(2):123–125, 2020.
  * [15] Daniel Sonntag, Fabrizio Nunnari, and Hans-Jürgen Profitlich.  The Skincare project, an interactive deep learning system for differential diagnosis of malignant skin lesions. Technical Report.  CoRR, abs/2005.09448, 2020.
  * [16] Daniel Sonntag and Hans-Jürgen Profitlich.  An architecture of open-source tools to combine textual information extraction, faceted search and information visualisation.  Artificial Intelligence in Medicine, 93:13–28, 2019.
  * [17] Daniel Sonntag, Volker Tresp, Sonja Zillner, Alexander Cavallaro, Matthias Hammon, André Reis, Peter A. Fasching, Martin Sedlmayr, Thomas Ganslandt, Hans-Ulrich Prokosch, Klemens Budde, Danilo Schmidt, Carl Hinrichs, Thomas Wittenberg, Philipp Daumke, and Patricia G. Oppelt.  The clinical data intelligence project - A smart data initiative.  Informatik Spektrum, 39(4):290–300, 2016.
  * [18] Grigorios Tsoumakas and Ioannis Katakis.  Multi-Label Classification: An Overview.  International Journal of Data Warehousing and Mining, 3(3):1–13, July 2007.


11institutetext: University College London 22institutetext: iRobot

# Matching Multiple Perspectives for Efficient Representation Learning

Omiros Pantazis Mathew Salvaris 1*1*22

###### Abstract

Representation learning approaches typically rely on images of objects
captured from a single perspective that are transformed using affine
transformations. Additionally, self-supervised learning, a successful paradigm
of representation learning, relies on instance discrimination and self-
augmentations which cannot always bridge the gap between observations of the
same object viewed from a different perspective. Viewing an object from
multiple perspectives aids holistic understanding of an object which is
particularly important in situations where data annotations are limited. In
this paper, we present an approach that combines self-supervised learning with
a multi-perspective matching technique and demonstrate its effectiveness on
learning higher quality representations on data captured by a robotic vacuum
with an embedded camera. We show that the availability of multiple views of
the same object combined with a variety of self-supervised pretraining
algorithms can lead to improved object classification performance without
extra labels.

##  1 Introduction

Figure 1: The agent with the embedded camera traverses the space and captures
images. The images with overlapping projected polygons in the room map
represent different views of the same object (match) and can be used as pairs
for self-supervised learning. Here we illustrate views of shoes (green) and a
robot dock (red) observed from various perspectives.

Mobile robots are increasingly asked to recognize objects to inform their
decision making process under a variety of real-world situations. For example,
a home robot may need to automatically identify obstacles to stay clear of
them and more critically a self-driving car must identify road signs without
fault[24], whereas the above challenges can present themselves under different
lighting conditions, levels of view obstruction or difficult angles. Having
said that, robots nowadays have started to rely on embedded cameras and deep
learning systems that help them navigate [33] and discriminate between targets
or obstacles [23]. Of course, there are challenges arising from the
computational restrictions of the hardware, the fact that the agent is moving
while taking pictures and the endless variation in conditions presented across
natural environments. Tackling these difficulties while solving challenging
visual tasks would typically demand a long and expensive annotation procedure
and despite the fact that the emergence of benchmark datasets [7, 15] enabled
rapid progress in computer vision, neither generalization from them nor
repeating their annotation procedure in every real-world task dataset are
options. Thus, learning transferable representations of visual data without
requiring explicit semantic supervision at training time is an important and
open problem in computer vision.

On that note, efficient representation learning has been a key object of
studies in the recent years, with multiple self-supervised learning (SSL)
methods reporting remarkable performance while using a fraction of the data
labels [12, 5, 11, 4, 9, 28]. The first wave of successful SSL techniques that
were able to compete with fully-supervised benchmarks involved a pretext task
that produces high quality representations by forcing invariance to a
predefined set of transformations [12, 5, 11]. This is mainly achieved by
forming pairs of images and their augmented versions (i.e. creating positive
pairs) and pushing them together through a loss function. However, subsequent
works devised ways to retrieve positives that turned to be more informative
for self-supervised pretraining when compared with traditional self-
augmentations [9, 2, 18, 3]. For example, in [9], the authors mined positives
based on the image’s nearest neighbours in representation space. Moving away
from ImageNet, beneficial and diverse positives were also mined by exploiting
the contextual information that ordinarily comes with naturally collected
data. More specifically, [2, 18] take advantage of the spatiotemporal
information that comes with images collected from satellites or wild cameras
to mine effective positives for the self-supervised pretext task. In another
work [3], positive pairs were formed through the availability of multiple
photos for a patient’s medical case. Natural variation in the self-supervised
learning pair formation can also be introduced in sequential frames by pushing
together frames that have temporal proximity within a video [22, 19, 20]. The
common denominator of the aforementioned successful positive retrieval
techniques is that a static viewpoint (camera) is able to capture a dynamic
set of views and a subset of these views can be linked in some cases through
the availability of metadata. Nevertheless, there are numerous scenarios where
a dynamic agent collects data about a static environment. These scenarios can
be encountered in cameras that are embodied in agents (robots) and can range
from robotic vacuums and agricultural robots to space and ocean exploration
robots [29, 1, 23]. To the best of our knowledge, there is no exploration of
self-supervised representation learning for object classification where the
viewpoint is dynamic and the objects of interest are static, and we attempt to
demonstrate and tackle such a scenario with the use-case of this paper.

The fact that moving agents with embedded cameras capture data while
navigating through an environment enables collection of all-around views of
objects. However, simply using meta-information such as 2-Dimensional
coordinates or time cannot ensure retrieval of positives that come from the
same class. For this reason, we exploit the camera intrinsics and extrinsics
as well as the robot position to gain an understanding of a) where the agent
is located within the environment, b) where the agent is looking and finally
for c) mapping the 2-D images to the 3-Dimensional space. The aforementioned
mapping and camera localization enables us to understand whether two boxes
that come from a different photo, enclose the same object. This exercise will
provide for each image or detected box within an image, a list of candidate
images that can be selected to form a positive pair in the self-supervised
pretext task. An illustration of this process can be observed in Figure 1. The
assumption is that by pushing together embeddings that come up from different
perspectives of the same object we can form a SSL task that leads to better
representations. We test our assumption in data collected from a robotic
vacuum across multiple episodes, where each episode corresponds to a set of
photos captured within the same house through a period of time. The downstream
task we employ for evaluation is object classification, where each class
corresponds to an object the vacuum needs to avoid.

The key contribution of this paper is twofold:

  1. 1.

We show how recent self-supervised learning techniques can assist image
classification in a challenging dataset with images collected in a home
environment by a navigating robot. Exploitation of self-supervised learning
proved to be more effective than transfer learning from an ImageNet pretrained
network.

  2. 2.

We propose Polygon Matching, a novel way to mine informative positives for
self-supervised learning in data collected from robot agents. We utilize our
approach on top of three different self-supervised learning techniques and
report consistent gains in accuracy.

##  2 Related Work

###  2.1 Self-Supervised Learning

Self-supervised learning for computer vision applications can be broken down
into pretext learning, where a model is trained on unlabeled image collections
given a self-supervisory signal and the downstream task where the
representations learnt from the pretext task are exploited in tandem with
available annotations. Earlier works in self-supervised representation
learning involved the meticulous design of pretext tasks that are capable to
generate supervisory signal out of unlabeled image collections [8, 17, 30,
10]. Then, researchers were able to successfully exploit strong image
augmentations to generate alternative views of images that are capable of
introducing visual variance while preserving the key content of images [12, 5,
11, 28]. At first, suggested approaches used contrastive learning [12, 5, 9]
to push together the embeddings that are associated with a different
augmentation while pushing them away from the embeddings of other images in a
memory bank [12] or within the same batch [5, 9, 28]. The success of the
contrastive learning framework in SSL led researchers to also exploit it and
record notable gains in the supervised domain [14]. Subsequent works in SSL,
produced networks that do not rely on negative pairs [11, 6] and proved
capable of learning high quality representations that lead to high downstream
task performance. In our experiments, we use the simple contrastive learning
framework (SimCLR) proposed by [5] as the basis of our multi-perspective
mining approach because it is intuitive and quite representative of most of
the recent state-of-the-art techniques. In addition, we also explore SimSiam
[6] as a representative of approaches that do not use negatives and a simple
Triplet loss [26].

Most of the aforementioned self-supervised approaches benchmark their
contributions on ImageNet [7], but the successes of self-supervised learning
are not limited there. Self-supervised learning also proved to be effective on
learning high quality representations of data across a variety of challenging
tasks such as species classification from camera traps [18], medical image
analysis [3] or remote sensing from satellite data [2]. Similarly in our work,
we use a challenging dataset comprising of images collected from the embedded
camera of a robotic agent while traversing home environments.

###  2.2 Positive Mining

The relationship between self-supervised learning and various real-world
applications has not been restricted to simple application of the most
advanced self-supervised algorithms. On the contrary, the contextual
information that comes from real-world imagery proved to be suitable for the
properties of self-supervised learning, leading to further improvements.
Indicatively, multiple medical images taken from a patient on the same day
[3], spatio-temporal proximity of camera trap frames [18], or exploitation of
consecutive frames within videos [22, 19, 20, 27] proved to be informative
proxies to mine positives for the pretext learning stage. In the
aforementioned cases, positive mining led to higher performance in the
respective downstream tasks by using widely available metadata to introduce
variation during pretext learning, which could not simply be introduced by
self-augmentations. Even though, these scenarios help deepen the variability
within the contrastive learning task, they lack variation in perspective that
stems from a different point-of-view or lighting conditions that may be
associated with different parts of the day.

Here, we propose an approach that exploits the positioning of an agent with an
embedded camera within a room along with the direction of the camera to figure
out what subspace of the room is captured by the photo. This information is
exploited to construct positive pairs of images for self-supervised learning,
where positives are defined as images that capture similar points in space.
With the above, we aspire to learn better representations by bringing closer
embeddings of different perspectives of each object.

##  3 Multi-Perspective Views in Self-Supervised Learning

###  3.1 Simple Contrastive Learning framework and Variants

Our approach is built on top of SSL approaches that exploit instance
discrimination as signal for efficient representation learning. Indicatively,
SimCLR [5], one of the approaches we explore, is a simple contrastive self-
supervised framework that for each image $x_{i}\in\mathcal{X}$ uses its
strongly augmented version ${x}_{p}$ as its positive pair and the rest of the
images ${x}_{n}$ in the same batch as negatives. Specifically, an image
$x_{i}$ and its transformation go through a feature extractor $f$ (e.g. ResNet
[13]) and then through a Multilayer Perceptron (MLP) $g$ that projects them
into a lower-dimensional embedding vector $z_{i}=g(f(x_{i}))$.

Then, learning takes place through a normalized temperature-based softmax
cross-entropy loss [5] that is defined for every image ${x}_{i}$ within a
batch $\mathcal{B}$ as

$L_{i}=-\log\left(\frac{\exp({sim(z_{i},z_{p})/\tau})}{\sum_{n\in\mathcal{B}}\mathds{1}_{[n\neq
i]}exp({-sim(z_{i},z_{n})/\tau})}\right),$ (1)

where $sim(.)$ is the cosine similarity between the projected image
embeddings, temperature $\tau$ regulates the scale of the distances and
$\mathds{1}_{[n\neq i]}$ ensures that each image embedding is not compared
with itself. Basically, the loss function $L$ pushes the embedding of the
query image close to the embedding of its augmented self while repulsing it
from the negatives. The above procedure teaches the network to be invariant to
appearance variations that do not affect the key content of the image.

We also examine the more recent SimSiam [6] that maintains satisfactory
performance and avoids learning a degenerate solution without the need for
negatives, by using $h$, an additional predictor MLP between the projections
$z_{i}$ of the different augmentations of each image. In addition, we also try
a simple triplet loss [26] that selects positives in a similar way while
randomly sampling a negative for each query image.

###  3.2 Polygon Matching

We build our task on top of the aforementioned SSL approaches, which we also
compare against. The difference is on the images that are considered positive
pairs and are pushed close to each other during the self-supervised task. We
propose a mechanism that aspires to add variation in the signal used as
supervision for SSL by selecting positives that cover different perspectives
of the same object.

Bounding Box Extraction. First, we extract bounding boxes from the photo
captured by the robot by using a pretrained object detection network, in this
case CenterNet [32] that has been trained on a separate dataset. One could
easily replace it with a class agnostic network such as from [31]. We run
inference on the images using this from a run in a number of environments.

Algorithm 1 Polygon Matching Algorithm

1:for each Agent do

2: for each Image $x_{i}$ captured by the Agent do

3: Gather robot position, camera parameters ($K,\left[R|t\right]$) and polygon

4: Project polygon from image plane $x_{i}\in\mathbb{R}^{2}$ to world frame
$x_{w}\in\mathbb{R}^{3}$

5: end for

6: for each Bounding Box do

7: Use world map view to find bounding boxes that overlap with it

8: Record images with overlapping bounding boxes (to be retrieved during SSL
as positives)

9: end for

10:end for

Polygon Projection and Matching. We know the robot’s position in the 3D space
from the VSLAM system [25] employed by the robot. We then simply project the
bounding box on to the floor based on Inverse Perspective Mapping [16]. So if
viewed from a birds-eye view, the floor would be littered with various
polygons that correspond to the individual images, many of which would
overlap. To take the polygon from the image plane and project it into the real
world we need four pieces of information, the camera intrinsics ($K$) and
extrinsics ($[R|t]$), robot position and finally the polygon coordinates in
the image. The process by which the information is used to project from the
robot camera frame to the map frame (or bird’s eye view) is briefly described
below.

In order to transform the homogeneous image coordinates
$x_{i}\in\mathbb{R}^{3}$ to the homogeneous world coordinates
$x_{w}\in\mathbb{R}^{4}$ we use the projection matrix $P\in\mathbb{R}^{3\times
4}$ as given by:

$x_{i}=Px_{w}\,.$ (2)

With the projection matrix $P$ encoding the camera’s intrinsic parameters $K$
and extrinsics (rotation $R$ and translation $t$ with respect to the world
frame):

$P=K[R|t]\,.$ (3)

Assuming there is a transformation $M\in\mathbb{R}^{4\times 3}$ that
transforms from the floor plane $x_{f}\in\mathbb{R}^{3}$ to the world frame

$x_{w}=Mx_{f}\,,$ (4)

we can obtain a transformation from image coordinates to the floor plane with
the following:

$x_{f}=\left(PM\right)^{-1}x_{i}\,.$ (5)

Then, we check if the polygons that are projected in the map with the
aforementioned process have an overlap. The assumption is that the projected
bounding boxes that overlap will likely correspond to the same object captured
from various viewpoints. Thus, instead of formulating a self-supervised signal
by just pushing together an image with an augmented version that arises
through a limited combination of transformations, we explore the natural
variation that exists in the world by looking at the same object from a
different perspective. In practice, for each image $x_{i}\in\mathcal{X}$ we
store the images they overlap with and retrieve them during the SSL task as
their positive pair ${x}_{p}$. The proposed positive retrieval method is
summarized in Algorithm 1. It is fair to assume that the suggested method, as
a positive mining mechanism, can be considered orthogonal to the underlying
self-supervised method and thus, robust against future methodological
developments in the field. This has been confirmed by preceding positive
mining approaches that have consistently reported additive gains under various
self-supervised settings and datasets [18].

##  4 Use-Case: Robotic Vacuum Agent

The scenario in which the data is collected for this work is that of a robot
vacuum navigating around a house and cleaning. The trajectories of the robot
are not modified in any way and therefore the viewpoints of the object are
coincidental with the robots path and not planned in any way. During robot
operation we use a VSLAM system that localises the robot in the environment.
Through this localisation as well as camera projection we can identify whether
the images contain the same object or not. The dataset has been collected from
hundreds of homes of iRobot employees and therefore includes a considerable
amount of variation. Data is collected at around 10Hz at a resolution of 640
by 480. For the purpose of the paper, we refer to this dataset as _Vacuum
Objects_. The _Vaccum Objects_ dataset covers 12 common objects that can be
found inside a typical home environment and the vacuum needs to avoid while it
navigates and cleans the house. These include among others: cables, socks,
shoes, clothes and pet waste. The challenges of the dataset can be realized by
the variety between different views of the same object illustrated in Figure
2. In standard operation the robot may see the same object from 3 to 20 times
in a specific mission. It should be noted that the dataset used for these
experiments cannot be released but a similar setup can be achieved given any
agent with embedded camera that collects data by navigating through space or
with simulators such as Meta’s Habitat platform [21].

Figure 2: Here we show some images of the _Vaccum Objects_ use-case we
examined in this work. Specifically, we see multiple images of the same
cables, towel, shoes and robot dock as captured from various perspectives. The
variety among the different views of the same object stress the need for an
objective function that bridges these differences.

##  5 Experiments

###  5.1 Implementations Details

The input images correspond to detections around the object of interest and
are acquired using a pretrained CenterNet detector [32] without the use of any
specific labels for the _Vacuum Objects_ dataset. The CenterNet model was
trained using the standard CenterNet hyperparameters [32]. The model was
trained using previously gathered data where 12 classes were labeled. In
total, there are 261,695 training image boxes generated by CenterNet which we
use for self-supervised pretraining and downstream task tuning and 27,626
manually annotated boxes that constitute our test set.

As a pre-processing step, for each image collected with the robot-embedded
camera during its navigation, we build a set of matching images using the
polygon matching approach described in Section 3.2, i.e. photos collected from
the same agent whose polygons intersect with the given image. The image
pairings generated in this step are later on retrieved and used during the SSL
step, serving as candidate positives for their paired image. This step also
involves a depth parameter that defines the maximum depth away from the robot
that we consider polygons for, e.g. if an object is too far then its projected
polygon will not be taken into account as candidate positive for self-
supervised learning. Unless specified differently, the experiments in the
paper use 0.7 as the maximum meters away from the camera that we consider.

A ResNet18 [13] convolutional neural network is used as the backbone feature
extractor $f$ for both the self-supervised pretext and the supervised
downstream tasks. Across all the SSL approaches tested, the ResNet18 feature
extractor is followed by an MLP projector $g$ that maps the features to lower
dimensions (128) while for SimSiam we also append an additional predictor MLP
$h$ as described in the paper [6]. The input image size we used for the
experiments of this paper is $112\times 112$. Similar to SimCLR [5], we
introduce variance by using transformations such as random cropping,
horizontal flipping and color distortion. The initial weights of the ResNet18
used in the self-supervised task come from ImageNet pretraining as it is
reported that this decision gives a performance boost in the downstream task
[18] while requiring less epochs to converge. Indicatively, for the
experiments of this paper we train for 200 epochs and batch size 256.

To evaluate our proposed methodology, we use the linear evaluation protocol
[5], i.e. training a linear classifier for object classification, on top of
the frozen layers of the feature extractor $f$ informed by self-supervised
pretraining. The aforementioned procedure is typical for evaluating the
quality of representations learnt by unsupervised approaches. The classifier
is trained with 1%, 10% and 100% of the available data to illustrate the point
that self-supervised learning can do well in the lower data regime. These
percentages correspond to number of training images that range between
approximately 3,549 and 261,695 images, and can be important to illustrate the
usefulness of the suggested approach across a range of data sizes. It’s
important to note that sampling takes place per class to maintain the class-
imbalance as this exists in the original dataset.

###  5.2 Results

####  5.2.1 Mining Positives with Polygon Matching

As our main baseline, we used the three aforementioned standard self-
supervised approaches with a ResNet18 [13] as backbone. For completeness, we
also train a ResNet18 as our supervised baseline initialized either randomly
or with ImageNet pretrained weights. Initially, we find out that self-
supervised learning is a promising approach for learning high quality
representation in images collected from robots navigating home environments.
Specifically in Table 1, we observe that the Top-1 accuracy boost we get by
using the “Standard” self-supervised learning frameworks instead of ImageNet
pretrained features is between $12\%$ and $17\%$, with the biggest gains
recorded in the low data regime. In addition, our “Polygon Matching”
suggestion for efficient self-supervised learning, further boosts the
performance of the SSL paradigm, reporting gains consistently across all
approaches and amounts of supervision. On average, the best results are
reported with SimSiam, showing that negatives may not be necessary in a
scenario where we the visual diversity of the selected positives has increased
in a natural way such as matching multiple views of an object.

Top-1 Accuracy
---
Approach | Method | 1% | 10% | 100%
Supervised | Random Init. | 37.7 | 38.5 | 39.4
Supervised | ImageNet Init. | 57.8 | 63.1 | 65.2
SimCLR | Standard | 74.6 | 75.7 | 77.0
| Polygon Matching | 75.0 | 76.5 | 77.5
SimSiam | Standard | 74.0 | 76.1 | 77.8
| Polygon Matching | 75.5 | 77.7 | 78.5
Triplet | Standard | 62.8 | 68.3 | 70.3
| Polygon Matching | 70.6 | 73.1 | 73.9
Table 1: Top-1 accuracy of “Standard” and “Polygon Matching” variants of the
SSL along with supervised baselines of the same architecture (ResNet18) either
initialized randomly or with ImageNet features. The reported performance
corresponds to the accuracy after linearly evaluating with various amounts of
supervision. Gains are reported for both vanila SSL when compared with an
established transfer-learning baseline and the “Polygon Matching” positive
mining technique when compared to “Standard” approaches.

####  5.2.2 Consistency of Gains across Classes

Moreover, the improvements achieved with “Polygon Matching” versus “Standard”
are more significant when averaged per class. Given the imbalances that exist
in any real-world dataset such as _Vacuum Objects_ , we want to make sure that
the reported gains do not only represent a boost in the majority classes. From
the results in Table 2, we see that the increase in the downstream task
performance is again consistent across all techniques and amounts of data. An
assumption for the aforementioned finding is that by taking into account
different perspectives of the same object during the self-supervised task
helps especially the underrepresented classes, i.e. when a class is rarely
observed we can make sure that we exploit all of the captured perspectives in
the best way possible.

Balanced Top-1 Accuracy
---
Approach | Method | 1% | 10% | 100%
SimCLR | Standard | 52.0 | 53.5 | 55.0
| Polygon Matching | 53.1 | 54.5 | 56.4
SimSiam | Standard | 50.5 | 54.5 | 56.5
| Polygon Matching | 53.0 | 56.7 | 58.2
Triplet | Standard | 41.9 | 46.6 | 48.7
| Polygon Matching | 48.7 | 50.8 | 52.8
Table 2: Balanced Top-1 accuracy of “Standard” and “Polygon Matching” variants
of the SSL approaches after linearly evaluating with various amounts of
supervision. The performance improvements are not only consistent across the
underlying techniques but also of greater magnitude compared to the overall
Top-1 Accuracy discussed above.

####  5.2.3 The Impact of the Maximum Depth considered

When a mobile agent goes around and captures images, there can be cases where
the objects detected lie far away from the camera. The suggested ‘Polygon
Matching” approach for mining positives can be parameterized with the maximum
distance (meters) away from the camera that is considered during candidate
selection. It is fair to assume that considering images captured in distance,
will give us more candidate positives for self-supervised learning. We examine
the effect of the maximum depth by varying its value between 0.5 and 1 meter
away from the center of the robot. As we can see in Figure 3, the best
performance across various levels of supervision is observed when the maximum
distance is about 0.7-0.8 meters away from the camera. Thus, we can infer that
considering objects that are too close to the camera may not give us enough
positives, while a large threshold can hurt performance, maybe because of a
the potential increase in the amount of false positives. For computational
purposes, we sampled a smaller amount of training images for the experiments
investigating the effect of maximum depth compared to other experiments in the
paper and that is why the results in Figure 3 should be observed
independently.

Figure 3: We varied the maximum distance away from of the camera that we
consider for our suggested “Polygon Matching” approach. Top-1 Accuracy of the
maximum distance variants ranging from 0.5 to 1.0 show that setting the
maximum depth parameter around 0.7-0.8 meters gives the best performance.

##  6 Conclusion

In this work, we examined and proved the potential of self-supervised learning
for efficient representation learning for data collected under the setting of
a navigating robotic agent with an embedded camera. In addition, we
demonstrated the effectiveness of retrieving more informative positives for
self-supervised learning under the aforementioned setting by simply exploiting
camera and robot information that typically comes with any navigating mobile
robot. In particular, we show that a self-supervised learning task that learns
to push together different perspectives of the same object instead of simply
relying on self-augmentations leads to consistent gains across various
representation learning techniques. Given the fact that the boost becomes even
more significant when we average across classes, we expect our findings to be
useful in real-world imbalanced data scenarios where an agent collects vast
amounts of data and the annotation budget is limited. Finally, we believe that
similar approaches can aid holistic understanding of objects, especially on
tasks when on top of the object’s variations we have to deal with a huge
variety of real-world settings.

## References

  * [1] Arm, P., Zenkl, R., Barton, P., Beglinger, L., Dietsche, A., Ferrazzini, L., Hampp, E., Hinder, J., Huber, C., Schaufelberger, D., et al.: Spacebok: A dynamic legged robot for space exploration. In: 2019 international conference on robotics and automation (ICRA). pp. 6288–6294. IEEE (2019)
  * [2] Ayush, K., Uzkent, B., Meng, C., Tanmay, K., Burke, M., Lobell, D., Ermon, S.: Geography-aware self-supervised learning. arXiv preprint arXiv:2011.09980 (2020)
  * [3] Azizi, S., Mustafa, B., Ryan, F., Beaver, Z., Freyberg, J., Deaton, J., Loh, A., Karthikesalingam, A., Kornblith, S., Chen, T., et al.: Big self-supervised models advance medical image classification. arXiv preprint arXiv:2101.05224 (2021)
  * [4] Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., Joulin, A.: Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems 33, 9912–9924 (2020)
  * [5] Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International conference on machine learning. pp. 1597–1607. PMLR (2020)
  * [6] Chen, X., He, K.: Exploring simple siamese representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 15750–15758 (2021)
  * [7] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: CVPR (2009)
  * [8] Doersch, C., Gupta, A., Efros, A.A.: Unsupervised visual representation learning by context prediction. In: Proceedings of the IEEE international conference on computer vision. pp. 1422–1430 (2015)
  * [9] Dwibedi, D., Aytar, Y., Tompson, J., Sermanet, P., Zisserman, A.: With a little help from my friends: Nearest-neighbor contrastive learning of visual representations. arXiv preprint arXiv:2104.14548 (2021)
  * [10] Gidaris, S., Singh, P., Komodakis, N.: Unsupervised representation learning by predicting image rotations. arXiv:1803.07728 (2018)
  * [11] Grill, J.B., Strub, F., Altché, F., Tallec, C., Richemond, P.H., Buchatskaya, E., Doersch, C., Pires, B.A., Guo, Z.D., Azar, M.G., et al.: Bootstrap your own latent: A new approach to self-supervised learning. In: NeurIPS (2020)
  * [12] He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2020)
  * [13] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016)
  * [14] Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., Krishnan, D.: Supervised contrastive learning. Advances in Neural Information Processing Systems 33, 18661–18673 (2020)
  * [15] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: ECCV (2014)
  * [16] Mallot, H.A., Bülthoff, H.H., Little, J., Bohrer, S.: Inverse perspective mapping simplifies optical flow computation and obstacle detection. Biological cybernetics 64(3), 177–185 (1991)
  * [17] Noroozi, M., Favaro, P.: Unsupervised learning of visual representations by solving jigsaw puzzles. In: ECCV (2016)
  * [18] Pantazis, O., Brostow, G.J., Jones, K.E., Mac Aodha, O.: Focus on the positives: Self-supervised learning for biodiversity monitoring. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 10583–10592 (October 2021)
  * [19] Purushwalkam, S., Gupta, A.: Demystifying contrastive self-supervised learning: Invariances, augmentations and dataset biases. arXiv preprint arXiv:2007.13916 (2020)
  * [20] Qian, R., Meng, T., Gong, B., Yang, M.H., Wang, H., Belongie, S., Cui, Y.: Spatiotemporal contrastive video representation learning. In: CVPR (2021)
  * [21] Savva, M., Kadian, A., Maksymets, O., Zhao, Y., Wijmans, E., Jain, B., Straub, J., Liu, J., Koltun, V., Malik, J., et al.: Habitat: A platform for embodied ai research. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 9339–9347 (2019)
  * [22] Sermanet, P., Lynch, C., Chebotar, Y., Hsu, J., Jang, E., Schaal, S., Levine, S., Brain, G.: Time-contrastive networks: Self-supervised learning from video. In: 2018 IEEE international conference on robotics and automation (ICRA). pp. 1134–1141. IEEE (2018)
  * [23] Skoczeń, M., Ochman, M., Spyra, K., Nikodem, M., Krata, D., Panek, M., Pawłowski, A.: Obstacle detection system for agricultural mobile robot application using rgb-d cameras. Sensors 21(16), 5292 (2021)
  * [24] Swaminathan, V., Arora, S., Bansal, R., Rajalakshmi, R.: Autonomous driving system with road sign recognition using convolutional neural networks. In: 2019 International Conference on Computational Intelligence in Data Science (ICCIDS). pp. 1–4. IEEE (2019)
  * [25] Taketomi, T., Uchiyama, H., Ikeda, S.: Visual slam algorithms: A survey from 2010 to 2016. IPSJ Transactions on Computer Vision and Applications 9(1), 1–11 (2017)
  * [26] Weinberger, K.Q., Saul, L.K.: Distance metric learning for large margin nearest neighbor classification. JMLR (2009)
  * [27] Wu, C.E., Lai, F., Hu, Y.H., Kadav, A.: Self-supervised video representation learning with cascade positive retrieval. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4070–4079 (2022)
  * [28] Zbontar, J., Jing, L., Misra, I., LeCun, Y., Deny, S.: Barlow twins: Self-supervised learning via redundancy reduction. In: International Conference on Machine Learning. pp. 12310–12320. PMLR (2021)
  * [29] Zereik, E., Bibuli, M., Mišković, N., Ridao, P., Pascoal, A.: Challenges and future trends in marine robotics. Annual Reviews in Control 46, 350–368 (2018)
  * [30] Zhang, R., Isola, P., Efros, A.A.: Colorful image colorization. In: ECCV (2016)
  * [31] Zhou, X., Koltun, V., Krähenbühl, P.: Probabilistic two-stage detection. In: arXiv preprint arXiv:2103.07461 (2021)
  * [32] Zhou, X., Wang, D., Krähenbühl, P.: Objects as points. In: arXiv preprint arXiv:1904.07850 (2019)
  * [33] Zhu, K., Zhang, T.: Deep reinforcement learning based mobile robot navigation: A review. Tsinghua Science and Technology 26(5), 674–691 (2021)