# PCD2Vec: A Poisson Correction Distance Based Approach for Viral Host
Classification

Sarwan Ali∗, Taslim Murad∗, and Murray Patterson  Department of Computer
Science, Georgia State University, Atlanta, USA
{sali85<EMAIL_ADDRESS><EMAIL_ADDRESS>
∗ Equal Contribution

###### Abstract

Coronaviruses are membrane-enveloped, non-segmented positive-strand RNA
viruses belonging to the Coronaviridae family. They are primarily divided into
two subfamilies, Letovirinae and Coronavirinae, with the majority of these
viruses belonging to the latter subfamily. Various animal species, mainly
mammalian and avian, are severely infected by various coronaviruses, causing
serious concerns like the recent pandemic (COVID-19) — one example of the
impact of these viruses on human health as well as the global economy.
Therefore, building a deeper understanding of these viruses is essential to
devise prevention and mitigation mechanisms. Coronaviruses have an invariant
genome organization of $\approx$30KB, divided into regions that code for non-
structural and structural proteins. Among these, an essential structural
region is the spike region and its resulting protein which is responsible for
attaching the virus to the host cell membrane. Therefore, the usage of only
the spike protein, instead of the full genome, provides most of the essential
information for performing analyses such as host classification. In this
paper, we propose a novel method for predicting the host specificity of
coronaviruses by analyzing spike protein sequences from different viral
subgenera and species. Our method involves using the Poisson correction
distance to generate a distance matrix, followed by using a radial basis
function (RBF) kernel and kernel principal component analysis (PCA) to
generate a low-dimensional embedding. Finally, we apply classification
algorithms to the low-dimensional embedding to generate the resulting
predictions of the host specificity of coronaviruses. We provide theoretical
proofs for the non-negativity, symmetry, and triangle inequality properties of
the Poisson correction distance metric, which are important properties in a
machine-learning setting. By encoding the spike protein structure and
sequences using this comprehensive approach, we aim to uncover hidden patterns
in the biological sequences to make accurate predictions about host
specificity. Finally, our classification results illustrate that our method
can achieve higher predictive accuracy and improve performance over existing
baselines.

###### Index Terms:

Host Classification, Spike Sequence, Coronaviruses, Sequence Analysis,
Classification

##  I Introduction

Coronaviruses are well-known for causing pandemics throughout history. For
instance, in 2003, a SARS-CoV (severe acute respiratory syndrome coronavirus)
pandemic originated in China, probably from bats, spread worldwide, and caused
8,000 cases and 800 deaths [23]. Then, in 2012, a MERS-CoV (Middle East
respiratory syndrome coronavirus) was reported in Saudi Arabia, spread from
camels to humans, causing 2,500 cases and 800 deaths [1]. Finally, in 2019, a
SARS-CoV-2 originated in China and causing the COVID-19 pandemic, with
102,171,644 cases and 1,103,615 deaths reported so far only in the United
States by CDC (Centers for Disease Control and Prevention).

Moreover, coronaviruses have recently affected many living organisms,
including humans, animals, and birds. Coronaviruses are positive-sense
enveloped RNA viruses consisting of two major subfamilies, namely
Coronavirinae and Letovirinae, comprising five genera, namely
Alphacoronavirus, Betacoronavirus, Gammacoronavirus, Deltacoronavirus, and
Alphaletovirus. We can also divide certain major lineages of coronaviruses,
such as SARS-CoV-2 — the virus responsible for the COVID-19 pandemic — into
different variant categories such as Alpha, Beta, Delta, Omicron, etc. The
taxonomy of the major groups within the Coronaviridae family is given in
Figure 1

Figure 1: Taxonomic classification scheme for different Coronaviridae.

Here, our main research question is “based on a coronavirus sequence, can we
train a model to perform efficient host classification?”. Since we have dozens
of hosts in our data, it is a multi-class classification problem. As the full
genome sequence (of $\approx$30KB) of the virus is long, it is more efficient
to extract the most useful part of this sequence (the region having the
largest effect on the host classification) to do further analysis, and for
this purpose, we only use the spike region of the virus, or more accurately,
the protein (amino acid sequence) that it produces, as it is most important
for host specificity [13]. This is because virus-cell interaction is done
through binding the spike protein to the ACE2 cell receptor, a key step prior
to virus-cell membrane fusion and viral entry into the host cell. As a result,
a disproportionate number of mutations (with respect to the length of the
genome) occur in the spike region. Therefore, we extract only the spike
protein sequences from the raw data and used them to perform host
classification. A detailed structure of the SARS-CoV-2 genome can be seen in
Figure 2.

Figure 2: The SARS-CoV-2 genome consists of a structural (S, E, M, N) and a
non-structural (ORF1ab) part, forming a length of almost $30K$ bp. Among them,
the spike (S) region is used to attach the virus to the host cell membrane,
and it also contains the major mutations related to viral transmissibility.
This S region is used to do further host classification.

Various methods to predict the host specificity of coronaviruses have been
applied so far. Some of them use complete viral genomes while others prefer
individual genes to do the analysis. For instance, [13] reported the use of
four different machine-learning methods to analyze spike protein sequences
from several coronaviruses. The study’s major conclusion was that the host
specificity of coronaviruses could be predicted with accuracy using only the
spike protein sequences.

In this work, we formulate the method of host classification of coronaviruses
by using the concept of Poisson Correction Distance (PCD). PCD is a measure of
the difference in amino acid composition between two protein sequences and is
specifically designed for the analysis of biological sequences. The
theoretical basis for this distance measure is the Poisson distribution, which
models the number of events occurring in a fixed interval of time. The PCD
formula uses the observed and expected frequencies of each amino acid in two
sequences and the Poisson distribution to calculate the distance between the
sequences. This distance is a good measure because it takes into account both
the observed and expected frequencies of each amino acid in the sequences, and
it also considers the variability in the frequencies of the amino acids by
using the Poisson distribution. Therefore, given the spike protein sequences
of SARS-CoV-2 virus, we follow PCD to classify the sequences into their
respective hosts using machine learning (ML) models. However, before the
classification step, we reduce the distance matrix dimensionality by employing
RBF kernel and kernel PCA. As RBF kernel is known to perform well in non-
linear classification tasks and is often used in the analysis of biological
sequences. Similarly, kernel PCA allows us to perform dimensionality reduction
while preserving the non-linear structure of the data, which can lead to
better separation between different classes and improve classification
performance. Additionally, our method retains the maximum information for
classification because it operates on raw spike sequences, instead of any
prior embeddings generated out of the sequences.

Our contributions to this paper are as follows:

  1. 1.

Efficient Prediction: We show that coronavirus hosts can be efficiently
predicted using spike sequences only.

  2. 2.

Incorporation of biological knowledge: Our method to generate a low-
dimensional embedding, based on the Poisson correction distance (PCD), better
captures the biological relationships between the spike protein sequences in
the classification task, which general distance measures / representation
learning methods may not consider.

  3. 3.

Use of RBF kernel: We used the RBF kernel to project the data into high
dimensional space, which has been proven to perform well in non-linear
classification tasks and is often used in the analysis of biological
sequences.

  4. 4.

Use of kernel PCA: We used Kernel PCA, which allows us to perform
dimensionality reduction while preserving the non-linear structure of the
data. This can lead to better separation between the different classes and
improved classification performance.

  5. 5.

Theoretical proofs for three properties: We provide theoretical proofs for the
triangle inequality, symmetry, and non-negativity properties to ensure the
validity of the distance metric used in our method, which can add confidence
to the results obtained from our method.

The rest of the paper is organized as follows: Section II contains the
detailed literature review of the paper. The proposed approach is described in
Section III. Section IV discusses the data source, evaluation metrics, and
baseline methods used for performing the experiments. The results are reported
in Section V. Finally, we conclude the paper in Section VI.

##  II Literature Review

The availability of a huge amount of COVID-19 data has enabled researchers to
put more effort into analyzing the behavior of the Coronavirus. The motivation
behind this kind of study is to build prevention mechanisms, like vaccines,
etc, and control the spread of the virus. Sequence analysis [12, 11] is a
popular approach used to conduct such studies, especially in the
bioinformatics domain. For instance, the authors in [13] used different ML
algorithms to study the spike sequences by converting the sequences to one-hot
encoded vectors for the purpose of predicting the host specificity of
Coronavirus. However, this technique undergoes sparsity and a curse of
dimensionality challenges. Likewise, [22] used Generalized Inverse Wei-bull
distribution to predict the growth and trend of the COVID-19 pandemic. In [3]
they have compared various research papers that are using machine learning-
based approaches for predicting confirmed cases of COVID-19 worldwide.

Furthermore, more traditional methods for analyzing sequencing data typically
employ a phylogenetic approach [9, 20, 16]. However, the number of sequences
currently available for viruses, such as SARS-CoV-2, is several orders of
magnitude which is beyond the capabilities of the traditional methods, making
ML approaches a more attractive alternative.

Another category of sequence investigation strategies is based on $k$-mers [6,
21] (sub-strings of length $k$). The use of $k$-mer frequencies for
phylogenetic applications was first explored in [7], which proposed
constructing accurate phylogenetic trees from several coding and non-coding
nDNA sequences. Similarly, authors in [4] provided an alignment-free $k$-mers-
based approach for classifying SARS-CoV-2 sequences. PWKmer [14] technique
performs phylogenetic analysis of HIV-1 viruses by utilizing $k$-mers counts
and position distribution information. Although $k$-mers-based approaches
achieve reasonable performance, their applications are inherently limited. As
$k$ increases and quickly approaches a sparse matrix of (binary) counts, the
likelihood of seeing any particular $k$-mer decreases [19]. Moreover, any
statistical method that uses $k$-mers is vulnerable to inaccurate $k$-mer
frequency estimates. Another domain of research involves using kernel methods
for sequence classification [10, 5]. However, most of these methods only work
with traditional $k$-mers spectrum, which may not capture all (hidden)
biological patterns from the sequences, and hence could under-perform.

##  III Proposed Approach

This section discusses our proposed approach, Poisson Correction Distance
Based Embedding (PCD2Vec), to perform host classification of Coronavirus based
on the virus’s spike sequences. It also highlights the theoretical proofs of
the triangle inequality, symmetry, and non-negativity properties of the
distance matrix used by our method.

The pseudo-code for PCD2Vec is illustrated in Algorithm 1. We can observe that
given the set of spike protein sequences, it returns the embeddings, which are
further utilized for doing the classification. The algorithm starts with
initializing a distance matrix by zeros, as shown in line 1. Then we are
computing only the upper triangular values of this matrix (from line
$2$-$23$), as it is a symmetric matrix with diagonal elements being equal to
zero. Each element in the matrix corresponds to a distance value between the
pair of sequences. This distance value is calculated first by counting the
occurrences of each amino acid for both sequences respectively to get observed
frequency vectors ($obs\\_freq1$ and $obs\\_freq1$), as shown in lines
$5$-$6$. Then an expected frequency ($exp\\_freq$) vector is obtained by
averaging the observed frequencies in line $8$. To eliminate any zero value in
the observed frequencies, $0.0001$ is added to them. After that, the distance
$d$ between the sequence pair is calculated using the PCD formula (as
illustrated in lines $10$-$19$), and the formula is as follows,

$\displaystyle d=2\times
exp\\_freq\times(\ln(\frac{obs\\_freq1}{exp\\_freq})+\ln(\frac{obs\\_freq2}{exp\\_freq}))$
(1)

Once the distance values are computed, the distance matrix is updated with
these values at their respective positions, as given in lines $20$-$21$. For
$n$ number of sequences, we get an $n$x$n$ symmetric distance matrix. We then
pass this distance matrix to RBF kernel having $\sigma=0.5$ to get the kernel
matrix (line $24$). This kernel matrix is further given to kernel PCA to
reduce its dimensions to $500$ components (line $25$) and it outputs the final
embeddings of the given sequences.

Algorithm 1 The algorithm for PCD-based embedding generation for spike
sequences.

1:Input: Set of Spike Sequences ($seqs$)

2:Output: Embeddings

3:$distances$ $\leftarrow$ $zeros(len(seqs),len(seqs))$

4:for $i$ in $|seqs|-1$ do

5: for $j$ in $(i+1,|seqs|)$ do $\triangleright$ Upper Triangle Only

6: /* Compute the observed frequencies of each amino acid */

7: $obs\\_freq1$ $\leftarrow$ AminoAcidFreq($seqs[i]$)

8: $obs\\_freq2$ $\leftarrow$ AminoAcidFreq($seqs[j]$)

9: /* Compute the expected frequencies */

10: $exp\\_freq$ $\leftarrow$ $0.5\times(obs\\_freq1+obs\\_freq2)$

11: $d$ $\leftarrow$ 0

12: for $k$ in $|20|$ do $\triangleright$ $20$ Amino Acids

13: if $exp\\_freq[k]$ $>$ 0 then

14: $\epsilon\leftarrow 0.0001$ $\triangleright$ to avoid divided by 0 error

15: $obs\\_freq1[k]$ $\leftarrow$ $obs\\_freq1[k]+\epsilon$

16: $obs\\_freq2[k]$ $\leftarrow$ $obs\\_freq2[k]+\epsilon$

17: Freq_1 $\leftarrow$ $ln(\frac{obs\\_freq1[k]}{exp\\_freq[k]})$

18: Freq_2 $\leftarrow$ $ln(\frac{obs\\_freq2[k]}{exp\\_freq[k]})$

19: Freq $\leftarrow$ Freq_1 + Freq_2

20: $d$ $\leftarrow$ d + 2 $\times$ exp_freq[k] $\times$ Freq

21: end if

22: end for

23: $distances[i,j]$ $\leftarrow$ $d$

24: $distances[j,i]$ $\leftarrow$ $d$

25: end for

26:end for

27:$kernelMatrix$ $\leftarrow$ RbfKernel(distances)

28:$Embedding$ $\leftarrow$ KernelPCA($kernelMatrix$)

Figure 3: The workflow of PCD2Vec. Given a set of spike sequences, it
generates the corresponding numerical embeddings using the Poisson correlation
distance, RBF kernel and kernel PCA (steps $a$-$f$).

Moreover, the workflow of PCD2Vec is illustrated in Figure 3. Given a set of
spike sequences, it iteratively follows the step ($a$-$b$) to get the
respective distance values for each pair of sequences. These values are used
to update the distance matrix in ($d$). This updated distance matrix is given
to RBF kernel to get a kernel matrix in ($e$), and then the final embeddings
are generated by passing the kernel matrix to kernel PCA in ($f$).

Furthermore, the details of theoretical proofs of certain properties of the
used distance metric are given as follows:

###  III-A Theoretical Properties

The distance matrix generated using Poison Correction Distance should hold
three properties namely “triangle inequality,” “symmetry,” and “non-
negativity”. These properties are important for a distance matrix in a
machine-learning setting for several reasons:

  1. 1.

Triangle inequality: The triangle inequality property ensures that the
distance between two points via a third point is always equal to or greater
than the direct distance between the two points. This property ensures that
the distance metric is consistent and well-defined.

  2. 2.

Symmetry: The symmetry property ensures that the distance between two points
is the same regardless of the order in which the points are considered. This
property ensures that the distance metric is consistent and unbiased.

  3. 3.

Non-negativity: The non-negativity property ensures that the distance between
any two points is always non-negative, i.e., it is either zero or a positive
number. This property ensures that the distance metric is well-defined and
that it has a clear meaning.

These properties ensure that the distance matrix generated is a valid distance
metric that can be used for various machine-learning tasks, such as
clustering, classification, and dimensionality reduction. The proofs for these
properties are given below.

####  III-A1 Triangle inequality

Let $d(i,j)$ be the distance between two protein sequences $i$ and $j$, and
let $d(i,k)$ be the distance between sequences $i$ and $k$. The triangle
inequality states that $d(i,j)+d(j,k)\geq d(i,k)$ for any three sequences $i$,
$j$, and $k$.

Proof: The triangle inequality for the Poisson correction distance matrix can
be shown as follows:

$\displaystyle d(i,j)+d(j,k)=2\times\Sigma(\text{exp\\_freq}[m]\times$ (2)
$\displaystyle(\ln(\frac{\text{obs\\_freq1}[m]}{\text{exp\\_freq}[m]})+\ln(\frac{\text{obs\\_freq2}[m]}{\text{exp\\_freq}[m]})))$

where obs_freq1 and obs_freq2 are the observed frequencies of amino acids in
sequences $i$ and $j$, and exp_freq is the average of obs_freq1 and obs_freq2.

$\displaystyle d(i,k)=2\times\Sigma(\text{exp\\_freq}[m]\times$ (3)
$\displaystyle(\ln(\frac{\text{obs\\_freq1}[m]}{\text{exp\\_freq}[m]})+\ln(\frac{\text{obs\\_freq3}[m]}{\text{exp\\_freq}[m])})))$

where observed_freq1 and observed_freq3 are the observed frequencies of amino
acids in sequences $i$ and $k$, and expected_freq is the average of
observed_freq1 and observed_freq3.

Since $\ln(x)$ is a strictly increasing function, it follows that
$d(i,j)+d(j,k)\geq d(i,k)$ for any three sequences $i$, $j$, and $k$.

####  III-A2 Symmetry

Let $d(i,j)$ be the distance between two protein sequences $i$ and $j$, then
$d(i,j)=d(j,i)$ for any two sequences $i$ and $j$.

Proof: Since the Poisson correction distance formula is symmetric with respect
to the observed frequencies of the two sequences, it follows that
$d(i,j)=d(j,i)$ for any two sequences $i$ and $j$.

####  III-A3 Non-negativity

Let $d(i,j)$ be the distance between two protein sequences $i$ and $j$, then
$d(i,j)\geq 0$ for any two sequences $i$ and $j$.

Proof: The Poisson correction distance formula is based on the logarithm of
the ratio of observed and expected frequencies. Since the logarithm of a
positive number is always positive, it follows that the Poisson correction
distance is always non-negative.

In conclusion, the Poisson correction distance matrix generated in Algorithm 1
satisfies the triangle inequality, symmetry, and non-negativity properties,
which are important properties for a distance matrix in a machine learning
setting.

##  IV Experimental Evaluation

In this section, we present the experimental evaluation of our proposed method
to predict the host specificity of coronaviruses using spike protein
sequences. We begin by describing the datasets used in our experiments,
including the statistics of the datasets. Then, we explain the feature
selection methods used to extract relevant information from the spike protein
sequences. Finally, we outline the machine learning classification algorithms
used to predict the host specificity of coronaviruses and compare their
performance in terms of average accuracy, precision, recall, weighted
F1-score, macro F1 score, and ROC-AUC. Since our dataset is imbalanced, so
reporting performance using various metrics will provide more meaningful
insights to it therefore we are employing 5 different evaluation metrics.
Experiments are performed on a Core i5 system with $32$ GB memory and Windows
10 OS with a 2.4 GHz processor. Our code is implemented in Python and is
available online 111https://github.com/sarwanpasha/PCD2Vec.

###  IV-A Dataset Analysis

The protein sequences ($2576$ sequences in total) from different species of
mammalian and avian coronaviruses were obtained from the Virus Pathogen
Database 222https://www.viprbrc.org/. Amino acid sequences from each host were
stored in a different fasta file (one fasta file per host). Multiple sequence
alignments were performed using Clustal Omega
333https://www.ebi.ac.uk/Tools/msa/clustalo/. The distribution of different
hosts is given in Table I.

We can observe that $6$ hosts are dominating while the presence of other hosts
is very low. Initially, there was also a problem with some missing information
in the sequences. Those sequences were discarded. We selected the sequences
having one of the first $6$ hosts (humans, swine, chicken, camel, bat, cat)
for the classification and analysis part. The rest of the $7$ hosts were
assigned a new class “others,” because their presence is very low in the data
(i.e., $\leq 10$ sequences for each of these $7$ hosts).

Host | Count | Host | Count
---|---|---|---
human | 957 | pangolin | 5
swine | 785 | duck | 3
chicken | 309 | chimpanzee | 3
camel | 265 | goose | 2
bat | 181 | beluga Whale | 2
cat | 57 | falcon | 1
civet | 5 | - | -
Total | 2575 |  |
TABLE I: Host (class label) distribution in data.

To use the data for classification, we have to convert the sequences into
feature vectors (in euclidean space). For this purpose, we used one hot
encoding approach. The overall size of the one-hot encoding vector for each
sequence is $24700$. Since this one-hot encoding vector is very high
dimensional, it creates the problem of the “curse of dimensionality”. We used
different feature selection methods to deal with this problem, which we will
be discussing in the next section.

###  IV-B Baseline Methods

We selected baseline methods from different domains, such as feature
engineering (PCA, Lasso Regression, Ridge Regression, OHE), neural network
(Auto Encoder), kernel function (string kernel), hyperbolic embedding
(Poincaré embedding), and pre-trained language model for proteins (Protein
Bert), so that we can test the performance of the proposed PCD2Vec more
rigorously.

####  IV-B1 One-Hot Encoding (OHE)

It is an algorithm to generate a fixed-length numerical representation of
given sequences [13]. It creates a binary vector for each character of the
sequence where this vector is of length equal to total number of possible
unique values and the corresponding location of the appeared character is
marked $1$ in this binary vector and the rest is $0$. Then all the binary
vectors corresponding to every character of the given sequence is concatenated
to form the final embedding.

####  IV-B2 One-Hot Encoding + PCA

This method is proposed in [13] where the authors perform host classification
by first designing one-hot encoding-based representation for the protein
sequences and then applying Principal Component Analysis (PCA) [2] to compute
low dimensional embedding of the data. We selected $40$ principal components
out of a total of $24700$ features. The reason for selecting only $40$
components is that most of the information was preserved in these components.
For reference, we call this method as PCA.

####  IV-B3 Ridge Regression

It is a famous parameter estimation approach used to address the collinearity
problem that arises in multiple linear regression frequently [15]. Ridge
Regression works by introducing a Bias term. Its goal is to increase the bias
to improve the variance (i.e., the model’s generalization capability). It
changes the slope of the line and tries to make it more horizontal like. Ridge
regression is very useful for feature selection because it gives insights into
which independent variables are useless (can reduce the slope close to zero).
We can eliminate some of the useless independent variables to reduce the
dimensions of the overall dataset. The objective function of ridge regression
is the following:

$min(\text{Sum of square residuals}+\alpha\times\text{slope}^{2})$ (4)

where $\alpha\times{slope}^{2}$ is called penalty terms. After applying ridge
regression, we selected $7079$ features out of $24700$.

####  IV-B4 Lasso Regression

Lasso regression works similarly to ridge regression [18]. The only difference
is that it can reduce the slope exactly $=0$ rather than close to zero (which
is the case with ridge regression). The objective function of ridge regression
is the following

$min(\text{Sum of square residuals}+\alpha\times|\text{slope}|)$ (5)

where $\alpha\times|{slope}|$ is called penalty terms. Note that in the
penalty term, the cardinality of the slope is taken rather than the square.
This helps to reduce the slope of useless variables exactly equal to zero.
After applying lasso regression, we selected $813$ features out of $24700$.

####  IV-B5 Autoencoder

It is an unsupervised artificial neural network-based approach [24] (also
called self-supervised approach). It performs two operations, namely
compression, and reconstruction. One property of autoencoder is that it
ignores the features that are noisy and/or redundant. In this way, only those
features are considered (in low dimensions) that are important. It is also
called the self-supervised approach because it uses reconstruction loss to
decompress the data. When the loss is minimum, it stops. At that stage, the
middle (encoded layer) contains the reduced dimensional information that we
can use as input for the classification algorithm. The number of dimensions
that we selected after applying the autoencoder is $200$.

####  IV-B6 Poincaré Embeddings

The Poincaré embedding [17] is a technique for embedding high-dimensional data
in hyperbolic space. The goal of embedding is to preserve the geometric
structure of the data while reducing its dimensionality. Unlike classical
embedding techniques such as PCA, which embeds data in Euclidean space,
Poincaré embedding embeds data in a space with non-Euclidean geometry
(hyperbolic space), where distances between points are measured using the
Poincaré distance. In our case, the goal of using this method is to map the
sequence of amino acids (which are used as input) into a hyperbolic space
using the Poincaré distance formula. We start by counting the number of
occurrences of each amino acid in the sequence and then normalizing the counts
to obtain amino acid frequencies. After normalization, We map the normalized
frequencies into a hyperbolic space using gradient descent and the Poincaré
distance formula. The Poincaré distance formula is as follows:

$d(\mathbf{x},\mathbf{y})=\operatorname{arcCosh}\left(1+\frac{2\left|\mathbf{x}-\mathbf{y}\right|^{2}}{(1-\left|\mathbf{x}\right|^{2})(1-\left|\mathbf{y}\right|^{2})}\right)$
(6)

where $\mathbf{x}$ and $\mathbf{y}$ are points in the hyperbolic space, and
$\left|\cdot\right|$ represents the Euclidean norm. We first initialize the
embedding as a random point in the hyperbolic space and sets the learning rate
($0.001$) and the number of iterations ($1000$) for optimization. It then
updates the embedding using gradient descent until the desired number of
iterations is reached. The final result (output) of this method is an
embedding that represents the input sequence into the hyperbolic space. We
selected $50$ as the length of the final embedding. The hyperparameters such
as learning rate, number of iterations, and embedding length are tuned using
the standard validation set approach.

####  IV-B7 String Kernel

This method [5] has proposed a technique to compute similarity between two
sequences by using the number of matched and mis-matched $k$-mers. The matches
between 2 $k$-mers is determine by having them at a distance $m$. It also
claims to provide a computationally efficient closed-form solution for the
size of the intersection of d-mismatch neighborhoods. String kernel yields a
square kernel matrix of dimensions equal to the length of sequences, and this
matrix is given to kernel PCA to reduce its dimensions. We used $k=3$ for our
experiments.

####  IV-B8 Protein Bert

Protein Bert [8] provides a deep language model to analyze protein sequences.
This model is pretrained by combining Gene Ontology (GO) annotation prediction
task with language modeling. To make the model flexible to deal with long
sequences efficiently, new architectural elements are introduced in it.

###  IV-C Classification Task

For classification, we used the following ML algorithms: Support Vector
Machine (SVM), Naive Bayes (NB), Multilayer Perceptron (MLP), k-Nearest
Neighbor (k-NN) (where $k=3$), Random Forest (RF), Logistic Regression (LR).
For experiments, we used a 10-fold cross-validation approach. We first
randomly divide the data into 70-30% (where 30% is the held-out set). On the
70% data, we use 10-fold cross-validation to optimize the hyperparameters. In
the end, we validate the results on the 30% held-out testing set. This 70-30%
split process is repeated 5 times, and we compute averages and variances over
these runs and computed results for different performance metrics. The
performance metrics include Average accuracy, precision, recall, weighted F1,
F1 macro, and ROC AUC macro. One of the reasons for the selection that many
performance metrics was to analyze the underlying classifiers in detail. Since
we are dealing with a class imbalance problem, reporting only the accuracy
does not give us a detailed overview of a classifier’s actual performance.

###### Remark 1

For binary evaluation metrics, we use the one-vs-rest approach.

##  V Results and Discussion

This section provides insight into the classification performance result
obtained by our proposed system and compares it with the other baselines.

The results in Table II show average and variance values for various
performance metrics using different baseline methods and our proposed system.
For all the results, we can observe that random forest consistently
outperforms all other machine learning classifiers. We can also note that the
values for variance are very low.

###  V-A Comparison with feature engineering methods

Overall, our method is outperforming all the feature engineering-based
baselines in terms of every evaluation metric. Like, PCD2Vec achieves $2\%$
average accuracy improvement over PCA, OHE, Ridge Regression and Lasso
Regression for RF classifier. As RF model is showcasing the best performance
results for all the methods as compared to the other classifiers.

###  V-B Comparison with Neural network baseline

We can observe that our method is showing better performance than the NN-based
baseline (Autoencoder). For instance, PCD2Vec has $2\%$ accuracy and $1\%$
precision improvement over Autoencoder for RF classifier.

###  V-C Comparison with kernel function based baseline

The String Kernel technique also exhibits lower performance then PCD2Vec. For
example, it has $2\%$ and $1\%$ lower accuracy and precision then PCD2Vec.

###  V-D Comparison with hyperbolic embedding based baseline

PCD2Vec portrays huge performance improvement over Poincaré Embedding method
for all evaluation metrics. Like, it achieves $19\%$ higher accuracy and
$23\%$ higher precision then the Poincaré Embedding method for RF classifier.

###  V-E Comparison with pre-trained language model baseline

Comparing PCD2Vec to a more advance pre-trained language model based-baseline
(Protein Bert) illustrates that PCD2Vec is able to maintain higher performance
results. For instance, PCD2Vec has $5\%$ accuracy and $4\%$ precision
improvement over Protein Bert. A reason for this behavior can be that Protein
Bert may not have been trained on a similar dataset as breast cancer therefore
it is unable to generalize well on our used breast cancer dataset.

Furthermore, for the majority of the methods and classifiers, the average ROC-
AUC values are close to 0.9, indicating that the performance of the models is
good in terms of separating the positive and negative classes. The variance of
the ROC-AUC values is small, implying that the results are consistent across
different runs of the models.

Method |  Classifier | Accuracy | Precision | Recall | F1 Weigh. | F1 Macro |  ROC AUC
---|---|---|---|---|---|---|---
Avg. | Var. | Avg. | Var. | Avg. | Var. | Avg. | Var. | Avg. | Var. | Avg. |  Var.
PCA |  SVM | 0.90 | 0.0001 | 0.92 | 0.0002 | 0.90 | 0.0001 | 0.88 | 0.0003 | 0.82 | 0.0002 | 0.90 |  0.0125
NB | 0.86 | 0.0002 | 0.93 | 0.0001 | 0.86 | 0.0004 | 0.87 | 0.0002 | 0.77 | 0.0005 | 0.93 |  0.0114
MLP | 0.91 | 0.0001 | 0.91 | 0.0001 | 0.91 | 0.0002 | 0.90 | 0.0001 | 0.84 | 0.0004 | 0.90 |  0.0212
KNN | 0.94 | 0.0003 | 0.94 | 0.0001 | 0.94 | 0.0003 | 0.93 | 0.0002 | 0.86 | 0.0001 | 0.92 |  0.0152
RF | 0.95 | 0.0002 | 0.96 | 0.0001 | 0.95 | 0.0003 | 0.95 | 0.0002 | 0.95 | 0.0002 | 0.97 |  0.0099
LR | 0.91 | 0.0003 | 0.91 | 0.0001 | 0.91 | 0.0002 | 0.90 | 0.0001 | 0.84 | 0.0002 | 0.90 |  0.0104
AutoEncoder |  SVM | 0.92 | 0.0003 | 0.92 | 0.0003 | 0.92 | 0.0002 | 0.90 | 0.0001 | 0.82 | 0.0002 | 0.89 |  0.0099
NB | 0.78 | 0.0001 | 0.87 | 0.0002 | 0.78 | 0.0001 | 0.80 | 0.0001 | 0.69 | 0.0003 | 0.89 |  0.0107
MLP | 0.93 | 0.0001 | 0.94 | 0.0003 | 0.93 | 0.0001 | 0.93 | 0.0002 | 0.89 | 0.0002 | 0.94 |  0.0097
KNN | 0.93 | 0.0004 | 0.94 | 0.0002 | 0.93 | 0.0004 | 0.93 | 0.0001 | 0.86 | 0.0002 | 0.93 |  0.0092
RF | 0.95 | 0.0002 | 0.96 | 0.0001 | 0.95 | 0.0002 | 0.95 | 0.0003 | 0.94 | 0.0001 | 0.97 |  0.0114
LR | 0.91 | 0.0002 | 0.91 | 0.0004 | 0.91 | 0.0002 | 0.90 | 0.0002 | 0.80 | 0.0004 | 0.88 |  0.0012
Lasso Regression |  SVM | 0.95 | 0.0003 | 0.96 | 0.0004 | 0.95 | 0.0003 | 0.95 | 0.0001 | 0.94 | 0.0002 | 0.97 |  0.0094
NB | 0.92 | 0.0001 | 0.95 | 0.0005 | 0.92 | 0.0003 | 0.92 | 0.0004 | 0.91 | 0.0005 | 0.96 |  0.0059
MLP | 0.94 | 0.0003 | 0.95 | 0.0005 | 0.94 | 0.0002 | 0.94 | 0.0004 | 0.90 | 0.0005 | 0.94 |  0.0098
KNN | 0.93 | 0.0001 | 0.92 | 0.0003 | 0.93 | 0.0002 | 0.92 | 0.0003 | 0.88 | 0.0002 | 0.92 |  0.0047
RF | 0.95 | 0.0001 | 0.96 | 0.0001 | 0.95 | 0.0002 | 0.95 | 0.0001 | 0.95 | 0.0003 | 0.97 |  0.0098
LR | 0.94 | 0.0003 | 0.94 | 0.0004 | 0.94 | 0.0003 | 0.93 | 0.0001 | 0.91 | 0.0002 | 0.94 |  0.0025
Ridge Regression |  SVM | 0.95 | 0.0001 | 0.96 | 0.0002 | 0.95 | 0.0001 | 0.95 | 0.0005 | 0.94 | 0.0004 | 0.97 |  0.0075
NB | 0.94 | 0.0004 | 0.96 | 0.0005 | 0.94 | 0.0001 | 0.94 | 0.0002 | 0.92 | 0.0005 | 0.96 |  0.0071
MLP | 0.93 | 0.0002 | 0.94 | 0.0003 | 0.93 | 0.0002 | 0.93 | 0.0005 | 0.88 | 0.0006 | 0.93 |  0.0064
KNN | 0.92 | 0.0001 | 0.92 | 0.0002 | 0.92 | 0.0001 | 0.92 | 0.0005 | 0.86 | 0.0001 | 0.91 |  0.0027
RF | 0.95 | 0.0002 | 0.96 | 0.0004 | 0.95 | 0.0002 | 0.95 | 0.0003 | 0.94 | 0.0005 | 0.97 |  0.0028
LR | 0.94 | 0.0003 | 0.94 | 0.0002 | 0.94 | 0.0003 | 0.94 | 0.0001 | 0.91 | 0.0002 | 0.95 |  0.0046
OHE |  SVM | 0.95 | 0.0001 | 0.96 | 0.0003 | 0.95 | 0.0002 | 0.95 | 0.0005 | 0.94 | 0.0004 | 0.97 |  0.0078
NB | 0.94 | 0.0002 | 0.96 | 0.0001 | 0.94 | 0.0002 | 0.94 | 0.0005 | 0.93 | 0.0001 | 0.97 |  0.0051
MLP | 0.94 | 0.0003 | 0.94 | 0.0002 | 0.94 | 0.0003 | 0.93 | 0.0004 | 0.89 | 0.0001 | 0.94 |  0.0069
KNN | 0.93 | 0.0001 | 0.95 | 0.0003 | 0.93 | 0.0002 | 0.93 | 0.0001 | 0.90 | 0.0004 | 0.95 |  0.0074
RF | 0.95 | 0.0004 | 0.96 | 0.0002 | 0.95 | 0.0004 | 0.95 | 0.0003 | 0.94 | 0.0001 | 0.97 |  0.0028
LR | 0.94 | 0.0002 | 0.95 | 0.0004 | 0.94 | 0.0002 | 0.94 | 0.0001 | 0.93 | 0.0003 | 0.96 |  0.0088
String Kernel |  SVM | 0.94 | 0.0007 | 0.95 | 0.0002 | 0.94 | 0.0007 | 0.94 | 0.0014 | 0.90 | 0.0006 | 0.95 |  0.0019
NB | 0.69 | 0.0019 | 0.86 | 0.0017 | 0.69 | 0.0019 | 0.72 | 0.0011 | 0.70 | 0.0019 | 0.86 |  0.0004
MLP | 0.82 | 0.0011 | 0.81 | 0.0030 | 0.82 | 0.0031 | 0.81 | 0.0025 | 0.44 | 0.0040 | 0.71 |  0.0023
KNN | 0.93 | 0.0007 | 0.93 | 0.0055 | 0.93 | 0.0097 | 0.92 | 0.0092 | 0.61 | 0.0023 | 0.82 |  0.0030
RF | 0.95 | 0.0010 | 0.96 | 0.0025 | 0.95 | 0.0010 | 0.95 | 0.0063 | 0.91 | 0.0059 | 0.95 |  0.0083
LR | 0.94 | 0.0008 | 0.95 | 0.0017 | 0.94 | 0.0071 | 0.94 | 0.0018 | 0.90 | 0.0067 | 0.95 |  0.0015
Poincaré Embedding |  SVM | 0.39 | 0.0001 | 0.34 | 0.0001 | 0.39 | 0.0001 | 0.33 | 0.0001 | 0.10 | 0.0008 | 0.52 |  0.0004
NB | 0.74 | 0.0001 | 0.71 | 0.0008 | 0.74 | 0.0001 | 0.72 | 0.0004 | 0.30 | 0.0022 | 0.64 |  0.0005
MLP | 0.64 | 0.0001 | 0.56 | 0.0002 | 0.64 | 0.0001 | 0.59 | 0.0002 | 0.21 | 0.0008 | 0.58 |  0.0001
KNN | 0.60 | 0.0002 | 0.57 | 0.0002 | 0.60 | 0.0002 | 0.57 | 0.0003 | 0.21 | 0.0005 | 0.58 |  0.0001
RF | 0.78 | 0.0005 | 0.74 | 0.0009 | 0.78 | 0.0005 | 0.73 | 0.0008 | 0.31 | 0.0019 | 0.64 |  0.0004
LR | 0.34 | 0.0003 | 0.30 | 0.0013 | 0.34 | 0.0003 | 0.27 | 0.0003 | 0.08 | 0.0001 | 0.50 |  0.0000
Protein Bert |  - | 0.92 | 0.0004 | 0.93 | 0.0002 | 0.92 | 0.0003 | 0.91 | 0.0001 | 0.86 | 0.0002 | 0.92 |  0.0003
PCD2Vec (Ours) |  SVM | 0.87 | 0.0098 | 0.90 | 0.0508 | 0.87 | 0.0098 | 0.86 | 0.0193 | 0.74 | 0.1030 | 0.87 |  0.0505
NB | 0.68 | 0.0470 | 0.87 | 0.0227 | 0.68 | 0.0470 | 0.71 | 0.0450 | 0.75 | 0.0724 | 0.90 |  0.0304
MLP | 0.84 | 0.0209 | 0.85 | 0.0219 | 0.84 | 0.0209 | 0.84 | 0.0236 | 0.64 | 0.0839 | 0.79 |  0.0392
KNN | 0.93 | 0.0107 | 0.94 | 0.0153 | 0.93 | 0.0107 | 0.93 | 0.0136 | 0.70 | 0.0663 | 0.90 |  0.0430
RF | 0.97 | 0.0085 | 0.97 | 0.0099 | 0.96 | 0.0085 | 0.96 | 0.0090 | 0.98 | 0.0842 | 0.99 |  0.0454
LR | 0.86 | 0.0116 | 0.87 | 0.0580 | 0.86 | 0.0116 | 0.84 | 0.0248 | 0.62 | 0.0802 | 0.80 |  0.0395

TABLE II: Average and variance results (of $5$ runs) for different methods.
The best average values are shown in bold.

###  V-F Discussion

The proposed PCD2Vec achieved a 2% improvement in average accuracy compared to
the second-best baseline method. Since the baseline accuracy is above 90% in
most cases, a small increase in accuracy could have significant implications
in the fight against the spread of the coronavirus, leading to early detection
and containment, reducing false positives and unnecessary quarantines, and
improving resource allocation. In conclusion, even a small 2% improvement in
performance can have a big impact on coronavirus host classification.

The class imbalance issue in the dataset was evaluated by measuring precision,
recall, weighted F1-score, macro F1-score, and ROC-AUC of different models
(using the one-vs.-rest approach). High scores of these metrics indicate good
average performance across classes, and the results show that the class
imbalance was handled effectively by all methods.

###  V-G Statistical Significance

We also used the student t-test to evaluate if the classification results were
statistically significant. We noted that the P-values were $<0.05$ in all
cases, confirming the significance of the results. We have not reported the
P-values in this paper due to limited space.

##  VI Conclusion and Future Work

In this paper, we presented a novel method for predicting the host specificity
of coronaviruses by analyzing spike protein sequences. Our method involves the
use of Poisson correction distance, radial basis function kernel, and kernel
PCA to generate low-dimensional embeddings of the spike protein sequences. Our
method was tested on a real-world dataset of spike protein sequences from
different coronavirus species, and our results showed that it outperforms
other existing methods in terms of predictive accuracy. Our method has a solid
theoretical foundation as we have provided proofs for the three important
properties of a distance matrix (triangle inequality, symmetry, and non-
negativity) which can be used to justify its use in a machine-learning
setting. Our results demonstrate the potential of our method to be used as a
tool for predicting the host specificity of coronaviruses, a critical step in
understanding the distribution, evolution, and spread of these diseases. In
conclusion, our proposed method provides an effective, efficient, and reliable
way to predict the host specificity of coronaviruses by analyzing spike
protein sequences. Future work will focus on refining and improving our
method, as well as testing it on larger and more diverse datasets.

## References

  * [1] Mers-cov 2012 outbreak.  https://www.who.int/health-topics/middle-east-respiratory-syndrome-coronavirus-mers#tab=tab_1, 2012\.
  * [2] Hervé Abdi and Lynne J Williams.  Principal component analysis.  Wiley interdisciplinary reviews: computational statistics, 2(4):433–459, 2010.
  * [3] Amir Ahmad, Sunita Garhwal, Santosh Kumar Ray, Gagan Kumar, Sharaf Jameel Malebary, and Omar Mohammed Barukab.  The number of confirmed cases of covid-19 by using machine learning: Methods and challenges.  Archives of Computational Methods in Engineering, pages 1–9, 2020\.
  * [4] Sarwan Ali and Murray Patterson.  Spike2vec: An efficient and scalable embedding approach for covid-19 spike sequences.  In International Conference on Big Data (Big Data), pages 1533–1540, 2021.
  * [5] Sarwan Ali, Bikram Sahoo, Muhammad Asad Khan, Alexander Zelikovsky, Imdad Ullah Khan, and Murray Patterson.  Efficient approximate kernel based spike sequence classification.  IEEE/ACM Transactions on Computational Biology and Bioinformatics, 2022.
  * [6] Sarwan Ali, Bikram Sahoo, Naimat Ullah, Alexander Zelikovskiy, Murray Patterson, and Imdadullah Khan.  A k-mer based approach for sars-cov-2 variant identification.  In International Symposium on Bioinformatics Research and Applications, pages 153–164, 2021.
  * [7] B. Blaisdell.  A measure of the similarity of sets of sequences not requiring sequence alignment.  Proceedings of the National Academy of Sciences, 83:5155–5159, 1986\.
  * [8] Nadav Brandes, Dan Ofer, Yam Peleg, Nadav Rappoport, and Michal Linial.  ProteinBERT: a universal deep-learning model of protein sequence and function.  Bioinformatics, 38(8):2102–2110, 2022.
  * [9] W Ford Doolittle.  Phylogenetic classification and the universal tree.  Science, 284(5423):2124–2128, 1999.
  * [10] M. Farhan, J. Tariq, A. Zaman, M. Shabbir, and I. Khan.  Efficient approximation algorithms for strings kernel based sequence classification.  In Advances in neural information processing systems (NeurIPS), pages 6935–6945. ., 2017.
  * [11] Michael Heinzinger, Ahmed Elnaggar, Yu Wang, Christian Dallago, Dmitrii Nechaev, Florian Matthes, and Burkhard Rost.  Modeling aspects of the language of life through transfer-learning protein sequences.  BMC bioinformatics, 20(1):1–17, 2019.
  * [12] G. Krishnan, S. Kamath, and V. Sugumaran.  Predicting vaccine hesitancy and vaccine sentiment using topic modeling and evolutionary optimization.  In International Conference on Applications of Natural Language to Information Systems (NLDB), pages 255–263, 2021.
  * [13] Kiril Kuzmin et al.  Machine learning methods accurately predict host specificity of coronaviruses based on spike sequences alone.  Biochemical and Biophysical Research Communications, 533(3):553–558, 2020.
  * [14] Yuanlin Ma, Zuguo Yu, Runbin Tang, Xianhua Xie, Guosheng Han, and Vo V Anh.  Phylogenetic analysis of hiv-1 genomes based on the position-weighted k-mers method.  Entropy, 22(2):255, 2020.
  * [15] Gary C McDonald.  Ridge regression.  Wiley Interdisciplinary Reviews: Computational Statistics, 1(1):93–100, 2009.
  * [16] B. Q. Minh et al.  Iq-tree 2: New models and efficient methods for phylogenetic inference in the genomic era.  Molecular Biology and Evolution, 37(5):1530–1534, 2020.
  * [17] Maximillian Nickel and Douwe Kiela.  Poincaré embeddings for learning hierarchical representations.  Advances in neural information processing systems, 30, 2017.
  * [18] J Ranstam and JA Cook.  Lasso regression.  Journal of British Surgery, 105(10):1348–1348, 2018.
  * [19] Ritambhara Singh, Arshdeep Sekhon, et al.  Gakco: a fast gapped k-mer string kernel using counting.  In Joint ECML and Knowledge Discovery in Databases, pages 356–373, 2017.
  * [20] Jamie R Stevens and Christopher J Schofield.  Phylogenetics and sequence analysis–some problems for the unwary.  TRENDS in Parasitology, 19(12):582–588, 2003.
  * [21] Zahra Tayebi, Sarwan Ali, and Murray Patterson.  Robust representation and efficient feature selection allows for effective clustering of sars-cov-2 variants.  Algorithms, 14(12):348, 2021.
  * [22] Shreshth Tuli, Shikhar Tuli, Rakesh Tuli, and Sukhpal Singh Gill.  Predicting the growth and trend of covid-19 pandemic using machine learning and cloud computing.  Internet of Things, 11:100222, 2020.
  * [23] CDC Website.  SARS-CoV 2003 Outbreak.  https://www.cdc.gov/mmwr/preview/mmwrhtml/mm5249a2.htm, 2003.  Online; accessed 12-December-2003.
  * [24] Junyuan Xie, Ross Girshick, and Ali Farhadi.  Unsupervised deep embedding for clustering analysis.  In International conference on machine learning, pages 478–487, 2016.


# Contrastive Object-level Pre-training with Spatial Noise Curriculum Learning

Chenhongyi Yang1 Lichao Huang2 Elliot J. Crowley1
1School of Engineering, University of Edinburgh
2Horizon Robotics
Corresponding Author. Email<EMAIL_ADDRESS>
###### Abstract

The goal of contrastive learning based pre-training is to leverage large
quantities of unlabeled data to produce a model that can be readily adapted
downstream. Current approaches revolve around solving an image discrimination
task: given an anchor image, an augmented counterpart of that image, and some
other images, the model must produce representations such that the distance
between the anchor and its counterpart is small, and the distances between the
anchor and the other images are large. There are two significant problems with
this approach: (i) by contrasting representations at the image-level, it is
hard to generate detailed object-sensitive features that are beneficial to
downstream object-level tasks such as instance segmentation; (ii) the
augmentation strategy of producing an augmented counterpart is fixed, making
learning less effective at the later stages of pre-training. In this work, we
introduce Curricular Contrastive Object-level Pre-training (CCOP) to tackle
these problems: (i) we use selective search to find rough object regions and
use them to build an inter-image object-level contrastive loss and an intra-
image object-level discrimination loss into our pre-training objective; (ii)
we present a curriculum learning mechanism that adaptively augments the
generated regions, which allows the model to consistently acquire a useful
learning signal, even in the later stages of pre-training. Our experiments
show that our approach improves on the MoCo v2 baseline by a large margin on
multiple object-level tasks when pre-training on multi-object scene image
datasets. Code is available at https://github.com/ChenhongyiYang/CCOP.

##  1 Introduction

Imagine that you are given an  _object-level task_ such as object detection
[16, 31, 25], or instance segmentation [19], but you do not have enough data
or compute resources to warrant training a network from scratch. A popular
solution is to take a pre-trained network trained on a existing large-scale
dataset, and then use it for the downstream object-level task [16]. This
allows the network to benefit from an existing store of data, helping it
converges faster and generalise better than training from scratch. But which
pre-trained network should you use?

Figure 1: Image-level contrastive pre-training versus object-level contrastive
pre-training. The former computes a single representation for each image and
uses it in an instance discrimination task. This is not ideal for multi-object
scene images. The latter addresses this problem by computing an embedding for
each object in the input image and applies object-level instance
discrimination.

For several years, the most popular pre-trained networks were those that had
been trained with full supervision on ImageNet [33]; however, recent works [6,
18, 37, 35, 41, 21, 39, 42] have demonstrated that self-supervised
_contrastive learning_ can be used to produce comparable, or better networks
without requiring any manual annotation. In contrastive learning, a deep model
is trained to perform an instance discrimination task, in which two augmented
views of an image are produced: the anchor, and its counterpart. Given the
anchor, the model must find its counterpart from a huge number of noisy
(negative) samples. This is achieved by mapping each image into a low-
dimensional embedding space, and using a contrastive loss to maximise the
similarity between the anchor image and its positive counterpart while
minimising the similarity between the anchor and the noisy samples.
Intuitively, this enables the model to discriminate images based on high-level
semantic features, rather than low-visual level cues.

However, there are two notable problems with this approach. Firstly, as
illustrated in Figure 1, the contrastive loss is applied to distinguish whole
images using a global representation, making it hard for the model to produce
detailed object-level features; as a consequence, the resulting model may
perform poorly when adapted downstream for object-level tasks such as
detection and instance segmentation [31, 19]. The second problem lies in how
the augmented view of the anchor is built [35]; current approaches use a fixed
augmentation strategy throughout the whole pre-training process. However, in
the later stages of pre-training, the model is strong enough to easily
maximise the similarity between the anchor and its counterpart, saturating the
loss gradient provided by the positive pair.

In this work, we present Curricular Contrastive Object-level Pre-training
(CCOP) to address these problems. To learn regional spatially-sensitive
features we introduce two additional loss terms utilising object-level
features; we use selective search [13] to obtain rough object-level bounding
boxes, and then extract features through the RoIAlign [19] operation. These
features are fed into a MLP to produce object-level embeddings. We use a
contrastive loss to allow the model to distinguish between object embeddings
across different images, and an intra-image discrimination loss to allow the
model to differentiate between objects from the same image. These two losses
together with a standard image-level contrastive loss enable the model to
learn not only global features but also local spatially-sensitive features,
and excel when applied downstream to object-level tasks. To solve the loss
saturation problem, we introduce a curriculum learning mechanism [2] to make
the instance discrimination task harder as pre-training progresses.
Specifically, in order to construct positive pairs of a suitable difficulty,
we propose Spatial Noise Curriculum Learning (SNCL), which uses a Spatial
Curriculum Selection (SCS) module that adaptively creates several augmented
boxes and selects the most suitable proposal for the loss based on a dynamic
difficulty-controlling mechanism. This allows the model to consistently
receive a useful learning signal through the whole pre-training process.

In summary, our contributions are threefold:

  * •

We present a simple and effective object-level self-supervised pre-training
framework called Curricular Contrastive Object-level Pre-training (CCOP), in
which rough object locations are obtained without supervision, and an object-
level inter-image contrastive loss and an intra-image discrimination loss are
combined to improve pre-training for downstream object-level tasks when
training on multi-object scene images.

  * •

We propose Spatial Noise Curriculum Learning (SNCL), which uses a Spatial
Curriculum Selection (SCS) module to select the most informative positive
pairs at an appropriate difficulty to aid learning.

  * •

Through extensive experiments, we show our approach achieves state-of-the-art
transfer learning performance on several object-level tasks when pre-training
on MS-COCO [26].

##  2 Related Work

Figure 2: (a) Our main framework. We first compute a set of rough bounding
boxes for objects in the input image using selective search. Two different
views are fed to the query network (upper branch) and key network (lower
branch) respectively. Each box in the key branch is then jittered by our
Spatial Curriculum Selection module to yield augmented boxes of appropriate
difficulty, and the hardest of these is selected to be matched with its
counterpart in the query branch. (b) In more detail, the Spatial Curriculum
Selection module takes a box in the key branch, and produces several jittered
boxes where the extent of jittering is proportional to how far we are into
pre-training. A feature is computed for each of these boxes, and the feature
that is the least similar to the corresponding feature in the query branch is
used for learning.

###  2.1 Self-Supervised Pre-training

Self-supervised methods [1, 10, 6, 18, 5, 17, 8] manipulate input data to
extract a surrogate supervision signal in the form of a pretext task. There
exist numerous pretext tasks including context prediction [30], grayscale
image colourisation [44], jigsaw puzzle solving [3], rotation prediction [14],
and image clustering [4].

Recently, instance discrimination as a pretext task has achieved promising
results [6]. It considers each image in a dataset as its own class, and uses
an InfoNCE loss [29] to match the same instance from among a large number of
negative samples. However, this is extremely expensive. MoCo [18, 7] remedies
this by using a memory bank that stores previously-computed representations as
negative samples. They rely on noise contrastive estimation to compare
instances, which is a special form of contrastive learning. [9] shows that the
memory bank can be entirely replaced with elements from the same batch if the
batch is large enough. More recently, BYOL [17], SimSiam [8], and SWAV [5]
avoid comparison with negative samples. BYOL and SimSiam directly draw
together features from the same instance, while SWAV maps image features to a
set of trainable prototype vectors.

However, models produced through contrasting global image-level
representations are unable to capture the inherent differences between image-
level representations and object-level, and pixel-level representations.
Several works have been proposed to remedy this. In PixPro [42], pixel-level
contrastive learning is performed by exploiting the geometry between pixels in
different views. DetCon [21] uses unsupervised segmentation algorithms to find
pixels belonging to the same object and then contrastively learns pixel
representations. DenseCL [37] adopts a grid matching mechanism between
different images to compute similarity. Self-EMD [27] improves on this by
computing similarity with the earth mover’s distance. For object-level pre-
training, Ins-Loc [43] combines images to produce pseudo-objects whose
regional features are used for contrastive learning. In DetCo [41], a global
image representation is composed of several local grid representations so that
local features can explicitly contribute to the loss. In SCRL [32], random
regions are selected from images for use in contrastive learning. The closest
work to our approach is SoCo [39]; this also utilises selective search to find
rough objects and forces embeddings of regional representations to be close.
However, it uses a fixed strategy for box augmentation, and does not provide a
mechanism for the model to distinguish between different objects within an
image.

Several works show that constructing good positive and negative samples is
essential for contrastive pre-training. In [45], hard positive extrapolation
is used to increase the distance between positive pairs and negative
interpolation is used to improve the diversity of negative samples. A hard-
negative mixing mechanism was proposed in [22] to create a hard negative set
for each query in MoCo. InfoMin [35] uses an adversarial training strategy to
reduce the mutual information between positive pairs.

###  2.2 Curriculum Learning

Curriculum learning was first proposed in [2] to mimic the easy-to-hard
learning practices that are commonly used to help with human and animal
learning. Recently [34] used a transferable curriculum learning framework to
allow information transfer between source and target domains; DeepSDF [11] was
proposed to help reconstruct shapes through gradually increasing difficulty;
DCL [38] uses an adaptive sampling strategy with loss re-weighting to aid
learning when given class-imbalanced data.

Following the core easy-to-hard philosophy of curriculum learning, our
proposed SNCL gradually decrease the similarity between positive object pairs
as our model gets stronger.

##  3 Method

In this section we describe our object-level self-supervising training
framework: Curricular Contrastive Object-level Pre-training (CCOP). It is
adapted from MoCo v2, which we review in Section 3.1. The key innovations of
CCOP are (i) the incorporation of object-level objectives, and (ii) spatial
noise curriculum learning, which we describe in Sections 3.2 and Section 3.3
respectively.

###  3.1 Description of the Baseline Model

Contrastive learning allows models to be trained so that different views of
the same image are mapped to be close in some representation space, while
different images are mapped to be far away from each other.

MoCo v2 [7] is a successful contrastive pre-training framework and is adopted
as our baseline approach. It comprises a query network $f_{q}^{\theta_{q}}$
and a key network $f_{k}^{\theta_{k}}$ that share the same architecture, where
$\theta_{q}$ and $\theta_{k}$ are the model parameters. Each network consists
of two parts: a backbone network e.g. ResNet-50 [20], and a MLP head that maps
the backbone features to an embedding vector. For training, an image $x$ is
augmented to produce two views $x_{q}$ and $x_{k}$ that are fed into $f_{q}$
and $f_{k}$ respectively to get their embeddings $z_{q}$ and $z_{k}$, and a
memory queue $Q$ is built to store the embeddings generated in previous
iterations, which are used as negative samples. A contrastive loss is employed
so that the model can match $z_{q}$ and $z_{k}$ against a huge number of noisy
negative samples $z_{k}^{-}$:

$\displaystyle\mathcal{L}_{con}=-\log{\frac{\exp{(z_{q}\cdot
z_{k}/\tau)}}{\exp{(z_{q}\cdot z_{k}/\tau)}+\sum_{z_{k}^{-}\in
Q}{\exp{(z_{q}\cdot z_{k}^{-}/\tau)}}}}$ (1)

where $\tau$ is a temperature hyper-parameter. The query network is updated
using the gradient of this loss, and the key network is the exponential moving
average (EMA) of the query network:

$\displaystyle\theta_{k}\leftarrow m\theta_{k}^{\prime}+(1-m)\theta_{q}.$ (2)

When pre-training is complete, the backbone of the query network is extracted
for downstream use.

###  3.2 Object-level Objectives

Our goal is to pre-train a model that will excel when applied to downstream
object-level tasks such as instance segmentation and object detection. This
pre-training should be conducted without any supervision, so that we are able
to leverage large amounts of unlabelled data wherever available in the wild.
We would like such a model to produce object-sensitive features. Because of
this we choose to use multi-object scene images for pre-training. However,
identifying the object regions to produce these features is challenging; this
typically requires supervision.

To circumvent this, we take inspiration from [15] and use selective search
[13] to obtain rough bounding boxes for the objects without supervision. We
use RoIAlign [19] to extract the regional features in a box from the feature
map generated by the backbone network. These are fed into a 2-layer MLP to
yield object embeddings. Please see Figure 2 for an illustration. We use a
separate memory queue $Q_{obj}$ to store object embeddings. Bounding boxes
that correspond to the same object in the two augmented image views are
treated as positive pairs, and the regional representations stored in
$Q_{obj}$ are used as negative samples in a constrative loss
$\mathcal{L}_{con}^{obj}$, which assumes the same form as Equation 1.

However, contrasting box embeddings against the negative samples stored in
$Q_{obj}$ may allow the model to find a shortcut where it could leverage the
global image-level information to avoid learning detailed object-level
features. To prevent this from happening, we introduce an intra-image
discrimination loss to force the model to compute diverse embeddings for
objects in the same image. Given bounding boxes in an image
$\\{B_{i}\\}_{i=1}^{N}$ with corresponding embeddings $\\{z_{i}\\}_{i=1}^{N}$,
we use a hinge loss

$\displaystyle\mathcal{L}_{intra}=\frac{1}{\left\|\mathbf{P}\right\|_{1}}\sum_{i=1}^{N}{\sum_{j\neq
i}{\mathbf{P}_{ij}\max\\{(z_{i}\cdot z_{j}-\alpha),0\\}}}.$ (3)

Here $\mathbf{P}$ is an indicator matrix where $\mathbf{P}_{ij}$ is 1 only if
the IoU between $B_{i}$ and $B_{j}$ is less than 0.05. Note that we compute
$\mathcal{L}_{intra}$ using the original boxes instead of the augmented boxes.
This loss reduces the similarity between box embeddings that do not overlap in
an image, and can be seen as a form of regularisation which prevents the model
from mapping distinct objects close together.

The final loss function used to train the model is combination of the original
image contrastive loss in MoCo v2, the object contrastive loss, and the intra-
image discrimination loss:

$\displaystyle\mathcal{L}_{total}=\mathcal{L}_{con}^{img}+\mathcal{L}_{con}^{obj}+\mathcal{L}_{intra}$
(4)

###  3.3 Spatial Noise Curriculum Learning

In contrastive learning, a fixed data augmentation strategy is often deployed
for the whole training process. As a result, the learning signal that the
model receives from the contrastive loss will decrease at the later stages of
pre-training; as the model becomes stronger it can more easily match positive
pairs with a high similarity from the noisy negative samples, which reduces
the magnitude of the gradients provided by the positive pair. Let’s consider
the gradient of $\mathcal{L}_{con}$ with respect to $z_{q}$. For $\tau=1$ it
is given by

$\displaystyle\nabla_{z_{q}}\mathcal{L}_{con}=\frac{\sum_{i=1}^{M}{(z^{-}_{i}-z_{k})\cdot\exp{(z_{q}\cdot
z^{-}_{i})}}}{\exp{(z_{q}\cdot z_{k})}+\sum_{i=1}^{M}\exp{(z_{q}\cdot
z^{-}_{i})}}$ (5)

where $\\{z^{-}_{i}\\}_{i=1}^{M}$ are noisy samples stored in the memory bank,
and $M$ is the size of the memory bank (65,335 in our case). In the later
stages of training, the $z_{q}\cdot z^{-}_{i}$ terms tend to zero [45], and
this gradient can be approximated as

$\displaystyle\nabla_{z_{q}}\mathcal{L}_{con}\approx(z_{q}-z_{k})+\frac{1}{M}\sum_{i}^{M}{(z_{i}^{-}-z_{q})}.$
(6)

We can see that when $z_{k}$ is very close to $z_{q}$, the first term in
Equation 6 disappears and only negative samples are contributing to the
gradient this loss.

To address this problem, we employ curriculum learning to gradually raise the
difficulty of the instance-discrimination task so that the model can
consistently receive a useful learning signal. Here, we define the difficulty
as the similarity between $z_{q}$ and $z_{k}$. For image-level contrastive
pre-training, one could generate several augmentations and adaptively select
the most suitable augmented image for the loss. However, this would be
expensive to compute. Fortunately, at the object-level we can augment a
bounding box multiple times and quickly acquire their features through the
effective RoIAlign operation [19]. This faciliates Spatial Noise Curriculum
Learning (SNCL).

Figure 3: Our Spatial Curriculum Selection module augments object bounding
boxes so that they are an appropriate difficulty for the model throughout
training (note that colour distortions are not shown here).

SNCL aims to gradually decrease the similarity of positive object pairs by
jittering the bounding boxes as training progresses; we hypothesise that this
will allow the model to attend to fine-grained visual cues in order to connect
these bounding boxes. In our framework, we achieve this by introducing a
Spatial Curriculum Selection (SCS) module, shown in Figure 2, that adaptively
creates a set of jittered boxes of a dynamically-controlled difficulty, before
selecting the hardest of these for learning. In detail, for a box
$B^{k}=(x,y,w,h)$111$(x,y)$ are the coordinates for the top-left corner of a
box with width and height $w$ and $h$ respectively. in the key branch, the SCS
module generates a set of augmented boxes $\\{\hat{B^{k}_{i}}\\}$ through:

$\displaystyle x^{\prime}=\sigma_{x}w+x,\ y^{\prime}=\sigma_{y}h+y,$
$\displaystyle w^{\prime}=w\cdot\mathrm{exp}(\sigma_{w}),\
h^{\prime}=h\cdot\mathrm{exp}(\sigma_{h}),$ (7)

where $\sigma_{x},\sigma_{y},\sigma_{w},\sigma_{h}$ are noise coefficient
drawn from a uniform distribution $(-\zeta,\zeta)$. In order to increase the
difficulty, we linearly scale $\zeta$ from 0.3 to 1.0 as training proceeds. We
ensure that the IoU between the jittered boxes and the original box is larger
than a threshold $\beta_{t}$, which is dynamically lowered from 0.8 to 0.3 in
our SNCLwith respect to the training iteration $t$. Then, the SCS module
select the hardest box $\hat{B^{k}_{j}}$ to compute $z_{k}$ for contrastive
learning where $j=argmin_{i}\\{z_{q}\cdot\hat{z_{i}}\\}$ . Here $\hat{z_{i}}$
is the object embedding corresponding to $\hat{B^{k}_{i}}$. This enlarges the
first term in Equation 6, allowing $z_{k}$ to contribute to the loss gradient.
We visualise the boxes produced by this process for different epochs in Figure
3.

##  4 Experiments

###  4.1 Experimental Setup

Datasets. CCOP utilises multi-object scene images as pre-training data to
produce a model that excels at object-level downstream tasks. For our
experiments we use MS-COCO [26] as the pre-training dataset; specifically, the
train2017 subset for pre-training, which contains 118k images. Note that we do
not use any annotations.

Model settings. We use MoCo v2 [7] as our baseline. We use ResNet-50 [20] as
the default backbone network. We follow [39] to add a feature pyramid network
(FPN) [24] to the backbone network, whose generated feature maps are from
$\frac{1}{4}$ to $\frac{1}{32}$ of the input image size. Following [18] we use
the $C_{5}$ feature map for image embeddings, and the features for object-
level embeddings are pooled by a RoIAlign operation from the FPN feature maps
according to their size. We use two separate MLPs with intermediate dimensions
of 2048 to get 128-D embeddings. Both $Q_{img}$ and $Q_{obj}$ are of size
65,335.

Method | APbb | AP${}^{bb}_{50}$ | AP${}^{bb}_{75}$ | APmk | AP${}^{mk}_{50}$ | AP${}^{mk}_{75}$
---|---|---|---|---|---|---
Rand Init | 31.0 | 49.6 | 33.3 | 28.6 | 46.7 | 30.5
IN Sup | 38.7 | 59.4 | 42.2 | 35.2 | 56.2 | 37.6
SimCLR [6] | 38.0 | 57.3 | 41.5 | 34.3 | 54.5 | 36.6
BYOL [17] | 38.0 | 57.4 | 41.5 | 34.5 | 54.6 | 36.9
SimSiam [8] | 38.3 | 58.0 | 41.8 | 34.8 | 55.2 | 37.2
MoCo v2 [7] | 38.5 | 58.0 | 41.9 | 34.9 | 55.2 | 37.3
Self-EMD [27] | 39.3 | 60.1 | 42.8 | - | - | -
DenseCL [37] | 39.6 | 59.3 | 43.3 | 35.7 | 56.8 | 38.4
CCOP (Ours) | 39.9 | 59.8 | 43.9 | 36.2 | 56.8 | 38.8

Table 1: COCO Object detection and instance segmentation using Mask R-CNN FPN
with 1$\times$ schedule. ‘Rand Init’ refers to training from scratch and ‘IN
Sup’ refers to ImageNet supervised pre-training. All unsupervised models are
pre-trained using COCO train2017 for 800 epochs.

Method | APbb | AP${}^{bb}_{50}$ | AP${}^{bb}_{75}$ | APmk | AP${}^{mk}_{50}$ | AP${}^{mk}_{75}$
---|---|---|---|---|---|---
Rand Init | 17.7 | 31.0 | 17.7 | 16.5 | 28.8 | 16.6
IN Sup | 25.5 | 42.4 | 27.0 | 23.3 | 39.4 | 24.3
SimCLR | 24.1 | 40.0 | 25.3 | 22.0 | 37.2 | 22.8
BYOL | 24.2 | 41.6 | 24.9 | 22.3 | 38.5 | 22.9
SimSiam | 24.5 | 40.5 | 26.0 | 22.4 | 37.8 | 23.4
MoCo v2 | 24.2 | 40.1 | 25.5 | 22.2 | 37.6 | 22.9
DenseCL | 26.1 | 42.7 | 27.2 | 23.8 | 40.1 | 24.9
CCOP (Ours) | 26.4 | 43.3 | 27.9 | 24.0 | 40.4 |  25.1 †

Table 2: Semi-supervised object detection and instance segmentation on COCO
using Mask R-CNN FPN. All models are fine-tuned for 27k iterations using only
10% of the COCO train2017 annotated data.

Selective search and data augmentation. Selective search [13] is performed
offline before pre-training to produce bounding box proposals for each image.
Boxes whose IoU are larger than 0.5 are merged. We remove bounding boxes whose
areas are smaller than $12^{2}$ and whose aspect ratios are outside of the
range $(0.33,3)$. During training, images are augmented as in MoCo v2: a
random patch of the image is cropped and resized to 224$\times$224, followed
by colour distortion, random Gaussian blur, random grayscale and random
horizontal flip operations. If after augmentation, a bounding box is clipped
by more than 40% of its original area, it is removed.

| COCO RetinaNet | PascalVOC Faster R-CNN | COCO Keypoint R-CNN
---|---|---|---
Method | APbb | AP${}^{bb}_{50}$ | AP${}^{bb}_{70}$ | APbb | AP${}^{bb}_{50}$ | AP${}^{bb}_{75}$ | APkp | AP${}^{kp}_{50}$ | AP${}^{bb}_{kp}$
Rand Init | 25.5 | 40.9 | 26.7 | 39.8 | 65.7 | 41.4 | 63.0 | 85.2 | 69.0
IN Sup | 37.9 | 57.9 | 40.4 | 54.4 | 80.4 | 59.7 | 65.5 | 87.1 | 71.8
SimCLR | 36.4 | 55.8 | 38.4 | 53.1 | 78.9 | 58.4 | 65.2 | 86.8 | 70.7
BYOL | 36.7 | 56.1 | 39.3 | 52.4 | 79.6 | 57.1 | 65.1 | 86.6 | 70.8
SimSiam | 36.9 | 56.6 | 39.1 | 53.2 | 79.3 | 58.3 | 65.6 | 86.8 | 71.5
MoCo v2 | 36.6 | 56.3 | 38.8 | 53.6 | 79.9 | 58.9 | 65.3 | 86.8 | 71.3
DenseCL | 38.3 | 58.1 | 41.0 | 55.3 | 80.5 | 60.8 | 65.5 | 87.2 | 71.5
CCOP (Ours) | 38.4 | 58.4 | 41.2 | 55.8 | 80.7 | 61.9 | 65.7 | 87.4 | 71.6
Table 3: Transfer learning performance of COCO object detection using
RetinaNet, Pascal VOC object detection using Faster R-CNN FPN, and COCO
Keypoint detection using Keypoint R-CNN FPN. All models are pre-trained on
COCO train2017 for 800 epochs.

Optimisation. We closely follow the pre-training settings in [37]. We train
the model for 800 epochs with an initial learning rate of 0.3, which is
cosine-annealed [28] to zero. We use an SGD optimiser with weight decay and
momentum of 0.0001 and 0.9 respectively. The margin parameter $\alpha$ in
Equation 3 is set to $0.4$. We use 8 Nvidia 2080Ti GPUs for pre-training with
a mini-batch size of 256.

Implementation. Our code is based on OpenSelfSup222https://github.com/open-
mmlab/OpenSelfSup and is available at https://github.com/ChenhongyiYang/CCOP.
We performed pre-training for competing methods locally333The exception to
this is for Self-EMD [27] as we could not find any source code. The numbers
for Self-EMD that appear in the sequel are as presented in the original paper.
across 8 Nvidia 2080Ti GPUs, using the training schemes given in their
respective papers, and the config files in the OpenSelfSup toolkit where
available. Downstream models in Section 4.2 were trained locally using
Detectron2 [40].

###  4.2 Main Results

Here, we apply our pre-trained model to a suite of downstream object-level
tasks and compare against existing pre-training approaches.

COCO object detection and segmentation. In Table 1, we compare the performance
of our approach to existing pre-training schemes by fine-tuning a Mask R-CNN
FPN model. We followed the training settings in [37]: the model is trained on
the COCO train2017 subset and tested on the mini-val subset. Synchronised
batch normalisation (SyncBN) is used in the backbone, FPN, and detection
heads. CCOP achieves a 1.4 and 1.3 absolute improvement on APbb and APmk
compared to the MoCo v2 baseline, and surpasses other recent pre-training
approaches including those designed for object-level and pixel-level tasks
such as DenseCL [37] and Self-EMD [27].

Semi-supervised detection and segmentation. In Table 2, we report the semi-
supervised object detection and instance-segmentation performance on COCO
where only 10% of images are used for fine-tuning444We note that this is not
technically semi-supervised learning as it does not utilise unlabelled data
points, but we use the term to be consistent with several of the works we
compare to.. This experiment allows us to gauge how well CCOP works in a
regime where we have limited downstream annotation. Our approach achieves a
2.4 APbb and 1.8 APmk improvement over the baseline MoCo v2 model and is
state-of-the-art.

COCO single-stage object detection. Here, we show that the model pre-trained
using CCOP can be used in a single stage object detector, demonstrating its
flexibility. We transfer the pre-trained model to a RetinaNet [25] detector
using 1$\times$ training schedule. SyncBN is used in the backbone and FPN, and
group normalisation is used in the detection heads. Results are given in Table
3. Our model attains 1.8 absolute APbb improvement compared to the baseline of
MoCo v2, which is larger than the improvement for the two-stage Mask R-CNN.
This could be because our method is able to pre-train not only the backbone
network, but also the FPN neck that plays an more important role in single
stage detectors [23].

Pascal VOC object detection. To verify whether our model can generalise to
datasets other than COCO, we transfer the pre-trained model to a Faster R-CNN
FPN detector and fine-tune it using the Pascal VOC [12] dataset. Following
[18], we train the model for 24k iterations using the trainval2007+2012 sets
and evaluate it on the test2007 set. The results are provided in Table 3; our
methods achieves a 2.2 improvement in AP over the baseline and surpasses other
state-of-the-art approaches, indicating that our model is able to generalise
to new data.

COCO human keypoint detection. Our pre-trained model also works well when used
downstream for human keypoint detection. In Table 3, we provide the results
obtained when adapting our pre-trained model to a Keypoint R-CNN FPN [19]
model. Our approach achieves 65.7 APkp, broadly improving on alternative pre-
training methods.

###  4.3 Ablation Studies

In this section, we examine how the different aspects of CCOP contribute to
the overall performance. We use COCO object detection and instance
segmentation to evaluate different models, which are fine-tuned using a Mask
R-CNN FPN model with 1$\times$ training schedule.

Obj | Intra | SNCL | APbb | AP${}^{bb}_{50}$ | AP${}^{bb}_{75}$ | APmk | AP${}^{mk}_{50}$ | AP${}^{mk}_{75}$
---|---|---|---|---|---|---|---|---
|  |  | 38.5 | 58.0 | 41.9 | 34.9 | 55.2 | 37.3
✓ |  |  | 39.3 | 59.3 | 42.9 | 35.6 | 56.5 | 38.2
✓ | ✓ |  | 39.6 | 59.5 | 43.2 | 35.8 | 56.2 | 38.5
✓ |  | ✓ | 39.7 | 59.9 | 43.4 | 36.0 | 56.9 | 38.5
✓ | ✓ | ✓ | 39.9 | 59.8 | 43.9 | 36.2 | 56.8 | 38.8

Table 4: Ablation Studies of our contributions. ‘Obj’ stands for object-level
contrastive learning; ‘Intra’ stands for intra-image discrimination loss; SNCL
is our Spatial Noise Curriculum Learning approach.

Method | APbb | AP${}^{bb}_{50}$ | AP${}^{bb}_{75}$ | APmk | AP${}^{mk}_{50}$ | AP${}^{mk}_{75}$
---|---|---|---|---|---|---
MoCo v2 | 38.5 | 58.0 | 41.9 | 34.9 | 55.2 | 37.3
Random | 39.1 | 58.9 | 42.8 | 35.1 | 55.8 | 37.8
GT | 39.4 | 59.3 | 43.2 | 35.5 | 56.4 | 38.2
SS | 39.9 | 59.8 | 43.9 | 36.2 | 56.8 | 38.8

Table 5: Comparison of different object generation approaches. ‘Random’ stands
for random generated boxes; ‘GT’ stands for human annotated boxes; ‘SS’ stands
for selective search.

$\beta$ | APbb | AP${}^{bb}_{50}$ | AP${}^{bb}_{75}$ | APmk | AP${}^{mk}_{50}$ | AP${}^{mk}_{75}$
---|---|---|---|---|---|---
1.0 | 39.6 | 59.5 | 43.2 | 35.8 | 56.2 | 38.5
0.6 | 39.5 | 56.5 | 42.9 | 35.7 | 56.5 | 38.0
0.3 | 39.2 | 58.9 | 42.7 | 35.5 | 56.1 | 38.2
SNCL | 39.9 | 59.8 | 43.9 | 36.2 | 56.8 | 38.8

Table 6: Comparison of box augment strategies with different IoU lower-bound
$\beta$. Note that 1.0 stands for no augmentation.

Role of each module. In Table 4 we examine how object-level contrastive
learning, the intra-image discrimination loss, and Spatial Noise Curriculum
Learning contribute to the overall performance. Our baseline MoCo v2 model
achieves 38.5 APbb and 34.9 APmk. When we include the object-level contrastive
learning loss, the performance increases significantly to 39.3 APbb and 35.6
APmk. This shows that incorporating object-level information in the pre-
training stage can help with downstream object-level tasks. When we add the
intra-image discrimination loss, the performance is further enhanced to 39.6
APbb and 35.8 APmk. Separately adding SNCL without the intra-image loss also
improves performance to 39.7 APbb and 36.0 APmk. Finally, when all three
modules are adopted, we achieve 39.9 APbb and 36.2 APmk.

Object Generation. We compare proposing regions using selective search to two
alternative strategies: randomly generating proposals, and using ground-truth
bounding boxes. For random generation, we produce 12 bounding boxes per view
(this is on average how many boxes selective search produces) and we remove
boxes whose areas are smaller than $12^{2}$ and whose aspect ratios are out of
range $(0.33,3)$. For ground-truth, we use the COCO annotations. The
performance for these strategies is given in Table 5. Using randomly generated
boxes gives us a marginal improvement over the baseline but is worse than
using ground-truth boxes or those obtained through selective search. Our
intuition is that random boxes allow the model to learn spatially-sensitive
features, but do not give any notion of what an object is. We found it
intriguing that selective search boxes outperform ground-truth. A possible
explanation is that selective search introduces more diversity into training.

$\alpha$ | APbb | AP${}^{bb}_{50}$ | AP${}^{bb}_{75}$ | APmk | AP${}^{mk}_{50}$ | AP${}^{mk}_{75}$
---|---|---|---|---|---|---
0.0 | 38.8 | 58.1 | 41.9 | 35.2 | 55.0 | 37.4
0.2 | 39.5 | 59.5 | 42.9 | 35.6 | 56.5 | 38.0
0.4 | 39.9 | 59.8 | 43.9 | 36.2 | 56.8 | 38.8
0.6 | 39.9 | 59.8 | 43.4 | 36.0 | 56.8 | 38.5

Table 7: Ablation study of the margin parameter $\alpha$ in the intra-image
discrimination loss.

Backbone | Method | APbb | AP${}^{bb}_{50}$ | AP${}^{bb}_{75}$ | APmk | AP${}^{mk}_{50}$ | AP${}^{mk}_{75}$
---|---|---|---|---|---|---|---
ResNet-18 | MoCo v2 | 35.0 | 53.9 | 38.2 | 31.8 | 51.2 | 34.0
CCOP | 35.6 | 54.7 | 38.6 | 32.3 | 52.0 | 34.4
ResNet-50 | MoCo v2 | 38.5 | 58.0 | 41.9 | 34.9 | 55.2 | 37.3
CCOP | 39.9 | 59.8 | 43.9 | 36.2 | 56.8 | 38.8
ResNet-101 | MoCo v2 | 41.0 | 60.7 | 44.5 | 36.7 | 57.6 | 39.5
CCOP | 42.1 | 61.9 | 45.9 | 37.6 | 58.7 | 40.3

Table 8: Performance comparison between our CCOP and the baseline MoCo v2 on
different backbone sizes.

Curriculum Learning. We conduct experiments to study the extent to which
curriculum learning benefits our approach. We compare SNCL to box generations
with fixed magnitude noise and with a fixed IoU lower-bound $\beta$. The
results are presented in Table 6. Note that when $\beta=1$ there is no
augmentation applied. We observe that even if no noise is added, the pre-
trained model is quite good achieving a 39.6 APbb and 35.8 APmk. Lowering the
IoU lower-bound to 0.6 barely changes the performance. However when we lower
it to 0.3 this harms performance. We observe that Spatial Noise Curriculum
Learning can avoid such problems by gradually enlarging the extent of
augmentation. This stabilises training for the early stages of learning and
raise the difficulty in later stages, leading to a superior performance of
39.9 APbb and 36.2 APmk.

Intra-image Margin In Table 7 we explore the influence of the margin $\alpha$
parameter in the intra-image discrimination loss. We can see that the
performance is robust for large margins, but low for small values. A possible
reason is that a small margin makes it difficult for the model to learn good
embeddings as it introduce too much conflict in the image.

Figure 4: Performance comparison between the baseline MoCo v2 and our CCOP on
Mask R-CNN FPN with 1$\times$ training schedule under different sizes of pre-
training dataset. Note that 200% means that we add the extra COCO unlabeled
set for pre-training. As we increase the dataset size, CCOP performance
continues to increase.

Model sizes. In Table 8, we compare the performance of CCOP and the baseline
MoCo v2 using ResNets of different sizes. There are performance boosts when
using our approach in all cases. However, these are more significant as the
models get larger.

Pre-train dataset size. In Figure 4, we show the performance of our approach
and the baseline model for different sizes of the pre-training dataset. We
take subsets with different percentages of the COCO training set and use them
to pre-train the model for 800 epochs. We also test the performance when we
add the COCO unlabeled set into pre-training, which contains an extra 123k
images. We observe that the performance of the baseline starts to saturate
earlier than our method. This suggests that our method has the potential for
large-scale pre-training.

Pre-train dataset size. In Figure 4, we show the performance of our approach
and the baseline model for different sizes of the pre-training dataset. We
take subsets with different percentages of the COCO training set and use them
to pre-train the model for 800 epochs. We also test the performance when we
add the COCO unlabeled set into pre-training, which contains an extra 123k
images. We observe that the performance of the baseline starts to saturate
earlier than our method. This suggests that our method has the potential for
large-scale pre-training.

###  4.4 Qualitative Studies

We qualitatively study how CCOP learns useful features from bounding boxes
generated in an unsupervised manner on multi-object input images and how this
helps with downstream object-level tasks.

In Figure 5, we visualise the learnt object embeddings using T-SNE [36]. In
detail, we forward the COCO mini-val images through the pre-trained model and
compute the object embeddings of the ground-truth boxes, then pick 20 common
classes (denoted with different colours) that appear in both COCO and Pascal
VOC for visualisation. We can see that the box embeddings in the same class
form a series of clear clusters in the embedding space even though the model
is trained without supervision, demonstrating the model’s ability to capture
high-level semantic information. As only the pre-trained backbone is used for
fine-tuning, we also visualise the regional features that are directed pooled
and averaged from the backbone network. The clustering structure can also be
clearly observed in these features even before they are transformed using an
MLP head. Further, we query the $k$ nearest neighbours of each object in the
representation space and check how many of them are in the same class, which
we denote as kNN recall. These values are given in Table 9. The performance
using backbone features is only slightly worse than the performance using the
actual embeddings, indicating that the pre-trained model has strong potential
for transfer learning.

(a) Backbone

(b) Prediction

Figure 5: Visualisation of regional backbone features and predicted box embeddings on COCO mini-val set using T-SNE for dimension reduction. The objects are acquired by using oracle annotations. We only draw the common 20 classes, shown in different colors, that appears in both COCO and Pascal VOC. Topk | 1 | 5 | 10 | 20 | 50 | 100
---|---|---|---|---|---|---
Prediction | 50.1 | 44.4 | 40.8 | 37.5 | 33.6 | 30.5
Backbone | 48.1 | 43.2 | 39.4 | 35.9 | 31.7 | 28.3
Table 9: Comparison of kNN recall between the predicted object embeddings and
the backbone regional features on COCO mini-val set using oracle boxes and
class annotations.

Finally, we observe how good our approach is at learning spatially-sensitive
and scale-sensitive features that are important to object-level tasks. First,
two augmented views of an image are fed into the query and key branch
respectively. Then we pool and average an object’s regional features from the
query network and use this it as a convolution kernel to compute the cross-
correlation with the backbone features computed in the key branch. The results
are in Figure 6. We can see that the same object in the key image is
highlighted; this suggests that the pre-trained model is able to extract
features that are suitable for object-level tasks.

Figure 6: Visualisation of box query results. We pool the box features
directly from the query network’s backbone features and used it as convolution
kernel to compute the cross-correlation results with key image’s features in
different FPN layers.

##  5 Conclusion

In this work, we propose an object-level contrastive pre-training framework:
Curricular Contrastive Object-level Pre-training (CCOP) to improve transfer
learning performance for downstream object-level tasks. Our approach is
designed for pre-training on multi-object scene images that take both inter-
image and intra-image instance discrimination into consideration. To further
improve the performance, we designed Spatial Noise Curriculum Learning that
adaptively selects the positive pairs of an appropriate difficulty for the
current stage of training. Extensive experimental results verify that a model
pre-trained using our method is able to transfer well to different tasks,
different downstream model architectures, and different datasets.

#### Acknowledgements.

The authors would like to thank Adam Jelly, Antreas Antoniou, Jiawei He, Joe
Mellor, Justin Engelmann, and Zehui Chen for their helpful feedback and
suggestions. Chenhongyi Yang was supported by a PhD studentship provided by
the School of Engineering, University of Edinburgh.

## References

  * [1] Pulkit Agrawal, João Carreira, and Jitendra Malik.  Learning to See by Moving.  In ICCV, 2015.
  * [2] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston.  Curriculum Learning.  In ICML, 2009.
  * [3] Fabio M. Carlucci, Antonio D’Innocente, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi.  Domain Generalization by Solving Jigsaw Puzzles.  In CVPR, 2019.
  * [4] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze.  Deep Clustering for Unsupervised Learning of Visual Features.  In ECCV, 2018.
  * [5] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised Learning of Visual Features by Contrasting Cluster Assignments.  In NeurIPS, 2020.
  * [6] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A Simple Framework for Contrastive Learning of Visual Representations.  In ICML, 2020.
  * [7] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He.  Improved baselines with momentum contrastive learning.  arXiv preprint arXiv:2003.04297, 2020.
  * [8] Xinlei Chen and Kaiming He.  Exploring Simple Siamese Representation Learning.  CVPR, 2021.
  * [9] Xinlei Chen, Saining Xie, and Kaiming He.  An Empirical Study of Training Self-Supervised Vision Transformers.  In ICCV, 2021.
  * [10] Carl Doersch, Abhinav Gupta, and Alexei A Efros.  Unsupervised visual representation learning by context prediction.  In ICCV, 2015.
  * [11] Yueqi Duan, Haidong Zhu, He Wang, Li Yi, Ram Nevatia, and Leonidas J Guibas.  Curriculum DeepSDF.  In ECCV, 2020.
  * [12] Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The Pascal Visual Object Classes Challenge: A Retrospective.  IJCV, 2015.
  * [13] Pedro F Felzenszwalb and Daniel P Huttenlocher.  Efficient graph-based image segmentation.  IJCV, 2004.
  * [14] Zeyu Feng, Chang Xu, and Dacheng Tao.  Self-Supervised Representation Learning by Rotation Feature Decouplin.  In CVPR, 2019.
  * [15] Ross Girshick.  Fast r-cnn.  In ICCV, 2015.
  * [16] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik.  Rich feature hierarchies for accurate object detection and semantic segmentation.  In CVPR, 2014.
  * [17] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning.  In NeurIPS, 2020.
  * [18] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum Contrast for Unsupervised Visual Representation Learning.  In CVPR, 2020.
  * [19] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In ICCV, 2017.
  * [20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [21] Olivier J. Hénaff, Skanda Koppula, Jean-Baptiste Alayrac, Aaron van den Oord, Oriol Vinyals, and João Carreira.  Efficient Visual Pretraining With Contrastive Detection.  In ICCV, 2021.
  * [22] Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus.  Hard negative mixing for contrastive learning.  NeurIPS, 2020.
  * [23] Yanghao Li, Yuntao Chen, Naiyan Wang, and Zhaoxiang Zhang.  Scale-aware trident networks for object detection.  In ICCV, 2019.
  * [24] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In CVPR, 2017.
  * [25] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In ICCV, 2017.
  * [26] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In ECCV, 2014.
  * [27] Songtao Liu, Zeming Li, and Jian Sun.  Self-emd: Self-supervised object detection without imagenet.  arXiv preprint arXiv:2011.13677, 2020.
  * [28] Ilya Loshchilov and Frank Hutter.  Sgdr: Stochastic gradient descent with warm restarts.  In ICLR, 2017.
  * [29] Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation Learning with Contrastive Predictive Coding.  arXiv preprint arXiv:1807.03748, 2018.
  * [30] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros.  Context Encoders: Feature Learning by Inpainting.  In CVPR, 2016.
  * [31] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In NeurIPS, 2015.
  * [32] Byungseok Roh, Wuhyun Shin, Ildoo Kim, and Sungwoong Kim.  Spatially Consistent Representation Learning.  In CVPR, 2021.
  * [33] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei.  ImageNet Large Scale Visual Recognition Challenge.  IJCV, 2015.
  * [34] Yang Shu, Zhangjie Cao, Mingsheng Long, and Jianmin Wang.  Transferable Curriculum for Weakly-supervised Domain Adaptation.  In AAAI, 2019.
  * [35] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola.  What Makes for Good Views for Contrastive Learning?  In NeurIPS, 2020.
  * [36] Laurens Van der Maaten and Geoffrey Hinton.  Visualizing Data Using T-SNE.  JMLR, 2008.
  * [37] Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li.  Dense contrastive learning for self-supervised visual pre-training.  In CVPR, 2021.
  * [38] Yiru Wang, Weihao Gan, Jie Yang, Wei Wu, and Junjie Yan.  Dynamic Curriculum Learning for Imbalanced Data Classification.  In ICCV, 2019.
  * [39] Fangyun Wei, Yue Gao, Zhirong Wu, Han Hu, and Stephen Lin.  Aligning Pretraining for Detection via Object-Level Contrastive Learning.  arXiv preprint arXiv:2106.02637, 2021.
  * [40] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick.  Detectron2.  https://github.com/facebookresearch/detectron2, 2019.
  * [41] Enze Xie, Jian Ding, Wenhai Wang, Xiaohang Zhan, Hang Xu, Peize Sun, Zhenguo Li, and Ping Luo.  DetCo: Unsupervised Contrastive Learning for Object Detection.  In ICCV, 2021.
  * [42] Zhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao, Stephen Lin, and Han Hu.  Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learning.  In CVPR, 2021.
  * [43] Ceyuan Yang, Zhirong Wu, Bolei Zhou, and Stephen Lin.  Instance localization for self-supervised detection pretraining.  In CVPR, 2021.
  * [44] Richard Zhang, Phillip Isola, and Alexei A Efros.  Colorful Image Colorization.  In ECCV, 2016.
  * [45] Rui Zhu, Bingchen Zhao, Jingen Liu, Zhenglong Sun, and Chang Wen Chen.  Improving Contrastive Learning by Visualizing Feature Transformation.  In ICCV, 2021.