# Distributional Robustness Loss for Long-tail Learning

Dvir Samuel
Bar-Ilan University, Israel
<EMAIL_ADDRESS>Gal Chechik
Bar-Ilan University, Israel
NVIDIA Research, Israel
<EMAIL_ADDRESS>
###### Abstract

Real-world data is often unbalanced and long-tailed, but deep models struggle
to recognize rare classes in the presence of frequent classes. To address
unbalanced data, most studies try balancing the data, the loss, or the
classifier to reduce classification bias towards head classes. Far less
attention has been given to the latent representations learned with unbalanced
data. We show that the feature extractor part of deep networks suffers greatly
from this bias. We propose a new loss based on robustness theory, which
encourages the model to learn high-quality representations for both head and
tail classes. While the general form of the robustness loss may be hard to
compute, we further derive an easy-to-compute upper bound that can be
minimized efficiently. This procedure reduces representation bias towards head
classes in the feature space and achieves new SOTA results on CIFAR100-LT,
ImageNet-LT, and iNaturalist long-tail benchmarks. We find that training with
robustness increases recognition accuracy of tail classes while largely
maintaining the accuracy of head classes. The new robustness loss can be
combined with various classifier balancing techniques and can be applied to
representations at several layers of the deep model.

##  1 Introduction

Real-world data typically has a long-tailed distribution over semantic
classes: few classes are highly frequent, while many classes are only rarely
encountered. When trained with such unbalanced data, deep models tend to
produce predictions that are biased and over-confident towards head classes
and fail to recognize tail classes.

Early approaches for handling unbalanced data used resampling [12, 17] or loss
reweighing [18, 27] aiming to re-balance the training process. Other
approaches address unbalanced data by transferring information from head to
tail classes [31, 46, 41, 29], or by applying an adaptive loss [6] or
regularization of classifiers [21]. The main focus of these techniques is on
balancing the multi-class classifier.

Figure 1:  Our distributional robustness loss is designed for learning a
representation where samples are kept close to the centroid of their class.
Here, the empirical centroid $\widehat{\mu_{2}}$ (framed pink triangle) is
estimated based only on few samples (four pink triangles) and as a result it
deviates far from the true centroid ($\mu_{2}$). Our loss pulls the same-class
samples (green arrow), and pushes away other-class samples (red arrows). The
loss takes into account the estimation error by pushing and pulling towards a
worst-case possible distribution within an uncertainty area around the
estimated centroid (dashed red line). Uncertainty areas are typically larger
for tail classes, compared with head classes that have many samples (blue
dashed line around $\widehat{\mu_{1}}$).

Far less attention has been given to the latent representations learned with
unbalanced data. Intuitively, head classes are encountered more often during
training and are expected to dominate the latent representation at the top
layer of a deep model. Counter to this intuition, [21] compared a series of
techniques for rebalancing representations and concluded that ”data imbalance
might not be an issue in learning high-quality representations”, and that
strong long-tailed recognition can be achieved by only adjusting the
classifier. However, the effect of unbalanced data on the learned
representations is far from being understood, and the extent to which it may
hurt classification accuracy is unknown. While existing rebalancing methods do
not improve representations, the question remains if better representations
could substantially improve recognition with unbalanced data.

The current paper focuses on improving the representation layer of deep models
trained with unbalanced data. We show that large gains in accuracy can be
achieved simply by balancing the representation at the last layer. The key
insight is that the distribution of training samples of tail classes does not
represent well the true distribution of the data. This yields representations
that hinder the classifier applied to that representation.

To address this problem, we introduce ideas from robust optimization theory.
We design a loss that is more robust to the uncertainty and variability of
tail representations. Standard training follows Empirical Risk Minimization
(ERM), which is designed to learn models that perform well on the training
distribution. However, ERM assumes that the test distribution is the same as
the training distribution, and this assumption often breaks with tail classes.
In comparison, Distributionally Robust Optimization (DRO) [28, 14, 4] is
designed to be robust against likely shifts of the test distribution. It
learns classifiers that can handle the worst-case distribution within a
neighborhood of the training distribution. Figure 1 illustrates this idea.

In the general case, computing a loss based on a worst-case distribution may
be computationally hard. Here, we show how the worst-case loss can be bounded
from above, with a bound that has an intuitive form and can be easily
minimized. The resulting bound allows us to minimize the DRO loss, which only
affects the representation, and to combine it with a standard classification
loss, which tunes a classifier on top of the representation.

The main contributions of this paper are:

  1. 1.

We formulate learning with unbalanced data as a problem of robust optimization
and highlight the role of high variance in hindering tail accuracy.

  2. 2.

We develop a new loss, DRO-LT Loss, based on distributional robustness
optimization for learning a balanced feature extractor. Training with DRO-LT
yields representations that capture well both head and tail classes.

  3. 3.

We derive an upper bound of the DRO-LT loss that can be computed and optimized
efficiently. We further show how to learn the robustness safety margin for
each class, jointly with the task, avoiding additional hyper-parameter tuning.

  4. 4.

We evaluate our approach on four long-tailed visual recognition benchmarks:
CIFAR100-LT, CIFAR10-LT, ImageNet-LT, and iNaturalist. Our proposed method
consistently achieve superior performance over previous models.

##  2 Related Work

###  2.1 Long-tail recognition

Real-world data usually follows a long-tailed distribution, which causes
models to favor head classes and overfit tail classes [5, 35]. Previous
efforts to address this effect can be divided into four main approaches: Data-
manipulation approaches, Loss-manipulation approaches, Two-stage fine-tuning,
and ensemble methods.

Data-manipulation (re-sampling): Data-manipulation approaches aim to balance
long-tail data. There are three popular techniques of resampling strategies:
(1) Over-sampling minority (tail) classes by simply copying samples [7, 17].
(2) Under-sampling majority (head) classes by removing samples [12, 20]. (3)
Generating augmented samples to supplement tail classes [2, 8, 24, 7]. While
simple and intuitive, over-sampling methods suffer from heavy over-fitting on
the tail classes, under-sampling methods degrade the generalization of models,
and data augmentation methods are expensive to develop.

Loss-manipulation (re-weighting): Loss reweighting approaches encourage
learning of tail classes by setting costs to be non-uniform across classes.
For instance, [18] scaled the loss by inverse class frequency. An alternative
strategy down-weighs the loss of well-classified examples, preventing easy
negatives from dominating the loss [27] or dynamically rescale the cross-
entropy loss based on the difficulty to classify a sample [34]. [6] proposed
to encourage larger margins for rare classes. [22] use class-uncertainty
information, using Bayesian uncertainty estimates, to learn robust features,
and [10] reweighs using the effective number of samples instead of
proportional frequency.

Two-stage fine-tuning: Two-stage methods separate the training process into
representation learning and classifier learning [32, 21, 30, 35]. The first
stage aims to learn good representations from unmodified long-tailed data,
training using conventional cross-entropy without re-sampling or re-weighting.
The second stage aims to balance the classifier by freezing the backbone and
finetuning the last fully connected layers with re-sampling techniques or by
learning to unbias the confidence of the classifier. Those methods basically
assume that the bias towards head classes is significant only in the
classifier (i.e, last fully connected layer), or that tweaking the classifier
layer can correct the underlying biases in the representation.

Ensemble-models: Ensemble models focus on generating a balanced model by
assembling and grouping models. Typically, classes are separated into groups,
where classes that contain similar training instances are grouped together.
Then, individual models focused on each group are assembled to form a multi-
expert framework. [46] learned one branch for head classes and another for
tail classes, then combine the branches using a soft-fusion procedure. [43]
distilled a unified model from multiple teacher classifiers. Each classifier
focuses on the classification of a small and relatively balanced group of
classes from the data. [40] described a shared architecture for multiple
classifiers, a distribution-aware loss, and an expert routing module. The
current paper proposes a two-stage approach for training one model with a
single classifier. Thus, we compare our method with non-ensemble approaches.

###  2.2 Distributionally robust optimization

Distributionally Robust Optimization (DRO) [28, 14, 4] considers the case
where the test distribution is different from the train distribution. It does
that by defining an uncertainty set of test distributions around the training
distribution and learning a classifier that minimizes the expected risk with
respect to the worst-case test distribution. DRO has been shown to be
equivalent to standard robust optimization (RO) [44] and has a strong
relationship with regularization methods [36], Risk-Aversion methods and game
theory [33]. For more details, see [33]. As far as we know, DRO theory was not
applied for long-tail learning.

Many studies looked into learning representations that are robust to
adversarial attacks [3, 1, 16] but this is outside the scope of the current
paper.

##  3 Overview of our approach

We start with an overview of the main idea of our approach and provide the
details in subsequent sections.

Our goal is to learn a representation at the last layer of a deep network,
such that the distributions of samples of different classes are well
separated. Then, they can later be correctly classified by different linear
classifiers. Deep networks can learn such representations efficiently when
trained with enough labeled data. However, when a class has only few samples,
the distribution of training samples may not represent well the true
distribution of the data, and the representations learned by the model hinder
the classifier.

To remedy this shortcoming, we design a loss that is applied to the
representation, which takes into account errors due to a small number of
samples. Figure 1 illustrates this idea. Our loss extends standard contrastive
losses, which pull samples closer to the centroid of their own class and push
away samples that belong to other classes. Our new loss accounts for the fact
that the true centroids are unknown, and their estimate is noisy. It,
therefore, optimizes against the worst possible centroids within a safety
hyper ball around the empirical centroid.

In the general case, computing such a worst-case loss may be computationally
hard. We further derive an upper bound of that loss that can be computed
easily and show that using that bound as a surrogate loss, yields
significantly better representations. The resulting loss has a simple general
form, yielding that the loss for a sample $z$ is

$\displaystyle\mathcal{L}(z)_{Robust}=-\log{\frac{e^{-d(\widehat{\mu_{c}},z)-\Delta}}{\sum_{z^{\prime}}{e^{-d(\widehat{\mu_{c}},z^{\prime})-\Delta^{\prime}}}}}.$
(1)

where $d(\widehat{\mu_{c}},z)$ measures the distance in feature space between
a sample $z$ and the estimated centroid of its class $\widehat{\mu_{c}}$,
$\Delta$ and $\Delta^{\prime}$ are robustness margins that we describe below.

##  4 Distributional Robust Optimization

When learning a classifier, we seek a model $f$ that minimizes the expected
loss for the data distribution $P(x,y)$.

$\min_{f}E_{x\sim P}[l_{f}(x)]=\int l(f(x),y)dP(x,y).$ (2)

Since the data distribution is not known, Empirical Risk Minimization (ERM)
[39] proposes to use the training data for an empirical estimate of the data
distribution $P_{\delta}=\frac{1}{n}\sum\delta(x=x_{i},y=y_{i})$, where
$\delta$ is the kronecker delta.

$\textbf{ERM: \quad}\min_{f}E_{(x,y)\sim P_{\delta}}\left[l_{f}(x)\right]$ (3)

Unfortunately, using $P_{\delta}$ to approximate $P$ makes the naive
assumption that the test distribution would be close to the empirical train
distribution. That assumption may be far from true when the training data is
small. In those cases, it is beneficial to choose other estimates of $P$ that
are more likely to reduce the loss over the test distribution.

One such solution is given by Distributionally robust optimization (DRO) [14,
4]. It suggests learning a model $f$ that minimizes the loss within a family
of possible distributions

$\textbf{DRO: \quad}\min_{f}\sup_{Q\in U}{E_{(x,y)\sim Q}[l_{f}(x)]}.$ (4)

DRO aims to perform well simultaneously for a set of test distributions, each
in an uncertainty set $U$. The set of distributions $U$ is typically chosen as
a hyper ball of radius $\epsilon$ ($\epsilon$-ball) around the empirical
training distribution $\widehat{P}$:

$U:=\\{Q:D(Q,\widehat{P})\leq\epsilon\\},$ (5)

where $D$ is a discrepancy measure between distributions, usually chosen to be
the KL-divergence or the Wasserstein distance.

##  5 Our approach

We now formally describe our approach: DRO-LT Loss.

###  5.1 Preliminaries

We are given $n$ labeled samples $(x_{i},y_{i})$ $i=1,\ldots,n$, where $y_{i}$
is one of $k$ classes $\\{c_{1},\ldots,c_{k}\\}$. Let $f_{\theta}$ be a
feature extractor function $f$ with learnable parameters $\theta$, which maps
any given input sample $x_{i}$ to $z_{i}=f_{\theta}(x_{i})$. The set of mapped
input $Z={z_{1},...z_{n}}$ resides in some latent vector space.

Let $S_{c}=\\{z_{i}|y_{i}=c\\}$ be the set of feature vectors representing
samples whose label is $c$. We denote by $\widehat{\mu_{c}}$ the empirical
centroid of that set, namely the mean of all feature vectors $z$ of class $c$,
$\widehat{\mu_{c}}=\frac{1}{|c|}\sum_{i=1}^{|c|}{z_{i}}$. We also denote by
$\mu_{c}$ the mean of the true data distribution $P$ of samples from class
$c$, $\mu_{c}=E_{x\sim P|y=c}[z]$.

###  5.2 A loss for representation learning

We adopt ideas from metric learning and representation learning [42, 26, 37]
for designing a representation-learning loss.

Given a representation $z_{i}$ of a sample $x_{i}$ that has a label $y_{i}=c$,
we wish to design a loss that maps $z_{i}$ close in feature space to other
samples from the same class, and far from samples of other classes. We start
by developing a contrastive loss and later extend it to a robust one.

Consider a sample $(x_{i},y_{i})$ from a class $y_{i}=c$, whose feature
representation is $z_{i}$. We model the samples of class $c$ as if they are
distributed around a centroid $\mu_{c}$, and the likelihood of a sample decays
exponentially with the distance from the centroid $\mu_{c}$ of class $c$. Such
exponential decay has been long used in metric learning and can be viewed as
reflecting a random walk over the representation space [15, 13]. The
normalized likelihood of a sample is, therefore:

$P(z_{i}|\mu_{c})=\frac{e^{-d(\mu_{c},z_{i})}}{\sum_{z^{\prime}\in
Z}{e^{-d(\mu_{c},z^{\prime})}}},$ (6)

where $d$ is a measure of divergence or distance between the centroid
$\mu_{c}$ and a feature representation $z_{i}$. $d$ is typically set to be the
Euclidean distance, but can also be modeled with heavier tails as when using a
student t-distribution [38].

Similarly, for a set $S_{c}=\\{z_{i}|y_{i}=c\\}$, the log-likelihood is:

$\log P(S_{c}|\mu_{c})=\sum_{z_{i}\in
S_{c}}\log\frac{e^{-d(\mu_{c},z_{i})}}{\sum_{z^{\prime}\in
Z}{e^{-d(\mu_{c},z^{\prime})}}}.$ (7)

We define a negative log-likelihood loss as a weighted average over per-class
losses:

$\displaystyle\mathcal{L_{NLL}}$ $\displaystyle=$ $\displaystyle\sum_{c\in
C}w(c)\left(-\log P(S_{c}|\mu_{c})\right)$ $\displaystyle=$
$\displaystyle-\sum_{c\in C}w(c)\sum_{z_{i}\in
S_{c}}\log{\frac{e^{-d(\mu_{c},z_{i})}}{\sum_{z^{\prime}\in
Z}{e^{-d(\mu_{c},z^{\prime})}}}},$

where $w(c)$ are class weights. Setting $w(c)=\frac{1}{|S_{c}|}$ gives equal
weighting to all classes and prevents head classes from dominating the loss.

###  5.3 A robust loss

The log-likelihood loss of Eq. (7) operates under the assumption that the
centroid of each class $\mu_{c}$ is known. In reality, it is not available to
us. Following Empirical-risk minimization, we could naively plug in the
empirical estimate $\widehat{\mu_{c}}$ in Eq. (7), but $\widehat{\mu_{c}}$ may
be a poor approximation of $\mu_{c}$, and estimation error grows as the number
of samples $|S_{c}|$ decreases. As a result, the log-likelihood and the loss
would also be badly estimated.

Instead of approximating the NLL loss ($-\log P(S_{c}|\mu_{c})$), we show that
we can bound it with high probability by computing the worst-case loss over a
set of candidate centroids. For that purpose, we take an approach based on
distributionally robust optimization.

Let $\widehat{p_{c}}$ be the empirical distribution of samples of class $c$.
We define the uncertainty set of candidate distributions in its neighborhood
to be

$U_{c}:=\\{q|D(q||\widehat{p_{c}})\leq\epsilon_{c}\\},$ (9)

where $D$ is a measure of divergence of two distributions. Specifically, we
consider here the case where $D$ is the Kullback-Leibler divergence, and the
distributions under consideration are same-variance spherical Gaussians. In
this case, their divergence equals
$D_{KL}(q||\widehat{p_{c}})=\frac{1}{2\sigma^{2}}d(\mu_{q},\mu_{\widehat{p_{c}}})^{2}$[9],
where $d$ is the Euclidean distance. Hence for any $q\in U_{c}$, we have

$d(\mu_{q},\mu_{\widehat{p_{c}}})\leq\sigma_{c}\sqrt{2\epsilon_{c}}\equiv\varepsilon_{c},$
(10)

where we define $\varepsilon_{c}\equiv\sigma_{c}\sqrt{2\epsilon_{c}}$ for
convenience.

We now derive an upper bound on the NLL loss that we can compute using the
estimated centroids $\widehat{\mu_{c}}$. The form of the bound is very
intuitive, it can be viewed as a modification of the NLL loss, where the
distance between a sample and an empirical centroid, is increased by a factor
that depends on the radius of the uncertainty ball.

######  Theorem 1.

Let $\epsilon_{c}$ be the radius of the uncertainty set $U_{c}$ and let
$\sigma_{c}$ be the variance of the distribution of a sample in class $c$. Let
$p(\epsilon)$ be the probability that the true distribution whose centroid is
$\mu_{c}$ is within $U_{c}$. The per-class negative log-likelihood is bounded
by

$\displaystyle-\log
P(z|\mu_{c})\leq-\log\frac{e^{-d(\widehat{\mu_{c}},z)-2\varepsilon_{c}}}{\sum_{z^{\prime}\in
Z}e^{-d(\widehat{\mu_{c}},z^{\prime})-2\varepsilon_{c}\delta(z^{\prime},c)}}$
(11)

with probability $p(\epsilon)$, where $\delta(z,c)=1$ if $z$ is of the class
$c$ and 0 otherwise and $\varepsilon_{c}=\sigma_{c}\sqrt{2\epsilon_{c}}$.

###### Proof.

Denote the negative log-likelihood of a given sample $z$ and class
distribution $p$ with a centroid $\mu$ by $\mathcal{L}(z,p)=-\log P(z|\mu)$.
With probability $p(\epsilon)$, the true distribution $p_{c}$ is within the
uncertainty ball $p_{c}\in U_{c}$. Therefore

$\mathcal{L}(z,p_{c})\leq\max_{q\in U_{c}}{\mathcal{L}(z,q)}=\max_{q\in
U_{c}}(-\log P(z|\mu_{c}^{q})).$ (12)

Here, $\mu_{c}^{q}$ is the centroid of a distribution $q\in U_{c}$. To bound
$-\log P(z|\mu_{c}^{q})$, we use the triangle inequality and write:

$\displaystyle d(\mu_{c}^{q},z)$ $\displaystyle\leq$ $\displaystyle
d(\widehat{\mu_{c}},z)+d(\widehat{\mu_{c}},\mu_{c}^{q})$ (13) $\displaystyle
d(\widehat{\mu_{c}},z)$ $\displaystyle\leq$ $\displaystyle
d(\mu_{c}^{q},z)+d(\widehat{\mu_{c}},\mu_{c}^{q})$

yielding

$\displaystyle-d(\mu_{c}^{q},z)\\!\\!\\!\\!\\!\\!$
$\displaystyle\geq\\!\\!-d(\widehat{\mu_{c}},z)-d(\widehat{\mu_{c}},\mu_{c}^{q})\\!\\!\geq\\!\\!-d(\widehat{\mu_{c}},z)-\varepsilon_{c}$
(14) $\displaystyle-d(\mu_{c}^{q},z)\\!\\!\\!\\!\\!\\!$
$\displaystyle\leq\\!\\!-d(\widehat{\mu_{c}},z)+d(\widehat{\mu_{c}},\mu_{c}^{q})\\!\\!\leq\\!\\!-d(\widehat{\mu_{c}},z)+\varepsilon_{c}$
(15)

Applying Eq. (14) to the numerator of Eq. (6), and applying Eq. (15) to its
denominator, we obtain:

$P(z|\mu_{c}^{q})={\frac{e^{-d(\mu_{c}^{q},z)}}{\sum_{z^{\prime}\in
Z}{e^{-d(\mu_{c}^{q},z^{\prime})}}}}\geq\frac{e^{-d(\widehat{\mu_{c}},z)-2\varepsilon_{c}}}{\sum_{z^{\prime}\in
Z}e^{-d(\widehat{\mu_{c}},z^{\prime})-2\varepsilon_{c}\delta(z^{\prime},c)}}$
(16)

where $\delta(z,c)=1$ when $c$ is the class of $z$ and $0$ otherwise. This
inequality holds for all distributions $q\in U_{c}$, and also for the true
distribution $\mu_{c}$ as a special case (with probability $p(\epsilon)$). The
negative log-likelihood is therefore bounded by

$-\log
P(z|\mu_{c})\leq-\log\frac{e^{-d(\widehat{\mu_{c}},z)-2\varepsilon_{c}}}{\sum_{z^{\prime}\in
Z}e^{-d(\widehat{\mu_{c}},z^{\prime})-2\varepsilon_{c}\delta(z^{\prime},c)}}.$
(17)

This completes the proof. ∎

Based on the theorem, define a surrogate robustness loss:

$\displaystyle\mathcal{L}_{Robust}=-\sum_{c\in C}w(c)\sum_{z\in
S_{c}}\log{\frac{e^{-d(\widehat{\mu_{c}},z)-2\varepsilon_{c}}}{\sum_{z^{\prime}}{e^{-d(\widehat{\mu_{c}},z^{\prime})-2\varepsilon_{c}\delta(z^{\prime},c)}}}}$
(18)

This surrogate loss amends the loss of Eq. (8) in a simple and intuitive way.
It increases the distance $d(\widehat{\mu_{c}},z)$ between a sample and an
empirical centroid in a way that depends on the radius of the uncertainty ball
of a class.

#### Joint loss.

In practice, we train the deep network with a combination of two losses. A
standard cross-entropy loss is applied to the output of the classification
layer, and the robust loss is applied to the latent representation of the
penultimate layer. We linearly combine these two losses

$\mathcal{L}=\lambda\mathcal{L}_{CE}+(1-\lambda)\mathcal{L}_{Robust}.$ (19)

The trade-off parameter $\lambda$ can be tuned using a validation set. See
implementation details below.

####  $p(\varepsilon)$ and Lower bound.

Appendix A provides a formal definition of $p(\varepsilon)$. Appendix B
derives a lower bound of our loss and also shows empirically that the bounds
are tight in our experiments.

##  6 Training

Uncertainty radius: The size of the uncertainty $\epsilon$-ball around each
class plays an important role. When the uncertainty radius is too small, the
probability that the true centroid is within the uncertainty area decreases,
together with the probability that the bound holds. When the radius is too
large, the bound is more likely to hold, but it becomes less tight.
Furthermore, since tail classes have fewer samples, the estimate of class
centroids is expected to be noisier and a larger radius is needed.

We explored three ways to determine the radius:

  1. 1.

Shared $\varepsilon$: All classes share the same uncertainty radius.

  2. 2.

Sample count $\varepsilon/\sqrt{n}$: The class radius scales with
$1/\sqrt{n}$, where $n$ is the number of training samples. This scaling is
based on the fact that the standard-error-of-the-mean decays as $\sqrt{n}$,
and leads to tail classes having a larger safety radius. See Appendix C for
more details.

  3. 3.

Learned $\varepsilon$: We treated the radius as a learnable parameter and
tuned its value during training.

In the first two cases, the radius parameter $\varepsilon$ was treated as a
hyper-parameter and tuned using a validation set.

#### Training process:

To compute the robustness loss, an initial feature representation of the data
is required for estimating class centroids. As a result, during training, we
first train the model with standard cross-entropy loss ($\lambda=1$) to learn
initial feature representations and centroids. Then, we add the DRO loss to
the training by setting $\lambda<1$. Finally, as in [21], we learn a balanced
classifier by freezing the feature extractor and fine-tune only the classifier
with balanced sampling.

#### Estimating centroids:

Calculating the centroids for each class is computationally expensive if
computed often using the full dataset. One could estimate the centroids within
each batch, but with unbalanced data, minority classes would hardly have any
samples and their centroid estimates would be very poor. Instead, we compute
the features $z_{i}$ for every sample $x_{i}$ at the beginning of every epoch,
compute the centroids for each class, and keep them fixed in memory for the
duration of the epoch.

##  7 Experiments

###  7.1 Datasets

We evaluated our proposed method using experiments on three major long-tailed
recognition benchmarks.

(1) CIFAR100-LT [6]: A long-tailed version of CIFAR100 [25]. CIFAR100 consists
of 60K images from 100 categories. Following [6], we control the degree of
data imbalance with an imbalance factor $\beta$.
$\beta=\frac{N_{max}}{N_{min}}$, where $N_{max}$ and $N_{min}$ are the number
of training samples for the most frequent and the least frequent classes,
respectively. We conduct experiments with $\beta=$ 100, 50, and 10.
(2) ImageNet-LT [31]: A long-tailed version of the large-scale object
classification dataset ImageNet [11] by sampling a subset following the Pareto
distribution with power value $\alpha=6$. Consists of 115.8K images from 1000
categories with 1280 to 5 images per class. (3) iNaturalist [19]: A large-
scale dataset for species classification. It is long-tailed by nature, with an
extremely unbalanced label distribution. Has 437.5K images from 8,142
categories.

###  7.2 Compared methods

We compared our approach with the following methods.

(A) Baselines: CE: Naive training with a cross-entropy loss; ReSample: Over-
sampling classes to reach a uniform distribution as in [6]. (B) Loss-
manipulation: Reweight: Reweight the loss as in [6, 46]; Focal Loss [27] and
LDAM Loss [6]. (C) Two-stage fine-tuning: $\tau$-norm [21], cRT [21] and
smDragon [35]. (D) Representation learning: SSL the method by [23] and SSP,
the method by [45].

(E) DRO-LT variants: We compared four ways to set the radii of $U_{c}$. Set
$\varepsilon=0$ (ERM); $\varepsilon$ value shared across all classes; A shared
value divided by the square root of class size ($\varepsilon/\sqrt{n}$); and a
learned value per class.

Imbalance Type | Long-tailed CIFAR-100
---|---
Imbalance Ratio | 100 | 50 | 10
CE* | 38.32 | 43.85 | 55.71
ReSample [10] | 33.44 | - | 55.06
ReWeight [10] | 33.99 | - | 57.12
Focal Loss [27] | 38.41 | 44.32 | 55.78
LDAM Loss [6] | 39.60 | 44.97 | 56.91
$\tau$-norm* [21] | 41.11 | 46.74 | 57.06
cRT* [21] | 41.24 | 46.83 | 57.93
smDRAGON [35] | 43.55 | 46.85 | 58.01
SSL* [23] | 37.51 | 44.02 | 56.70
SSP [45] | 43.43 | 47.11 | 58.91
DRO-LT (Ours) |  |  |
$\varepsilon=0$ (ERM) | 43.92 | 52.31 | 59.54
Shared $\varepsilon$ | 45.66 | 55.32 | 61.22
$\varepsilon/\sqrt{n}$ | 46.92 | 57.20 | 63.10
Learned $\varepsilon$ | 47.31 | 57.57 | 63.41

Table 1: Top-1 accuracy of ResNet32 on long-tailed CIFAR-100 [6], comparing
our method and SoTA techniques. Asterisks * denote reproduced results. DRO-LT
variants achieve best results on all imbalance ratios.

###  7.3 Evaluation Protocol

Following [31] and [21], we report the top-1 accuracy over all classes on
class balanced test sets. This metric is denoted by ”Acc”. For CIFAR-100 and
ImageNet-LT, we further report accuracy on three splits of the set of classes.
”Many”: classes with more than 100 train samples; ”Med”: classes with 20 – 100
train samples; and ”Few”:  classes with less than 20 train samples.

###  7.4 Implementation details

In all experiments, we use an SGD optimizer with a momentum of 0.9 to optimize
the network. For long-tailed CIFAR-100, we follow [6] and train a ResNet-32
backbone on one GPU with a multistep learning-rate schedule. For ImageNet-LT
and iNaturalist, we follow [21] and use the cosine learning-rate schedule to
train a ResNet-50 backbone on 4 GPUs.

Hyper-parameter tuning: We determined the number of training epochs (early-
stopping), and tuned hyperparameters using the validation set. We optimized
the following hyper-parameters: (1) Radius parameter
$\varepsilon\in\\{1,2,5,10,30,70\\}$ for ”Shared $\varepsilon$” and ”Sample
Count $\varepsilon/\sqrt{n}$”. (2) Trade-off parameter
$\lambda\in{\\{0,0.3,0.5,0.7,1\\}}$. (3) Learning rate
$\in{\\{10^{-4},10^{-3},10^{-2}\\}}$. we studied the sensitivity of the
accuracy to the values of $\varepsilon$ and $\lambda$, and found that high
accuracy is obtained for a wide range of $\epsilon$ values. We also found
accuracy to be quite stable when tuning $\lambda$ and selected $\lambda=0.5$.
See Appendix D for details.

Figure 2: Learned $\varepsilon$ values for each class of a ResNet-32 trained
on CIFAR100-LT with imbalance ratio 50. Blue: Polynomial fit.

| Long-tailed CIFAR-100 | Long-tailed ImageNet
---|---|---
Methods | Many | Med | Few | Acc | Many | Med | Few | Acc
CE* | 65.5 | 37.9 | 7.4 | 38.3 | 64.0 | 38.8 | 5.8 | 41.6
LDAM Loss [6] | 61.0 | 41.6 | 19.8 | 39.6 | - | - | - | -
OLTR [30] | 61.8 | 41.4 | 17.6 | 41.2 | - | - | - | -
$\tau$-norm [21] | 61.4 | 42.5 | 15.7 | 41.1 | 56.6 | 44.2 | 27.4 | 46.7
smDragon [35] | 60.5 | 44.3 | 23.5 | 43.5 | 59.7 | 44.2 | 25.3 | 47.4
SSL * [23] | 64.1 | 36.9 | 7.1 | 37.5 | 61.4 | 47.0 | 28.2 | 49.8
DRO-LT (Ours) |  |  |  |  |  |  |  |
$\varepsilon=0$ (ERM) | 61.9 | 43.7 | 22.2 | 43.9 | 61.0 | 45.5 | 26.8 | 48.1
Shared $\varepsilon$ | 64.1 | 47.9 | 21.5 | 45.7 $\pm$ 0.2 | 62.6 | 45.2 | 30.5 | 51.6 $\pm$ 0.4
$\varepsilon/\sqrt{n}$ | 65.0 | 49.8 | 22.3 | 46.9 $\pm$ 0.2 | 63.8 | 49.5 | 32.7 | 53.0 $\pm$ 0.3
Learnable $\varepsilon$ | 64.7 | 50.0 | 23.8 | 47.3 $\pm$ 0.1 | 64.0 | 49.8 | 33.1 | 53.5 $\pm$ 0.2

Table 2: Top-1 accuracy on long-tailed CIFAR-100 [6] with imbalance factor
100, and ImageNet-LT[31]. We also report many-shot (”Many”), medium-shot
(”Med”) and few-show (”Few”) performance separately. Our method performs well
on tail classes without sacrificing head accuracy. Asterisks * denote results
reproduced using code published by authors.

##  8 Results

CIFAR100-LT:  Table 1 compares DRO-LT with common long-tail methods on
CIFAR100-LT. It shows that all our robust loss variants consistently achieve
the best results on all imbalance factors, emphasizing the importance of
robust learning in unbalanced data. ”Learned $\varepsilon$” outperforms other
variants of our method. CIFAR-10-LT: Appendix G provides results for
CIFAR-10-LT with imbalanced ratio 100. ImageNet-LT:  Table 2 further evaluates
our approach on ImageNet-LT and CIFAR100-LT (imbalance factor=100) reporting
accuracy for different test splits. DRO-LT performs well on tail classes
(”Few”) as well as head classes (”Many”). This contrasts with previous methods
that sacrifice head accuracy for better tail classification. This also
suggests that DRO-LT learns high-quality features for all classes.
iNaturalist: Table 3 evaluates our approach on the large-scale iNaturalist.
DRO-LT slightly improves the accuracy compared with SoTA baselines. See
Appendix F for more results and further analysis.

| iNaturalist
---|---
CE | 61.7
LDAM Loss [6] | 68.0
$\tau-$norm [21] | 69.5
CB LWS [21] | 69.5
smDragon [35] | 69.1
SSL * [23] | 66.4
DRO-LT (ours) |
Learned $\varepsilon$ | 69.7 $\pm$ 0.1

Table 3: Top-1 accuracy on long-tailed iNaturalist. DRO-LT achieves slightly
better results compared to previous methods. Asterisks * denote reproduced
results by us.

Classifier vs feature extractor:  Our method focuses on improving the learned
representation at the penultimate layer. Other methods focused on improving
the classifier applied to that representation. Therefore, it is interesting to
explore the relation between these two tasks (compare with [46, 21]). We,
therefore, compared different feature extractors and classifier training
methods.

Figure 3:  Accuracy of a nearest-centroid neighbor classifier when applied to
convolutional layers 0, 10, 20, and 30 of a ResNet-32. Top: The model was
trained on balanced CIFAR-100. The validation accuracy grows when using higher
layers. Bottom: The model was trained on CIFAR-100-LT. We report balanced
validation accuracy for head classes (blue), medium classes (orange) and tail
classes (green). The accuracy gap between head and tail classes is substantial
even at all layers.

For representation learning, we employ plain training with a cross-entropy
loss (CE), balanced resampling of the data (RS) and our method (DRO-LT). For
classifier learning, we freeze the parameters of the feature extractor and
fine-tune the classifier in three ways: cross-entropy loss (CE), re-sampling
(RS) following the protocol of [6], and balanced classifier (LWS) [21]. Table
4 provides the top-1 accuracy of all combinations. It shows that our
representation learning approach enables all types of classifiers to reach
good performance, compared to other representation learning approaches. This
strongly suggests that our method learns: (1) good feature representations for
both head and tail classes, and (2) to separate them in a way that
discriminative classifiers can easily distinguish between classes.

| Classifier learning
---|---
Representation |  |  |
learning | CE | RS (cRT) | LWS [21]
CE | 38.5 | 41.2 | 41.4
RS | 34.9 | 37.6 | 36.3
DRO-LT (ours) | 41.2 | 47.3 | 46.8

Table 4: Top-1 accuracy of different representation learning manners and
classifier learning manners, on CIFAR100-LT. CE refers to Cross-Entropy, RS
refers resampling and LWS refers to balanced classifier according to [21]. The
results suggest that simple resampling methods achieve good results when the
learned features are good for both head and tail. Figure 4: Comparing train
and test error between a model trained with vanilla cross-entropy loss and a
model trained with our DRO-LT. The gap between the train error and test error
for tail classes is much lower in DRO-LT compared with vanilla, while
remaining the same for head classes.

Adaptive robustness:  Figure 2 shows the values of uncertainty per-class radii
($\varepsilon$) that were learned by training a ResNet-32 on CIFAR100-LT with
an imbalance ratio of 50. The model learned slightly larger radii for tail
classes compared with head classes. See the supplemental for details about the
effect of the radius on accuracy.

Robustness:  Our loss is expected to improve recognition mostly at tail
classes. Figure 4 compares train error and test error between a model trained
with cross-entropy loss (red) and a model trained with our approach (blue).
Using a robustness loss cuts down errors substantially, in tail classes,
without hurting head classes.

Feature space visualization:  To gain additional insight, we look at the t-SNE
projection of learned representations and compared vanilla cross-entropy loss
with our proposed method. Figure 5 shows that our learned feature space is
more compact with margins around head and tail classes. Tail classes have
larger margins since the estimation of their features is less accurate.

How unbalanced are latent representations?  The above analysis focused on
correcting the representation at the penultimate layer. But how biased are the
representations at lower layers? Intuitively, the low layers represent more
”physical” properties of inputs, while higher layers capture properties that
correspond more to semantic classes. One would expect that early layers would
be quite balanced.

We tested class imbalance in several layers of a ResNet-32 in the following
way. We first trained a ResNet-32 on the unbalanced CIFAR100-LT dataset. Then,
for each latent representation, we computed the centroids of each class and
used them to classify all samples using a nearest-centroid classifier. Figure
3 shows the accuracy obtained with this classifier when applied to layers 0,
10, 20, and 30 of the ResNet-32. When training with balanced data (up), the
accuracy grows when using higher layers, as expected. Surprisingly, when
training with unbalanced data (down), there is a substantial difference in
accuracy achieved for head and tail classes in every layer that we tested.
While accuracy grows when using higher layers, accuracy differences between
head and tail classes are maintained, even in low layers that are believed to
represent class agnostic features. In Appendix H, we compare the accuracy of a
nearest-centroid neighbor classifier between a model trained with standard
cross-entropy loss and one trained with DRO-LT. We show that DRO-LT narrows
the accuracy gap between head and tail.

Figure 5: t-SNE visualization of embedding space of CIFAR100-LT obtained using
cross-entropy loss and DRO-LT Loss method. The feature embedding of our model
is more compact for both head (blue) and tail (green and red) classes and
better separated. The number of samples for each class is written in
parentheses.

##  9 Discussion

This paper investigates the feature representations learned by deep models
trained on long-tail data. We find that such models suffer greatly from bias
towards head classes in their feature extractor (backbone), which hurts
recognition. This is in contrast to previous studies suggesting that
unbalanced data does not hurt representation learning and re-balancing the
classifier layer is sufficient. To learn a balanced representation, we take a
robustness approach and develop a novel loss based on Distributionally Robust
Optimization (DRO) theory. We further derive an upper bound of that loss that
can be minimized efficiently. We show how the robustness safety margin can be
learned during training, and do not require additional hyper-parameter tuning.

Training with a combination of the DRO-LT loss and a standard classifier sets
new state-of-the-art results on three long-tail benchmarks: CIFAR100-LT,
ImageNet-LT, and iNaturalist. Our method not only improves the performance of
tail classes but also maintains high accuracy at the head. These results
suggest that proper training of representations for unbalanced data can have a
large impact on downstream accuracy. We believe that our finding not only
contributes to a deeper understanding of the long-tailed recognition task but
can offer inspiration for future work.

Acknowledgments: This work was funded by the Israeli innovation authority
through the AVATAR consortium; by the Israel Science Foundation (ISF grant
737/2018); and by an equipment grant to GC and Bar Ilan University (ISF grant
2332/18).

## References

  * [1] Anurag Arnab, Ondrej Miksik, and Philip HS Torr.  On the robustness of semantic segmentation models to adversarial attacks.  In CVPR, 2018.
  * [2] S. Beery, Y. Liu, D. Morris, J. Piavis, A. Kapoor, M. Meister, and P. Perona.  Synthetic examples improve generalization for rare classes.  Preprint arXiv:1904.05916, 2019.
  * [3] Dimitris Bertsimas, Jack Dunn, Colin Pawlowski, and Ying Daisy Zhuo.  Robust classification.  INFORMS Journal on Optimization, 2019.
  * [4] Dimitris Bertsimas, Vishal Gupta, and Nathan Kallus.  Data-driven robust optimization.  Mathematical Programming, 2018.
  * [5] M. Buda, A. Maki, and M. Mazurowski.  A systematic study of the class imbalance problem in convolutional neural networks.  Neural Networks, 2018.
  * [6] K. Cao, C. Wei, A. Gaidon, N. Arechiga, and T. Ma.  Learning imbalanced datasets with label-distribution-aware margin loss.  NeurIPS, 2019.
  * [7] Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer.  Smote: synthetic minority over-sampling technique.  Journal of artificial intelligence research, 2002.
  * [8] Peng Chu, Xiao Bian, Shaopeng Liu, and Haibin Ling.  Feature space augmentation for long-tailed data.  arXiv preprint arXiv:2008.03673, 2020.
  * [9] T. Cover and J. Thomas.  Elements of information theory.  1991\.
  * [10] Y. Cui, M. Jia, T. Lin, Y. Song, and S. Belongie.  Class-balanced loss based on effective number of samples.  CVPR, 2019.
  * [11] J. Deng, W. Dong, R. Socher, L. Li, K. Li, and F. Li.  Imagenet: A large-scale hierarchical image database.  CVPR, 2009.
  * [12] Chris Drummond.  C 4 . 5 , class imbalance , and cost sensitivity : Why under-sampling beats oversampling.  2003\.
  * [13] Amir Globerson, Gal Chechik, Fernando CN Pereira, and Naftali Tishby.  Euclidean embedding of co-occurrence data.  In NIPS, 2004.
  * [14] Joel Goh and Melvyn Sim.  Distributionally robust optimization and its tractable approximations.  Operations research, 2010.
  * [15] Jacob Goldberger, Geoffrey E Hinton, Sam Roweis, and Russ R Salakhutdinov.  Neighbourhood components analysis.  Advances in neural information processing systems, 2004.
  * [16] Gaurav Goswami, Nalini Ratha, Akshay Agarwal, Richa Singh, and Mayank Vatsa.  Unravelling robustness of deep learning based face recognition against adversarial attacks.  In AAAI, 2018.
  * [17] H. Han, W. Wang, and B. Mao.  Borderline-smote: A new over-sampling method in imbalanced data sets learning.  In ICIC, 2005.
  * [18] H. He and E. A. Garcia.  Learning from imbalanced data.  IEEE Transactions on Knowledge and Data Engineering, 2009.
  * [19] G. Horn, O. Aodha, Y. Song, A. Shepard, H. Adam, P. Perona, and S. Belongie.  The inaturalist challenge 2017 dataset.  ArXiv, 2017.
  * [20] Xinting Hu, Yi Jiang, Kaihua Tang, Jingyuan Chen, Chunyan Miao, and Hanwang Zhang.  Learning to segment the tail.  In CVPR, 2020.
  * [21] B. Kang, S. Xie, M. Rohrbach, M. Yan, A. Gordo, J. Feng, and Y. Kalantidis.  Decoupling representation and classifier for long-tailed recognition.  ICLR, 2020.
  * [22] Salman Khan, Munawar Hayat, Syed Waqas Zamir, Jianbing Shen, and Ling Shao.  Striking the right balance with uncertainty.  In CVPR, 2019.
  * [23] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, A. Maschinot, Ce Liu, and Dilip Krishnan.  Supervised contrastive learning.  Nuerips, abs/2004.11362, 2020.
  * [24] Jaehyung Kim, Jongheon Jeong, and Jinwoo Shin.  M2m: Imbalanced classification via major-to-minor translation.  CVPR, 2020.
  * [25] Alex Krizhevsky.  Learning multiple layers of features from tiny images.  2009\.
  * [26] Brian Kulis et al.  Metric learning: A survey.  Foundations and trends in machine learning, 2012.
  * [27] T. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár.  Focal loss for dense object detection.  ICCV, 2017.
  * [28] Anqi Liu and Brian D. Ziebart.  Robust classification under sample selection bias.  In NIPS, 2014.
  * [29] Jialun Liu, Yifan Sun, Chuchu Han, Zhaopeng Dou, and Wenhui Li.  Deep representation learning on long-tailed data: A learnable embedding augmentation perspective.  In CVPR, 2020.
  * [30] Z. Liu, Z. Miao, X. Zhan, J. Wang, B. Gong, and S. Yu.  Large-scale long-tailed recognition in an open world.  In CVPR, 2019.
  * [31] Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu.  Large-scale long-tailed recognition in an open world.  In CVPR, 2019.
  * [32] Wanli Ouyang, X. Wang, Cong Zhang, and X. Yang.  Factors in finetuning deep model for object detection with long-tail distribution.  CVPR, 2016.
  * [33] H. Rahimian and S. Mehrotra.  Distributionally robust optimization: A review.  ArXiv, 2019.
  * [34] S. Ryou, S. Jeong, and P. Perona.  Anchor loss: Modulating loss scale based on prediction difficulty.  In ICCV, 2019.
  * [35] Dvir Samuel, Yuval Atzmon, and Gal Chechik.  From generalized zero-shot learning to long-tail with class descriptors.  In WACV, 2021.
  * [36] Soroosh Shafieezadeh-Abadeh, Peyman Mohajerin Esfahani, and D. Kuhn.  Distributionally robust logistic regression.  In NIPS, 2015.
  * [37] Jake Snell, Kevin Swersky, and Richard S Zemel.  Prototypical networks for few-shot learning.  NIPS, 2017.
  * [38] Laurens van der Maaten and Geoffrey Hinton.  Visualizing data using (t-sne).  Journal of Machine Learning Research, 2008.
  * [39] Vladimir Vapnik.  The nature of statistical learning theory.  Springer science & business media, 2013.
  * [40] Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella X Yu.  Long-tailed recognition by routing diverse distribution-aware experts.  arXiv preprint arXiv:2010.01809, 2020.
  * [41] Yiru Wang, Weihao Gan, Jie Yang, Wei Wu, and Junjie Yan.  Dynamic curriculum learning for imbalanced data classification.  In ICCV, 2019.
  * [42] Kilian Q Weinberger, John Blitzer, and Lawrence K Saul.  Distance metric learning for large margin nearest neighbor classification.  In Advances in neural information processing systems, 2006.
  * [43] Liuyu Xiang and G. Ding.  Learning from multiple experts: Self-paced knowledge distillation for long-tailed classification.  In ECCV, 2020.
  * [44] Huan Xu, Constantine Caramanis, and Shie Mannor.  A distributional interpretation of robust optimization.  Mathematics of Operations Research, 2012.
  * [45] Yuzhe Yang and Zhi Xu.  Rethinking the value of labels for improving class-imbalanced learning.  NeurIPS, 2020.
  * [46] Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen.  Bbn: Bilateral-branch network with cumulative learning for long-tailed visual recognition.  In CVPR, 2020.

Supplemental Material

##  Appendix A Additional information about $p(\epsilon)$

We provide a more formal definition of the probability $p(\epsilon)$. Consider
the empirical distribution $\widehat{p_{c}(z)}$ of $n$ samples
$z_{1},.\ldots,z_{n}$ from class $c$. This empirical distribution $\hat{p}$
can be viewed by itself as a random variable because it receives different
values for different instantiations of the random sample. $\hat{p}$ is
distributed over the simplex $\Delta^{d}$ where $d$ is the dimension of the
representation of $z$. As a result, the $\epsilon$-ball around
$\widehat{\mu_{c}}$ is also a random variable, since different random samples
yield different centroids and balls.

For any given $\epsilon$, some of these balls may cover the true distribution
$p$ and some may not, depending on the instances $z_{i}$ drawn. $p(\epsilon)$
denotes the probability that such an $\epsilon$-ball covers the true
distribution.

##  Appendix B Tightness of the bound

We derive a lower bound of our loss and evaluate empirically that the bounds
are tight.

The lower bound is easily derived in a very similar way to the derivation of
the upper bound in Theorem 1 using the triangle inequality. The bound has the
following form

$-\log
P(z|\mu_{c})\geq-\log\frac{e^{-d(\widehat{\mu_{c}},z)+2\varepsilon_{c}}}{\sum_{z^{\prime}\in
Z}e^{-d(\widehat{\mu_{c}},z^{\prime})+2\varepsilon_{c}\delta(z^{\prime},c)}}$
(20)

We further estimated empirically the relative magnitude of the bound gap
$\frac{|Eq.\ref{eq_upper}-Eq.\ref{eq_lower}|}{Eq.\ref{robust_loss}}$, averaged
over all samples. For CIFAR100-LT it was $0.07$, suggesting that the bounds
are quite tight in our case.

##  Appendix C DRO-LT Sample count $\epsilon/\sqrt{n}$

Given a set of samples $z_{i}$, their mean is known to have a standard
deviation of $\sigma/\sqrt{n}$, where $\sigma$ is the standard deviation of
the sample distribution $p(z)$. In our case, $\sigma$ is not known.

The DRO-LT variant that we call Sample count $\epsilon/\sqrt{n}$, can be
viewed as assuming that all classes share the same standard deviation of their
sample distribution $\sigma$, which we tune as a hyperparameter, and the
uncertainty about class centroid only varies by the number of samples.

##  Appendix D Loss trade-off parameter $\lambda$

Figure 6 quantifies the effect of the trade-off parameter $\lambda$ (Eq. 19)
on the validation accuracy. The model was trained on CIFAR100-LT with an
imbalance factor of 100 and with our DRO-LT loss (Learnable $\varepsilon$
variant). It shows that training with DRO-LT alone ($\lambda=0$) is not enough
and leads to poor accuracy. Combining the robustness loss with a
discriminative loss (cross-entropy) gives the best results and suggests high-
quality feature representations and a discriminative classifier.

##  Appendix E Robustness vs Performance

Figure 6:  Validation accuracy of a model trained on CIFAR100-LT (imbalance
factor 100) with different loss trade-off parameter ($\lambda$) values between
standard cross-entropy loss and our DRO-LT loss (Learnable $\varepsilon$
variant).  Figure 7:  Validation accuracy of a model trained on CIFAR100-LT
(imbalance factor 100) with different uncertainty radius $\varepsilon$. The
radius is set to be equal for all classes in our shared $\varepsilon$ variant.

Figure 7 explores the effect of different uncertainty radii ($\varepsilon$) on
the validation accuracy of a model trained on CIFAR100-LT with an imbalance
factor 100. The radius is set to be equal for all classes in our DRO-LT loss
(Shared $\varepsilon$ variant). It shows that the accuracy is maintained over
a large range of $\varepsilon$ values. Setting $\varepsilon$ to very small
values nullifies the robustness and reduces accuracy. At the same time, very
large values of $\varepsilon$ cause the worst-case centroid in the
$\epsilon$-ball to be too far from $\mu_{c}$ making the bound too loose and
again reduces the accuracy.

Figure 8: Accuracy of a nearest-centroid neighbor classifier when applied to
convolutional layers 0, 10, 20, and 30 of a ResNet-32. Our model narrows the
accuracy gap between head and tail classes. The model was trained on
CIFAR-100-LT with an imbalance factor of 100. We compare a model trained with
standard cross-entropy loss (solid line) and a model trained with DRO-LT
(dashed line) where the loss is applied to the last convolutional layer
(conv30). We report balanced validation accuracy for head classes (blue),
medium classes (orange), and tail classes (green).

##  Appendix F More analysis on iNaturalist:

iNaturalist | Many | Med | Few | Acc
---|---|---|---|---
CE* | 72.2 | 63.0 | 57.2 | 61.7
CB LWS | 71.0 | 69.8 | 68.8 | 69.5
DRO-LT (ours) |  |  |  |
Shared $\varepsilon$ | 78.2 | 70.6 | 64.7 |  69.0 $\pm$ 0.2
$\varepsilon/\sqrt{n}$ | 71.0 | 68.9 | 69.3 |  69.1 $\pm$ 0.2
Learned $\varepsilon$ | 73.9 | 70.6 | 68.9 | 69.7 $\pm$ 0.1

Table 5: Top-1 accuracy on long-tailed iNaturalist with accuracy broken to
Many-shot, Medium-shot and Few-shot classes. Our approaches improves the
performance on head and tail classes.

Table 5 compares DRO-LT with common long-tail methods on iNaturalist with
accuracy broken by class frequency: many-shot (”Many”), medium-shot (”Med”)
and few-show (”Few”). It shows that improvement is larger at the head (Many)
and tail (Few), but relatively small for most of the classes in this dataset
(Med). We also provide the variance for 10 runs.

##  Appendix G Results for CIFAR-10-LT:

Table 6 compares our approach with common long-tail methods on CIFAR-10-LT
[6]. Our method outperforms all baselines.

##  Appendix H Balancing latent representations

Here, we provide more analysis on the imbalance of latent representation and
compare our approach with a standard baseline.

Figure 8 shows the accuracy obtained with a nearest-centroid classifier when
applied to layers 0, 10, 20, and 30 of a ResNet-32. We compare a model trained
with standard cross-entropy loss (solid line) and a model trained with DRO-LT
(dashed line) where the loss is applied to the last convolutional layer
(conv30). We show that our model narrows the accuracy gap between head classes
(blue) and tail classes (green), mostly in the last layer. This shows the
effectiveness of our approach on the latent representations of the deep models
and suggests that applying our loss to the rest of the layers might result in
a more balanced model overall.

CIFAR-10-LT | Acc
---|---
CE | 72.1
CB LWS [21] | 73.5
LDAM DRW [6] | 77.03
smDragon [35] | 79.6
Learned $\varepsilon$ (ours) | 82.6

Table 6: Top-1 accuracy on long-tailed CIFAR-10 [6] with imbalance factor 100.


11institutetext: ISIA Lab (UMONS)
11email: {omar.seddati, stéphane.dupont, saïd.mahmoudi,
<EMAIL_ADDRESS>
# Transformers and CNNs both Beat Humans on SBIR

Omar Seddati  Stéphane Dupont  Saïd Mahmoudi  Thierry Dutoit

###### Abstract

Sketch-based image retrieval (SBIR) is the task of retrieving natural images
(photos) that match the semantics and the spatial configuration of hand-drawn
sketch queries. The universality of sketches extends the scope of possible
applications and increases the demand for efficient SBIR solutions. In this
paper, we study classic triplet based SBIR solutions and show that a
persistent invariance to horizontal flip (even after model finetuning) is
harming performance. To overcome this limitation, we propose several
approaches and evaluate in depth each of them to check their effectiveness.
Our main contributions are twofold: We propose and evaluate several intuitive
modifications to build SBIR solutions with better flip equivariance. We show
that vision transformers are more suited for the SBIR task, and that they
outperform CNNs with a large margin. We carried out numerous experiments and
introduce the first models to outperform human performance on a large scale
SBIR benchmark (Sketchy). Our best model achieves a recall of 62.25% (at
$k=1$) on the sketchy benchmark compared to previous state-of-the-art methods
46.2%.

###### Keywords:

Sketch-based image retrieval, Triplet Networks, Vision Transformers

##  1 Introduction

The possible applications for SBIR solutions are multiple and especially with
the large availability of touchscreens everywhere. In addition, the
universality of sketches and the efficiency of modern SBIR solutions that are
able to deal with badly drawn sketches make the use of such solutions less
burdensome. For example, a SBIR solution may be integrated in an e-commerce
system, where the user draws a sketch to find a specific product. Sketching
will offer a powerful solution to communicate details that goes beyond the
category of the object. These details could go from the global product design
to detailed patterns present on the product and with respect to their actual
spatial configuration. It would be difficult to reach such level of details
using a query-by-text paradigm.

In order to build SBIR solutions, researchers have conducted extensive studies
and proposed several solutions to manage the abstract nature of sketches and
extract cross-domain features. In this work, we use the classic triplet
training pipeline to train our models on the Sketchy database. Then, we
conduct a qualitative inspection of the model’s errors and show that the model
struggles with horizontal flipping. In multiple cases, when the photo matching
the sketch query comes at the second position, the retrieved photos holding
first, and second place share the same semantics and even spatial
configuration if mirrored. Which indicates that the models keep preserving in
somehow flip invariance even after finetuning. Despite the value of such
property for classification problems, in the particular case of SBIR where we
seek for a fine-grained matching, such property represents a major weakness
because object pose is not represented well enough. To overcome this
limitation, we propose several methods (custom mini-batch sampling, adapting
pooling layer, using a vision transformer) and conduct for each an additional
finetuning to assess if it is beneficial. During the whole study process, we
use intermediate assumptions and experiments to draw conclusions and build
step-by-step our final end-to-end enhanced SBIR solution. Through this study,
we show that a simple modification to a triplet CNN based SBIR solution at the
last pooling layer is enough to bring a significant improvement. In addition,
we also show that a vision transformer does not share the limitation observed
with CNNs, which makes it more suited for SBIR.

The rest of the paper is organized as follows: we start with a brief review of
different approaches used in the field of SBIR. Next, we present the method
that we selected as a baseline, and we proceed to the study and the evaluation
of the different approaches that propose to enhance existing solutions.

##  2 Related Work

In computer vision, it has been a few years since supervised learning with CNN
achieves state-of-the-art results [12, 31, 9]. CNNs are known for their
ability to learn to extract relevant features directly from pixels without
requiring classic feature extraction approaches. This feature makes them a
convenient and powerful tool for multiple computer vision tasks. In addition,
compared to hand crafted features (shallow features), CNN features reach
higher performance [7, 33, 28] when it comes to generating representations for
tasks like content-based image retrieval (CBIR). Sketch recognition and SBIR
have also benefited from the power of CNN and outperformed previous solutions
based on hand crafted features (e.g. [13, 17, 3, 14, 36]). In several CNN
based works [18, 30, 19, 25], the authors use a CNN trained for sketch
recognition to extract features and build a sketch-edge map matching solution
(under the assumption that photos edge maps are visually closer to sketches).
In [18], Qi et al. used a Siamese CNN architecture, for category-level SBIR
(which means that the result is considered as correct if the retrieved photo
belongs to the same category as the sketch query). In such architecture, two
branches are used, one for the sketches and the other for the edge maps.
During training, sketch-edge map pairs are passed to the model and a binary
label indicates if the sketch and the edge map share the same category. Then
the loss is computed and the model parameters are updated to extract better
representations in the future. In a more general fashion, pair losses are used
with the objective to learn to project inputs in a feature space where the
distance between positive pairs (similar inputs) is smaller than a margin
$m_{p}$, while the distance for negative pairs is larger than a second margin
$m_{n}$. Using the same margin for all pairs constitutes a major disadvantage
since it does not consider the variance of (dis)similarity between different
pairs. This limitation is avoided when a Triplet CNN is used. A similar
approach where inputs are presented as triplets. In each triplet we have a
reference sample, a positive sample (similar to the reference) and a negative
sample (dissimilar to the reference). During training, the model learns to
project inputs into a space where a positive example is closer to the
reference than a negative one. This approach based on relative distance
measure gives the model the ability to manage arbitrary feature space
distortions. This property makes Triplet CNN more suited for CBIR/SBIR
applications and attracted a lot of attention [2, 15, 38, 34, 29, 5, 4, 27] in
the last recent years. In [2], Bui et al. made several experiments with
triplet CNN to explore weight sharing strategies and analyze its ability to
generalize. In [38], the authors added a deformable CNN layer (changeable
receptive field) to deal with sketch changes. In [15], in addition to the
triplet loss, the authors experimented with a combination three loss functions
(SoftMax loss, Spherical loss, and Center loss). In [29, 5, 4, 27], the
authors added attention modules to enhance fine information capture. In [1],
Bhunia et al. addressed the problem related to the lack of annotated sketches.
They proposed an approach where a photo-to-sketch generator (a sort of GAN) is
used to automatically generate sketches for unlabeled photos. The generated
synthetic sketch-photo pairs are then used as additional data to train a
triplet CNN. In [26], the authors introduced the quadruplet networks, a new
architecture for SBIR. It uses an additional branch compared to a triplet
architecture to ensure that for each reference, there are two negatives. The
first one belongs to the same object category, while the other is from a
different category. The goal of such architectures is to encode semantic
information in a similar fashion as what triplet does for local information.
In [34], Wang et al. proposed a sophisticated SBIR solution where textual
descriptions (collected from MSCOCO and Flickr30k databases) are used as
additional input to the pipeline to reduce the gap between sketches and
images. They propose a three-stage solution, where a deep multimodal embedding
model is used to extract features for the three modalities. Then, a candidate
selection based on the extracted embeddings is applied. In the final stage, a
triplet is used for ranking optimization. In [23], a cross-modal variational
autoencoder is proposed to disentangle sketch into semantic content (an
information shared with the corresponding photo) and sketcher style to build a
style-agnostic model.

All these works have brought various adaptations and new solutions for SBIR.
In this work, we take a closer look to the errors that occurs when using a
triplet CNN approach and how we can overcome its limitations.

##  3 Methodology and Experiments

In order to build an efficient SBIR system, we need a solution able to analyze
in depth both sketches and images to generate representations that seize the
semantics and spatial configurations. Before we proceed to the core of this
study, let introduce how a triplet network works and what makes it suitable
for the SBIR task.

###  3.1 Triplet Network

In this kind of architecture, we generally have three copies of the same
network with shared parameters. The input is in the form of Anchor, Positive
and Negative. In the context of SBIR, if we use photos as anchors, the goal
will be minimizing the distance between an anchor (a photo) and a positive (a
sketch corresponding to the anchor), while maximizing the distance between
this anchor and a negative (a sketch not corresponding to the anchor). In
order to do that, we train our triplet network with the following loss
function:

$\sum_{i}^{N}[\left\|f(x_{i}^{a})-f(x_{i}^{p})\right\|_{2}^{2}-\left\|f(x_{i}^{a})-f(x_{i}^{n})\right\|_{2}^{2}+\alpha]_{+}$
(1)

where :
$x_{i}^{a}$: the feature vector of the anchor photo.
$x_{i}^{p}$: the feature vector of the positive sketch.
$x_{i}^{n}$: the feature vector of the negative sketch.
$\alpha$: by using this margin we try to ensure a minimal margin between the
negative pair and positive pair distances.

###  3.2 The Sketchy Dataset

Figure 1: Samples from the sketchy database [13].

For the whole study, we will use the Sketchy benchmark [24]. It is a large-
scale collection for SBIR. It contains 125 categories and 75,471 sketches of
12,500 objects (some samples are shown in Figure 1). The crowd workers are
asked to sketch particular photographic objects in order to give a fine-
grained association between sketches and photos. For each category we have 100
images and at least 5 sketches per image. We followed the guidelines of the
authors [24] and used the provided testset list to split data into training
and testset. 90% of the data are used for the training and the rest are used
for test.

###  3.3 Baseline Models and Practical Details

We began our study by training triplet CNNs with three different architectures
[10] (ResNet18, ResNet34, ResNet50) on the Sketchy dataset. We use the pre-
trained (pre-trained on ImageNet [6]) versions provided by the torchvision
library. We select the last pooling layer (adaptive average pooling layer) to
extract features (in this work, we do not normalize the extracted embeddings
and we use the Euclidean distance), and we keep the classification branch (we
change the number of neurons of the fc layer to correspond to the 125
categories of Sketchy) to build a multi-task triplet network. During training,
we use two different models, one for sketches and the other for photos. We
train each architecture for 100 epochs using the Adam optimizer [11] and we
set the initial learning rate to $lr=10^{-4}$. The learning rate is changed to
$lr=10^{-5}$ after 30 epochs. The batch-size (bs) and the triplet margin (m)
remain the same for the whole study ($bs=128$ and $m=3$). The best baseline
models’ performances are reported on the first line of Table 1. We can notice
that replacing a ResNet18 with a ResNet34 improves significantly the results.
But the opposite happens when replacing a ResNet34 with a ResNet50. This may
come from the fact that in ResNet50 the basic blocks are replaced with
bottleneck blocks to reduce memory usage. In these blocks, pointwise
convolution is used to reduce the number of channels before $3\times 3$
convolutions. And the loss of performance might be a side effect of this form
of compression.

Table 1: The top results ($recall@1$) achieved on the Sketchy benchmark using different architectures and methods. Method | ResNet18 | ResNet34 | ResNet50
---|---|---|---
Baseline | 52.98 | 56.10 | 54.89
Finetuning: Continue | 52.98 | 56.10 | 55.08
Finetuning: Flip sampling | 53.78 | 57.19 | 55.91
Finetuning: Category sampling | 53.83 | 56.92 | 56.24
Finetuning: Flip + Category sampling | 53.61 | 57.43 | 56.29
Finetuning: Pooling $2\times 2$ | 55.10 | 58.23 | 58.37

###  3.4 Qualitative Inspection

Figure 2: Some examples showing that ResNet18 baseline still struggle in some
cases with horizontal flip.

When using the Sketchy benchmark, the recall metric, and especially the
$recall@1$ is used to evaluate and compare different solutions. Since our
models have reached an exceptionally high level of performance (human
performance $recall@1$ is 54.27% as reported in [24]) and in order to see if
there is much room for improvement, we did also compute the $recall@2$ (the
results are reported in Table 2). The fact that the chances to retrieve the
wanted photo increase significantly when we retrieve more than one image,
motivated us to look closer to the errors made by the models, hoping to find a
way to push $recall@1$ to get closer to $recall@2$. During this qualitative
inspection, we noticed that in many cases, both, the first and second
retrieved images exhibit an obvious similarity, but the models seem to
struggle with the orientation as shown in figure 2. Which indicates that the
models keep preserving in somehow flip invariance even after finetuning.
Despite the value of such property for classification problems, in the
particular case of SBIR where we seek for a fine-grained matching, such
property represent a major weakness because object pose is not represented
well enough. In order to overcome this limitation, in the next subsections, we
explore several approaches and show which of them bring a significant
improvement.

In addition to the flip invariance problem, we have also noticed during the
aforementioned qualitative inspection that in some cases (in $18.5\%$ of the
cases for the ResNet18 baseline model) the category of the image retrieved at
the first position does not match the query’s category. This is another aspect
of the model that we thought that we should be able to improve through a
better mini-batch sampling during the training phase. More details about the
approach that we propose will be presented in the subsection 3.6.

Table 2: The top results ($recall@2$) achieved on the Sketchy benchmark using different architectures and methods. Method | ResNet18 | ResNet34 | ResNet50
---|---|---|---
Baseline | 69.24 | 73.75 | 71.88

###  3.5 Flip Sampling

Before moving forward to the evaluation of the different approaches we have
considered in this work, we have begun with finetuning the baseline models for
an additional 100 epochs on the same task. For each architecture (e.g.,
ResNet18), we reload the model achieving the best performance, we set the
learning rate to $lr=10^{-6}$ and we continue the training following the same
procedure. This provides us a fair ground for future comparisons and
improvements analysis. The results are shown in the second line 111If the
value at the second line is the same as the corresponding one from the first
line, this means that the additional 100 epochs finetuning did not brought any
improvement. of Table 1.

Flipping the inputs, simultaneously the sketch and the photo, is the only data
augmentation technique used during the classic training procedure adopted in
this work. For the flip invariance problem, the first intuitive approach that
we came out with was the adaptation of the mini-batch sampler. We modify the
latter to include both, the flipped and non-flipped versions of some samples
in the same mini-batch. With this modification, we expect the model to focus
beyond the global semantics and pay more attention to the spatial
configuration systematically within each mini-batch. As reported in the third
line of Table 1, we can notice that the proposed approach brings a substantial
improvement.

###  3.6 Category Sampling

Like the solution described in the previous subsection, here we modify the
mini-batch sampler to make sure that inside the same mini-batch, multiple
samples with the same category appears a random number of times (a random
number is selected between 2 and 5). The aim of this modification is to build
harder training samples (it is defensible to think that it is harder to
differentiate two objects from the same category compared to objects from
different categories) and strengthen the model’s ability to manage intra-
categorical information. In order to evaluate the improvement brought with our
approach, we compare the evolution of the number of times that the first
retrieved photo is from a category not matching the one of the query (while
the second photo is the wanted image). In Table 3 we report the percentage
with which this number drops for each architecture when using the proposed
approach. For example, if the number of errors for the baseline model is 100,
and this number drops to 90 after using category sampling, the reported
percentage will be 10%.

Table 3: The percentage of improvement noticed when using our category sampling approach. Method | ResNet18 | ResNet34 | ResNet50
---|---|---|---
Baseline | 8.95 | 6.47 | 5.49

###  3.7 Task Specific Pooling

Here, we present an approach that differs fundamentally from the previous
ones. We propose to introduce a tiny modification to the model’s architecture
in order to extract richer embeddings preserving more information about
spatial configuration. We keep using the output of the original average
pooling layer for the classification branch of our models. And we add an
additional $2\times 2$ adaptive average pooling for the embedding branch to
replace the previous one. The motivation behind such modification, is that
with the original $1\times 1$ average pooling, the only option left to the
model to encode spatial information is through channels. So, we continue to
use it for the classification branch, since we think that it is enough for
this task. But for the other branch, we pass the outputs feature maps of the
last convolutional layer through a new $2\times 2$ adaptive average pooling
(which produces 4 times longer vectors than before) to form our new
embeddings. The rest of the training follows the same procedure. In the last
line of Table 1, we show how this tiny modification brings a significant
improvement to the different architectures compared to the baseline models.

###  3.8 Vision Transformer

Since a few years now, CNNs have been singled out for their lack of relative
position encoding for different features because of the use of pooling layers
[22]. Vision Transformers (VT) on the other hand, are known for their ability
to track long-range dependencies within an image. Thanks to self-attention
layers, VT models are able to efficiently aggregate global information. This
property drew our attention to VT models and their usage for the SBIR task. We
have looked for other works on the field of SBIR using VT models, and we were
surprised to find out that only three works [8, 21, 32] have used VT. In
addition, in two of them [8, 32], the proposed solutions are for Zero-shot
SBIR. While in the other [21], no one of the common SBIR benchmarks is used to
evaluate the performance. Therefore, we have followed the same approach as
previously, we start with training a baseline VT model. To this end, we use
the Hugging face library [35] and the pre-trained VT model under the name
“openai/clip-vit-base-patch32” based on the work of Radford et al. [20]. The
initial learning rate we used to train the baseline VT model is $lr=10^{-5}$.
We have also finetuned the baseline VT model for an additional 100 epochs
(with $lr=10^{-6}$) for each of the proposed approaches (except for the
pooling-based one). The outcomes of the different experiments with the VT
model are listed in Table 4 (side by side with those of ResNet for ease of
comparison). We can see that the VT model outperforms by a large margin
ResNets even when the pooling approach is used. We can also notice that in the
case of VT models, the proposed approaches do not bring a significant
improvement 222In Table 4, we show that finetuning the VT model just with
setting the learning rate to $lr=10^{-6}$ is enough to reach a $recall@1$ of
61.74%. Adding the approaches that we proposed and validated previously for
ResNets, does not seem to provide additional benefit.. Which may suggest that
as expected, VT models do not share the same limitations as ResNets and makes
them more suited for the SBIR task. In Table 5, we compare our best models
results with others from different papers and show that this study enabled us
to outperform existing approaches with a huge margin, and even exceed for the
first-time human performance on a large-scale SBIR benchmark.

Table 4: The top results ($recall@1$) achieved on the Sketchy benchmark using different architectures, including the VT model. Method | ResNet18 | ResNet34 | ResNet50 | VT
---|---|---|---|---
Baseline | 52.98 | 56.10 | 54.89 | 56.24
Finetuning: Continue | 52.98 | 56.10 | 55.08 | 61.78
Finetuning: Flip sampling | 53.78 | 57.19 | 55.91 | 60.56
Finetuning: Category sampling | 53.83 | 56.92 | 56.24 | 62.08
Finetuning: Flip + Category sampling | 53.61 | 57.43 | 56.29 | 62.25
Finetuning: Pooling $2\times 2$ | 55.10 | 58.23 | 58.37 | -
Table 5: Comparison with some SBIR solutions on Sketchy [24]. Methods | Recall@1
---|---
Chance [24] | 0.01%
Sketch me that shoe [37] | 25.87%
Siamese Network [24] | 27.36%
Triplet Network [24] | 37.10%
Quadruplet_MT [26] | 38.21%
DCCRM(S+I) [34] | 40.16%
DeepTCNet [16] | 40.81%
Triplet attention [27] | 41.66%
Quadruplet_MT_v2 [26] | 42.16%
DCCRM(S+I+D) [34] | 46.20%
Human [24] | 54.27%
Ours (ResNet18 3.7) | 55.10%
Ours (ResNet34 3.7) | 58.23%
Ours (ResNet50 3.7) | 58.37%
Ours (VT 3.8) | 62.25%

##  4 Conclusions

In this work, we have selected a classic pipeline for building SBIR solutions.
We started with training multiple architectures in a classic fashion to serve
as a baseline for future comparisons. Then, we have conducted a qualitative
inspection of the errors made by the model on the testset. Based on the
observations, we identified a few limitations that we can address in an effort
to improve existing solutions. To this aim, we proposed and motivated several
approaches that we explored and evaluated on a large-scale SBIR benchmark. We
have been able to show that the different approaches proposed implying the use
of adapted mini batch sampling bring a substantial improvement. Furthermore,
we demonstrate that with a tiny modification to the architecture at the
pooling layer results on significant improvements. In addition, we did also
show that VT models represent better candidates than CNNs for tasks with SBIR
needs and how they surpassed remarkably the latter, and even human performance
on the Sketchy benchmark.

## References

  * [1] Bhunia, A.K., Chowdhury, P.N., Sain, A., Yang, Y., Xiang, T., Song, Y.Z.: More photos are all you need: Semi-supervised learning for fine-grained sketch based image retrieval. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4247–4256 (2021)
  * [2] Bui, T., Ribeiro, L., Ponti, M., Collomosse, J.: Generalisation and sharing in triplet convnets for sketch based visual search. arXiv preprint arXiv:1611.05301 (2016)
  * [3] Cao, X., Zhang, H., Liu, S., Guo, X., Lin, L.: Sym-fish: A symmetry-aware flip invariant sketch histogram shape descriptor. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 313–320 (2013)
  * [4] Chaudhuri, U., Banerjee, B., Bhattacharya, A., Datcu, M.: Crossatnet-a novel cross-attention based framework for sketch-based image retrieval. Image and Vision Computing 104, 104003 (2020)
  * [5] Chen, Y., Zhang, Z., Wang, Y., Zhang, Y., Feng, R., Zhang, T., Fan, W.: Ae-net: Fine-grained sketch-based image retrieval via attention-enhanced network. Pattern Recognition 122, 108291 (2022)
  * [6] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 248–255. Ieee (2009)
  * [7] Gordo, A., Almazan, J., Revaud, J., Larlus, D.: End-to-end learning of deep visual representations for image retrieval. International Journal of Computer Vision 124(2), 237–254 (2017)
  * [8] Gupta, S., Chaudhuri, U., Banerjee, B.: Zero-shot sketch based image retrieval using graph transformer. arXiv preprint arXiv:2201.10185 (2022)
  * [9] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 770–778 (2016)
  * [10] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 770–778 (2016)
  * [11] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  * [12] Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems. pp. 1097–1105 (2012)
  * [13] LaViola Jr, J.J., Zeleznik, R.C.: Mathpad 2: a system for the creation and exploration of mathematical sketches. In: ACM SIGGRAPH 2007 courses. p. 46. ACM (2007)
  * [14] Li, Y., Song, Y.Z., Gong, S.: Sketch recognition by ensemble matching of structured features. In: BMVC. vol. 1, p. 2 (2013)
  * [15] Lin, H., Fu, Y., Lu, P., Gong, S., Xue, X., Jiang, Y.G.: Tc-net for isbir: Triplet classification network for instance-level sketch based image retrieval. In: Proceedings of the 27th ACM International Conference on Multimedia. pp. 1676–1684 (2019)
  * [16] Lu, P., Lin, H., Fu, Y., Gong, S., Jiang, Y.G., Xue, X.: Instance-level sketch-based retrieval by deep triplet classification siamese network. arXiv preprint arXiv:1811.11375 (2018)
  * [17] Ouyang, T.Y., Davis, R.: Chemink: a natural real-time recognition system for chemical drawings. In: Proceedings of the 16th international conference on Intelligent user interfaces. pp. 267–276. ACM (2011)
  * [18] Qi, Y., Song, Y.Z., Zhang, H., Liu, J.: Sketch-based image retrieval via siamese convolutional neural network. In: 2016 IEEE International Conference on Image Processing (ICIP). pp. 2460–2464. IEEE (2016)
  * [19] Radenovic, F., Tolias, G., Chum, O.: Deep shape matching. In: Proceedings of the european conference on computer vision (eccv). pp. 751–767 (2018)
  * [20] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning. pp. 8748–8763. PMLR (2021)
  * [21] Ribeiro, L.S.F., Bui, T., Collomosse, J., Ponti, M.: Sketchformer: Transformer-based representation for sketched structure. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 14153–14162 (2020)
  * [22] Sabour, S., Frosst, N., Hinton, G.E.: Dynamic routing between capsules. Advances in neural information processing systems 30 (2017)
  * [23] Sain, A., Bhunia, A.K., Yang, Y., Xiang, T., Song, Y.Z.: Stylemeup: Towards style-agnostic sketch-based image retrieval. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8504–8513 (2021)
  * [24] Sangkloy, P., Burnell, N., Ham, C., Hays, J.: The sketchy database: learning to retrieve badly drawn bunnies. ACM Transactions on Graphics (TOG) 35(4), 119 (2016)
  * [25] Seddati, O., Dupont, S., Mahmoudi, S.: Deepsketch2image: deep convolutional neural networks for partial sketch recognition and image retrieval. In: Proceedings of the 24th ACM international conference on Multimedia. pp. 739–741 (2016)
  * [26] Seddati, O., Dupont, S., Mahmoudi, S.: Quadruplet networks for sketch-based image retrieval. In: Proceedings of the 2017 ACM on International Conference on Multimedia Retrieval. pp. 184–191 (2017)
  * [27] Seddati, O., Dupont, S., Mahmoudi, S.: Triplet networks feature masking for sketch-based image retrieval. In: International Conference Image Analysis and Recognition. pp. 296–303. Springer (2017)
  * [28] Seddati, O., Dupont, S., Mahmoudi, S., Parian, M.: Towards good practices for image retrieval based on cnn features. In: Proceedings of the IEEE international conference on computer vision workshops. pp. 1246–1255 (2017)
  * [29] Song, J., Yu, Q., Song, Y.Z., Xiang, T., Hospedales, T.M.: Deep spatial-semantic attention for fine-grained sketch-based image retrieval. In: Proceedings of the IEEE international conference on computer vision. pp. 5551–5560 (2017)
  * [30] Song, Y., Lei, J., Peng, B., Zheng, K., Yang, B., Jia, Y.: Edge-guided cross-domain learning with shape regression for sketch-based image retrieval. IEEE Access 7, 32393–32399 (2019)
  * [31] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 1–9 (2015)
  * [32] Tian, J., Xu, X., Shen, F., Yang, Y., Shen, H.T.: Tvt: Three-way vision transformer through multi-modal hypersphere learning for zero-shot sketch-based image retrieval (2022)
  * [33] Tolias, G., Sicre, R., Jégou, H.: Particular object retrieval with integral max-pooling of cnn activations. arXiv preprint arXiv:1511.05879 (2015)
  * [34] Wang, Y., Huang, F., Zhang, Y., Feng, R., Zhang, T., Fan, W.: Deep cascaded cross-modal correlation learning for fine-grained sketch-based image retrieval. Pattern Recognition 100, 107148 (2020)
  * [35] Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Scao, T.L., Gugger, S., Drame, M., Lhoest, Q., Rush, A.M.: Transformers: State-of-the-art natural language processing. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. pp. 38–45. Association for Computational Linguistics, Online (Oct 2020), https://www.aclweb.org/anthology/2020.emnlp-demos.6
  * [36] Yesilbek, K.T., Sen, C., Cakmak, S., Sezgin, T.M.: Svm-based sketch recognition: which hyperparameter interval to try? In: Proceedings of the workshop on Sketch-Based Interfaces and Modeling. pp. 117–121. Eurographics Association (2015)
  * [37] Yu, Q., Liu, F., Song, Y.Z., Xiang, T., Hospedales, T.M., Loy, C.C.: Sketch me that shoe. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 799–807 (2016)
  * [38] Zhang, X., Shen, M., Li, X., Feng, F.: A deformable cnn-based triplet model for fine-grained sketch-based image retrieval. Pattern Recognition p. 108508 (2021)