# Deep Bregman Divergence for
Contrastive Learning of Visual Representations

Mina Rezaei1, Farzin Soleymani2*, Bernd Bischl1, Shekoofeh Azizi3 Equal
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Deep Bregman divergence measures divergence of data points using neural
networks which is beyond Euclidean distance and capable of capturing
divergence over distributions. In this paper, we propose deep Bregman
divergences for contrastive learning of visual representation and we aim to
enhance contrastive loss used in self-supervised learning by training
additional networks based on functional Bregman divergence. In contrast to the
conventional contrastive learning methods which are solely based on
divergences between single points, our framework can capture the divergence
between distributions which improves the quality of learned representation. By
combining conventional contrastive loss with the proposed divergence loss, our
method outperforms baseline and most of previous methods for self-supervised
and semi-supervised learning on multiple classifications and object detection
tasks and datasets. The source code of the method and of all the experiments
are available at supplementary.

## Introduction

Metric learning aims to construct task-specific distance from supervised data,
where learned distance metrics can be used to perform various tasks such as
classification, clustering, and information retrieval. We can divide the
extension of conventional metric learning or Mahalanobis metric learning
algorithms into two directions: those models based on deep metric learning and
those based on Bregman divergence learning. Deep metric learning uses neural
networks to automatically learn discriminative features from samples and then
compute the metric such as contrastive loss in Siamese network (Koch, Zemel,
and Salakhutdinov 2015) or triplet loss in the triplet network (Hoffer and
Ailon 2015). Bregman divergences learning aims to generalize measures such as
Euclidean distance (Bregman 1967) and KL divergences (Painsky and Wornell
2019). These methods learn the underlying generating function of the Bregman
divergence using piecewise linear approximation (Siahkamari et al. 2020), or
by adding quantization rates to the existing basis of functional Bregman for
clustering (Liu and Belkin 2016).

Recently, deep divergence learning (Cilingir, Manzelli, and Kulis 2020;
Kampffmeyer et al. 2019) was introduced to learn and parameterize functional
Bregman divergence using linear neural networks. This method measures
divergences of data by approximating functional Bregman divergences by
training deep neural networks where it aims to reduce the distance between
feature vectors corresponding to the same class and increase the distance
between the feature vectors corresponding to different classes A key advantage
of this method is to shift the divergence learning focus from learning
divergence between single points to capturing the divergence between the
distribution of a vector of points (Cilingir, Manzelli, and Kulis 2020). This
can potentially introduce new desirable properties to a learned feature space.
Nevertheless, learning Bregman divergence has not yet seen such widespread
adoption and it remains a challenging endeavor for the representation learning
and self-supervision based on distance reduction (i.e. similarity
maximization).

Figure 1: Illustration of our proposed framework. First two augmented samples
are generated by the augmentation function $t$ and $t^{\prime}$. The augmented
samples are transformed into vectors with the base encoder $f_{\theta}$. Next,
they are projected onto a lower-dimensional space $z$. Then, $\kappa$
different subnetworks map the vector $z$ into arbitrary Bregman divergence. In
these configurations, the underlying Bregman divergence is parameterized by a
convex set of linear functions. Finally, contrastive divergence loss measures
the underlying Bregman divergence of distribution for current mini-batch
samples.

Self-supervised learning is amongst the most promising approaches for learning
from limited labeled data. In contrast to supervised methods, these techniques
learn a representation of the data without relying on human annotation. Early
self-supervised methods focused on solving a pretext task such as predicting
the rotation of images, the relative position of patches, and colorization
(Lee et al. 2017; Noroozi and Favaro 2016; Pathak et al. 2016; Zhang, Isola,
and Efros 2016). Most recent successful self-supervised methods use
contrastive loss and maximize the similarity of representation obtained from
different distorted versions of a sample. Instance discrimination (Wu et al.
2018), CPC (Hénaff et al. 2019; Oord, Li, and Vinyals 2018), Deep InfoMax
(Hjelm et al. 2019), AMDIM (Bachman, Hjelm, and Buchwalter 2019), CMC (Stojnic
and Risojevic 2021), PIRL (Misra and Maaten 2020), MoCo (He et al. 2020), and
SimCLR (Chen et al. 2020a, b) are examples of such contrastive learning
methods that produce representations that are competitive with supervised
ones. However, existing deep contrastive learning methods are not directly
amenable to compare the data distributions.

In this paper, we extend existing contrastive learning approaches in a more
profound way using the _contrastive divergence learning_ strategy which learns
a generalized divergence of the data distribution by jointly adopting a
functional Bregman divergence along with a contrastive learning strategy. In
summary, we train our framework end-to-end in four stages: (1) Transformation:
we sequentially apply random color distortions, random rotations, random
cropping followed by resizing back to the original size, and random Gaussian
blur. Then, we train our network with two distorted images only. (2) Base
network: learns representations on the top of distorted samples using deep
neural networks. These representations are projected onto a lower-dimensional
space by the projection head. We call the output of the base network the
representations and the output of the projector the embeddings. (3) Bregman
divergence network: the $\kappa$ different deep linear neural subnetworks
create the convex generating function and are able to compute divergence
between two distributions given embeddings. (4) Contrastive divergence
learning: Our network is trained end-to-end by combining a traditional
contrastive loss and a new divergence loss. As depicted in Fig. 1, the
contrastive loss computed over embeddings and our novel divergence loss
formulated over the output of subnetworks. Our main findings and contributions
can be summarized as follows:

  * •

We propose a novel framework for self-supervised learning of Bregman
divergence of visual representations. The proposed method learns deep Bregman
divergences which are beyond Euclidean distance and capable of capturing
divergence over distributions while also benefiting from contrastive learning
representation.

  * •

We propose a contrastive divergence loss which encourages each subnetwork to
focus on different attributes of an input. The combination of new contrastive
divergence loss with recent modified contrastive loss (i.e. NT-Xent, NT-
Logistic) improves the performance of contrastive learning for multiple tasks
including image classification and object detection over multiple baselines.

  * •

We show empirical results that highlight the benefits of learning
representation from our deep Bregman divergence and contrastive divergence
loss.

## Related Work

#### Divergence learning

Our method is comparable with methods involving measuring or learning
divergence over distributions such as contrastive divergence (Carreira-
Perpinan and Hinton 2005), stochastic Bregman divergence (Dragomir, Even, and
Hendrikx 2021), maximum mean discrepancy (Xing et al. 2002), and most recently
deep divergence learning (Cilingir, Manzelli, and Kulis 2020; Kampffmeyer et
al. 2019; Kong et al. 2020; Kato and Teshima 2021). Another related work is
Rankmax (Kong et al. 2020) which studies an adaptive projection alternative to
the softmax function that is based on a projection on the $(n,k)-$simplex with
application in multi-class classification. However, our proposed framework
differs in representation learning strategy and applications.

#### Self-supervised learning

Much of the early works on self-supervised learning focused on the problem of
learning embeddings without labels such that the linear classifier operating
on the learned embeddings from self-supervision could achieve high
classification accuracy (Doersch, Gupta, and Efros 2015). Later, some models
aim to learn the representation using auxiliary handcrafted prediction tasks.
Examples are image jigsaw puzzle (Noroozi and Favaro 2016), relative patch
prediction (Doersch, Gupta, and Efros 2015; Doersch and Zisserman 2017), image
inpainting (Pathak et al. 2016), image super-resolution (Ledig et al. 2017;
Yüksel et al. 2021).

Contrastive learning of is amongst the most successful self-supervised method
to achieve linear classification accuracy and outperforming supervised
learning tasks by suitable architectures and loss (Caron et al. 2020; Chen et
al. 2020a, b; Chen and Li 2020; Zbontar et al. 2021), using pretraining in a
task-agnostic fashion (Kolesnikov, Zhai, and Beyer 2019; Shen et al. 2020),
and fine-tuning on the labeled subset in a task-specific fashion (Wu et al.
2018; Hénaff et al. 2019). However, Hjelm et al. (Hjelm et al. 2019) showed
the accuracy depends on a large number of negative samples in the training
batch. Self-supervised training with a large batch size is computationally
expensive for high-resolution images. BYOL (Grill et al. 2020) and SimSiam
(Chen and He 2021) mitigated this issue by an additional prediction head and
learning the latent representations of positive samples only. Robinson et al.
(Robinson et al. 2021) proposed a technique for selecting negative samples and
the whitening procedure of (Ermolov et al. 2021) despite success is still
sensitive to batch size. MoCo (He et al. 2020; Chen et al. 2020c) alleviates
this problem using a memory-efficient queue of the last visited negatives,
together with a momentum encoder that preserves the intra-queue representation
consistency. Most recently, SSL-HSIC (Li et al. 2021) proposed a method to
maximize dependence between representations of transformed versions of an
image and the image identity, while minimizing the kernelized variance of
those features. VICReg (Bardes, Ponce, and LeCun 2021) proposed a
regularization term on the variance of the embeddings to explicitly avoid the
collapse problem. In this paper, we developed and studied the impact of
$\kappa$-different parameterized convex linear neural networks with functional
Bregman divergence on top of a simple contrastive learning framework (Chen et
al. 2020a, c).

## Problem Formulation and Approach

Our goal is two fold: First, we aim to learn representations using contrastive
loss on the top of embeddings via representation network. Second, we learn a
deep Bregman divergence by minimizing divergence between samples from the same
distribution and maximizing divergence for samples from different classes and
different distributions. As depicted in Fig. 1, our proposed method includes
two sequentially connected neural networks: the _representation network_ and
the _deep Bregman divergence network_. In the following sections, we first
describe contrastive learning in the context of visual representation. Then we
discuss Bergman divergence, functional Bregman, and deep Bregman divergence
network. Next, we describe our framework and the proposed loss.

Figure 2: Illustration of the functional Bregman divergence optimization
process.

### Contrastive Learning

Given a randomly sampled mini-batch of images $X=\\{x_{i}\\}_{i=1}^{n}$ with
$N$ samples, contrastive learning aims to learn an embedding function
$f(\cdot)$ by contrasting positive pairs $(x,x^{+})$ against negative pairs
$(x,x^{-})$. First, we generate a positive pair sharing the same semantics
from each sample in a mini-batch $X$ by applying standard image transformation
techniques. For each positive pair, there exists $2(N-1)$ negative examples in
a mini-batch. The encoder network (e.g. ResNet-50 (He et al. 2016)) encodes
distorted positive and negative samples to a set of corresponding features.
These features are then transformed with a projection MLP head (Chen et al.
2020a) which results in $Z_{1}$ and $Z_{2}$. The contrastive estimation for a
positive pair of examples $(x_{i},x_{j})$ is defined as:

$\ell^{\mathrm{NT}\text{-}\mathrm{Xent}}_{x_{i},x_{j}}=-log~{}\frac{\exp(\mathrm{sim}(\bm{z}_{i},\bm{z}_{j})/\tau)}{\sum_{m=1}^{2N}\mathbbm{1}_{[m\neq
i]}\exp(\mathrm{sim}(\bm{z}_{i},\bm{z}_{m})/\tau)}$ (1)

Where $\mathrm{sim}(\cdot,\cdot)$ is cosine similarity between two vectors,
$N$ is the number of samples in a mini-batch, and $\tau$ is a temperature
scalar. Loss over all the pairs formulated as:

$\mathcal{L}^{\mathrm{Contrastive}}=\frac{1}{2N}{\sum\nolimits_{m=1}^{2N}\Bigg{[}\ell(2m-1,2m)+\ell(2m,2m-1)\Bigg{]}}$
(2)

### Bregman Divergence Learning

Bregman divergence parametrizes by a strictly convex function $\phi$ on convex
set $\Omega\subseteq\mathbb{R}^{d}$, where $\phi$ is continuously-
differentiable on relative interior of $\Omega$. The Bregman divergence
associated with $\phi$ for data point $x_{i},x_{j}\in\Omega$ calculated by:

$D_{\phi}(x_{i},x_{j}):\phi(x_{i})-\phi(x_{j})-(x_{i}-x_{j})^{T}\nabla\phi(x_{j})$
(3)

The well-known examples of Bregman divergence is the squared Euclidean
distance parametrized by $\phi(x)=\frac{1}{2}\|x_{i}\|_{2}^{2}$; the KL-
divergence parameterized by $\phi(x)=\sum_{i}x_{i}logx_{i}$; and the Itakura-
Saito distance parametrized by $\phi(x)=-\sum_{i}logx_{i}$. Bregman
divergences appear in various settings in machine learning and statistical
learning. In optimization, Bregman et. al (Bregman 1967) proposed Bregman
divergences as part of constrained optimization. In the unsupervised
clustering, Bregman divergences provide a solution to extend the K-means
algorithm beyond the convenience of the squared Euclidean distance (Banerjee
et al. 2005). In this paper, we use an extension of standard Bregman
divergences called functional Bregman divergences.

A functional Bregman divergence (Frigyik, Srivastava, and Gupta 2008; Ovcharov
et al. 2018) generalizes the standard Bregman divergence for vectors and it
measures the divergence between two functions or distributions. Given two
functions $p$ and $q$, and a strictly convex functional $\phi$ defined on a
convex set of functions which output in $\mathbb{R}$, the functional Bregman
divergence formulated as:

$D_{\phi}(p,q)=\phi(p)-\phi(q)-\int[p(x)-q(x)]\delta\phi(q)(x)d\mu(x)$ (4)

Same as the vector Bregman divergence, the functional Bregman divergence holds
several properties including convexity, non-negativity, linearity, equivalence
classes, linear separation, dual divergences, and a generalized Pythagorean
inequality.

Deep Bregman divergence (Cilingir, Manzelli, and Kulis 2020) parametrize the
functional Bregman divergence by weight functions $w_{1},w_{2},...,w_{\kappa}$
and biases $b_{1},b_{2},...,b_{\kappa}$ with assumption that every generating
convex functional $\phi(p)$ can be expressed in terms of linear functional.
For the set of linear functional $A$, the $\phi$ defines as:

$\phi(p)=\underset{(w,b_{w})\in A}{sup}\int p(x)w(x)dx+b_{w}$ (5)

and based on the underlying generating convex functional $\phi$, functional
Bregman divergence can be expressed as:

$D_{\phi}(p,q)=\\\ \bigg{(}\int
p(x)w_{p^{*}}(x)dx+b_{p^{*}}\bigg{)}-\bigg{(}\int
p(x)w_{q^{*}}(x)dx+b_{q^{*}}\bigg{)}$ (6)

with $p$, $q$ are given by empirical distributions over input points;
$p^{*}=\operatorname*{argmax}_{\kappa}[\int p(x)w_{\kappa}(x)dx+b_{\kappa}]$;
and $q^{*}$ is defined same as $p^{*}$.

Therefore, we can train deep functional Bregman divergence if each of the
weights and bias functions is given by $\kappa$ separate linear neural
networks (Fig. 2).

### Contrastive Divergence Learning

Given a randomly sampled mini-batch of images $X$, our method takes an
augmentation set $\bm{T}$ and draw two random augmentation $t$ and
$t^{\prime}$ to produces two distorted images $x_{i}=t(x)$ and
$x_{j}=t^{\prime}(x)$ for a single image $x$. The distorted samples are
encoded via base network $f_{\theta}$ to generate corresponding
representations, $z_{i}=f_{\theta}(x_{i})$ and $z_{j}=f_{\theta}(x_{j})$.

Next, we perform divergence learning by adopting a functional Bregman
$D_{\phi}$ ( Fig. 2). Consider $p$ and $q$ as empirical distributions over
$z_{i}$ and $z_{j}$, respectively. We parametrize our deep divergence with
weight function $w_{1},w_{2},...,w_{\kappa}$ and biases
$b_{1},b_{2},...,b_{\kappa}$. Each sub-network $d=\\{1,2,...,\kappa\\}$ takes
$z_{i},z_{j}$ and produces a single output $w_{d}(z)+b_{d}$. Consider $p^{*}$
as the index of the maximum output and $q^{*}$ index of the maximum output
across the $\kappa$ subnetworks. Now, the divergence is the difference between
the output of $z_{i}$ at $p^{*}$ and the output of $z_{i}$ at $q^{*}$.

Considering each of the $\kappa$ outputs corresponds to a different class: the
divergence is zero when both points achieve a maximum value for the same
class, and it is non-zero otherwise. The divergence increases as the two
outputs become more separated.

Our method is trained with a combination of two losses; one based on
discriminative features by representation network and another based on the
Bregman divergence output by $\kappa$ subnetworks. In this paper, we estimate
noise contrastive loss on top of representation vectors similar to Eq. 2. The
output of the representation network results in well-separated inter-product
differences while the deep features learned by $\kappa$ different Bregman
divergences result in well discriminative features with compact intra-product
variance. Therefore, the combination of these is key to have better visual
search engines. In addition, learning such discriminative features enables the
network to generalize well on unseen images.

We convert Bregman divergence $D$ (Eq. 6) to similarity using a Gaussian
kernel $\psi_{i,j}=1/exp(-D/{2{\sigma}^{2}})$ (where $\sigma$ is adjustable
parameter). All the divergences obtained with the same network are viewed as
positive pairs while all other divergences obtained with a different network
are considered as negative pairs. We enforce each subnetwork to have a
consistent but also orthogonal effect on the feature. The divergence loss
between $p(z_{i})$ and $q(z_{j})$ for a mini-batch of the size of $N$ define
as:

$\mathcal{\ell}_{div(p(z_{i}),q(z_{j}))}=-log\Big{(}exp(\psi_{i,j})/\sum\nolimits_{m=1}^{N}exp(\psi_{i,m})\Big{)}$
(7)

$\mathcal{L}^{\mathrm{divergence}}=\frac{1}{N}{\sum\nolimits_{m=1}^{N}\Bigg{[}\ell_{div}(2m-1,2m)\Bigg{]}}$
(8)

and the total loss calculated by:

$\mathcal{L}_{total}=\mathcal{L}_{contrastive}+\mathcal{L}_{divergence}$ (9)

At the end of training, similar to (Chen et al. 2020a) we only keep the
encoder $f_{\theta}$. Algorithm 1 shows PyTorch-style pseudo-code for our
proposed method.

⬇ \par# model: network f + network b # f: base network + projection mlp
(Nxf_out)  # d: Bregman subnetworks # D: Bregman divergence # t: temperature
\par# load a minibatch x with N samples for x in loader(dataset):  x1, x2 =
tau(x), tau_bar(x) # Augmentation z1, z2 = f(x1), f(x2) # Projection: Nxf_out
o1, o2 = d(z1), d(z2) # Subnetworks output z = cat(z1, z2) # concatenation #
part 1: calculating contrastive loss # cosine similarity matrix: 2Nxf_out
sim_f = cos(z.unsqueeze(1), z.unsqueeze(0))  pos_ab = diag(sim_f, N)  pos_ba =
diag(sim_f, -N)  pos_f = cat(pos_ab, pos_ba) # positive logits neg_f =
sim_f[~pos] # negative logits logits_f = cat(pos_f, neg_f)  labels_f =
zeros(2*N)  # contrastive loss loss1 = CrossEntropyLoss(logits_f/t, labels_f)
# (Eq.2)  # part 2: calculating Bregman loss div_matrix = D(o1, o2) # Bregman
divergence # converting divergence to similarity sim_b =
div_to_sim(div_matrix)  pos_b = diag(sim_b) # positive logits neg_b =
sim_b[~pos_b] # negative logits logits_b = cat(pos_b, neg_b)  labels_b =
zeros(N)  # Bregman loss loss2 = CrossEntropyLoss(logits_b, labels_b)  #
accumulation of two losses loss = loss1 + loss2 # (Eq.8)  # SGD update
loss.backward()  update(model.params)  def D(o1, o2):  p_star = argmax(o1)
q_star = argmax(o2)  # Bregman divergence (Eq.6)  return o1[p_star] -
o1[q_star]  \par

Algorithm 1 PyTorch-style pseudo-code for our method

## Experiment Setup

One advantage of our framework is learning over distributions and we do not
restrict ourselves only to divergences between single points. We can also
capture divergences between distributions of points similar to the maximum-
mean discrepancy and the Wasserstein distance. Example applications are data
generation, semi-supervised learning, unsupervised clustering, information
retrieval, and Ranking. To empirically compare our proposed framework to
existing contrastive models, we follow standard protocols by self-supervised
learning and evaluate the learned representation by linear classification and
semi-supervised tasks as well as transfer learning to different datasets and
different computer vision tasks.

Image augmentation. We define a random transformation function $\bm{T}$ that
applies a combination of crop, horizontal flip, color jitter, and grayscale.
Similar to (Chen et al. 2020a), we perform crops with a random size from $0.2$
to $1.0$ of the original area and a random aspect ratio from $3/4$ to $4/3$ of
the original aspect ratio. We also apply horizontal mirroring with a
probability of $0.5$. Then, we apply grayscale with probability $0.2$, and
color jittering with probability $0.8$ and with configuration
$(0.4,0.4,0.4,0.1)$. However, for ImageNet, we define the stronger jittering
$(0.8,0.8,0.8,0.2)$, crop size from $0.08$ to $1.0$, grayscale probability
$0.2$, and Gaussian blurring with $0.5$ probability and $\sigma=(0.1,2.0)$. In
all the experiments, at the testing phase, we apply only resize and center
crop (standard protocol).

Deep representation network architecture Our base encoder consists of a
convolutional residual network (He et al. 2016) with 18 layers and 50 layers
with minor changes. The network is without the final classification layer
instead it has two nonlinear multi-layer perceptrons (MLP) with rectified
linear unit activation in between. This MLP consists of a linear layer with
input size 1024 followed by batch normalization, rectified linear unit
activation, and a final linear layer with output dimension 128 as embedding
space. The embeddings are fed to the contrastive loss and used as an input for
our deep divergence networks.

Deep divergence network architecture (subnetworks) We implemented
$\kappa$-adaptive subnetworks on top of the MLP projection head. Many possible
architectures are suitable to capture this type of network; we consider a
simple architecture where each network includes 2-layer MLPs with 128, 64, 1
hidden nodes, follow by batch normalization. We do not include activation
between the layers to maintain Bregman properties and convex network. Each
subnetwork own independent set of weights. We perform a Bayesian
hyperparameter search to find the best $\kappa$.

Optimization We use the Adam optimizer (Kingma and Ba 2015) with a learning
rate $0.005$, $\beta_{1}=0.5$, $\beta_{2}=0.999$, and weight decay $0.0001$
without restarts, over 500 epochs (similar to MoCo configuration). We convert
Bregman divergences to similarity score using a strictly monotone decreasing
function $\psi$. As explained in Method, we use Gaussian kernel and set
$\sigma=0.9$ for all experiments. Temperature $\tau$ sets equal to 0.1. We
train our model with a mini-batch size of 512 and 2 GPUs on all small
datasets, considering our primary objective is to verify the impact of our
proposed method rather than to suppress state-of-the-art results. For
ImageNet, we use a mini-batch size of 256 in 4 GPUs (Tesla A-100), and an
initial learning rate of 0.003. We train for 500 epochs with the learning rate
multiplied by 0.1 at 120 and 360 epochs, taking around five days of training
ResNet-50.

Datasets and tasks We use the following datasets in our experiments: CIFAR
10/100 (Krizhevsky, Hinton et al. 2009) are subsets of the tiny images
dataset. Both datasets include 50,000 images for training and 10,000
validation images of size $32\times 32$ with 10 and 100 classes, respectively.
STL 10 (Coates, Ng, and Lee 2011) consists of 5000 training images and 8000
test images in 10 classes with size of $96\times 96$. This dataset includes
100,000 unlabeled images for unsupervised learning task. ImageNet (Deng et al.
2009), aka ILSVRC 2012, contains 1000 classes, with 1.28 million training
images and 50,000 validation images. ISCI-2018 (Codella et al. 2019) is a
challenge on the detection of seven different skin cancer and part of the
MICCAI-2018 conference. The organizers released 10,015 dermatology scans with
a size of $128\times 128$ pixels collected from different clinics.

## Experiments and Results

### Linear Evaluation

The common evaluation protocol for self-supervised learning is based on
freezing the base encoder $f(\cdot)$ after unsupervised pretraining, and then
train a supervised linear classifier on top of it. The linear classifier is a
fully connected layer followed by softmax, which is plugged on top of
$f(\cdot)$ after removing the MLP’s head and divergence network $D(\cdot)$.
Our linear evaluation consists of studies on small and large datasets
including STL-10, CIFAR-10/100, ISIC-7, ImageNet with different approaches to
explore and compare the effectiveness of our proposed method. Top-1 accuracy
in $\%$ reported on the test set of ImageNet in Table 1 and Table 2 shows the
comparison of our model against the self-supervised baselines on small
datasets. Based on the performance reported in Table 1 and Table 2, our method
obtains up to 1.5% improvement compared with baselines MoCo-V2 or SimCLR. We
also achieves a top-1 accuracy of 71.3% on ImageNet which is comparable to the
state-of-the-art methods.

Table 1: Top-1 accuracy (%) under linear evaluation for ImageNet with the standard ResNet-50 architecture (24M parameters). Top five best methods are underlined. Method | Top-1 Accuracy
---|---
PIRL (Misra and Maaten 2020) | 63.6
CPC-v2 (Henaff 2020) | 63.8
CMC (Caron et al. 2020) | 66.2
SimCLR (Chen et al. 2020b) | 69.3
SimSiam (Chen and He 2021) | 70.8
MoCo-v2 (Chen et al. 2020c) | 71.1
SimCLR-v2 (Chen et al. 2020b) | 71.7
SwAV (Caron et al. 2020) (w/o multi-crop) | 71.8
Barlow Twins (Zbontar et al. 2021) | 73.2
BYOL (Grill et al. 2020) | 74.3
Our method | 71.3
Table 2: Top-1 accuracy (in %) under linear evaluation for small datasets compared to SimCLR and MoCo-v2 (baselines). Method | CIFAR-10 | STL-10 | CIFAR-100 | ISIC-7
---|---|---|---|---
SimCLR | 91.6 | 91.8 | 68.2 | 82.6
MoCo-v2 | 92.8 | 91.6 | 68.4 | 82.9
Our method | 92.9 | 92.2 | 68.8 | 83.4

### Semi-supervised Training

We evaluate the performance of our method’s representation on a semi-
supervised image classification task. In this task, we pre-train the ResNet-50
model on unlabeled ImageNet examples and fine-tune a classification model
using a subset of ImageNet examples with labels. We follow the semi-supervised
protocol of (Chen et al. 2020b) and use the same fixed splits of respectively
1% and 10% of ImageNet labeled training data. Table 3 shows the comparison of
our performance against state-of-the-art methods and baselines. The result
indicates using the proposed method we can outperform the baseline SimCLR and
MoCo-v2. We also have a slightly better or comparable results compared to
sate-of-the-art methods such as BYOL.

Table 3: Comparison of achieved accuracy (top-1) for semi-supervised classification of ImageNet dataset. We use our pretrained ResNet-50 model, and fine-tune it on a classification task by using only 1% and 10% of ImageNet labeled training data. Method | 1% | 10%
---|---|---
SimCLR (Chen et al. 2020b) | 48.3 | 65.6
MoCo-v2 (Chen et al. 2020c) | 49.1 | 66.4
BYOL (Grill et al. 2020) | 53.2 | 68.8
W-MSE (Ermolov et al. 2021) | 48.2 | 66.1
Our method | 50.5 | 66.9
Table 4: Transfer learning results from ImageNet with standard ResNet-50 architecture on the object detection task on VOC07+12. Results of (*) methods are taken from (Zbontar et al. 2021). | VOC07+12 det
---|---
Method | $AP_{all}$ | $AP_{50}$ | $AP_{75}$
*Barlow Twins | 56.8 | 82.6 | 63.4
MoCo-v2 | 57.4 | 82.5 | 64.0
SimSiam | 57.0 | 82.4 | 63.7
SwAV | 56.1 | 82.6 | 62.7
Ours | 57.8 | 82.7 | 64.1

### Transfer to Other Tasks

We further assess the generalization capacity of the learned representation on
object detection. We train a Faster R-CNN (Faster 2015) model on Pascal VOC
2007 and Pascal VOC 2012 and evaluate on the test set of Pascal VOC. Table 4
provides a comparison of transfer learning performance of our self-supervised
approach for the task of object detection. We use pre-trained ResNet-50 models
on ImageNet and perform object detection on Pascal VOC07+12 dataset
(Everingham et al. 2010).

### Ablation Studies and Discussions

To build intuition around the behavior and the observed performance of the
proposed method, we further investigate the following aspects of our approach
in multiple ablation studies: (i) the impact of the Bregman divergence network
on the quality of representation; (ii) the number of $\kappa$ subnetworks;
(iii) robustness of our algorithm in invariance to augmentations and
transformations. Further analysis about the combination of loss functions and
training behavior is available in the Appendix.

Table 5: Top-1 accuracy (in %) under linear evaluation when our method is trained on top of different numbers of subnetworks and using CIFAR-10 ($\kappa$) data. The performance of the divergence learning framework depends on the representation feature space while our method significantly outperforms deep divergence. Number of subnetworks $\bm{k}$ | 5 | 20 | 50 | 100 | 200 | 500 | 1000
---|---|---|---|---|---|---|---
Deep Divergence (Cilingir, Manzelli, and Kulis 2020) | 71.9 | 77.8 | 79.4 | 80.0 | 77.4 | 74.1 | 70.8
Our Method | 84.3 | 88.6 | 89.5 | 92.1 | 92.4 | 88.4 | 85.1

Impact of Bregman divergence network on quality of representations We
visualize the representation features using t-SNE with the last convolution
layer from ResNet-18 to explore the quality of learned features using our
proposed method. Figure 3 compares the representation space learned by our
proposed method (a) and SimCLR (b). As depicted in Fig. 3, our model shows
better separation on clusters, especially for classes 2, 4, 6, and 7.

(a)

(b)

Figure 3: t-SNE visualization of embeddings with ResNet-50 on CIFAR-10: (a)
SimCLR, (b) our method. Classes are well-clustered using our method especially
for classes 2, 4, 6, and 7.

Number of $\kappa$-subnetworks We trained $\kappa$ individual deep neural
networks on top of the embedding space. The input of each network was similar
but they parameterized with different weights and biases. Here, we provide
more details regarding our classification experiments by considering different
$\kappa$. Fig. 4 shows the performance in term of top-1 accuracy for CIFAR-10
and Tiny ImageNet (Le and Yang 2015). Based on quantitative results shown in
Fig. 4, the performance improves in both CIFAR-10 and Tiny ImageNet datasets
by increasing the number of subnetworks ($\kappa$) until a certain point, then
it starts dropping possibly due to over parameterization. With a small
$\kappa$, the performance of our network is more similar to contrastive loss.
For example in case of CIFAR-10 shown in Fig. 4, when $\kappa=10$ our
performance is almost 90% and the performance is increased to 92.9% for larger
$\kappa$ ($\kappa=150$). This shows training our network with a correct number
of $\kappa$ can learn a better representation of data.

Table 5 compares the performance of our method with Cilingir et al. (Cilingir,
Manzelli, and Kulis 2020). Based on reported results in Table 5, the
performance of the divergence learning framework depends on the representation
feature space while our method significantly outperforms deep divergence.

Figure 4: Impact of increasing subnetworks ($\kappa$) on accuracy (%): The
performance improves in both CIFAR-10 and Tiny ImageNet datasets by increasing
($\kappa$) until a certain point, then it starts dropping possibly due to over
parameterization of deep Bregman divergence sub-networks.

Image augmentations Since contrastive loss is sensitive to the choice of
augmentation technique and learned representations can get controlled by the
specific set of distortions (Grill et al. 2020), we also examined how robust
our method is to remove some of data augmentations. Figure 5 presents decrease
in top-1 accuracy (in % points) of our method and SimCLR at 300 epochs and
under linear evaluation on ImageNet (SimCLR numbers are extracted from (Grill
et al. 2020)). This figure shows that the representations learned by our
proposed contrastive divergence are more robust to removing certain
augmentations in comparison to SimCLR. For instance, SimCLR does not work well
when removing image crop from its transformation set.

Figure 5: Impact of progressively removing augmentations measured by the
decrease in top-1 accuracy (%) at 300 epochs and under linear evaluation on
ImageNet: our method is more robust to the effect of removing augmentation in
comparison to traditional contrastive methods.

## Conclusion

In this paper, we proposed and examined deep divergence for contrastive
learning of visual representation. Our framework is composed of the
representation learning network followed by multiple divergence learning
networks. We train functional Bregman divergence on top of the representation
network using $\kappa$-adaptive convex neural networks. The similarity matrix
is formulated according to Bregman distance output by an ensemble of the
networks. Then networks are optimized end-to-end using our novel contrastive
divergence loss. We successfully improve over previous methods for deep metric
learning, deep divergence learning, self-supervised, semi-supervised, and
transfer learning. Empirical experiments demonstrate the efficacy of the
proposed method on standard benchmarks as well as recent clinical datasets on
both classification and object detection tasks.

## Acknowledgment

This work has been funded in part by the German Federal Ministry of Education
and Research (BMBF) under Grant No.01IS18036A, Munich Center for Machine
Learning (MCML). F. S., M. R., and B. B. were supported by the German Federal
Ministry of Education and Research (BMBF) under Grant No. 01IS18036A. We would
like to thank Ali Siahkamari for helpful discussions. We are also grateful to
Simon Kornblith for valuable feedback on the manuscript.

## References

  * Bachman, Hjelm, and Buchwalter (2019) Bachman, P.; Hjelm, R. D.; and Buchwalter, W. 2019.  Learning representations by maximizing mutual information across views.  In _Advances in Neural Information Processing Systems_ , 15535–15545.
  * Banerjee et al. (2005) Banerjee, A.; Merugu, S.; Dhillon, I. S.; Ghosh, J.; and Lafferty, J. 2005.  Clustering with Bregman divergences.  _Journal of machine learning research_ , 6(10).
  * Bardes, Ponce, and LeCun (2021) Bardes, A.; Ponce, J.; and LeCun, Y. 2021.  Vicreg: Variance-invariance-covariance regularization for self-supervised learning.  _arXiv preprint arXiv:2105.04906_.
  * Bregman (1967) Bregman, L. M. 1967.  The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming.  _USSR computational mathematics and mathematical physics_ , 7(3): 200–217.
  * Caron et al. (2020) Caron, M.; Misra, I.; Mairal, J.; Goyal, P.; Bojanowski, P.; and Joulin, A. 2020\.  Unsupervised learning of visual features by contrasting cluster assignments.  _NeurIPS_.
  * Carreira-Perpinan and Hinton (2005) Carreira-Perpinan, M. A.; and Hinton, G. E. 2005.  On contrastive divergence learning.  In _Aistats_ , volume 10, 33–40. Citeseer.
  * Chen et al. (2020a) Chen, T.; Kornblith, S.; Norouzi, M.; and Hinton, G. 2020a.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , 1597–1607. PMLR.
  * Chen et al. (2020b) Chen, T.; Kornblith, S.; Swersky, K.; Norouzi, M.; and Hinton, G. 2020b.  Big self-supervised models are strong semi-supervised learners.  _arXiv preprint arXiv:2006.10029_.
  * Chen and Li (2020) Chen, T.; and Li, L. 2020.  Intriguing properties of contrastive losses.  _arXiv preprint arXiv:2011.02803_.
  * Chen et al. (2020c) Chen, X.; Fan, H.; Girshick, R.; and He, K. 2020c.  Improved baselines with momentum contrastive learning.  _arXiv preprint arXiv:2003.04297_.
  * Chen and He (2021) Chen, X.; and He, K. 2021.  Exploring simple siamese representation learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 15750–15758.
  * Cilingir, Manzelli, and Kulis (2020) Cilingir, H. K.; Manzelli, R.; and Kulis, B. 2020.  Deep divergence learning.  In _International Conference on Machine Learning_ , 2027–2037. PMLR.
  * Coates, Ng, and Lee (2011) Coates, A.; Ng, A.; and Lee, H. 2011.  An analysis of single-layer networks in unsupervised feature learning.  In _Proceedings of the fourteenth international conference on artificial intelligence and statistics_ , 215–223. JMLR Workshop and Conference Proceedings.
  * Codella et al. (2019) Codella, N.; Rotemberg, V.; Tschandl, P.; Celebi, M. E.; Dusza, S.; Gutman, D.; Helba, B.; Kalloo, A.; Liopyris, K.; Marchetti, M.; et al. 2019.  Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic).  _arXiv preprint arXiv:1902.03368_.
  * Deng et al. (2009) Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , 248–255. Ieee.
  * Doersch, Gupta, and Efros (2015) Doersch, C.; Gupta, A.; and Efros, A. A. 2015.  Unsupervised visual representation learning by context prediction.  In _Proceedings of the IEEE international conference on computer vision_ , 1422–1430.
  * Doersch and Zisserman (2017) Doersch, C.; and Zisserman, A. 2017.  Multi-task self-supervised visual learning.  In _Proceedings of the IEEE International Conference on Computer Vision_ , 2051–2060.
  * Dragomir, Even, and Hendrikx (2021) Dragomir, R.-A.; Even, M.; and Hendrikx, H. 2021.  Fast Stochastic Bregman Gradient Methods: Sharp Analysis and Variance Reduction.  _International Conference on Machine Learning_ , 5300–5311.
  * Ermolov et al. (2021) Ermolov, A.; Siarohin, A.; Sangineto, E.; and Sebe, N. 2021.  Whitening for self-supervised representation learning.  In _International Conference on Machine Learning_ , 3015–3024. PMLR.
  * Everingham et al. (2010) Everingham, M.; Van Gool, L.; Williams, C. K.; Winn, J.; and Zisserman, A. 2010\.  The pascal visual object classes (voc) challenge.  _International journal of computer vision_ , 88(2): 303–338.
  * Faster (2015) Faster, R. 2015.  Towards real-time object detection with region proposal networks.  _Advances in neural information processing systems_ , 9199.
  * Frigyik, Srivastava, and Gupta (2008) Frigyik, B. A.; Srivastava, S.; and Gupta, M. R. 2008.  Functional Bregman divergence.  In _2008 IEEE International Symposium on Information Theory_ , 1681–1685. IEEE.
  * Grill et al. (2020) Grill, J.-B.; Strub, F.; Altché, F.; Tallec, C.; Richemond, P. H.; Buchatskaya, E.; Doersch, C.; Pires, B. A.; Guo, Z. D.; Azar, M. G.; et al. 2020\.  Bootstrap your own latent: A new approach to self-supervised learning.  _arXiv preprint arXiv:2006.07733_.
  * He et al. (2020) He, K.; Fan, H.; Wu, Y.; Xie, S.; and Girshick, R. 2020.  Momentum contrast for unsupervised visual representation learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 9729–9738.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 770–778.
  * Henaff (2020) Henaff, O. 2020.  Data-efficient image recognition with contrastive predictive coding.  In _International Conference on Machine Learning_ , 4182–4192. PMLR.
  * Hénaff et al. (2019) Hénaff, O. J.; Srinivas, A.; De Fauw, J.; Razavi, A.; Doersch, C.; Eslami, S.; and Oord, A. v. d. 2019.  Data-efficient image recognition with contrastive predictive coding.  _arXiv preprint arXiv:1905.09272_.
  * Hjelm et al. (2019) Hjelm, R. D.; Fedorov, A.; Lavoie-Marchildon, S.; Grewal, K.; Bachman, P.; Trischler, A.; and Bengio, Y. 2019.  Learning deep representations by mutual information estimation and maximization.  In _ICLR_.
  * Hoffer and Ailon (2015) Hoffer, E.; and Ailon, N. 2015.  Deep metric learning using triplet network.  In _International workshop on similarity-based pattern recognition_ , 84–92. Springer.
  * Kampffmeyer et al. (2019) Kampffmeyer, M.; Løkse, S.; Bianchi, F. M.; Livi, L.; Salberg, A.-B.; and Jenssen, R. 2019.  Deep divergence-based approach to clustering.  _Neural Networks_ , 113: 91–101.
  * Kato and Teshima (2021) Kato, M.; and Teshima, T. 2021.  Non-negative bregman divergence minimization for deep direct density ratio estimation.  In _International Conference on Machine Learning_ , 5320–5333. PMLR.
  * Kingma and Ba (2015) Kingma, D. P.; and Ba, J. 2015.  Adam: A method for stochastic optimization.  _ICLR, International Conference Learning Representation_.
  * Koch, Zemel, and Salakhutdinov (2015) Koch, G.; Zemel, R.; and Salakhutdinov, R. 2015.  Siamese neural networks for one-shot image recognition.  In _ICML deep learning workshop_ , volume 2. Lille.
  * Kolesnikov, Zhai, and Beyer (2019) Kolesnikov, A.; Zhai, X.; and Beyer, L. 2019.  Revisiting self-supervised visual representation learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 1920–1929.
  * Kong et al. (2020) Kong, W.; Krichene, W.; Mayoraz, N.; Rendle, S.; and Zhang, L. 2020.  Rankmax: An Adaptive Projection Alternative to the Softmax Function.  _Advances in Neural Information Processing Systems_ , 33.
  * Krizhevsky, Hinton et al. (2009) Krizhevsky, A.; Hinton, G.; et al. 2009.  Learning multiple layers of features from tiny images.
  * Le and Yang (2015) Le, Y.; and Yang, X. 2015.  Tiny imagenet visual recognition challenge.  _CS 231N_ , 7(7): 3.
  * Ledig et al. (2017) Ledig, C.; Theis, L.; Huszár, F.; Caballero, J.; Cunningham, A.; Acosta, A.; Aitken, A.; Tejani, A.; Totz, J.; Wang, Z.; et al. 2017.  Photo-realistic single image super-resolution using a generative adversarial network.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 4681–4690.
  * Lee et al. (2017) Lee, H.-Y.; Huang, J.-B.; Singh, M.; and Yang, M.-H. 2017.  Unsupervised representation learning by sorting sequences.  In _Proceedings of the IEEE International Conference on Computer Vision_ , 667–676.
  * Li et al. (2021) Li, Y.; Pogodin, R.; Sutherland, D. J.; and Gretton, A. 2021.  Self-Supervised Learning with Kernel Dependence Maximization.  _arXiv preprint arXiv:2106.08320_.
  * Liu and Belkin (2016) Liu, C.; and Belkin, M. 2016.  Clustering with Bregman divergences: An asymptotic analysis.  In _Proceedings of the 30th International Conference on Neural Information Processing Systems_ , 2351–2359. Citeseer.
  * Misra and Maaten (2020) Misra, I.; and Maaten, L. v. d. 2020.  Self-supervised learning of pretext-invariant representations.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 6707–6717.
  * Noroozi and Favaro (2016) Noroozi, M.; and Favaro, P. 2016.  Unsupervised learning of visual representations by solving jigsaw puzzles.  In _European conference on computer vision_ , 69–84. Springer.
  * Oord, Li, and Vinyals (2018) Oord, A. v. d.; Li, Y.; and Vinyals, O. 2018.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_.
  * Ovcharov et al. (2018) Ovcharov, E. Y.; et al. 2018.  Proper scoring rules and Bregman divergence.  _Bernoulli_ , 24(1): 53–79.
  * Painsky and Wornell (2019) Painsky, A.; and Wornell, G. W. 2019.  Bregman divergence bounds and universality properties of the logarithmic loss.  _IEEE Transactions on Information Theory_ , 66(3): 1658–1673.
  * Pathak et al. (2016) Pathak, D.; Krahenbuhl, P.; Donahue, J.; Darrell, T.; and Efros, A. A. 2016.  Context encoders: Feature learning by inpainting.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2536–2544.
  * Robinson et al. (2021) Robinson, J.; Chuang, C.-Y.; Sra, S.; and Jegelka, S. 2021.  Contrastive learning with hard negative samples.  _ICLR_.
  * Shen et al. (2020) Shen, Z.; Liu, Z.; Liu, Z.; Savvides, M.; Darrell, T.; and Xing, E. 2020.  Un-mix: Rethinking image mixtures for unsupervised visual representation learning.  _arXiv preprint arXiv:2003.05438_.
  * Siahkamari et al. (2019) Siahkamari, A.; Xia, X.; Saligrama, V.; Castañón, D.; and Kulis, B. 2019\.  Learning to Approximate a Bregman Divergence.  _arXiv preprint arXiv:1905.11545_.
  * Siahkamari et al. (2020) Siahkamari, A.; Xia, X.; Saligrama, V.; Castañón, D.; and Kulis, B. 2020\.  Learning to Approximate a Bregman Divergence.  _Advances in Neural Information Processing Systems_ , 33: 3603–3612.
  * Stojnic and Risojevic (2021) Stojnic, V.; and Risojevic, V. 2021.  Self-Supervised Learning of Remote Sensing Scene Representations Using Contrastive Multiview Coding.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 1182–1191.
  * Wu et al. (2018) Wu, Z.; Xiong, Y.; Yu, S. X.; and Lin, D. 2018.  Unsupervised feature learning via non-parametric instance discrimination.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 3733–3742.
  * Xing et al. (2002) Xing, E. P.; Ng, A. Y.; Jordan, M. I.; and Russell, S. 2002.  Distance metric learning with application to clustering with side-information.  In _NIPS_ , 505–512, 12. Citeseer.
  * Yüksel et al. (2021) Yüksel, O. K.; Simsar, E.; Er, E. G.; and Yanardag, P. 2021.  LatentCLR: A Contrastive Learning Approach for Unsupervised Discovery of Interpretable Directions.  _arXiv preprint arXiv:2104.00820_.
  * Zbontar et al. (2021) Zbontar, J.; Jing, L.; Misra, I.; LeCun, Y.; and Deny, S. 2021.  Barlow twins: Self-supervised learning via redundancy reduction.  _arXiv preprint arXiv:2103.03230_.
  * Zhang, Isola, and Efros (2016) Zhang, R.; Isola, P.; and Efros, A. A. 2016.  Colorful image colorization.  In _European conference on computer vision_ , 649–666. Springer.

## APPENDIX

## Contrastive Divergence Loss

In this section, we provide more training details related to our proposed
contrastive divergence loss. As it has been mentioned previously, the
divergence loss between $p(z_{i})$ and $q(z_{j})$ for a mini-batch of the size
of $N$ define as:

$\mathcal{\ell}_{div(p(z_{i}),q(z_{j}))}=-log\Big{(}exp(\psi_{i,j})/\sum\nolimits_{m=1}^{N}exp(\psi_{i,m})\Big{)}$
(A.1)

$\mathcal{L}^{\mathrm{divergence}}=\frac{1}{N}{\sum\nolimits_{m=1}^{N}\Bigg{[}\ell_{div}(2m-1,2m)\Bigg{]}}$
(A.2)

and the total loss calculated by:

$\mathcal{L}_{total}=\mathcal{L}_{contrastive}+\mathcal{L}_{divergence}$ (A.3)

where we convert Bregman divergence $D$ (Eq. 6) to similarity using
$\psi_{i,j}$. The conversion function $\psi_{i,j}$, must be defined as an
strictly monotone function and can have many forms. In general there are
multiple options to define a $\psi$ function to convert divergences to
similarity scores ranging from a simple inverse function to a more complex
Gaussian kernel. Table A.1 shows the functions that has been examined in our
experiments where we achieved the best performance when the Gaussian kernel is
used. Our chosen conversion function, $\psi_{i,j}=1/exp(-D/{2{\sigma}^{2}})$
can be adjusted using a $\sigma$ parameter. We set $\sigma$ in our experiments
to 0.9.

Table A.1: Converting functional Bregman divergences ($D$) to similarity
($\psi$). There are multiple options to define a function to convert
divergences to similarity scores ranging from a simple inverse function to a
more complex Gaussian kernel. We achieve the best results using a Gaussian
Kernel with adjustable term $\sigma=0.9$. Strictly monotone function $\psi$
---
$\psi_{i,j}=\frac{1}{1+D}$
$\psi_{i,j}=\sqrt{(1-D)}$
$\psi_{i,j}=1-D/max(D)$
$\psi_{i,j}=arccot(\alpha D)(\alpha>0)$
$\psi_{i,j}=\frac{1}{exp(-D/{2{\sigma}^{2}})}(\sigma=0.9$)

Figure A.1 demonstrated approximating divergences between different classes.
In this example, we show that the divergence of a random sample $(x)$ to a
specific sample $I$ in the data is equal to zero when they are from same
category. In general, the outputs of the underlying convex generating
functional of the input samples that are from the same category will lay on
the same hyper-plane. We use the same strategy as (Siahkamari et al. 2019) to
produce the graph.

Figure A.1: Visualization of the learnt Bregman divergence on synthetic 2-D
samples. In this example, we show that the divergence of a random sample $(x)$
to a specific sample $I$ in the data is equal to zero when they are from same
category. In general, the outputs of the underlying convex generating
functional of the input samples that are from the same category will lay on
the same hyper-plane.

## Training Details

We use Adam optimizer (Kingma and Ba 2015) with a learning rate $0.005$, and
weight decay $0.0001$ without restarts, over 300 epochs for all of the
pretraining. Figure A.2 shows a sample learning curve of our model using the
ResNet-18 base network over 100 epochs and the corresponding test loss and
top-1 and top-5 accuracy on CIFAR-10 dataset. For the pre-training we use with
a mini-batch size of 512 and 2 GPUs.

Figure A.2: The learning curves of deep Bregman divergence and the
corresponding test loss, top-1 and top-5 accuracy for linear evaluation on
CIFAR-10.

## Other Applications

The metric learning and divergence learning problems are the fundamental
problem in machine learning, attracting considerable research and
applications. These applications include (but not limited to) uncertainty
quantification, density estimation, image retrieval, unsupervised image
clustering, program debugging, image generation, music analysis, and ranking.
Fundamental studies in these problems will help to improve results in these
applications as well as direct to additional impact in new domains.