# Class Anchor Margin Loss for Content-Based Image Retrieval

Alexandru Ghiţă  Department of Computer Science, University of Bucharest14
AcademieiBucharestRomania010014<EMAIL_ADDRESS>and  Radu Tudor
Ionescu  Department of Computer Science, University of Bucharest14
AcademieiBucharestRomania010014<EMAIL_ADDRESS>
(2023)

###### Abstract.

The performance of neural networks in content-based image retrieval (CBIR) is
highly influenced by the chosen loss (objective) function. The majority of
objective functions for neural models can be divided into metric learning and
statistical learning. Metric learning approaches require a pair mining
strategy that often lacks efficiency, while statistical learning approaches
are not generating highly compact features due to their indirect feature
optimization. To this end, we propose a novel repeller-attractor loss that
falls in the metric learning paradigm, yet directly optimizes for the $L_{2}$
metric without the need of generating pairs. Our loss is formed of three
components. One leading objective ensures that the learned features are
attracted to each designated learnable class anchor. The second loss component
regulates the anchors and forces them to be separable by a margin, while the
third objective ensures that the anchors do not collapse to zero. Furthermore,
we develop a more efficient two-stage retrieval system by harnessing the
learned class anchors during the first stage of the retrieval process,
eliminating the need of comparing the query with every image in the database.
We establish a set of four datasets (CIFAR-100, Food-101, SVHN, and Tiny
ImageNet) and evaluate the proposed objective in the context of few-shot and
full-set training on the CBIR task, by using both convolutional and
transformer architectures. Compared to existing objective functions, our
empirical evidence shows that the proposed objective is generating superior
and more consistent results.

contrastive learning, contrastive loss, class anchors, content-based image
retrieval, object retrieval

††copyright: iw3c2w3††journalyear: 2023††doi: XXXXXXX.XXXXXXX††conference:
Arxiv; 2023; Arxiv††price: 15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs:
Computing methodologies Machine learning††ccs: Computing methodologies
Ranking††ccs: Computing methodologies Supervised learning by classification

##  1\. Introduction

Content-based image retrieval (CBIR) is a challenging task that aims at
retrieving images from a database that are most similar to a query image. The
state-of-the-art image retrieval systems addressing the aforementioned task
are currently based on deep neural networks (Cao et al., 2020; Dubey, 2021;
Lee et al., 2022; Radenović et al., 2019; Revaud et al., 2019; Wu et al.,
2021). Although neural models obtained impressive performance levels compared
to handcrafted CBIR models (Philbin et al., 2007), one of the main challenges
that remain to be solved in training neural networks for retrieval problems is
the choice of the objective function. Indeed, a well-chosen objective function
should enhance the discriminative power of the learned embeddings, i.e. the
resulting features should exhibit small differences for images representing
the same object and large differences for images representing different
objects. This would implicitly make the embeddings more suitable for image
retrieval.

The majority of loss functions used to optimize neural models can be separated
into two major categories: statistical learning and metric learning. The most
popular category is represented by statistical learning, which includes
objective functions such as the cross-entropy loss (Murphy, 2012) or the hinge
loss (Cortes and Vapnik, 1995). The optimization problem is usually based on
minimizing a particular probability distribution. Consequently, objective
functions in this category achieve the desired properties as an indirect
effect during optimization. Hence, objective functions based on statistical
learning are more suitable for modeling multi-class classification tasks
rather than retrieval tasks. The second category corresponds to metric
learning, which comprises objective functions such as contrastive loss
(Hadsell et al., 2006), triplet loss (Schroff et al., 2015), and quadruplet
loss (Chen et al., 2017). Objective functions in this category operate
directly in the embedding space and optimize for the desired distance metric.
However, they usually require forming tuples of examples to compute the loss,
which can be a costly step in terms of training time. To reduce the extra time
required to build tuples, researchers resorted to hard example mining schemes
(Georgescu and Ionescu, 2021; Georgescu et al., 2022; Harwood et al., 2017;
Schroff et al., 2015; Suh et al., 2019; Wu et al., 2017). Still, statistical
learning remains more time efficient.

In this context, we propose a novel repeller-attractor objective function that
falls in the metric learning paradigm. Our loss directly optimizes for the
$L_{2}$ metric without needing to generate pairs, thus alleviating the
necessity of performing the costly hard example mining. The proposed objective
function achieves this through three interacting components, which are
expressed with respect to a set of learnable embeddings, each representing a
class anchor. The leading loss function ensures that the data embeddings are
attracted to the designated class anchor. The second loss function regulates
the anchors and forces them to be separable by a margin, while the third
objective ensures that the anchors do not collapse to the origin. In addition,
we propose a two-stage retrieval method that compares the query embedding with
the class anchors in the first stage, then continues by comparing the query
embedding with image embeddings assigned to the nearest class anchors. By
harnessing the learned class anchors, our retrieval process becomes more
efficient and effective than the brute-force search.

We carry out experiments on four datasets (CIFAR-100 (Krizhevsky, 2009),
Food-101 (Bossard et al., 2014), SVHN (Netzer et al., 2011), and Tiny ImageNet
(Russakovsky et al., 2015)) to compare the proposed loss function against
representative statistical and deep metric learning objectives. We evaluate
the objectives in the context of few-shot and full-set training on the CBIR
task, by using both convolutional and transformer architectures, such as
residual networks (ResNets) (He et al., 2016) and shifted windows (Swin)
transformers (Liu et al., 2021). Moreover, we test the proposed losses on
various embedding space dimensions, ranging from $32$ to $2048$. Compared to
existing loss functions, our empirical results show that the proposed
objective is generating higher and more consistent performance levels across
the considered evaluation scenarios. Furthermore, we conduct an ablation study
to demonstrate the influence of each loss component on the overall
performance.

In summary, our contribution is threefold:

  * •

We introduce a novel repeller-attractor objective function that directly
optimizes for the $L_{2}$ metric, alleviating the need to generate pairs via
hard example mining or alternative mining strategies.

  * •

We propose a two-stage retrieval system that leverages the use of the learned
class anchors in the first stage of the retrieval process, leading to
significant speed and performance gains.

  * •

We conduct comprehensive experiments to compare the proposed loss function
with popular loss choices in multiple evaluation scenarios.

##  2\. Related Work

As related work, we refer to studies introducing new loss functions, which are
related to our first contribution, and new content-based image retrieval
methods, which are connected to our second contribution.

Loss functions. The problem of generating features that are tightly clustered
together for images representing the same objects and far apart for images
representing distinct objects is a challenging task usually pursued in the
context of CBIR. For retrieval systems based on neural networks, the choice of
the objective function is the most important factor determining the geometry
of the resulting feature space (Tang et al., 2022). We hereby discuss related
work proposing various loss functions aimed at generating effective embedding
spaces.

Metric learning objective functions directly optimize a desired metric and are
usually based on pairs or tuples of known data samples (Sohn, 2016; Vassileios
Balntas and Mikolajczyk, 2016; Yu and Tao, 2019; Chen et al., 2017). One of
the earliest works on metric learning proposed the contrastive loss (Hadsell
et al., 2006), which was introduced as a method to reduce the dimensionality
of the input space, while preserving the separation of feature clusters. The
idea behind contrastive loss is to generate an attractor-repeller system that
is trained on positive and negative pairs generated from the available data.
The repelling can happen only if the distance between a negative pair is
smaller than a margin $m$. In the context of face identification, another
successful metric learning approach is triplet loss (Schroff et al., 2015). It
obtains the desired properties of the embedding space by generating triplets
of anchor, positive and negative examples. Each image in a batch is considered
an anchor, while positive and negative examples are selected online from the
batch. For each triplet, the proposed objective enforces the distance between
the anchor and the positive example to be larger than the distance between the
anchor and the negative example, by a margin $m$. Other approaches introduced
objectives that directly optimize the AUC (Gajić et al., 2022), recall (Patel
et al., 2022) or AP (Revaud et al., 2019). The main issues when optimizing
with loss functions based on metric learning are the usually slow convergence
(Sohn, 2016) and the difficulty of generating useful example pairs or tuples
(Suh et al., 2019). In contrast, our method does not require mining strategies
and, as shown in the experiments, it converges much faster than competing
losses.

The usual example mining strategies are hard, semi-hard, and online negative
mining (Schroff et al., 2015; Wu et al., 2017). In hard negative mining, for
each anchor image, we need to construct pairs with the farthest positive
example and the closest negative example. This adds an extra computational
step at the beginning of every training epoch, extending the training time.
Similar problems arise in the context of semi-hard negative mining, while the
difference consists in the mode in which the negatives are sampled. Instead of
generating pairs with the farthest example, one can choose a negative example
that is slightly closer than a positive sample, thus balancing hardness and
stability. A more efficient approach is online negative mining, where negative
examples are sampled during training in each batch. In this approach, the pair
difficulty adjusts while the model is training, thus leading to a more
efficient strategy, but the main disadvantage is that the resulting pairs may
not be the most challenging for the model, due to the randomness of samples in
a batch.

Statistical learning objective functions indirectly optimize the learned
features of the neural network. Popular objectives are based on some variation
of the cross-entropy loss (Deng et al., 2019; Liu et al., 2017; Wang et al.,
2018b, a; Liu et al., 2016; Elezi et al., 2022) or the cosine loss (Barz and
Denzler, 2020). By optimizing such functions, the model is forced to generate
features that are close to the direction of the class center. For example,
ArcFace (Deng et al., 2019) reduces the optimization space to an
$n$-dimensional hypersphere by normalizing both the embedding generated by the
encoder, and the corresponding class weight from the classification head,
using their Euclidean norm. An additive penalty term is introduced, and with
the proposed normalization, the optimization is performed on the angle between
each feature vector and the corresponding class center.

Hybrid objective functions promise to obtain better embeddings by minimizing a
statistical objective function in conjunction with a metric learning objective
(Min et al., 2020; Khosla et al., 2020). For example, Center Loss (Wen et al.,
2016) minimizes the intra-class distances of the learned features by using
cross-entropy in conjunction with an attractor for each sample to its
corresponding class center. During training, the class centers are updated to
become the mean of the feature vectors for every class seen in a batch.
Another approach (Zhu et al., 2019) similar to Center Loss (Wen et al., 2016)
is based on predefined evenly distributed class centers. A more complex
approach (Cao et al., 2020) is to combine the standard cross-entropy with a
cosine classifier and a mean squared error regression term to jointly enhance
global and local features.

Both contrastive and triplet loss objectives suffer from the need of employing
pair mining strategies, but in our case, mining strategies are not required.
The positive pairs are built online for each batch, between each input feature
vector and the dedicated class anchor, the number of positive pairs thus being
equal to the number of examples. The negative pairs are constructed only
between the class centers, thus alleviating the need of searching for a good
negative mining strategy, while also significantly reducing the number of
negative pairs. To the best of our knowledge, there are no alternative
objective functions for CBIR that use dedicated self-repelling learnable class
anchors acting as attraction poles for feature vectors belonging to the
respective classes.

Content-based image retrieval methods. CBIR systems are aimed at finding
similar images with a given query image, matching the images based on the
similarity of their scenes or the contained objects. Images are encoded using
a descriptor (or encoder), and a system is used to sort a database of encoded
images based on a similarity measure between queries and images. In the
context of content-based image retrieval, there are two types of image
descriptors. On the one hand, there are general descriptors (Radenović et al.,
2019), where a whole image is represented as a feature vector. On the other
hand, there are local descriptors (Wu et al., 2021) where portions of an image
are represented as feature vectors. Hybrid descriptors (Cao et al., 2020) are
also used to combine both global and local features. To improve the quality of
the results retrieved by learned global descriptors, an additional
verification step is often employed. This step is meant to re-rank the
retrieved images by a precise evaluation of each candidate image (Polley et
al., 2022). The re-ranking step is usually performed with the help of an
additional system, and in some cases, it can be directly integrated with the
general descriptor (Lee et al., 2022). In the CBIR task, one can search for
visually similar images as a whole, or search for images that contain similar
regions (Philbin et al., 2007) of a query image. In this work, we focus on
building global descriptors that match whole images. Further approaches based
on metric learning, statistical learning, or hand-engineered features are
discussed in the recent survey of Dubey et al. (Dubey, 2021). Different from
other CBIR methods, we propose a novel two-stage retrieval system that
leverages the use of the class anchors learned through the proposed loss
function to make the retrieval process more efficient and effective.

##  3\. Method

Overview. Our objective function consists of three components, each playing a
different role in obtaining a discriminative embedding space. All three loss
components are formulated with respect to a set of learnable class anchors
(centroids). The first loss component acts as an attraction force between each
input embedding and its corresponding class anchor. Its main role is to draw
the embeddings representing the same object to the corresponding centroid,
thus creating embedding clusters of similar images. Each center can be seen as
a magnet with a positive charge and its associated embeddings as magnets with
negative charges, thus creating attraction forces between anchors and data
samples of the same class. The second loss component acts as a repelling force
between class anchors. In this case, the class centers can be seen as magnets
with similar charges. If brought together, they will repel each other, and if
they lie at a certain distance, the repelling force stops. The last component
acts similarly to the second one, with the difference that an additional
magnet is introduced and fixed at the origin of the embedding space. Its main
effect is to push the class centers away from the origin.

Notations. Let $\mathbf{x}_{i}\in\mathbb{R}^{h\times w\times c}$ be an input
image and $y_{i}\in\mathbb{N}$ its associated class label, $\forall
i\in\\{1,2,...,l\\}$. We aim to optimize a neural encoder $f_{\theta}$ which
is parameterized by the learnable weights $\theta$ to produce a discriminative
embedding space. Let $\mathbf{e}_{i}\in\mathbb{R}^{n}$ be the $n$-dimensional
embedding vector of the input image $\mathbf{x}_{i}$ generated by
$f_{\theta}$, i.e. $\mathbf{e}_{i}=f_{\theta}(\mathbf{x}_{i})$. In order to
employ our novel loss function, we need to introduce a set of learnable class
anchors $C=\\{\mathbf{c}_{1},\mathbf{c}_{2},...,\mathbf{c}_{t}\\}$, where
$\mathbf{c}_{j}\in\mathbb{R}^{n}$ resides in the embedding space of
$f_{\theta}$, and $t$ is the total number of classes.

Figure 1. An example showing the behavior of the attractor-repeller loss
components for three classes. The stars represent the class anchors $C$. Faded
circles around class anchors represent the sphere of radius $m$ around each
anchor. Solid circles represent feature vectors generated by the encoder
$f_{\theta}$. Dashed arrows between feature vectors and class anchors
represent the attraction of the force generated by the attractor
$\mathcal{L}_{A}$. The solid red arrow between class anchors represents the
repelling force generated by the repeller $\mathcal{L}_{R}$. Best viewed in
color.

Loss components. With the above notations, we can now formally define our
first component, the attractor loss $\mathcal{L}_{A}$, as follows:

(1)
$\mathcal{L}_{A}(\mathbf{x}_{i},C)=\frac{1}{2}\|\mathbf{e}_{i}-\mathbf{c}_{y_{i}}\|_{2}^{2}.$

The main goal of this component of the objective is to cluster feature vectors
as close as possible to their designated class anchor by minimizing the
distance between $\mathbf{e}_{i}$ and the corresponding class anchor
$\mathbf{c}_{y_{i}}$. Its effect is to enforce low intra-class variance.
However, obtaining low intra-class variance is only a part of what we aim to
achieve. The first objective has little influence over the inter-class
similarity, reducing it only indirectly. Therefore, another objective is
required to repel samples from different classes. As such, we introduce the
repeller loss $\mathcal{L}_{R}$, which is defined as follows:

(2) $\mathcal{L}_{R}(C)=\frac{1}{2}\sum_{y,y^{\prime}\in Y,y\neq
y^{\prime}}\left\\{\max\left(0,2\cdot
m-\|\mathbf{c}_{y}-\mathbf{c}_{y^{\prime}}\|\right)\right\\}^{2},$

where $y$ and $y^{\prime}$ are two distinct labels from the set of ground-
truth labels $Y$, and $m>0$ is a margin representing the radius of an
$n$-dimensional sphere around each anchor, in which no other anchor should
lie. The goal of this component is to push anchors away from each other during
training to ensure high inter-class distances. The margin $m$ is used to limit
the repelling force to an acceptable margin value. If we do not set a maximum
margin, then the repelling force can push the anchors too far apart, and the
encoder could struggle to learn features that satisfy the attractor loss
defined in Eq. (1).

Figure 2. Contribution of the minimum norm loss $\mathcal{L}_{N}$ imposed on
the class anchors. The blue star represents a class anchor. Solid circles
represent embedding vectors generated by the encoder $f_{\theta}$. Dashed
arrows represent the attraction force generated by the attractor
$\mathcal{L}_{A}$. The solid gray line represents the direction in which the
anchor is pushed away from the origin due to the component $\mathcal{L}_{N}$.
Best viewed in color.

Figure 3. Embedding vectors (left) and their distribution (right) for a
ResNet-18 model modified to output 2D features and trained with $(a)$ cross-
entropy loss, $(b)$ contrastive loss and $(c)$ class anchor margin loss
(ours). The top row represents training embeddings from the SVHN dataset,
while the bottom row shows test embeddings. Best viewed in color.

A toy example of the attractor-repeller mechanism is depicted in Figure 1.
Notice how the optimization based on the attractor-repeller objective tends to
pull data samples from the same class together (due to the attractor), and
push samples from different classes away (due to the repeller). However, when
the training begins, all data samples start from a location close to the
origin of the embedding space, essentially having a strong tendency to pull
the class anchors to the origin. To ensure that the anchors do not collapse to
the origin (as observed in some of our preliminary experiments), we introduce
an additional objective that imposes a minimum norm on the class anchors. The
minimum norm loss $\mathcal{L}_{N}$ is defined as:

(3) $\mathcal{L}_{N}(C)=\frac{1}{2}\sum_{y\in
Y}\left\\{\max\left(0,p-\|\mathbf{c}_{y}\|\right)\right\\}^{2},$

where $p$ is the minimum norm that each anchor must have. This objective
contributes to our full loss function as long as at least one class anchor is
within a distance of $p$ from the origin. Figure 2 provides a visual
interpretation of the effect induced by the minimum norm loss. Notice how the
depicted class anchor is pushed away from the origin (due to the minimum norm
loss), while the data samples belonging to the respective class move along
with their anchor (due to the attractor loss).

Assembling the three loss components presented above into a single objective
leads to the proposed class anchor margin (CAM) loss
$\mathcal{L}_{\scriptsize{\mbox{CAM}}}$, which is formally defined as follows:

(4)
$\mathcal{L}_{\scriptsize{\mbox{CAM}}}(\mathbf{x},C)=\mathcal{L}_{A}(\mathbf{x},C)+\mathcal{L}_{R}(C)+\mathcal{L}_{N}(C).$

Notice that only $\mathcal{L}_{A}$ is directly influenced by the training
examples, while $\mathcal{L}_{R}$ and $\mathcal{L}_{N}$ operate only on the
class anchors. Hence, negative mining strategies are not required at all.

Gradients of the proposed loss. We emphasize that the $L_{2}$ norm is
differentiable. Moreover, the updates for the weights $\theta$ of the encoder
$f_{\theta}$ are provided only by the attractor loss $\mathcal{L}_{A}$. Hence,
the weight updates (gradients of $\mathcal{L}_{\scriptsize{\mbox{CAM}}}$ with
respect to $\theta$) for some data sample $\mathbf{x}_{i}$ are computed as
follows:

(5)
$\frac{\partial\mathcal{L}_{\scriptsize{\mbox{CAM}}}(\mathbf{x}_{i},C)}{\partial\theta}=\frac{\partial\mathcal{L}_{A}(\mathbf{x}_{i},C)}{\partial\theta}=\left(f_{\theta}(\mathbf{x}_{i})-\mathbf{c}_{y_{i}}\right)\cdot\frac{\partial
f_{\theta}(\mathbf{x}_{i})^{T}}{\partial\theta}.$

The class anchors receive updates from all three loss components. For a class
anchor $\mathbf{c}_{y}$, the update received from the component
$\mathcal{L}_{A}$ of the joint objective is given by:

(6)
$\frac{\partial\mathcal{L}_{A}(\mathbf{x},C)}{\partial\mathbf{c}_{y}}=\mathbf{c}_{y}-f_{\theta}(\mathbf{x}).$

The contribution of the repeller to a class anchor $\mathbf{c}_{y}$ is null
when the $m$-margin sphere of the respective anchor is not intersecting
another class anchor sphere, i.e.:

(7) $\frac{\partial\mathcal{L}_{R}(C)}{\partial C}=0,$

when $2\cdot m-\|\mathbf{c}_{y}-\mathbf{c}_{y^{\prime}}\|>0$, $\forall
y,y^{\prime}\in Y,y\neq y^{\prime}$.

To simply the notation, let $d=\|\mathbf{c}_{y}-\mathbf{c}_{y^{\prime}}\|$ be
the distance between $\mathbf{c}_{y}$ and another class center
$\mathbf{c}_{y^{\prime}}$. The update for a class center $\mathbf{c}_{y}$ of
the repeller is given by:

(8)
$\frac{\partial\mathcal{L}_{R}(C)}{\partial\mathbf{c}_{y}}=\\!\sum_{y,y^{\prime}\in
Y,y\neq
y^{\prime}}\\!\delta(d<2\\!\cdot\\!m)\cdot\frac{-(2\\!\cdot\\!m-d)\\!\cdot\\!(\mathbf{c}_{y}-\mathbf{c}_{y^{\prime}})}{d},$

where $\delta(*)=1$ when $*$ is satisfied, and $0$ otherwise. Similarly, the
contribution of the minimum norm loss is given by:

(9) $\frac{\partial\mathcal{L}_{N}(C)}{\partial\mathbf{c}_{y}}=\sum_{y\in
Y}\delta\left(\|\mathbf{c}_{y}\|<p\right)\cdot\left(1-\frac{p}{\|\mathbf{c}_{y}\|}\right)\cdot\mathbf{c}_{y}.$

The final update of the learnable class centers $C$ is given by summing the
gradients given in Eq. (6), Eq. (8) and Eq. (9):

(10)
$\frac{\partial\mathcal{L}(\mathbf{x},C)}{\partial\mathbf{c}_{y}}=\frac{\partial\mathcal{L}_{A}(\mathbf{x},C)}{\partial\mathbf{c}_{y}}+\frac{\partial\mathcal{L}_{R}(C)}{\partial\mathbf{c}_{y}}+\frac{\partial\mathcal{L}_{N}(C)}{\partial\mathbf{c}_{y}}.$

Fast retrieval via two-stage system. During inference, we can employ the
$L_{2}$ measure between query and database embeddings to retrieve images
similar to the query. Aside from the brute-force search that compares the
query with every embedding in the database, we can harness the geometric
properties of the resulting embedding space and use the class anchors $C$ to
improve the speed of the retrieval system. Instead of computing the $L_{2}$
distances between a query feature vector and all the embedding vectors stored
in the database, we propose to employ a two-stage system. In the first stage,
distances are computed between the query feature vector and each class anchor.
Upon finding the closest class anchor, in the second stage, distances are
computed between the query feature vector and all the embeddings associated
with the established class. Distances are then sorted and the closest
corresponding items are retrieved. The main advantage of this approach is an
improved retrieval time due to the reduced number of required comparisons. A
possible disadvantage consists in retrieving wrong items when the retrieved
class anchor is not representative for the query image. We present results
with our faster alternative in the comparative experiments.

Visualizing the embedding space. To emphasize the geometric effect of the
proposed objective, we have modified the final encoder layer of a ResNet-18
model to output 2D feature vectors without the final non-linearity. The model
is trained from scratch on the SVHN dataset by alternatively using the cross-
entropy loss, the contrastive loss, and the proposed class anchor margin loss.
The resulting embeddings, together with the distribution of the embeddings for
each of the models, are presented in Figure 3. For the proposed objective, we
have set $m=4$ and $p=1$. From the distribution of the embeddings, it can be
noted that the proposed objective generates tighter and clearly separable
clusters.

Application to classification tasks. An important remark is that we can use
the class centers $C$ to apply the proposed system to classification tasks. In
this case, the predicted label $\hat{y}$ for a test sample $\mathbf{x}$ can be
obtained as follows:

(11) $\hat{y}=\arg\min_{j}\|f_{\theta}(\mathbf{x})-\mathbf{c}_{j}\|.$

We perform additional experiments to demonstrate the application of our class
anchor margin loss to classification tasks, with competitive results.

##  4\. Experiments

In the experiments, we compare our class anchor margin loss with the cross-
entropy and the contrastive learning losses on four datasets, considering both
convolutional and transformer models.

###  4.1. Datasets

We perform experiments on four datasets: CIFAR-100 (Krizhevsky, 2009),
Food-101 (Bossard et al., 2014), SVHN (Netzer et al., 2011), and Tiny ImageNet
(Russakovsky et al., 2015). CIFAR-100 contains 50,000 training images and
10,000 test images belonging to 100 classes. Food-101 is composed of 101,000
images from 101 food categories. The official split has $750$ training images
and $250$ test images per category. SVHN contains 73,257 digits for training,
26,032 digits for testing. Tiny ImageNet is a subset of ImageNet-1K, which
contains 100,000 training images, 25,000 validation images and 25,000 test
images from 200 classes.

###  4.2. Experimental setup

As underlying neural architectures, we employ three ResNet (He et al., 2016)
model variations (ResNet-18, ResNet-50, ResNet-101) and a Swin transformer
(Liu et al., 2021) model (Swin-T). We rely on the PyTorch (Paszke et al.,
2019) library together with Hydra (Yadan, 2019) to implement and test the
models.

We apply random weight initialization for all models, except for the Swin
transformer, which starts from the weights pre-trained on the ImageNet Large
Scale Visual Recognition Challenge (ILSVRC) dataset (Russakovsky et al.,
2015). We employ the Adam (Kingma and Ba, 2015) optimizer to train all models,
regardless of their architecture. For the residual neural models, we set the
learning rate to $10^{-3}$, while for the Swin transformer, we use a learning
rate of $10^{-4}$. The residual nets are trained from scratch for 100 epochs,
while the Swin transformer is fine-tuned for 30 epochs. For the lighter models
(ResNet-18 and ResNet-50), we use a mini-batch size of 512. Due to memory
constraints, the mini-batch size is set to 64 for the Swin transformer, and
128 for ResNet-101. Residual models are trained with a linear learning rate
decay with a factor of $0.5$. We use a patience of 6 epochs for the full-set
experiments, and a patience of 9 epochs for the few-shot experiments. Fine-
tuning the Swin transformer does not employ a learning rate decay. Input
images are normalized to have all pixel values in the range of $[0,1]$ by
dividing the values by 256. The inputs of the Swin transformer are
standardized with the image statistics from ILSVRC (Russakovsky et al., 2015).

Table 1. Retrieval performance levels of ResNet-18, ResNet-50, ResNet-101 and Swin-T models on the CIFAR-100 (Krizhevsky, 2009), Food-101 (Bossard et al., 2014), SVHN (Netzer et al., 2011), and Tiny ImageNet (Russakovsky et al., 2015) datasets, while comparing the cross-entropy (CE) loss, the contrastive learning (CL) loss, and the class anchor margin (CAM) loss. The best score for each architecture and each metric is highlighted in bold. | Loss | CIFAR-100 | Food-101 | SVHN | Tiny ImageNet
---|---|---|---|---|---
| mAP | P@20 | P@100 | mAP | P@20 | P@100 | mAP | P@20 | P@100 | mAP | P@20 | P@100
ResNet-18 | CE |  .249±.003 |  .473±.005 |  .396±.005 |  .234±.002 |  .547±.002 |  .459±.001 |  .895±.013 | .954±.003 |  .952±.003 |  .130±.001 |  .340±.001 |  .262±.001
CL |  .220±.013 |  .341±.010 |  .303±.011 |  .025±.001 |  .042±.004 |  .038±.003 |  .116±.000 |  .115±.002 |  .116±.001 |  .070±.012 |  .139±.017 |  .117±.016
CAM (ours) | .622±.005 | .560±.007 | .553±.007 | .751±.001 | .676±.003 | .669±.003 | .967±.001 | .954±.000 | .953±.001 | .508±.004 | .425±.004 | .418±.005
ResNet-50 | CE |  .211±.006 |  .454±.007 |  .366±.006 |  .158±.004 |  .471±.005 |  .370±.005 |  .909±.010 | .958±.001 |  .956±.002 |  .088±.003 |  .292±.006 |  .209±.005
CL |  .164±.016 |  .271±.017 |  .240±.016 |  .019±.000 |  .030±.000 |  .028±.001 |  .372±.310 |  .447±.361 |  .437±.364 |  .005±.000 |  .008±.001 |  .007±.000
CAM (ours) | .640±.008 | .581±.009 | .578±.009 | .765±.005 | .697±.008 | .697±.009 | .969±.001 |  .956±.001 | .957±.001 | .543±.003 | .472±.002 | .468±.002
ResNet-101 | CE |  .236±.009 |  .482±.008 |  .397±.009 |  .160±.003 |  .479±.008 |  .376±.007 |  .936±.003 | .958±.001 | .957±.001 |  .093±.002 |  .299±.002 |  .216±.002
CL |  .034±.028 |  .069±.051 |  .056±.044 |  .014±.002 |  .018±.004 |  .017±.003 |  .595±.242 |  .700±.295 |  .696±.293 |  .006±.001 |  .007±.002 |  .007±.002
CAM (ours) | .629±.006 | .575±.008 | .573±.008 | .758±.007 | .690±.007 | .693±.006 | .969±.001 |  .956±.001 |  .956±.001 | .529±.005 | .458±.006 | .455±.006
Swin-T | CE |  .661±.004 | .808±.001 |  .770±.001 |  .617±.006 |  .817±.002 |  .777±.003 |  .963±.001 | .968±.000 | .968±.000 |  .560±.006 |  .743±.001 |  .707±.003
CL |  .490±.010 |  .629±.007 |  .584±.008 |  .495±.004 |  .667±.001 |  .633±.002 |  .895±.000 |  .928±.001 |  .924±.000 |  .223±.002 |  .397±.001 |  .338±.001
CAM (ours) | .808±.004 |  .794±.003 | .795±.004 | .873±.001 | .858±.001 | .861±.001 | .967±.002 |  .944±.001 |  .946±.001 | .761±.002 | .745±.003 | .749±.004

Figure 4. Performance (mAP) of ResNet-50 models with different embedding
dimensions from 32 to 2048, trained on CIFAR-100 from scratch. Results are
shown for three losses: cross-entropy, contrastive, and class anchor margin
(ours). Best viewed in color.

Figure 5. Converge speed of ResNet-50 models trained on CIFAR-100 for 100
epochs using alternative losses: cross-entropy, contrastive, and class anchor
margin (ours). Best viewed in color.

We use several data augmentation techniques such as random crop with a padding
of 4 pixels, random horizontal flip (except for SVHN, where flipping the
digits horizontally does not make sense), color jitter, and random affine
transformations (rotations, translations). Moreover, for the Swin transformer,
we added the augmentations described in (Muller and Hutter, 2021).

For the models optimized either with the cross-entropy loss or our class
anchor margin loss, the target metric is the validation accuracy. When using
our loss, we set the parameter $m$ for the component $\mathcal{L}_{R}$ to $2$,
and the parameter $p$ for the component $\mathcal{L}_{N}$ to $1$, across all
datasets and models. Since the models based on the contrastive loss optimize
in the feature space, we have used the $1$-nearest neighbors ($1$-NN)
accuracy, which is computed for the closest feature vector retrieved from the
gallery.

As evaluation measures for the retrieval experiments, we report the mean
Average Precision (mAP) and the precision$@k$ on the test data, where
$k\in\\{20,100\\}$ is the retrieval rank. For the classification experiments,
we use the classification accuracy on the test set. We run each experiment in
5 trials and report the average score and the standard deviation.

###  4.3. Retrieval results with full training data

Results with various architectures. We first evaluate the performance of the
ResNet-18, ResNet-50, ResNet-101 and Swin-T models on the CBIR task, while
using the entire training set to optimize the respective models with three
alternative losses, including our own. The results obtained on the CIFAR-100
(Krizhevsky, 2009), Food-101 (Bossard et al., 2014), SVHN (Netzer et al.,
2011), and Tiny ImageNet (Russakovsky et al., 2015) datasets are reported in
Table 1. First, we observe that our loss function produces better results on
the majority of datasets and models. Furthermore, as the rank $k$ increases
from 20 to 100, we notice that our loss produces more consistent results,
essentially maintaining the performance level as $k$ increases.

Figure 6. Few-shot retrieval performance (mAP) of four models (ResNet-18,
ResNet-50, ResNet-101 and Swin-T) on four datasets (CIFAR-100, Food-101, SVHN
and Tiny ImageNet). On each dataset, the results are shown from one sample per
class (one-shot learning) to the maximum number of samples per class, by
doubling the number of training samples in each trial.

Results with various embedding dimensions. We train a ResNet-50 encoder from
scratch, where the last encoder layer is modified to output embeddings of
various dimensions from the set $\\{32,64,128,256,512,1024,2048\\}$ on the
CIFAR-100 (Krizhevsky, 2009) dataset. We illustrate the results with the
cross-entropy, contrastive learning and class anchor margin losses in Figure
4. We observe that the performance is degrading as the embedding size gets
larger for models trained with cross-entropy or contrastive losses. In
contrast, the presented results show that the performance is consistent for
the ResNet-50 based on our objective. This indicates that our model is more
robust to variations of the embedding space dimension $n$.

Convergence speed. We have also monitored the convergence speed of ResNet-50
models, while alternatively optimizing with cross-entropy, contrastive
learning and class anchor margin losses. In Figure 5, we show how the
ResNet-50 models converge over 100 training epochs. The model trained with
contrastive loss exhibits a very slow convergence speed. The model trained
with cross-entropy achieves its maximum performance faster than the model
trained with contrastive-loss, but the former model reaches a plateau after
about 60 epochs. The model trained with our CAM loss converges at a faster
pace compared with the other models.

###  4.4. Few-shot retrieval results

We next evaluate the neural models on the few-shot object retrieval task. For
each dataset, we sample a certain number of training images from each class,
starting from one example per class. We gradually increase the number of
training samples, doubling the number of training examples per class in each
experiment, until we reach the maximum amount of available images. In Figure
6, we present the corresponding results on CIFAR-100 (Krizhevsky, 2009),
Food-101 (Bossard et al., 2014), SVHN (Netzer et al., 2011), and Tiny ImageNet
(Russakovsky et al., 2015).

For all three ResNet models, our CAM loss leads to better results, once the
number of training samples becomes higher or equal to $4$ per class. In all
cases, the contrastive loss obtains the lowest performance levels, being
surpassed by both cross-entropy and class anchor margin losses. For the Swin-T
model, cross-entropy usually leads to better results when the number of
samples per class is in the lower range (below $64$). As the number of
training samples increases, our loss recovers the gap and even surpasses
cross-entropy after a certain point (usually when the number of samples per
class is higher than $128$). In general, all neural models obtain increasingly
better performance levels when more data samples are available for training.
With some exceptions, the models trained with our CAM loss achieve the best
performance. In summary, we conclude that the class anchor margin loss is
suitable for few-shot retrieval, but the number of samples per class should be
generally higher than $4$ to obtain optimal performance.

Figure 7. Top 6 retrieved items by a ResNet-50 model trained with one of three
losses: cross-entropy (CE), contrastive (CL), and class anchor margin (CAM).
We randomly selected two queries per dataset. Best viewed in color.

###  4.5. Qualitative results

We choose the ResNet-50 model and inspect the retrieved images for each of the
three loss functions. In Figure 7, we show a set of eight randomly sampled
queries from the four datasets (CIFAR-100, Food-101, SVHN, and Tiny ImageNet).
The model based on our loss seems to return more consistent results, with the
majority of images representing the same object as the query. In contrast,
models trained with the other losses can sometimes retrieve items that do not
always belong to the query category. Overall, the qualitative results confirm
the superiority of our class anchor margin loss.

Table 2. Retrieval performance for brute-force and two-stage retrieval systems on CIFAR-100, Food-101, SVHN and Tiny ImageNet. Results are reported for four models: ResNet-18, ResNet-50, ResNet-101 and Swin-T. The running times are measured on a machine with an Intel Xeon v4 3.0GHz CPU, 256GB of RAM, and an Nvidia GeForce GTX 1080Ti GPU with 11GB of VRAM. Model | Method | CIFAR-100 | Food-101 | SVHN | Tiny ImageNet
---|---|---|---|---|---
mAP | time (ms) | mAP | time (ms) | mAP | time (ms) | mAP | time (ms)
ResNet-18 | brute-force | .476±.021 | 6.4±0.4 | .534±.003 | 9.7±0.9 | .961±.001 | 8.8±0.9 | .328±.009 | 11.1±0.9
two-stage | .622±.005 | 2.5±0.7 | .751±.001 | 3.2±0.8 | .967±.001 | 3.2±0.8 | .508±.004 | 2.5±0.7
ResNet-50 | brute-force | .528±.016 | 14.4±1.8 | .609±.017 | 23.2±2.0 | .963±.001 | 19.0±2.1 | .414±.003 | 25.3±3.0
two-stage | .640±.008 | 5.1±0.7 | .765±.005 | 9.0±0.8 | .969±.001 | 6.8±1.2 | .543±.003 | 5.1±0.8
ResNet-101 | brute-force | .525±.014 | 18.8±1.8 | .618±.005 | 28.8±2.0 | .960±.003 | 23.3±2.1 | .400±.012 | 28.6±2.5
two-stage | .629±.006 | 9.4±0.7 | .758±.007 | 14.6±0.8 | .969±.001 | 11.2±1.2 | .529±.005 | 9.5±0.7
Swin-T | brute-force | .762±.005 | 11.8±0.9 | .811±.002 | 14.9±0.9 | .943±.001 | 14.3±0.9 | .684±.008 | 17.2±1.3
two-stage | .808±.004 | 6.9±0.8 | .873±.001 | 7.2±0.8 | .967±.002 | 7.7±0.9 | .761±.002 | 6.9±0.8
Table 3. Classification and retrieval results while ablating the proposed class anchor initialization and various components of our CAM loss. Results are shown for a ResNet-18 model on SVHN. $\mathcal{L}_{R}$ | $\mathcal{L}_{N}$ | Anchors init | Accuracy | mAP
---|---|---|---|---
$\times$ | $\times$ | random | 19.56% | 0.116
base vectors | 66.74% | 0.483
$\times$ | ✓ | random | 19.58% | 0.116
base vectors | 66.95% | 0.483
✓ | $\times$ | random | 93.34% | 0.889
base vectors | 94.05% | 0.901
✓ | ✓ | random | 93.64% | 0.892
base vectors | 96.41% | 0.967

###  4.6. Ablation study

Ablating the loss. In Table 3, we demonstrate the influence of each additional
loss component on the overall performance of ResNet-18 on the SVHN dataset, by
ablating the respective components from the proposed objective. We emphasize
that the component $\mathcal{L}_{A}$ is mandatory for our objective to work
properly. Hence, we only ablate the other loss components, namely
$\mathcal{L}_{R}$ and $\mathcal{L}_{N}$.

In addition, we investigate different class center initialization heuristics.
As such, we conduct experiments to compare the random initialization of class
centers and the base vector initialization. The latter strategy is based on
initializing class anchors as scaled base vectors, such that each class center
has no intersection with any other class center in the $n$-dimensional sphere
of radius $m$, where $n$ is the size of the embedding space.

As observed in Table 3, the class center initialization has a major impact on
the overall performance. For each conducted experiment, we notice a
significant performance gain for the base vector initialization strategy.
Regarding the loss components, we observe that removing both $\mathcal{L}_{R}$
and $\mathcal{L}_{N}$ from the objective leads to very low performance. Adding
only the component $\mathcal{L}_{N}$ influences only the overall accuracy, but
the mAP is still low, since $\mathcal{L}_{N}$ can only impact each anchor’s
position with respect to the origin. Adding only the component
$\mathcal{L}_{R}$ greatly improves the performance, proving that
$\mathcal{L}_{R}$ is crucial for learning the desired task. Using both
$\mathcal{L}_{R}$ and $\mathcal{L}_{N}$ further improves the results,
justifying the proposed design.

Ablating the two-stage retrieval system. As discussed in Section 3, we employ
a two-stage retrieval system to speed up the retrieval process. We hereby
ablate the proposed two-stage approach that leverages the class anchors,
essentially falling back to the brute-force retrieval process, in which the
query is compared with every embedding vector from the database. We compare
the ablated (brute-force) retrieval with the two-stage retrieval in Table 2.
Remarkably, we observe that our two-stage retrieval system not only improves
the retrieval speed, but also the mAP. In terms of time, the two-stage
retrieval improves the speed by a factor ranging between $2\times$ and
$3\times$. In terms of performance, the gains are higher than $10\%$ in 9 out
of 16 cases. These results illustrate the benefits of our two-stage retrieval
process based on leveraging the class anchors.

Table 4. Classification accuracy of class anchor margin loss vs. cross-entropy loss on CIFAR-100, Food-101, SVHN and Tiny ImageNet. Results are reported for two models, ResNet-18 and ResNet-50. Dataset | Loss | Model
---|---|---
ResNet-18 | ResNet-50
CIFAR-100 | CE | 60.53%±0.18 | 61.88%±0.66
CAM (ours) | 61.94%±0.49 | 63.80%±0.76
Food-101 | CE | 74.66%±0.20 | 76.84%±0.37
CAM (ours) | 75.02%±0.04 | 76.37%±0.52
SVHN | CE | 96.11%±0.11 | 96.30%±0.05
CAM (ours) | 96.41%±0.01 | 96.6%3±0.07
Tiny ImageNet | CE | 50.82%±0.33 | 52.68%±0.65
CAM (ours) | 50.67%±0.41 | 54.26%±0.29

###  4.7. Classification results

As earlier mentioned, we can leverage the use of the learned class centers and
the predictions generated with Eq. (11) to classify objects into the
corresponding categories. We present the classification accuracy rates of the
ResNet-18 and ResNet-50 models in Table 4, while alternating between the
cross-entropy and the class anchor margin losses. Our CAM loss provides
competitive results, surpassing cross-entropy in 6 out of 8 cases. This
confirms that our loss is also suitable for the classification task, even
though its performance gains are not as high as for the retrieval task.

##  5\. Conclusion

In this paper, we proposed $(i)$ a novel loss function based on class anchors
to optimize convolutional networks and transformers for object retrieval in
images, as well as $(ii)$ a two-stage retrieval system that leverages the
learned class anchors to speed up and increase the performance of the system.
We conducted comprehensive experiments using four neural models on four image
datasets, demonstrating the benefits of our loss function against conventional
losses based on statistical learning and contrastive learning. We also
performed ablation studies to showcase the influence of the proposed
components, empirically justifying our design choices.

In future work, we aim to extend the applicability of our approach to other
data types, beyond images. We also aim to explore new tasks and find out when
our loss is likely to outperform the commonly used cross-entropy.

## References

  * (1)
  * Barz and Denzler (2020) Björn Barz and Joachim Denzler. 2020.  Deep Learning on Small Datasets without Pre-Training using Cosine Loss. In _Proceedings of WACV_. IEEE, 1360–1369.
  * Bossard et al. (2014) Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. 2014\.  Food-101 – Mining Discriminative Components with Random Forests. In _Proceedings of ECCV_. Springer, 446–461.
  * Cao et al. (2020) Bingyi Cao, Andre Araujo, and Jack Sim. 2020.  Unifying deep local and global features for image search. In _Proceedings of ECCV_. Springer, 726–743.
  * Chen et al. (2017) Weihua Chen, Xiaotang Chen, Jianguo Zhang, and Kaiqi Huang. 2017\.  Beyond Triplet Loss: A Deep Quadruplet Network for Person Re-identification. In _Proceedings of CVPR_. IEEE, 1320–1329.
  * Cortes and Vapnik (1995) Corinna Cortes and Vladimir Vapnik. 1995.  Support-vector networks.  _Machine Learning_ 20, 3 (1995), 273–297.
  * Deng et al. (2019) Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. 2019\.  ArcFace: Additive Angular Margin Loss for Deep Face Recognition. In _Proceedings of CVPR_. IEEE, 4685–4694.
  * Dubey (2021) Shiv Ram Dubey. 2021\.  A decade survey of content based image retrieval using deep learning.  _IEEE Transactions on Circuits and Systems for Video Technology_ 32, 5 (2021), 2687–2704.
  * Elezi et al. (2022) Ismail Elezi, Jenny Seidenschwarz, Laurin Wagner, Sebastiano Vascon, Alessandro Torcinovich, Marcello Pelillo, and Laura Leal-Taixe. 2022\.  The Group Loss++: A deeper look into group loss for deep metric learning.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 45, 2 (2022), 2505–2518.
  * Gajić et al. (2022) Bojana Gajić, Ariel Amato, Ramon Baldrich, Joost van de Weijer, and Carlo Gatta. 2022\.  Area under the roc curve maximization for metric learning. In _Proceedings of CVPR_. IEEE, 2807–2816.
  * Georgescu et al. (2022) Mariana-Iuliana Georgescu, Georgian-Emilian Duţǎ, and Radu Tudor Ionescu. 2022.  Teacher-student training and triplet loss to reduce the effect of drastic face occlusion: Application to emotion recognition, gender identification and age estimation.  _Machine Vision and Applications_ 33, 1 (2022), 12\.
  * Georgescu and Ionescu (2021) Mariana-Iuliana Georgescu and Radu Tudor Ionescu. 2021.  Teacher-student training and triplet loss for facial expression recognition under occlusion. In _Proceedings of ICPR_. IEEE, 2288–2295.
  * Hadsell et al. (2006) Raia Hadsell, Sumit Chopra, and Yann LeCun. 2006\.  Dimensionality Reduction by Learning an Invariant Mapping. In _Proceedings of CVPR_ , Vol. 2. IEEE, 1735–1742.
  * Harwood et al. (2017) Ben Harwood, Vijay B.G. Kumar, Gustavo Carneiro, Ian Reid, and Tom Drummond. 2017.  Smart mining for deep metric learning. In _Proceedings of ICCV_. IEEE, 2821–2829.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016\.  Deep Residual Learning for Image Recognition. In _Proceedings of CVPR_. IEEE, 770–778.
  * Khosla et al. (2020) Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020.  Supervised Contrastive Learning. In _Proceedings of NeurIPS_ , Vol. 33. Curran Associates, Inc., 18661–18673.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A Method for Stochastic Optimization. In _Proceedings of ICLR_.
  * Krizhevsky (2009) Alex Krizhevsky. 2009\.  _Learning multiple layers of features from tiny images_.  Technical Report. University of Toronto.
  * Lee et al. (2022) Seongwon Lee, Hongje Seong, Suhyeon Lee, and Euntai Kim. 2022\.  Correlation verification for image retrieval. In _Proceedings of CVPR_. IEEE, 5374–5384.
  * Liu et al. (2017) Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. 2017\.  SphereFace: Deep Hypersphere Embedding for Face Recognition. In _Proceedings of CVPR_. IEEE, Los Alamitos, CA, USA, 6738–6746.
  * Liu et al. (2016) Weiyang Liu, Yandong Wen, Zhiding Yu, and Meng Yang. 2016\.  Large-Margin Softmax Loss for Convolutional Neural Networks. In _Proceedings of ICML_. JMLR.org, 507–516.
  * Liu et al. (2021) Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021.  Swin Transformer: Hierarchical vision transformer using shifted windows. In _Proceedings of ICCV_. IEEE, 10012–10022.
  * Min et al. (2020) Weiqing Min, Shuhuan Mei, Zhuo Li, and Shuqiang Jiang. 2020\.  A two-stage triplet network training framework for image retrieval.  _IEEE Transactions on Multimedia_ 22, 12 (2020), 3128–3138.
  * Muller and Hutter (2021) Samuel G. Muller and Frank Hutter. 2021.  TrivialAugment: Tuning-free Yet State-of-the-Art Data Augmentation. In _Proceedings of ICCV_. IEEE, Los Alamitos, CA, USA, 754–762.
  * Murphy (2012) Kevin P. Murphy. 2012\.  _Machine Learning: A Probabilistic Perspective_.  The MIT Press.
  * Netzer et al. (2011) Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. 2011\.  Reading Digits in Natural Images with Unsupervised Feature Learning. In _Proceedings of NIPS 2011 Workshop on Deep Learning and Unsupervised Feature Learning_. Curran Associates, Inc.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2019\.  PyTorch: An Imperative Style, High-Performance Deep Learning Library. In _Proceedings of NeurIPS_. Curran Associates, Inc., 8024–8035.
  * Patel et al. (2022) Yash Patel, Giorgos Tolias, and Jiří Matas. 2022\.  Recall@k surrogate loss with large batches and similarity mixup. In _Proceedings of CVPR_. IEEE, 7502–7511.
  * Philbin et al. (2007) James Philbin, Ondrej Chum, Michael Isard, Josef Sivic, and Andrew Zisserman. 2007.  Object retrieval with large vocabularies and fast spatial matching. In _Proceedings of CVPR_. IEEE, 1–8.
  * Polley et al. (2022) Sayantan Polley, Subhajit Mondal, Venkata Srinath Mannam, Kushagra Kumar, Subhankar Patra, and Andreas Nürnberger. 2022.  X-Vision: Explainable Image Retrieval by Re-Ranking in Semantic Space. In _Proceedings of CIKM_ (Atlanta, GA, USA). Association for Computing Machinery, New York, NY, USA, 4955–4959.
  * Radenović et al. (2019) Filip Radenović, Giorgos Tolias, and Ondřej Chum. 2019\.  Fine-Tuning CNN Image Retrieval with No Human Annotation.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 41, 7 (2019), 1655–1668.
  * Revaud et al. (2019) Jerome Revaud, Jon Almazán, Rafael S. Rezende, and Cesar Roberto de Souza. 2019.  Learning with Average Precision: Training Image Retrieval with a Listwise Loss. In _Proceedings of ICCV_. IEEE, 5107–5116.
  * Russakovsky et al. (2015) Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. 2015\.  ImageNet Large Scale Visual Recognition Challenge.  _International Journal of Computer Vision_ 115 (2015), 211–252.
  * Schroff et al. (2015) Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015\.  FaceNet: A unified embedding for face recognition and clustering. In _Proceedings of CVPR_. IEEE, 815–823.
  * Sohn (2016) Kihyuk Sohn. 2016\.  Improved Deep Metric Learning with Multi-class N-pair Loss Objective. In _Proceedings of NIPS_ , Vol. 29. Curran Associates, Inc.
  * Suh et al. (2019) Yumin Suh, Bohyung Han, Wonsik Kim, and Kyoung Mu Lee. 2019\.  Stochastic Class-Based Hard Example Mining for Deep Metric Learning. In _Proceedings of CVPR_. IEEE, 7244–7252.
  * Tang et al. (2022) Yongxiang Tang, Wentao Bai, Guilin Li, Xialong Liu, and Yu Zhang. 2022.  CROLoss: Towards a Customizable Loss for Retrieval Models in Recommender Systems. In _Proceedings of CIKM_ (Atlanta, GA, USA). Association for Computing Machinery, New York, NY, USA, 1916–1924.
  * Vassileios Balntas and Mikolajczyk (2016) Daniel Ponsa Vassileios Balntas, Edgar Riba and Krystian Mikolajczyk. 2016\.  Learning local feature descriptors with triplets and shallow convolutional neural networks. In _Proceedings of BMVC_. BMVA Press, Article 119, 11 pages.
  * Wang et al. (2018a) Feng Wang, Jian Cheng, Weiyang Liu, and Haijun Liu. 2018a.  Additive Margin Softmax for Face Verification.  _IEEE Signal Processing Letters_ 25, 7 (2018), 926–930.
  * Wang et al. (2018b) Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Zhifeng Li, Dihong Gong, Jin Zhou, and Wei Liu. 2018b.  CosFace: Large Margin Cosine Loss for Deep Face Recognition. In _Proceedings of CVPR_. IEEE, 5265–5274.
  * Wen et al. (2016) Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. 2016\.  A Discriminative Feature Learning Approach for Deep Face Recognition. In _Proceedings of ECCV_. Springer, 499–515.
  * Wu et al. (2017) Chao-Yuan Wu, R. Manmatha, Alexander J. Smola, and Philipp Krähenbühl. 2017.  Sampling Matters in Deep Embedding Learning. In _Proceedings of ICCV_. IEEE, 2859–2867.
  * Wu et al. (2021) Hui Wu, Min Wang, Wengang Zhou, and Houqiang Li. 2021\.  Learning deep local features with multiple dynamic attentions for large-scale image retrieval. In _Proceedings of ICCV_. IEEE, 11416–11425.
  * Yadan (2019) Omry Yadan. 2019\.  Hydra - A framework for elegantly configuring complex applications.  Github.  https://github.com/facebookresearch/hydra
  * Yu and Tao (2019) Baosheng Yu and Dacheng Tao. 2019.  Deep Metric Learning With Tuplet Margin Loss. In _Proceedings of ICCV_. IEEE, 6489–6498.
  * Zhu et al. (2019) Qiuyu Zhu, Pengju Zhang, Zhengyong Wang, and Xin Ye. 2019\.  A New Loss Function for CNN Classifier Based on Predefined Evenly-Distributed Class Centroids.  _IEEE Access_ 8 (12 2019), 10888–10895.


††thanks:

# Telecom Quantum Photonic Interface for a 40Ca+ Single-Ion Quantum Memory

Elena Arenskötter  Tobias Bauer  Stephan Kucera  Experimentalphysik,
Universität des Saarlandes, 66123 Saarbrücken, Germany  Matthias Bock
Experimentalphysik, Universität des Saarlandes, 66123 Saarbrücken, Germany
present address: Institut für Quantenoptik und Quanteninformation,
Österreichische Akademie der Wissenschaften, Technikerstraße 21a, 6020
Innsbruck, Austria  Jürgen Eschner<EMAIL_ADDRESS>Christoph Becher<EMAIL_ADDRESS>Experimentalphysik,
Universität des Saarlandes, 66123 Saarbrücken, Germany

###### Abstract

Entanglement-based quantum networks require quantum photonic interfaces
between stationary quantum memories and photons, enabling entanglement
distribution. Here we present such a photonic interface, designed for
connecting a 40Ca+ single-ion quantum memory to the telecom C-band. The
interface combines a memory-resonant, cavity-enhanced spontaneous parametric
down-conversion (SPDC) photon pair source with bi-directional polarization-
conserving quantum frequency conversion (QFC). We demonstrate preservation of
high-fidelity entanglement during conversion, fiber transmission over up to 40
km and back-conversion to the memory wavelength. Even for the longest distance
and bi-directional conversion the entanglement fidelity remains larger than
95% (98%) without (with) background correction.

##  I Introduction

Entanglement-based quantum networks are the backbone for many quantum
technology applications connecting remote partners, such as distributed
quantum computing [1], quantum repeaters [2], quantum key distribution (QKD)
with entangled photons [3] or networks of quantum sensors [4]. Here we
consider networks involving matter-based quantum memories communicating via
optical photons. To realize such networks, several requirements have to be
fulfilled simultaneously: (i) a source of high-rate and high-fidelity
entanglement is essential [5, 6]; (ii) the spectral characteristics of
photonic information carriers and matter-based quantum memories have to match;
and (iii) the communication wavelength has to be in a low-loss band of optical
fibers.

Requirement (i) may be addressed by photon pair sources based on spontaneous
parametric down-conversion (SPDC) that have shown immense potential as high-
fidelity sources of entanglement since their introduction [7]. For their
interfacing with atom-based quantum memories, requirement (ii) [8, 9, 10], a
narrow spectrum of the photon pairs is required. This can be attained by
spectral filtering [11] or by placing the nonlinear crystal inside a cavity
that is tuned to be resonant with the desired atomic transition [12, 13]. The
latter scheme also enables very high pair rates, but it may suffer from
reduced state purity due to distinguishability induced by the birefringence of
both crystal and resonator. An alternative approach with higher-quality
entanglement but with a much lower pair rate is to use single-pass conversion
in an interferometric configuration, which eliminates all distinguishability
and eliminates the 50% background of unsplit photon pairs [14, 15, 16].
Ideally, resonator-enhanced generation and interferometric configuration would
be combined.

For the distribution of entangled photon pairs over long distances in fiber
networks, requirement (iii), transmission loss is a crucial issue, limiting
the entanglement rate and thereby the secret key rate of QKD. Thus it is
advantageous to use wavelengths in the telecom bands (1260 nm - 1625 nm),
where fiber absorption is minimal. Most of the relevant transitions in atomic
or ionic quantum memories, however, are in the visible or near IR regime [17,
18, 19, 20]. One possible approach to address quantum memory transitions and
at the same time minimize fiber loss is to use non-degenerate SPDC entangled-
pair sources. In this case one photon is resonant to the memory transition
while the other has a wavelength in the telecom regime [21, 22, 23, 8, 24].
Alternatively, non-degenerate photon pairs may be generated from degenerate
pairs by quantum frequency conversion (QFC) of one of the photons [25] which
shifts the photon wavelength while preserving all other properties. Here, an
efficient conversion process is three-wave mixing in $\chi^{(2)}$-nonlinear
media [26], but its strong polarization dependence is adverse to the
conversion of polarization qubits. To overcome this limitation, several
schemes for polarization-preserving conversion have been developed [27, 28,
29, 30, 31, 32, 33] and employed to demonstrate photon-photon [28, 32] and
light-matter entanglement over short distances [30, 31] as well as over km-
long fibers [34, 33, 35, 36].

In this paper we present a photonic interface designed to connect a single-ion
quantum memory with the telecom C-band. Our interface combines a
40Ca+-resonant, cavity-enhanced SPDC photon-pair source at 854 nm in
interferometric configuration with highly efficient polarization-preserving
QFC to 1550 nm and back-conversion to the atomic wavelength. It thereby
facilitates bi-directional sender and receiver operations between atomic and
photonic quantum bits [17, 37], as needed for a quantum network environment
involving atom-based memories. We present four different setups and verify
entanglement for each case by quantum state tomography: First, we characterize
the source operation, then the combined system with QFC to generate
polarization entanglement between a 40Ca+-resonant photon and a telecom
photon. In the third step we show entanglement distribution over 20 km of
fiber. Finally we demonstrate the preservation of high-quality entanglement
after transmission over 40 km of fiber and consecutive back-conversion of the
telecom photon to the 40Ca+-wavelength. We thus show for the first time the
simultaneous conversion and back-conversion by a single bi-directional
frequency converter. Even for the longest distance and bi-directional
conversion the entanglement fidelity remains larger than 95% (98%) without
(with) background correction.

##  II Results

###  II.1 SPDC Photon Pair Source

Figure 1: Schematic of the photon pair source including filtering and
detection. SPDC source in interferometic configuration with PDH locking beams.
Stabilization of the resonator and the interferometer is achieved by piezo
driven mirrors (PM). During the stabilization runs, the photons are switched
off by an optial chopper in order to protect the APDs. HWP/QWP: half-/quarter-
wave plate, PBS: polarizing beamsplitter, BS: non-polarizing beamsplitter, DM:
dichroic mirror, Pol: polarizer, WP: Wollaston prism, GP: glass plate, PM:
piezo driven mirror, FPI: frequency filter

Our SPDC photon pair source combines the interferometric configuration of [14,
38] with cavity enhancement, enabling the generation of photons with very high
pair rate, narrow linewidth and high-fidelity, offset-free entanglement
simultaneously. Compared to an earlier version of the source [39, 40], we also
optimised the locking stability and the output mode structure. The
experimental setup is shown in Fig. 1 and will be discussed in the following.

In order to address the D5/2-P3/2 transition in 40Ca+ at 854 nm, we use a
frequency-stable laser at 427 nm to pump the SPDC process in a periodically
poled KTP crystal with type-II phase-matching. The polarized pump light is
split on a non-polarizing 50:50 beam splitter (BS), and the two beams are
coupled into the non-linear crystal in opposite directions. The crystal is
placed inside a signal- and idler-resonant 3-mirror ring resonator
(FSR${}_{H}=2\pi\cdot 1.85\,$GHz, FSR${}_{V}=2\pi\cdot 1.83\,$GHz). Down-
converted photons from both pump directions leave the resonator at the same
outcoupling mirror, $M_{\text{out}}$, under different angles. The
polarizations of the signal and idler photons in one of the output arms are
interchanged by a half-wave plate. Then, the photons of both output arms are
overlapped on a polarizing beam splitter (PBS). This arrangement erases all
distinguishability between the two photons of a pair. Moreover, it avoids the
background of unsplit pairs that is inherent in single-direction SPDC
generation. The photonic 2-qubit state at 854 nm produced by the source is a
polarization Bell state [41]

$\ket{\Psi}=\frac{1}{\sqrt{2}}\left(\ket{H_{\text{A}}V_{\text{B}}}-e^{i\varphi}\ket{V_{\text{A}}H_{\text{B}}}\right)$
(1)

where $A$ and $B$ refer to the output ports of the PBS.

As shown in Fig. 1 we use a chopped locking beam, injected via a glass plate,
to stabilize one mode of the resonator (the H-polarized mode) to the ion
transition by the Pound-Drever-Hall (PDH) technique. The frequency of the
corresponding V-polarized mode is shifted by $\sim 480$ MHz because of the
birefringence of the crystal. Due to the conversion bandwidth of the crystal
(200 GHz) and the small polarisation dispersion of the resonator, several
pairs of modes at different frequencies exhibit cavity-enhanced SPDC (see
Supplement). Filtering out the ion-resonant mode and its partner mode is
effected by two monolithic Fabry-Pérot filters (FPI), one in each output. The
filters are described in more detail in the Methods section.

The SHG light that is produced by the locking beam is detected at the BS that
splits the pump beam and is used for the stabilization of the Mach-Zehnder-
type interferometer formed between this BS and the PBS that combines the SPDC
output arms. By tuning the phase of this interferometer, we tailor the phase
$\varphi$ of the photonic 2-qubit state, Eq. (1). For the experiments
described below, we set this phase to $\varphi=270^{\circ}$. A more detailed
account of the relation between $\varphi$ and the interferometer phase is
given in the Supplement.

With the described setup we reach a generated photon pair rate of
$R_{\text{pair}}=4.7\cdot 10^{4}\frac{\text{pairs}}{\text{s mW}}\times
P_{\text{pump}}$. In the experiments below, we used a maximum pump power of
$P_{\text{pump}}=20$ mW. The SPDC photons are available, after spectral
filtering and fiber coupling, with 31% (21.4%) efficiency in port A (B).

For analysis of the photonic polarization state, we use a projection setup
consisting of half-wave plate, quarter-wave plate, and polarizer (film
polarizer or Wollaston prism) in each output arm. As single-photon detectors
we use two APDs (Excelitas Technologies) with approximately
$10~{}\text{s}^{-1}$ dark counts. Detected photons are time-tagged, and time-
resolved coincidence functions between the two arms are evaluated for various
combinations of polarization settings.

Figure 2: Photon wave packet. Measured signal-idler coincidence function with
a bin size of 1 ns. The red dashed line is a double exponential fit to the
data.

For the characterization of the temporal shape of the photon wavepacket, we
measure the polarization-indiscriminate coincidence function between the
photons of the two output arms [42]. Fig. 2 shows the result, measured by
summing the coincidences for the four settings HV, VH, HH, and VV. The decay
times (or wave packet widths) of the two photons differ slightly because of
different loss of the two polarization modes in the cavity: we get values of
$\tau_{H}=15.78$ ns for the H-polarized photons and $\tau_{V}=12.94$ ns for
the V-polarized photons.

An important figure of merit is the signal-to-background ratio (SBR) of the
coincidence functions. The grey-shaded area in Fig. 2, representing the
coincidences in the time-window $\Delta t$ around zero, is taken as the signal
($S$). The background ($B$) is determined using the same time window size but
at a delay $\tau>150$ ns (red shaded area). For our setup the background is
dominated by accidental coincidences, originating from the temporal overlap of
the generated photons.

Theoretically, the number of polarization-indiscriminate coincidence counts in
a time interval $T$ and for a given pair rate $R_{\text{pair}}$ is given by

$S=\eta_{1}\eta_{2}R_{\textrm{pair}}(1-e^{-\Delta
t/2\tau_{\text{mean}}})T~{},$ (2)

where $\eta_{1}$ and $\eta_{2}$ are the detection efficiencies for the two
outputs, and $\tau_{\text{mean}}=\frac{1}{2}(\tau_{H}+\tau_{V})$ is the mean
wavepacket width of the photons. The theoretical value for the background is

$B=\eta_{1}\eta_{2}R_{\text{pair}}^{2}\Delta t\,T~{}.$ (3)

Hence the SBR of the source is expected to be given by [40]

$\text{SBR}=\frac{S}{B}=\frac{1-e^{-\Delta
t/2\tau_{\text{mean}}}}{R_{\text{pair}}\Delta t}$ (4)

Fig. 3 shows a comparison between this theoretical expression and our
measurement, for fixed pump power of 20 mW and variable coincidence window
$\Delta t$. When $\Delta t$ is similar to the 1/e width of the photon
wavepacket, we reach an SBR of $\sim$30, whereas for a coincidence window that
covers 99.97% of the photon, we find an SBR of $\sim$10.

Fig. 3 also shows how the coincidence rate varies with $\Delta t$. From
fitting Eq. (2) to the data, with $\eta_{1,2}$ independently measured, we
derive the pair rate $R_{\text{pair}}$ as mentioned before.

The SBR enters into the maximally reachable fidelity of the photon pair state
to the ideal Bell state of eq. (1). The fidelity of a measured (i.e.,
reconstructed) density matrix $\rho$ to a given state $\ket{\Psi}$ is
calculated as

$F=\bra{\Psi}\rho\ket{\Psi}=\frac{\frac{1}{4}+F_{\text{w/o
BG}}\cdot\text{SBR}}{1+\text{SBR}}$ (5)

where $F_{\text{w/o BG}}$ corresponds to the fidelity of the photonic state
when the entire background is subtracted [40]. This formula is used below for
the theoretical curves in Fig. 6.

Figure 3: Signal-to-background ratio and coincidence rate of the photon pair
source. The SBR and the coincidence rate are plotted in dependence of the
coincidence time window $\Delta t$, for a pump power of 20 mW. The error bars
are calculated from the Poissonian noise of the measured coincidences. The
dashed lines show the theoretical calculations according to Eq. (4) for the
SBR and and Eq. (2) for the coincidence rate, with $R_{\text{pair}}$ the only
fit parameter.

###  II.2 Polarization Preserving Frequency Conversion

(a)

(b)

Figure 4: a) Schematic quantum frequency conversion setup. Details are found
in the main text. TDFL: Thulium-doped fiber laser, LPF: low-pass filter, BPF:
band-pass filter, VBG: volume-Bragg grating, DM: dichroic mirror, HWP: half-
wave plate, SNSPD: superconducting nanowire single photon detector; b) Device
efficiency of the converter for different pump powers and input polarizations.
The maximum efficiencies of the H- and V-polarizations are slightly different.

The setup for the polarization-preserving frequency conversion (Fig. 4(a)) is
located in a second lab and is connected to the photon-pair source via 90 m of
fiber. We use difference frequency generation (DFG) in a nonlinear PPLN
waveguide to convert the 854 nm photons to the telecom C-band at 1550 nm [29].
As the DFG efficiency in the waveguide is strongly polarization dependent, the
setup is realized in a Sagnac configuration [31, 33]. First, the 854 nm signal
is filtered by an input bandpass filter to prevent background photons being
coupled back into the source setup and overlapped with the strong, diagonally
polarized pump field at 1904 nm on a dichroic mirror. Both fields are then
split into their H- and V-polarization component on a PBS. To achieve
polarization preserving operation, the H-components are rotated to V by an
achromatic waveplate inside the Sagnac loop (HWP). All beams are now
V-polarized for optimum conversion and are coupled in a counterpropagating way
into the waveguide. After exiting the waveguide, the converted 1550-nm light
from the original V-component also passes the achromatic waveplate and is
thereby rotated to H. Finally, the two converted polarization components are
coherently overlapped on the PBS, which closes the Sagnac loop. The light then
passes the dichroic mirror a second time where unconverted (or back-converted)
light at 854 nm is split off. The same happens to SHG light from the pump that
is parasitically generated in the waveguide; it is then blocked by the input
bandpass filter. The converted light that is transmitted through the first DM
together with the pump light is separated with a second DM and coupled into a
telecom fiber, directing it to the detection setup.

Apart from enabling polarization-independent conversion, an additional
advantage of the Sagnac configuration is that all fields have the same optical
path, such that no phase difference between the split polarization components
occurs, and hence no active phase stabilization is needed. At the same time,
the configuration facilitates bi-directional conversion without compromising
the conversion efficiency. An experimental demonstration of bi-directional
operation will be presented below.

The conversion-induced background in this process mainly originates from anti-
Stokes Raman scattering of the pump field [43, 29, 44]. As this is spectrally
very broad compared to the converted signal, we can significantly reduce the
background count rate with a narrowband filtering stage. By combination of a
bandpass filter (transmission bandwidth $\Delta\nu_{\text{BPF}}=1500$ GHz), a
volume Bragg grating (VBG, $\Delta\nu_{\text{VBG}}=25$ GHz) and a Fabry-Pérot
etalon (FSR = 12.5 GHz, $\Delta\nu_{\text{FPE}}\,=\,250$ MHz) we achieve
broadband suppression outside a 250 MHz transmission bandwidth. To ensure high
transmission through VBG and Etalon, a clean gaussian spatial mode is needed.
That is why these filters are included in the detection setup, which is
separated from the conversion setup by 10 m of fiber (or later the fiber link)
acting as spatial filter. With this filtering stage, the total conversion-
induced background count rate is 24 s-1.

The device efficiency including all losses in optical components and spectral
filtering for H and V input polarizations and different pump powers in the
corresponding converter arm is shown in Fig. 4(b). The measurement agrees well
with the theoretical curve given by
$\eta_{\text{ext}}(P)=\eta_{\text{ext,max}}\sin^{2}(\sqrt{\eta_{\text{nor}}P}L)$
with the normalized power efficiency $\eta_{\text{nor}}$ and waveguide length
$L$ [45]. We achieve a maximum external conversion efficiency of
$\eta_{\text{ext,max}}=\,60.1\%\,(57.2\%)$ for H(V)-polarized input at 660 mW
(630 mW) pump power. The difference is explained by slightly different mode
overlaps between signal and pump field in the two arms. Since for
polarization-preserving operation both efficiencies need to be equal, the pump
power in the H-arm is reduced via the two HWPs in the pump laser arm to 485 mW
to match 57.2% conversion efficiency. To verify equal conversion efficiency
for arbitrary polarized input, the process matrix [46] was measured with
attenuated laser light, resulting in a value for the process fidelity of
99.947(2)%. Thus the preservation of the input polarization state is confirmed
with very high fidelity. Note that in fact the converter rotates the input
state by a constant factor $\pi/2$ due to the achromatic waveplate. This could
be compensated by an additional waveplate at the output, however the
polarization also gets arbitrarily rotated in the input and output fibers, so
we compensate for all rotations together by rotating the projection bases in
all measurements as described in the Methods section.
Further details on the conversion setup as well as the background and process
fidelity measurements are found in the supplement.

###  II.3 Experimental results

Figure 5: Overview of the different experimental configurations. a) The output
state is measured directly at the source. b) One of the photons is frequency-
converted before its detection. c) 20 km of fiber is inserted between
conversion and detection. d) The 20 km fiber gets terminated with a retro-
reflector. The back reflected photons get back-converted after 40 km of fiber
and are extracted with a 50:50 fiber beamsplitter before their detection. The
background colors correspond to the colors of the data points in Fig. 6.

To assess the performance of our quantum photonic interface we analyze the
photon pair states via quantum state tomography [47] for four different
configurations, as sketched in Fig. 5a-d.

In the first configuration (Fig. 5a) and for calibration purposes, we measure
the output state of the photon pair source itself, i.e., without conversion.
In the next configuration (Fig. 5b), we include the frequency converter in the
detuned output arm of the photon pair source (arm A in Fig. 1). Tomography is
then performed on the pairs of converted and unconverted photons.

In the third configuration (Fig. 5c), we extend the fiber link between
converter and detection setup to 20 km in order to evaluate the influence of
fiber transmission. We emphasize that the transmission of photons at a
wavelength of 854 nm via a suitable single mode fiber would suffer about -70
dB fiber attenuation [48] whereas conversion to 1550 nm and using a low-loss
telecom fiber results in a total fiber attenuation of -3.4 dB [49].

Finally, in the fourth configuration (Fig. 5d) bi-directional conversion is
implemented by terminating the 20 km fiber with a retroreflector. Thus, the
telecom photons are converted back to 854 nm at the second passage of the
converter, after 40 km of fiber transmission. To separate the returning back-
converted photons from the outgoing ones in the source lab, we use a 50:50
fiber beamsplitter that reflects 50% of the back-converted photons to the
second tomography setup. Note that in this configuration the converter filter
stage is not used, but the FPI filter in the 854 nm tomography setup, as
explained in the supplement.

(a) Fidelity for a detection window $\Delta t=1.5\tau_{\text{mean}}$

(b) SBR for a detection window $\Delta t=1.5\tau_{\text{mean}}$

(c) Fidelity for a detection window $\Delta t=5\tau_{\text{mean}}$

(d) SBR for a detection window $\Delta t=5\tau_{\text{mean}}$

Figure 6: Fidelity and SBR vs. SPDC pump power for the four configurations of
Fig. 5. a) and b) show the measurement results for a detection time window of
$1.5\tau_{\text{mean}}$ which corresponds to 78% of the photon temporal shape.
In a) the MLE results are represented by the data points, the dot-dashed line
shows the mean background corrected fidelity, while the dashed line shows the
theoretical expectation according to eq. (5). The uncertainties are derived
from a simulation; see Method section for further details. b) shows the
comparison of the SBR for the four different measurement settings. The dashed
line is calculated for the given pair rate according to eq. (4). The error
bars are calculated by including the $\sqrt{N}$-noise of the measured
coincidences [50]. c) and d) show the results when choosing a larger detection
window of $5\tau_{\text{mean}}$ for the evaluation.

In all four configurations, we use source pump powers of 5 mW, 10 mW, 15 mW
and 20 mW. The detected coincidence rates at 20 mW pump power are: 4168 s-1
for setup a), 974 s-1 for setup b), 428 s-1 for setup c) and 40 s-1 for setup
d) (also see Supplement).From the time-resolved coincidences in the different
detection bases, the density matrix is reconstructed by applying an iterative
maximum likelihood estimation (MLE) algorithm [51]. From the density matrix,
we then calculate all measures that characterize the quantum state, such as
fidelity and purity. For each configuration we show a representative density
matrix in the supplement. Before performing the tomography, the polarization
rotations in both arms, including the converter and the fiber link, were
calibrated; details on the procedure are found in the methods part.

The results of the four configurations are summarized in Fig. 6. For all
configurations and source pump powers, the fidelities with respect to the
maximally entangled state (1) are above 95% with and without background
correction, for a detection window of $\Delta t=1.5\,\tau_{\text{mean}}$
(subfigure a). Thus the entanglement is well-preserved for every scenario of
Fig. 5. For fixed source pump power, the fidelities of the individual
configurations differ only slightly outside the error bars, which confirms the
high process fidelity of the converter and the low detrimental effect of the
fiber link. With increasing source pump power, the fidelity decreases, which
is expected according to eq. (5). The black dashed line shows the
theoretically achievable fidelity for the given SBR, considering only
accidental coincidences. This line is expected to be an upper bound to the
measured values, as additional effects lead to further reductions: (i) errors
in the calibration of the polarisation rotation compensation; (ii) drifts in
the polarization rotation of the setup away from the calibration point during
the measurements; (iii) fluctuations of the photonic state phase resulting
from power fluctuations of the locking signal.

The SBR for each individual measurement is shown in Fig. 6b. Here the error
bars are calculated by taking Poissonian noise into account. As expected from
eq. (4), the SBR decreases for increasing pump power of the pair source. Again
we only see minor differences between the four configurations, as the
conversion-induced background rate is negligible compared to the pair rate and
source background rate in the relevant time window. For the back-conversion
part, the SBR is consistently lower, which is explained by the use of the FPI
filter instead of the converter filtering stage.

When choosing a larger detection window, fidelities and SBR decrease due to
the temporal overlap of the photons. The results for a detection window of
$\Delta t=5\,\tau_{\text{mean}}$, corresponding to 91% of the photon, are
shown in Fig. 6c-d. Due to the lower SBR, here the fidelities are lower by a
few percent, while the detected pair rate is increased by a factor of 1.7.

##  III Discussion

In summary, we have presented and characterized a combined system of entangled
photon pair source and bi-directional quantum frequency converter and use it
for entanglement distribution over a fiber link. Pair rate and entanglement
fidelity of the source are near the theoretical optimum, while efficiency,
polarization fidelity and noise performance of the converter are among the
best reported values [29, 31, 32]. The system is tailored as interface to the
40Ca+ single-ion quantum memory.

The presented measurements demonstrate the preservation of photon entanglement
with high fidelity through up to 2 conversion steps and distribution over up
to 40 km of fiber. The high-efficiency bi-directional quantum frequency
conversion adds low background to the photon pair source and thus enables a
proof-of-principle demonstration of a quantum photonic interface suitable to
connect remote quantum network nodes based on 40Ca+ single-ion memories.

Several future improvements are readily identified. The coincidence rates can
be enhanced by replacing lossy fiber-fiber couplings by direct splices and by
replacing the fiber beam splitter by an optical circulator. Given the low
unconditional background rate of the QFC process, we can further increase
photon count rates by lowering the finesse of the monolithic Fabry-Pérot
filters, which results in higher transmission. The conversion fidelity can be
improved near the theoretical maximum by repeating the polarization
calibration more frequently in a fully automated procedure. Furthermore, a
redesign of the source-cavity itself would make it possible to shorten the
photons to 8 ns, which corresponds to the transition linewidth of 23 MHz
between the D5/2 and P3/2 state. Due to less overlap between two subsequent
photons, the SBR and the fidelity would improve as well.

Recent developments in quantum networking operations with single ions [52, 53,
54], together with progress in ion-trap quantum processors [55, 56] confirm
the importance and the potential of quantum photonic interfaces for quantum
information technologies. The interface we demonstrate here enables extension
towards further fundamental operations in trapped-ion-based quantum networks.
In particular, bi-directional QFC allows for teleportation between remote
quantum memories, or for their entanglement via direct exchange of a photon
(see e.g. [57]), employing heralded absorption [40, 17] of a back-converted
telecom photon.

##  IV Methods

### State characterization

We perform full state tomography to reconstruct the photonic two-qubit state.
We project each photon to one of the six polarization basis states
(horizontal, vertical, diagonal, anti diagonal, left circular, or right
circular), which results in 36 possible measurement combinations. In order to
reconstruct the 2-qubit density matrix, 16 combinations are sufficient. We
therefore measure two-photon correlations in 16 independent polarization
settings [58] and reconstruct the density matrix by applying a maximum
likelihood algorithm to the count rates inside the detection window [51]. We
infer from the density matrix all characteristic measures such as purity and
fidelity. For calculating the error bars, we use a Monte Carlo simulation [50]
where we run the algorithm repeatedly on randomly generated, Poisson-
distributed coincidence rates based on the measured rates. From the
distribution of the resulting fidelities, we estimate the error bars.

### Polarization bases calibration

Polarization rotation due to the birefringence of optical components in the
setup, in particular the fibers, has to be accounted for to ensure projection
onto the correct basis states. We model the rotation in each arm by a unitary
rotation matrix $M$ that transforms the input polarization
$\lambda_{\text{in}}$ to an output polarisation
$\lambda_{\text{out}}=M\lambda_{\text{in}}$. Non-unitary effects such as
polarization-dependent loss play only a negligible role on the final state
fidelity.

By blocking one pump direction of the source, we deterministically generate
photons with linear (H) polarization. Additionally, we can insert a quarter-
wave plate directly behind the source to rotate the polarization to R. We then
perform single-qubit tomography with the detection setups to measure the
rotated polarization states. From this we calculate the rotation matrix $M$,
and the projection bases are rotated accordingly.

Depending on the experimental configuration (Fig. 5), we measure the rotation
matrix less or more frequently during the experiments: for the first two
configurations, short fibers are used, thus the polarization is very stable
over time and is measured only at the beginning of the experiment. In the last
two configurations, although the fiber spool is actively temperature
stabilized to minimize polarization drifts, the rotation matrix is measured
every 60 minutes.

### Fabry-Pérot filter

The Fabry-Pérot filters which we use for filtering the 854 nm photons to a
single frequency mode are built from single 2 mm thick NBK-7 lenses. Both
sides have a high-reflectivity coating ($R=0.9935$). This results in a finesse
of 481, a FSR of $\sim 50$ GHz and a FWHM of $\Delta\nu\approx 104$ MHz, which
is sufficient for filtering a single mode of the SPDC cavity (FSR $\sim 1.84$
GHz). Frequency tuning over a whole FSR is possible by changing the
temperature in the range between 20°C and 70°C. The stabilization of the
temperature with a precision of 1 mK (corresponding to 3 MHz) is sufficient
for stable operation; no further active stabilization is needed.

##  V Data availability

The underlying data for this manuscript is openly available in Zenodo at
https://doi.org/10.5281/zenodo.7313581. The evaluation algorithms are
available from the corresponding author upon reasonable request.

##  VI References

## References

  * [1] Jiang, L., Taylor, J. M., Sørensen, A. S. & Lukin, M. D.  Distributed quantum computation based on small quantum registers.  _Phys. Rev. A_ 76, 062323 (2007).  URL https://link.aps.org/doi/10.1103/PhysRevA.76.062323.
  * [2] Briegel, H.-J., Dür, W., Cirac, J. I. & Zoller, P.  Quantum repeaters: The role of imperfect local operations in quantum communication.  _Phys. Rev. Lett._ 81, 5932–5935 (1998).  URL https://link.aps.org/doi/10.1103/PhysRevLett.81.5932.
  * [3] Ekert, A. K.  Quantum cryptography based on bell’s theorem.  _Phys. Rev. Lett._ 67, 661–663 (1991).  URL https://link.aps.org/doi/10.1103/PhysRevLett.67.661.
  * [4] Proctor, T. J., Knott, P. A. & Dunningham, J. A.  Multiparameter estimation in networked quantum sensors.  _Phys. Rev. Lett._ 120, 080501 (2018).  URL https://link.aps.org/doi/10.1103/PhysRevLett.120.080501.
  * [5] Simon, C. _et al._ Quantum repeaters with photon pair sources and multimode memories.  _Phys. Rev. Lett._ 98, 190503 (2007).  URL https://link.aps.org/doi/10.1103/PhysRevLett.98.190503.
  * [6] Wehner, S., Elkouss, D. & Hanson, R.  Quantum internet: A vision for the road ahead.  _Science_ 362, eaam9288 (2018).  URL https://www.science.org/doi/abs/10.1126/science.aam9288.  eprint https://www.science.org/doi/pdf/10.1126/science.aam9288.
  * [7] Kwiat, P. G. _et al._ New high-intensity source of polarization-entangled photon pairs.  _Phys. Rev. Lett._ 75, 4337–4341 (1995).  URL https://link.aps.org/doi/10.1103/PhysRevLett.75.4337.
  * [8] Lenhard, A. _et al._ Telecom-heralded single-photon absorption by a single atom.  _Phys. Rev. A_ 92, 063827 (2015).  URL https://link.aps.org/doi/10.1103/PhysRevA.92.063827.
  * [9] Specht, H. P. _et al._ A single-atom quantum memory.  _Nature_ 473, 190–193 (2011).
  * [10] Rosenfeld, W., Volz, J., Weber, M. & Weinfurter, H.  Coherence of a qubit stored in zeeman levels of a single optically trapped atom.  _Phys. Rev. A_ 84, 022343 (2011).  URL https://link.aps.org/doi/10.1103/PhysRevA.84.022343.
  * [11] Brito, J., Kucera, S., Eich, P., Müller, P. & Eschner, J.  Doubly heralded single-photon absorption by a single atom.  _Applied Physics B_ 122, 36 (2016).
  * [12] Luo, K.-H. _et al._ Direct generation of genuine single-longitudinal-mode narrowband photon pairs.  _New Journal of Physics_ 17, 073039 (2015).
  * [13] Tsai, P.-J. & Chen, Y.-C.  Ultrabright, narrow-band photon-pair source for atomic quantum memories.  _Quantum Science and Technology_ 3, 034005 (2018).
  * [14] Fiorentino, M., Messin, G., Kuklewicz, C. E., Wong, F. N. C. & Shapiro, J. H.  Generation of ultrabright tunable polarization entanglement without spatial, spectral, or temporal constraints.  _Phys. Rev. A_ 69, 041801 (2004).  URL https://link.aps.org/doi/10.1103/PhysRevA.69.041801.
  * [15] Kuzucu, O. & Wong, F. N. C.  Pulsed sagnac source of narrow-band polarization-entangled photons.  _Phys. Rev. A_ 77, 032314 (2008).  URL https://link.aps.org/doi/10.1103/PhysRevA.77.032314.
  * [16] Kim, T., Fiorentino, M. & Wong, F. N. C.  Phase-stable source of polarization-entangled photons using a polarization sagnac interferometer.  _Phys. Rev. A_ 73, 012316 (2006).  URL https://link.aps.org/doi/10.1103/PhysRevA.73.012316.
  * [17] Kurz, C., Eich, P., Schug, M., Müller, P. & Eschner, J.  Programmable atom-photon quantum interface.  _Phys. Rev. A_ 93, 062348 (2016).  URL https://link.aps.org/doi/10.1103/PhysRevA.93.062348.
  * [18] Meyer, H. M. _et al._ Direct photonic coupling of a semiconductor quantum dot and a trapped ion.  _Phys. Rev. Lett._ 114, 123001 (2015).  URL https://link.aps.org/doi/10.1103/PhysRevLett.114.123001.
  * [19] Schupp, J. _et al._ Interface between trapped-ion qubits and traveling photons with close-to-optimal efficiency.  _PRX Quantum_ 2, 020331 (2021).  URL https://link.aps.org/doi/10.1103/PRXQuantum.2.020331.
  * [20] Mount, E. _et al._ Single qubit manipulation in a microfabricated surface electrode ion trap.  _New Journal of Physics_ 15, 093018 (2013).
  * [21] Saglamyurek, E. _et al._ Broadband waveguide quantum memory for entangled photons.  _Nature_ 469, 512–515 (2011).
  * [22] Fekete, J., Rieländer, D., Cristiani, M. & de Riedmatten, H.  Ultranarrow-band photon-pair source compatible with solid state quantum memories and telecommunication networks.  _Phys. Rev. Lett._ 110, 220502 (2013).  URL https://link.aps.org/doi/10.1103/PhysRevLett.110.220502.
  * [23] Bussières, F. _et al._ Quantum teleportation from a telecom-wavelength photon to a solid-state quantum memory.  _Nature Photonics_ 8, 775–778 (2014).
  * [24] Lago-Rivera, D., Grandi, S., Rakonjac, J. V., Seri, A. & de Riedmatten, H.  Telecom-heralded entanglement between multimode solid-state quantum memories.  _Nature_ 594, 37–40 (2021).
  * [25] Kumar, P.  Quantum frequency conversion.  _Optics letters_ 15, 1476–1478 (1990).
  * [26] Huang, J. & Kumar, P.  Observation of quantum frequency conversion.  _Phys. Rev. Lett._ 68, 2153–2156 (1992).  URL https://link.aps.org/doi/10.1103/PhysRevLett.68.2153.
  * [27] Albota, M. A., Wong, F. N. C. & Shapiro, J. H.  Polarization-independent frequency conversion for quantum optical communication.  _JOSA B_ 23, 918–924 (2006).
  * [28] Ramelow, S., Fedrizzi, A., Poppe, A., Langford, N. K. & Zeilinger, A.  Polarization-entanglement-conserving frequency conversion of photons.  _Phys. Rev. A_ 85, 013845 (2012).  URL https://link.aps.org/doi/10.1103/PhysRevA.85.013845.
  * [29] Krutyanskiy, V., Meraner, M., Schupp, J. & Lanyon, B. P.  Polarisation-preserving photon frequency conversion from a trapped-ion-compatible wavelength to the telecom c-band.  _Applied Physics B_ 123, 228 (2017).
  * [30] Bock, M. _et al._ High-fidelity entanglement between a trapped ion and a telecom photon via quantum frequency conversion.  _Nature communications_ 9, 1998 (2018).
  * [31] Ikuta, R. _et al._ Polarization insensitive frequency conversion for an atom-photon entanglement distribution via a telecom network.  _Nature communications_ 9, 1997 (2018).
  * [32] Kaiser, F. _et al._ Quantum optical frequency up-conversion for polarisation entangled qubits: towards interconnected quantum information devices.  _Optics Express_ 27, 25603–25610 (2019).
  * [33] van Leent, T. _et al._ Long-distance distribution of atom-photon entanglement at telecom wavelength.  _Phys. Rev. Lett._ 124, 010510 (2020).  URL https://link.aps.org/doi/10.1103/PhysRevLett.124.010510.
  * [34] Krutyanskiy, V. _et al._ Light-matter entanglement over 50 km of optical fibre.  _npj Quantum Information_ 5, 72 (2019).
  * [35] van Leent, T. _et al._ Entangling single atoms over 33 km telecom fibre.  _Nature_ 607, 69–73 (2022).
  * [36] Luo, X.-Y. _et al._ Postselected entanglement between two atomic ensembles separated by 12.5 km.  _Physical Review Letters_ 129, 050503 (2022).
  * [37] Brekenfeld, M., Niemietz, D., Christesen, J. D. & Rempe, G.  A quantum network node with crossed optical fibre cavities.  _Nature Physics_ 16, 647–651 (2020).  URL https://doi.org/10.1038/s41567-020-0855-3.
  * [38] Fedrizzi, A., Herbst, T., Poppe, A., Jennewein, T. & Zeilinger, A.  A wavelength-tunable fiber-coupled source of narrowband entangled photons.  _Opt. Express_ 15, 15377–15386 (2007).  URL https://opg.optica.org/oe/abstract.cfm?URI=oe-15-23-15377.
  * [39] Arenskoetter, J., Kucera, S. & Eschner, J.  Polarization-entangled photon pairs from a cavity-enhanced down-conversion source in sagnac configuration.  In _2017 Conference on Lasers and Electro-Optics Europe & European Quantum Electronics Conference (CLEO/Europe-EQEC)_, 1–1 (2017).
  * [40] Kucera, S.  _Experimental distribution of entanglement in ion-photon quantum networks : photon-pairs as resource_.  Ph.D. thesis, Saarland University (2019).
  * [41] Braunstein, S. L., Mann, A. & Revzen, M.  Maximal violation of bell inequalities for mixed states.  _Phys. Rev. Lett._ 68, 3259–3261 (1992).  URL https://link.aps.org/doi/10.1103/PhysRevLett.68.3259.
  * [42] Glauber, R. J.  Coherent and incoherent states of the radiation field.  _Phys. Rev._ 131, 2766–2788 (1963).  URL https://link.aps.org/doi/10.1103/PhysRev.131.2766.
  * [43] Zaske, S., Lenhard, A. & Becher, C.  Efficient frequency downconversion at the single photon level from the red spectral range to the telecommunications c-band.  _Optics express_ 19, 12825–12836 (2011).
  * [44] Kuo, P. S., Pelc, J. S., Langrock, C. & Fejer, M. M.  Using temperature to reduce noise in quantum frequency conversion.  _Optics letters_ 43, 2034–2037 (2018).
  * [45] Roussev, R. V., Langrock, C., Kurz, J. R. & Fejer, M. M.  Periodically poled lithium niobate waveguide sum-frequency generator for efficient single-photon detection at communication wavelengths.  _Optics letters_ 29, 1518–1520 (2004).
  * [46] Chuang, I. L. & Nielsen, M. A.  Prescription for experimental determination of the dynamics of a quantum black box.  _Journal of Modern Optics_ 44, 2455–2467 (1997).
  * [47] James, D. F. V., Kwiat, P. G., Munro, W. J. & White, A. G.  On the measurement of qubits.  In _Asymptotic Theory of Quantum Statistical Inference: Selected Papers_ , 509–538 (World Scientific, 2005).
  * [48] Nufern.  _Nufern 780 nm Select Cut-OffSingle-Mode Fiber_ (2006).  URL http://princetel.com/datasheets/nufern_780hp.pdf.
  * [49] Corning.  _Corning® SMF-28® ULL Optical Fiber_ (2014).  URL https://www.corning.com/media/worldwide/coc/documents/Fiber/SMF-28%20ULL.pdf.
  * [50] Altepeter, J. B., Jeffrey, E. R. & Kwiat, P. G.  Photonic state tomography.  _Advances in Atomic, Molecular, and Optical Physics_ 52, 105–159 (2005).
  * [51] Hradil, Z.  Quantum-state estimation.  _Phys. Rev. A_ 55, R1561–R1564 (1997).  URL https://link.aps.org/doi/10.1103/PhysRevA.55.R1561.
  * [52] Krutyanskiy, V. _et al._ Entanglement of trapped-ion qubits separated by 230 meters.  _arXiv preprint arXiv:2208.14907_ (2022).
  * [53] Krutyanskiy, V. _et al._ A telecom-wavelength quantum repeater node based on a trapped-ion processor.  _arXiv preprint arXiv:2210.05418_ (2022).
  * [54] Drmota, P. _et al._ Robust quantum memory in a trapped-ion quantum network node (2022).  URL https://arxiv.org/abs/2210.11447.
  * [55] Pogorelov, I. _et al._ Compact ion-trap quantum computing demonstrator.  _PRX Quantum_ 2, 020343 (2021).
  * [56] Noel, C. _et al._ Measurement-induced quantum phases realized in a trapped-ion quantum computer.  _Nature Physics_ 18, 760–764 (2022).
  * [57] Luo, X.-Y. _et al._ Entangling metropolitan-distance separated quantum memories.  _arXiv preprint arXiv:2201.11953_ (2022).
  * [58] James, D. F. V., Kwiat, P. G., Munro, W. J. & White, A. G.  Measurement of qubits.  _Phys. Rev. A_ 64, 052312 (2001).  URL https://link.aps.org/doi/10.1103/PhysRevA.64.052312.

##  VII Acknowledgements

We acknowledge support by the German Federal Ministry of Education and
Research (BMBF) through projects Q.Link.X (16KIS0864) and QR.X (16KISQ001K).

##  VIII Author contributions

E.A., T.B., S.K., and M.B. designed the experiments. E.A. and T.B. performed
the experiments. E.A. and T.B. analysed the data. E.A., T.B., S.K. and M.B.
prepared the experimental setup. E.A. and T.B. wrote the paper with input from
all authors. J.E. and C.B. supervised the project.

##  IX Competing interests

The authors declare no competing interests.