# Semi-Supervised Semantic Segmentation with
Pixel-Level Contrastive Learning from a Class-wise Memory Bank

Inigo Alonso1 Alberto Sabater1 David Ferstl2 Luis Montesano1, Ana C. Murillo1
1RoPeRT group, at DIIS - I3A, Universidad de Zaragoza, Spain
2Magic Leap, Zürich, Switzerland
3Bitbrain, Zaragoza, Spain
{inigo, asabater, montesano<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

This work presents a novel approach for semi-supervised semantic segmentation.
The key element of this approach is our contrastive learning module that
enforces the segmentation network to yield similar pixel-level feature
representations for same-class samples across the whole dataset. To achieve
this, we maintain a memory bank continuously updated with relevant and high-
quality feature vectors from labeled data. In an end-to-end training, the
features from both labeled and unlabeled data are optimized to be similar to
same-class samples from the memory bank. Our approach outperforms the current
state-of-the-art for semi-supervised semantic segmentation and semi-supervised
domain adaptation on well-known public benchmarks, with larger improvements on
the most challenging scenarios, i.e., less available labeled data.
https://github.com/Shathe/SemiSeg-Contrastive

##  1 Introduction

Semantic segmentation consists in assigning a semantic label to each pixel in
an image. It is an essential computer vision task for scene understanding that
plays a relevant role in many applications such as medical imaging [31] or
autonomous driving [2]. As for many other computer vision tasks, deep
convolutional neural networks have shown significant improvements in semantic
segmentation [2, 19, 1]. All these examples follow supervised approaches
requiring a large set of annotated data to generalize well. However, the
availability of labels is a common bottleneck in supervised learning,
especially for tasks such as semantic segmentation, which require expensive
per-pixel annotations.

Figure 1:  Proposed contrastive learning module overview. At each training
iteration, the teacher network $f_{\xi}$ updates the feature memory bank with
a subset of selected features from labeled samples. Then, the student network
$f_{\theta}$ extracts features $\triangle$ from both labeled and unlabeled
samples, which are optimized to be similar to same-class features from the
memory bank ○.

Semi-supervised learning assumes that only a small subset of the available
data is labeled. It tackles this limited labeled data issue by extracting
knowledge from unlabeled samples. Semi-supervised learning has been applied to
a wide range of applications [38], including semantic segmentation [11, 17,
27]. Previous semi-supervised segmentation works are mostly based on per-
sample entropy minimization [17, 22, 29] and per-sample consistency
regularization [11, 37, 29]. These segmentation methods do not enforce any
type of structure on the learned features to increase inter-class separability
across the whole dataset. Our hypothesis is that overcoming this limitation
can lead to better feature learning and performance, especially when the
amount of available labeled data is low.

This work presents a novel approach for semi-supervised semantic segmentation.
Our approach follows a teacher-student scheme where the main component is a
novel representation learning module (Figure 1). This module is based on
positive-only contrastive learning [5, 14] and enforces the class-separability
of pixel-level features across different samples. To achieve this, the teacher
network produces feature candidates, only from labeled data, to be stored in a
memory bank. Meanwhile, the student network learns to produce similar class-
wise features from both labeled and unlabeled data. The features stored in the
memory bank are selected based on their quality and learned relevance for the
contrastive optimization. Besides, our module enforces the alignment of
unlabeled and labeled data (memory bank) in the feature space, which is
another unexploited idea in semi-supervised semantic segmentation. Our main
contributions are the following:

  * •

A novel semi-supervised semantic segmentation framework.

  * •

The use of a memory bank for high-quality pixel-level features from labeled
data.

  * •

A pixel-level contrastive learning scheme where elements are weighted based on
their relevance.

The effectiveness of our method is demonstrated on well-known semi-supervised
semantic segmentation benchmarks, reaching the state-of-the-art on different
set-ups. Besides, our approach can naturally tackle the semi-supervised domain
adaptation task, obtaining state-of-the-art results too. In all cases, the
improvements upon comparable methods increase with the percentage of unlabeled
data.

##  2 Related Work

This section summarizes relevant related work for semi-supervised learning and
contrastive learning, with particular emphasis on work related to semantic
segmentation.

###  2.1 Semi-Supervised Learning

##### Pseudo-Labeling

Pseudo-labeling leverages the idea of creating artificial labels for unlabeled
data [25, 33] by keeping the most likely predicted class by an existing model
[22]. The use of pseudo-labels is motivated by entropy minimization [13],
encouraging the network to output highly confident probabilities on unlabeled
data. Both pseudo-labeling and direct entropy minimization methods are
commonly used in semi-supervised scenarios [9, 20, 34, 29] showing great
performance. Our approach makes use of both pseudo-labels and direct entropy
minimization.

##### Consistency Regularization

Consistency regularization relies on the assumption that the model should be
invariant to perturbations, _e.g_., data augmentation, made to the same image.
This regularization is commonly applied by using two different methods:
distribution alignment [3, 32, 36], or augmentation anchoring [34]. While
distribution alignment enforces the prediction of perturbed and non-perturbed
samples to have the same class distribution, augmentation anchoring enforces
them to have the same semantic label. To produce high-quality non-perturbed
class distribution or prediction on unlabeled data, the Mean Teacher method
[37], proposes a teacher-student scheme where the teacher network is an
exponential moving average (EMA) of model parameters, producing more robust
predictions.

###  2.2 Semi-Supervised Semantic Segmentation

One common approach for semi-supervised semantic segmentation is to make use
of Generative Adversarial Networks (GANs) [12]. Hung _et al_. [17] propose to
train the discriminator to distinguish between confidence maps from labeled
and unlabeled data predictions. Mittal _et al_. [27] make use of a two-branch
approach, one branch enforcing low entropy predictions using a GAN approach
and another branch for removing false-positive predictions using a Mean
Teacher method [36]. A similar idea was proposed by Feng _et al_. [10], a
recent work that introduces Dynamic Mutual Training (DMT). DMT uses two models
and the model’s disagreement is used to re-weight the loss. DMT method also
followed the multi-stage training protocol from CBC [9], where pseudo-labels
are generated in an offline curriculum fashion. Other works are based on data
augmentation methods for consistency regularization. French _et al_. [11]
focus on applying CutOut [7] and CutMix [46], while Olsson _et al_. [29]
propose a data augmentation technique specific for semantic segmentation.

###  2.3 Contrastive Learning

The core idea of contrastive learning [15] is to create positive and negative
data pairs, to attract the positive and repulse the negative pairs in the
feature space. This technique has been used in supervised and self-supervised
set-ups. However, recent self-supervised methods have shown similar accuracy
with contrastive learning using positive pairs only by performing similarity
maximization with distillation [5, 14] or redundancy reduction [47].

As for semantic segmentation, these techniques has been mainly used as pre-
training [41, 44, 45]. Very recently, Wang _et al_. [40] have shown
improvements in supervised scenarios applying standard contrastive learning in
a pixel and region level for same-class supervised samples. Van _et al_. [39]
have shown the advantages of contrastive learning in unsupervised set-ups,
applying it between features from different saliency masks. Lai _et al_. [21]
proposed to use contrastive learning in a self-supervised fashion where
positive samples were the same pixel from different views/crops and negative
samples were different pixels from a different view, making the model
invariant to context information.

In this work, we propose to follow a positive-only contrastive learning based
on similarity maximization and distillation [5, 14]. This way, we boost the
performance on semi-supervised semantic segmentation in a simpler and more
computationally efficient fashion than standard contrastive learning [40].
Differently from previous works, our contrastive learning module tackles a
semi-supervised scenario aligning class-wise and per-pixel features from both
labeled and unlabeled data to features from all over the labeled set that are
stored in a memory bank. Contrary to previous contrastive learning works [43,
16] that saved image-level features in a memory bank, our memory bank saves
per-pixel features for the different semantic classes. Besides, as there is
not infinite memory for all dataset pixels, we propose to only save features
with the highest quality.

##  3 Method

Semi-supervised semantic segmentation is a per-pixel classification task where
two different sources of data are available: a small set of labeled samples
$\mathcal{X}{{}_{l}}=\left\\{x_{l},y_{l}\right\\}$, where $x_{l}$ are images
and $y_{l}$ their corresponding annotations, and a large set of unlabeled
samples $\mathcal{X}{{}_{u}}=\left\\{x_{u}\right\\}$.

Figure 2:  Supervised and self-supervised optimization. The student
$f_{\theta}$ is optimized with $\mathcal{L}_{\mathrm{sup}}$ for labeled data
($x_{l},y_{l}$). For unlabeled data $x_{u}$, the teacher $f_{\xi}$ computes
the pseudo-labels $\hat{y}_{u}$ that are later used for optimizing
$\mathcal{L}_{\mathrm{pseudo}}$ for pairs of augmented samples and pseudo-
labels ($x_{u}^{a}$, $\hat{y}_{u}$). Finally, $\mathcal{L}_{\mathrm{ent}}$ is
optimized for predictions from $x_{u}^{a}$.

To tackle this task, we propose to use a teacher-student scheme. The teacher
network $f_{\xi}$ creates robust pseudo-labels from unlabeled samples and
memory bank entries from labeled samples to teach the student network
$f_{\theta}$ to improve its segmentation performance.

##### Teacher-student scheme.

The learned weights $\theta$ of the student network $f_{\theta}$ are optimized
using the following loss:

$\mathcal{L}=\lambda_{sup}\mathcal{L}_{\mathrm{sup}}+\lambda_{pseudo}\mathcal{L}_{\mathrm{pseudo}}+\lambda_{ent}\mathcal{L}_{\mathrm{ent}}+\lambda_{contr}\mathcal{L}_{\mathrm{contr}}.$
(1)

$\mathcal{L}_{\mathrm{sup}}$ is a supervised learning loss on labeled samples
(Section 3.1). $\mathcal{L}_{\mathrm{pseudo}}$ and
$\mathcal{L}_{\mathrm{ent}}$ tackle pseudo-labels (Section 3.2) and entropy
minimization (Section 3.3) techniques, respectively, where pseudo-labels are
generated by the teacher network $f_{\xi}$. Finally,
$\mathcal{L}_{\mathrm{contr}}$ is our proposed positive-only contrastive
learning loss (Section 3.4).

Weights $\xi$ of the teacher network $f_{\xi}$ are an exponential moving
average of weights $\theta$ of the student network $f_{\theta}$ with a decay
rate $\tau\in[0,1]$. The teacher model provides more accurate and robust
predictions [37]. Thus, at every training step, the teacher network $f_{\xi}$
is not optimized by a gradient descent but updated as follows:

$\xi=\tau\xi+(1-\tau)\theta.\vspace{-2mm}$ (2)

Figure 3:  Contrastive learning optimization. At every iteration, features are
extracted by $f_{\xi}$ from labeled samples (see right part). These features
are projected, filtered by their quality, and then, ranked to finally only
store the highest-quality features into the memory bank. Concurrently, feature
vectors from input samples extracted by $f_{\theta}$ are fed to the projection
and prediction heads (see left part). Then, feature vectors are passed to a
self-attention module in a class-wise fashion, getting a per-sample weight.
Finally, input feature vectors are enforced to be similar to same-class
features from the memory bank.

###  3.1 Supervised Segmentation: $\mathcal{L}_{\mathrm{sup}}$

Our supervised semantic segmentation optimization, applied to the labeled data
$\mathcal{X}{{}_{l}}$, follows the standard optimization with the weighted
cross-entropy loss. Let $\mathcal{H}$ be the weighted cross-entropy loss
between two lists of $N$ per-pixel class probability distributions
$y_{1},y_{2}$:

$\mathcal{H}(y_{1},y_{2})=-\frac{1}{N}\sum_{n=1}^{N}\sum_{c=1}^{C}y_{2}^{(n,c)}\log(y_{1}^{(n,c)})\alpha^{c}\beta^{n},$
(3)

where $C$ is the number of classes to classify, $N$ is the number of elements,
_i.e_., pixels in $y_{1}$, $\alpha^{c}$ is a per-class weight, and,
$\beta^{n}$ is a per-pixel weight. Specific values of $\alpha^{c}$ and
$\beta^{n}$ are detailed in Section 4.2. The supervised loss (see top part of
Figure 2) is defined as follows:

$\mathcal{L}_{\mathrm{sup}}=\mathcal{H}\left(f_{\theta}\left(x_{l}^{a}\right),y_{l}\right),$
(4)

where $x_{l}^{a}$ is a weak augmentation of $x_{l}$ (see Section 4.2 for
augmentation details).

###  3.2 Learning from Pseudo-labels: $\mathcal{L}_{\mathrm{pseudo}}$

The key to the success of semi-supervised learning is to learn from unlabeled
data. One idea our approach exploits is to learn from pseudo-labels. In our
case, pseudo-labels are generated by the teacher network $f_{\xi}$ (see Figure
2). For every unlabeled sample $x_{u}$, the pseudo-labels $\hat{y}_{u}$ are
computed following this equation:

$\hat{y}_{u}=\arg\max f_{\xi}\left(x_{u}\right),$ (5)

where $f_{\xi}$ predicts a class probability distribution. Note that pseudo-
labels are computed at each training iteration.

Consistency regularization is introduced with augmentation anchoring, _i.e_.,
computing different data augmentation for each sample $x_{u}$ on the same
batch, helping the model to converge to a better solution [34]. The pseudo-
labels loss for unlabeled data $\mathcal{X}{{}_{u}}$ is calculated by the
cross-entropy:

$\mathcal{L}_{\mathrm{pseudo}}=\frac{1}{A}\sum_{a=1}^{A}\mathcal{H}\left(f_{\theta}\left(x_{u}^{a}\right),\hat{y}_{u}\right),$
(6)

where $x_{u}^{a}$ is a strong augmentation of $x_{u}$ and $A$ is the number of
augmentations we apply to sample $x_{u}$ (see Section 4.2 for augmentation
details).

###  3.3 Direct Entropy Minimization: $\mathcal{L}_{\mathrm{ent}}$

Direct entropy minimization is applied on the class distributions predicted by
the student network from unlabeled samples $x_{u}$ as a regularization loss:

$\mathcal{L}_{ent}=-\frac{1}{A}\frac{1}{N}\sum_{a=1}^{A}\sum_{n=1}^{N}\sum_{c=1}^{C}f_{\theta}\left(x_{u}^{a,n,c}\right)\log
f_{\theta}\left(x_{u}^{a,n,c}\right),$ (7)

where $C$ is the number of classes to classify, $N$ is the number of pixels
and $A$ is the number of augmentations.

###  3.4 Contrastive Learning: $\mathcal{L}_{\mathrm{contr}}$

Figure 3 illustrates our proposed contrastive optimization inspired by
positive-only contrastive learning works based on similarity maximization and
distillation [5, 14]. In our approach, a memory bank is filled with high-
quality feature vectors from the teacher $f_{\xi}$ (right part of Figure 3).
Concurrently, the student $f_{\theta}$ extracts feature vectors from either
$\mathcal{X}{{}_{l}}$ or $\mathcal{X}{{}_{u}}$. In a per-class fashion, every
feature is passed through a simple self-attention module that serves as per-
feature weighting in the contrastive loss. Finally, the loss enforces the
weighted feature vectors from the student to be similar to feature vectors
from the memory bank. As the memory bank contains high-quality features from
all labeled samples, the contrastive loss helps to create a better class
separation in the feature space across the whole dataset as well as aligning
the unlabeled data distribution with the labeled data distribution.

##### Optimization.

Let $f_{\theta^{-}}$ be the student network without the classification layer
and $\left\\{{x,y}\right\\}$ a training sample either from
$\left\\{\mathcal{X}{{}_{l}},\mathrm{Y}{{}_{l}}\right\\}$ or
$\\{\mathcal{X}{{}_{u}},\mathrm{\hat{Y}}{{}_{u}}\\}$. The first step is to
extract all feature vectors: $V=f_{\theta^{-}}(x)$. The feature vectors $V$
are then fed to a projection head, $Z=g_{\theta}(V)$, and a prediction head,
$P=q_{\theta}(Z)$, following [5, 14], where $g_{\theta}$ and $q_{\theta}$ are
two different Multi-Layer Perceptrons (MLPs). Next, $P$ is grouped by the
different semantic classes in $y$.

Let $P_{c}=\left\\{p_{c}\right\\}$ be the set of prediction vectors from $P$
of a class $c$. Let $Z_{c}^{\prime}=\left\\{z_{c}^{\prime}\right\\}$ be the
set of projection vectors of class $c$ obtained by the teacher,
$Z^{\prime}=g_{\xi}(f_{\xi^{-}}(x))$ from the labeled examples stored in the
memory bank.

Next, we learn which feature vectors ($p_{c}$ and $z_{c}^{\prime}$) are
beneficial for the contrastive task, by assigning per-feature learned weights
(Equation 8) that will serve as a weighting factor (Equation 10) for the
contrastive loss function (Equation 11). These per-feature weights are
computed using class-specific attention modules $S_{c,\theta}$ (see Section
4.2 for further details) that generate a single value ($\mathrm{w}\in[0,1]$)
for every $z_{c}^{\prime}$ and $p_{c}$ feature. Following [35] we L1 normalize
these weights to prevent converging to the trivial all-zeros solution. For the
prediction vectors $P_{c}$ case, the weights $\mathrm{w}{{}_{p_{c}}}$ are then
computed as follows:

$\mathrm{w}{{}_{p_{c}}}=\frac{N_{P_{c}}}{\sum_{\mathrm{p_{i}}\in
P_{c}}S_{c,\theta}(p_{i})}S_{c,\theta}(p_{c}),$ (8)

where $N_{P_{c}}$ is the number of elements in $P_{c}$. Equation 8 is used to
compute $\mathrm{w}{{}_{z_{c}^{\prime}}}$ too, changing $Z_{c}^{\prime}$ and
$z_{c}^{\prime}$ for $P_{c}$ and $p_{c}^{\prime}$.

The contrastive loss enforces prediction vectors $p_{c}$ to be similar to
projection vectors $z_{c}^{\prime}$ as in [5, 14] (in our case, projection
vectors are in the memory bank). For that, we use the cosine similarity as the
similarity measure ${C}$:

$\mathcal{C}(p_{c},z_{c}^{\prime})=\frac{\left\langle
p_{c},z_{c}^{\prime}\right\rangle}{\left\|p_{c}\right\|_{2}\cdot\left\|z_{c}^{\prime}\right\|_{2}},$
(9)

where, the weighted distance between predictions and memory bank entry is
computed by:

$\mathcal{{D}}(p_{c},z_{c}^{\prime})=\mathrm{w}{{}_{p_{c}}}\mathrm{w}{{}_{z_{c}^{\prime}}}(1-\mathcal{C}(p_{c},z_{c}^{\prime})),$
(10)

and, our contrastive loss is computed as follows:

$\mathcal{L}_{contr}=\frac{1}{C}\frac{1}{N_{p_{c}}}\frac{1}{N_{z_{c}^{\prime}}}\sum_{c=1}^{C}\sum_{{p_{c}}\in
P_{c}}\sum_{{z_{c}^{\prime}}\in
Z_{c}^{\prime}}\mathcal{D}(p_{c},z_{c}^{\prime}).$ (11)

##### Memory Bank.

The memory bank is the data structure that maintains the target feature
vectors $z_{c}^{\prime}$, $\psi$ for each class $c$, used in the contrastive
loss. As there is not infinite space for saving all pixels of the labeled
data, we propose to store only a subset of the feature vectors from labeled
data with the highest quality. As shown in Figure 3, the memory bank is
updated on every training iteration with a subset of $z_{c}^{\prime}\in
Z^{\prime}$ generated by the teacher. To select what subset of $Z^{\prime}$ is
included in the memory bank, we first perform a Feature Quality Filter (FQF),
where we only keep features that lead to an accurate prediction when the
classification layer is applied, $y=\arg\max f_{\xi}(x_{l})$, having
confidence higher than a threshold, $f_{\xi}(x_{l})>\phi$. The remaining
$Z^{\prime}$ are grouped by classes $Z_{c}^{\prime}$. Finally, instead of
picking randomly a subset of every $Z_{c}^{\prime}$ to update the memory bank,
we make use of the class-specific attention modules $S_{c,\xi}$. We get
ranking scores $R_{c}=S_{c,\xi}(Z_{c}^{\prime})$ to sort $Z_{c}^{\prime}$ and
we update the memory bank only with the top-K highest-scoring vectors. The
memory bank is a First In First Out (FIFO) queue per class for computation and
time efficiency. This way it maintains recent high-quality feature vectors in
a very efficient fashion computation-wise and time-wise. Detailed information
about the hyper-parameters is included in Section 4.2.

##  4 Experiments

This section describes evaluation set-up and the evaluation of our method on
different benchmarks for semi-supervised semantic segmentation, including a
semi-supervised domain adaptation, and a detailed ablation study.

###  4.1 Datasets

  * •

Cityscapes [6]. It is a real urban scene dataset composed of $2975$ training
and $500$ validation samples, with $19$ semantic classes.

  * •

PASCAL VOC 2012 [8]. It is a natural scenes dataset with $21$ semantic
classes. The dataset has $10582$ and $1449$ images for training and validation
respectively.

  * •

GTA5 [30]. It is a synthetic dataset captured from a video game with realistic
urban-like scenarios with $24966$ images in total. The original dataset
provides $33$ different categories but, following [42], we only use the $19$
classes that are shared with Cityscapes.

###  4.2 Implementation details

##### Architecture.

We use DeepLab networks [4] in our experiments. For the ablation study and
most benchmarking experiments, DeepLabv2 with a ResNet101 backbone is used to
have similar settings to previous works [29, 9, 17, 27]. DeepLabv3+ with
Resnet50 backbone is also used to equal comparison with [26, 21]. $\tau$ is
set from 0.995 to 1 during training in (Equation 2).

The prediction and projection heads follow [14]: Linear (256) $\rightarrow$
BatchNorm [18] $\rightarrow$ Relu [28] $\rightarrow$ Linear (256). The
proposed class-specific attention modules follow a similar architecture:
Linear (256 )$\rightarrow$ BatchNorm $\rightarrow$ LeakyRelu [24]
$\rightarrow$ Linear (1) $\rightarrow$ Sigmoid. We use $2\times N_{classes}$
attention modules since they are used in a class-wise fashion. In particular,
two modules per class are used because we have different modules for
projection or prediction feature vectors.

##### Optimization.

For all experiments, we train for 150K iterations using the SGD optimizer with
a momentum of $0.9$. The learning rate is set to $2\times 10^{-4}$ for
DeepLabv2 and $4\times 10^{-4}$ for DeepLabv3+ with a poly learning rate
schedule. For the Cityscapes and GTA5 datasets, we use a crop size of
$512\times 512$ and batch sizes of 5 and 7 for Deeplabv2 and Deeplabv3+,
respectively. For Pascal VOC, we use a crop size of $321\times 321$ and batch
sizes of 14 and 20 for Deeplabv2 and Deeplabv3+, respectively. Cityscapes
images are resized to $512\times 1024$ before cropping when Deeplabv2 is used
for a fair comparison with [29, 9, 17, 27]. The different loss weights in
(Equation 1) are set as follows for all experiments: $\lambda_{sup}=1$,
$\lambda_{pseudo}=1$, $\lambda_{ent}=0.01$, $\lambda_{contr}=0.1$. An
exception is made for the first 2K training iterations where
$\lambda_{contr}=0$ and $\lambda_{pseudo}=0$ to make sure predictions have
some quality before being taken into account. Regarding the per-pixel weights
($\beta^{n}$) from $\mathcal{H}$ in (Equation 3), we set it to 1 for
$\mathcal{L}_{\mathrm{sup}}$. For $\mathcal{L}_{\mathrm{pseudo}}$, we follow
[9] weighting each pixel with its corresponding pseudo-label confidence with a
sharpening operation, $f_{\xi}\left(x_{u}\right)^{s}$, where we set $s=6$. As
for the per-class weights $\alpha^{c}$ in (Equation 3), we perform a class
balancing for the Cityscapes and GTA5 datasets by setting
$\alpha^{c}=\sqrt{\frac{f_{m}}{f_{c}}}$ with $f_{c}$ being the frequency of
class $c$ and $f_{m}$ the median of all class frequencies. In semi-supervised
settings the amount of labels, $\mathrm{Y}_{l}$, is usually small. For a more
meaningful estimation, we compute these frequencies not only from
$\mathrm{{Y}}{{}_{l}}$ but also from $\mathrm{\hat{Y}}{{}_{u}}$. For the
Pascal VOC we set $\alpha^{c}=1$ as the class balancing does not have a
beneficial effect.

##### Other details.

DeepLab’s output resolution is $\times 8$ lower than the input resolution. For
feature comparison during training, we keep the output resolution and
downsample the labels reducing memory requirements and computation.

The memory bank size is fixed to $\psi=256$ vectors per class (see Section 4.4
for more details). The confidence threshold $\phi$ for accepting features is
set to $0.95$. The number of vectors added to the memory bank at each
iteration, for each image, and for each class is set as
$\max(1,\frac{\psi}{|\mathcal{X}{{}_{l}}|})$, where $|\mathcal{X}{{}_{l}}|$ is
the number of labeled samples.

A single NVIDIA Tesla V100 GPU is used for all experiments. All our reported
results are the mean of three different runs with different labeled/unlabeled
data splits.

Following [29, 37], the segmentation is performed with the student
$f_{\theta}$ in the experimental validation, although the teacher would lead
to a slightly better performance [34].

##### Data augmentation.

We use two different augmentation set-ups, a weak one for labeled samples and
a strong set-up for unlabeled samples, following [29] with minor modifications
(Table 1 describes the followed data augmentation scheme in our method).
Besides, we set $A=2$ (Equation 6) as the number of augmentations for each
sample.

Table 1: Strong and weak data augmentation set-ups Parameter | Weak | Strong
---|---|---
Flip probability | $0.50$ | $0.50$
Resize $\times[0.75,1.75$] probability | $0.50$ | $0.80$
Color jittering probability | $0.20$ | $0.80$
Brightness adjustment max intensity | $0.15$ | $0.30$
Contrast adjustment max intensity | $0.15$ | $0.30$
Saturation adjustment max intensity | $0.075$ | $0.15$
Hue adjustment max intensity | $0.05$ | $0.10$
Gaussian blurring probability | $0$ | $0.20$
ClassMix probability | $0.20$ | $0.80$

###  4.3 Benchmark Experiments

Following experiments compare our method with state-of-the-art methods in
different semi-supervised scenarios.

####  4.3.1 Semi-supervised Semantic Segmentation

Figure 4: Qualitative results on Cityscapes. Models are trained with
$\frac{1}{8}$ of the labeled data using Deeplabv2 with ResNet-101. From left
to right: Image, manual annotations, ClassMix [29], DMT [10], our approach.

method | 1/30 | 1/8 | 1/4 | FS
---|---|---|---|---
Architecture: Deeplabv2 with ResNet-101 backbone
Adversarial [17]+ | — |  $58.8$ (-$7.6$) |  $62.3$ (-$4.1$) | $66.4$
s4GAN [27]* | — |  $59.3$ (-$6.7$) |  $61.9$ -($4.9$) | $66.0$
French _et al_. [11]* |  $51.2$ (-$16.3$) |  $60.3$ (-$7.2$) |  $63.9$ (-$3.6$) | $67.5$
CBC [9]+ |  $48.7$ (-$18.2$) |  $60.5$ (-$6.4$) |  $64.4$ (-$2.5$) | $66.9$
ClassMix [29]+ |  $54.1$ (-$12.1$) |  $61.4$ (-$4.8$) |  $63.6$ (-$2.6$) | $66.2$
DMT [10]*+ |  $54.8$ (-$13.4$) |  $63.0$ (-$5.2$) | — | $68.2$
Ours* |  $58.0$ (-$8.4$) |  $63.0$ (-$3.4$) |  $64.8$ (-$1.6$) | $66.4$
Ours+ | 59.4 (-7.9) | 64.4 (-2.9) | 65.9 (-1.4) | $67.3$
Architecture: Deeplabv3+ with ResNet-50 backbone
Error-corr [26]* | — |  $67.4$ (-$7.4$) |  $70.7$ (-$4.1$) | $74.8$
Lai _et al_. [21]* | — |  $69.7$ (-$7.8$) |  72.7 (-$4.8$) | $77.5$
Ours* |  $64.9$ (-$9.3$) | 70.1 (-4.1) | $71.7\textbf{ (-2.5)}$ | $74.2$

* ImageNet pre-training, + COCO pre-training

Table 2: Performance (Mean IoU) for the Cityscapes val set for different
labeled-unlabeled ratios and, in parentheses, the difference w.r.t. the
corresponding fully supervised (FS) result.

method | 1/50 | 1/20 | 1/8 | FS
---|---|---|---|---
Architecture: Deeplabv2 with ResNet-101 backbone
Adversarial [17]+ |  $57.2$ (-$17.7$) |  $64.7$ (-$10.2$) |  $69.5$ (-$5.4$) | $74.9$
s4GAN [27]+ |  $63.3$ (-$10.3$) |  $67.2$ (-$6.4$) |  $71.4$ (-$2.2$) | $73.6$
French _et al_. [11]* |  $64.8$ (-$7.7$) |  $66.5$ (-$6.0$) |  $67.6$ (-$4.9$) | $72.5$
CBC [9]+ |  $65.5$ (-$8.1$) |  $69.3$ (-$4.3$) |  $70.7$ (-$2.9$) | $73.6$
ClassMix [29]+ |  $66.2$ (-$7.9$) |  $67.8$ (-$6.3$) |  $71.0$ (-$3.1$) | $74.1$
DMT [10]*+ |  $67.2$ (-$7.6$) |  $69.9$ (-$4.9$) | 72.7 (-2.1) | $74.8$
Ours* |  $65.6$ (-$7.0$) |  $67.8$ (-$4.8$) |  $69.9$ (-$2.7$) | $72.6$
Ours+ | 68.2 (-5.9) | 70.1 (-4.0) |  $71.8$ (-$2.3$) | $74.1$
Architecture: Deeplabv3+ with ResNet-50 backbone
Error-corr [26]* | — | — |  $70.2$ (-$6.1$) | $76.3$
Lai _et al_. [21]* | — | — |  72.4 (-$4.1$) | $76.5$
Ours* |  $63.4$ (-$12.5$) |  $69.1$ (-$6.8$) | $72.0\textbf{ (-3.9)}$ | $75.9$

* ImageNet pre-training, + COCO pre-training

Table 3: Performance (Mean IoU) for the Pascal VOC val set for different
labeled-unlabeled ratios and, in parentheses, the difference w.r.t. the
corresponding fully supervised (FS) result.

##### Cityscapes.

Table 2 compares different methods on the Cityscapes benchmark for different
labeled-unlabeled rates: $\frac{1}{30}$, $\frac{1}{8}$ and, $\frac{1}{4}$.
Fully Supervised (FS) scenario, where all images are labeled, is also shown as
a reference. As shown in the table, our approach outperforms the current
state-of-the-art by a significant margin. The performance difference is
increasing as less labeled data is available, demonstrating the effectiveness
of our approach. This is particularly important since the goal of semi-
supervised learning is to learn with as little supervision as possible. Note
that the upper bound for each method is shown in the fully supervised setting
(FS). Figure 4 shows a visual comparison of the top-performing methods on
different relevant samples from Cityscapes. Note that Lai _et al_. [21] have a
higher FS baseline since they use a higher batch size and crop size among
other differences in the set-up.

##### Pascal VOC.

Table 3 shows the comparison of different methods on the Pascal VOC benchmark,
using different labeled-unlabeled rates: $\frac{1}{50}$, $\frac{1}{20}$ and,
$\frac{1}{8}$. Our proposed method outperforms previous methods for most of
the configurations. Like in the previous benchmark, our method presents larger
benefits for the more challenging cases, _i.e_., only a small fraction of data
is labeled ($\frac{1}{50}$). This demonstrates that the proposed approach is
especially effective to learn from unlabeled data.

City |  ASS [42] |  Liu _et al_. [23] | Ours | Ours
---|---|---|---|---
Labels | with domain adaptation | no adaptation
$1/30$ | $54.2$ | $55.2$ | 59.9 | $58.0$
$1/15$ | $56.0$ | $57.0$ | 62.0 | $59.9$
$1/6$ | $60.2$ | $60.4$ | 64.2 | $63.7$
$1/3$ | $64.5$ | $64.6$ | 65.6 | $65.1$
Table 4: Mean IoU in Cityscapes val set. Central columns evaluate the semi-
supervised domain adaptation task (GTA5 $\rightarrow$ Cityscapes). The last
column evaluates a semi-supervised setting in Cityscapes (no adaptation).
Different labeled-unlabeled ratios for Cityscapes are compared. All methods
use ImageNet pre-trained Deeplabv2 with ResNet-101 backbone.

####  4.3.2 Semi-supervised Domain Adaptation

Semi-supervised domain adaptation for semantic segmentation differs from the
semi-supervised set-up in the availability of labeled data from another
domain. That is, apart from having
$\mathcal{X}{{}_{l}}=\left\\{x_{l},y_{l}\right\\}$ and
$\mathcal{X}{{}_{u}}=\left\\{x_{u}\right\\}$ from the target domain, a large
set of labeled data from another domain is also available:
$\mathcal{X}{{}_{d}}=\left\\{x_{d},y_{d}\right\\}$.

Our method can naturally tackle this task by evenly sampling from both
$\mathcal{X}{{}_{l}}$ and $\mathcal{X}{{}_{d}}$ as our labeled data when
optimizing $\mathcal{L}_{\mathrm{sup}}$ and $\mathcal{L}_{\mathrm{contr}}$.
However, the memory bank only stores features from the target domain
$\mathcal{X}{{}_{l}}$. In this way, both the features from unlabeled data
$\mathcal{X}{{}_{u}}$, and the features from the other domain
$\mathcal{X}{{}_{d}}$ are aligned with those from $\mathcal{X}{{}_{l}}$.

Following ASS [42, 23], we take the GTA5 dataset as $\mathcal{X}{{}_{d}}$,
where all elements are labeled, and the Cityscapes is the target domain
consisting of a small set of labeled data $\mathcal{X}{{}_{l}}$ and a large
set of unlabeled samples $\mathcal{X}{{}_{u}}$. Table 4 compares the results
of our method with previous methods [42, 23] where all methods use ImageNet
pre-training. For reference, we also show the results of our approach with no
adaptation, _i.e_., only training on the target domain Cityscapes, as we do
for the semi-supervised set up from the previous experiment (Table 2). We can
see that our approach benefits from the use of the other domain data (GTA5),
especially where there is little labeled data available ($\frac{1}{30}$), as
it could be expected. Our method outperforms ASS by a large margin in all the
different set-ups. As in previous experiments, our improvement is more
significant when the amount of available labeled data is smaller.

###  4.4 Ablation Experiments

The following experiments study the impact of the different components of the
proposed approach. The evaluation is done on the Cityscapes data, since it
provides more complex scenes compared to Pascal VOC. We select the challenging
labeled data ratio of $\frac{1}{30}$.

$\mathcal{L}_{\mathrm{sup}}$ | $\mathcal{L}_{\mathrm{pseudo}}$ | $\mathcal{L}_{\mathrm{ent}}$ | $\mathcal{L}_{\mathrm{contr}}$ | mIoU
---|---|---|---|---
✓ |  |  |  | $49.5$
✓ | ✓ |  |  | $56.7$
✓ |  | ✓ |  | $52.2$
✓ |  |  | ✓ | $54.4$
✓ | ✓ | ✓ |  | $57.4$
✓ | ✓ |  | ✓ | $59.0$
✓ |  | ✓ | ✓ | $57.3$
✓ | ✓ | ✓ | ✓ | $59.4$
Table 5: Ablation study on the different losses included (Equation 1). Mean
IoU obtained on Cityscapes benchmark ($\frac{1}{30}$ available labels,
Deeplabv2-ResNet101 COCO pre-trained).

##### Losses impact.

Table 5 shows the impact of every loss used by the proposed method. We can
observe that the four losses are complementary, getting a $10$ mIoU increase
over our baseline model, using only the supervised training when
$\frac{1}{30}$ of the Cityscapes labeled data is available. Note that our
proposed contrastive learning module $\mathcal{L}_{\mathrm{contr}}$ is able to
get $54.32$ mIoU even without any other complementary loss, which is the
previous state-of-the-art for this set-up (see Table 2). Adding the
$\mathcal{L}_{\mathrm{pseudo}}$ significantly improves the performance and
then, adding $\mathcal{L}_{\mathrm{ent}}$ regularization loss gives a little
extra performance gain.

Note that at testing time, our approach only uses the student network
$f_{\theta}$, adding zero additional computational cost. At training time, for
the experiment of Table 5 having an input resolution of $512\times 512$ with a
forward pass cost of 372.04 GFLOPs, our method performs $1151.19$ GFLOPs for
one training step using one labeled image and one unlabeled image, compared to
the $1488.16$ GFLOPs from [10] or $1116.12$ GFLOPs from [29]. The total number
of GFLOPs come from $372.04$ for computing labeled image predictions, $372.04$
for the unlabeled image predictions, $372.04$ for computing the pseudo-labels
and, $35.07$ for our contrastive module, which mainly include the computation
of the prediction and projection heads ($8.59$), the class-specific attention
modules ($15.96$) and, the distance between the input features and memory bank
features ($10.52$).

##### Contrastive learning module.

Table 6 shows an ablation on the influence of the contrastive learning module
for different values of $\lambda_{contr}$ (Equation 1). As expected, if this
value is too low, the effect gets diluted, with similar performance as if the
proposed loss is not used at all (see Table 5). High values are also
detrimental, probably because it acts as increasing the learning rate vastly,
which hinders the optimization. The best performance is achieved when this
contrastive loss weight is a little lower than the segmentation losses
$\mathcal{L}_{sup}$ and $\mathcal{L}_{pseudo}$ ($\lambda_{contr}=10^{-1}$).

The effect of the size (per-class) of our memory bank is studied in Table 7.
As expected, higher values lead to stronger performances, although from 256 up
they tend to maintain similarly. Because all the elements from the memory bank
are used during the contrastive optimization (Equation 11) the larger the
memory bank is, the more computation and memory it requires. Therefore, we
select $256$ as our default value.

Table 8 studies the effect of the main components used in the proposed
contrastive learning module. The base configuration of the module which
includes our simplest implementation of the per-pixel contrastive learning
using the memory bank, still presents a performance gain compared to not using
the contrastive learning module (57.4 mIoU from 5). Generating and selecting
good quality prototypes is the most important factor. This is done both by the
Feature Quality Filter (FQF), _i.e_., checking that the feature leads to an
accurate and confident prediction, and extracting them with the teacher
network $f_{\xi}$. Then, using the class-specific attention $S_{c,\theta}$ to
weight every sample (both from the memory bank and input sample) is also
beneficial, acting as a learned sampling method.

$\lambda_{contr}$ | $10^{4}$ | $10^{2}$ | $10^{1}$ | $10^{0}$ | $10^{-1}$ | $10^{-2}$ | $10^{-4}$
---|---|---|---|---|---|---|---
mIoU | $50.3$ | $51.4$ | $54.8$ | $59.1$ | $59.4$ | $58.7$ | $57.6$

Table 6: Effect of different values for the factor $\lambda_{contr}$ (Equation 1) that weights the effect of the contrastive loss $\mathcal{L}_{\mathrm{contr}}$. Results on Cityscapes benchmark ($\frac{1}{30}$ available labels, Deeplabv2-ResNet101 COCO pre-trained).  $\psi$ | $32$ | $64$ | $128$ | $256$ | $512$
---|---|---|---|---|---
mIoU | $58.7$ | $58.9$ | $59.2$ | $59.4$ | $59.3$
Table 7: Effect of our memory bank size (features per-class), $\psi$. Results
on Cityscapes benchmark ($\frac{1}{30}$ available labels, Deeplabv2-ResNet101
COCO pre-trained).

Base | $f_{\xi}$ | $S_{c,\theta}$ | FQF | mIoU
---|---|---|---|---
✓ |  |  |  | $58.3$
✓ | ✓ |  |  | $58.7$
✓ |  | ✓ |  | $58.6$
✓ |  |  | ✓ | $59.0$
✓ | ✓ | ✓ | ✓ | $59.4$

$f_{\xi}$: Use teacher model $f_{\xi}$ to extract features instead of
$f_{\theta}$
$S_{c,\theta}$: Use class-specific attention $S_{c,\theta}$ to weight every
feature
FQF: Feature Quality Filter for Memory Bank update

Table 8: Ablation study of our contrastive learning module main components.
Results on Cityscapes benchmark ($\frac{1}{30}$ available labels, using
Deeplabv2-ResNet101 COCO pre-trained).

##### Future direction.

Our proposed approach could potentially be applied to other semi-supervised
tasks like object detection or instance segmentation. The straightforward way
is to perform the proposed contrastive learning using the features from the
semantic head of the detection or instance segmentation networks, i.e., the
part of the network that outputs the semantic class of the object or instance.
The method is currently restricted by the number of classes and number of
memory bank entries per class. A future step to solve this problem could be to
cluster the feature vectors per class and save only cluster centers of the
class features, similar to the very recent work from Zhang et. al [48] for
domain adaptation based on prototypical learning.

##  5 Conclusion

This paper presents a novel approach for semi-supervised semantic
segmentation. Our work shows the benefits of incorporating positive-only
contrastive learning techniques to solve this semi-supervised task. The
proposed contrastive learning module boosts the performance of semantic
segmentation in these settings. Our new module contains a memory bank that is
continuously updated with selected features from those produced by a teacher
network from labeled data. These features are selected based on their quality
and relevance for the contrastive learning. Our student network is optimized
for both labeled and unlabeled data to learn similar class-wise features to
those in the memory bank. The use of contrastive learning at a pixel-level has
been barely exploited and this work demonstrates the potential and benefits it
brings to semi-supervised semantic segmentation and semi-supervised domain
adaptation. Our results outperform state-of-the-art on several public
benchmarks, with particularly significant improvements on the more challenging
set-ups, _i.e_., when the amount of available labeled data is low.

##  6 Acknowledgments

This work was partially funded by FEDER/ Ministerio de Ciencia, Innovación y
Universidades/ Agencia Estatal de Investigación/RTC-2017-6421-7,
PGC2018-098817-A-I00 and PID2019-105390RB-I00, Aragón regional government (DGA
T45 17R/FSE) and the Office of Naval Research Global project ONRG-
NICOP-N62909-19-1-2027.

## References

  * [1] Iñigo Alonso, Luis Riazuelo, and Ana C Murillo.  Mininet: An efficient semantic segmentation convnet for real-time robotic applications.  IEEE Transactions on Robotics (T-RO), 2020.
  * [2] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla.  Segnet: A deep convolutional encoder-decoder architecture for image segmentation.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(12):2481–2495, 2017.
  * [3] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel.  Mixmatch: A holistic approach to semi-supervised learning.  In Advances in Neural Information Processing Systems, pages 5049–5059, 2019.
  * [4] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  IEEE transactions on pattern analysis and machine intelligence, 40(4):834–848, 2017.
  * [5] Xinlei Chen and Kaiming He.  Exploring simple siamese representation learning.  arXiv preprint arXiv:2011.10566, 2020.
  * [6] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In Proceedings of IEEE Conference on CVPR, pages 3213–3223, 2016\.
  * [7] Terrance DeVries and Graham W Taylor.  Improved regularization of convolutional neural networks with cutout.  arXiv preprint arXiv:1708.04552, 2017.
  * [8] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  International journal of computer vision, 88(2):303–338, 2010.
  * [9] Zhengyang Feng, Qianyu Zhou, Guangliang Cheng, Xin Tan, Jianping Shi, and Lizhuang Ma.  Semi-supervised semantic segmentation via dynamic self-training and class-balanced curriculum.  arXiv preprint arXiv:2004.08514, 2020.
  * [10] Zhengyang Feng, Qianyu Zhou, Qiqi Gu, Xin Tan, Guangliang Cheng, Xuequan Lu, Jianping Shi, and Lizhuang Ma.  Dmt: Dynamic mutual training for semi-supervised learning.  arXiv preprint arXiv:2004.08514, 2020.
  * [11] Geoff French, Samuli Laine, Timo Aila, and Michal Mackiewicz.  Semi-supervised semantic segmentation needs strong, varied perturbations.  In 29th British Machine Vision Conference, BMVC 2020, 2019.
  * [12] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  Advances in neural information processing systems, 27:2672–2680, 2014.
  * [13] Yves Grandvalet and Yoshua Bengio.  Semi-supervised learning by entropy minimization.  Advances in neural information processing systems, 17:529–536, 2004\.
  * [14] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap your own latent: A new approach to self-supervised learning.  arXiv preprint arXiv:2006.07733, 2020.
  * [15] Raia Hadsell, Sumit Chopra, and Yann LeCun.  Dimensionality reduction by learning an invariant mapping.  In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), volume 2, pages 1735–1742. IEEE, 2006.
  * [16] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729–9738, 2020.
  * [17] Wei-Chih Hung, Yi-Hsuan Tsai, Yan-Ting Liou, Yen-Yu Lin, and Ming-Hsuan Yang.  Adversarial learning for semi-supervised semantic segmentation.  arXiv preprint arXiv:1802.07934, 2018.
  * [18] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  arXiv preprint arXiv:1502.03167, 2015.
  * [19] Simon Jégou, Michal Drozdzal, David Vazquez, Adriana Romero, and Yoshua Bengio.  The one hundred layers tiramisu: Fully convolutional densenets for semantic segmentation.  In CVPR Workshops. IEEE, 2017.
  * [20] Tarun Kalluri, Girish Varma, Manmohan Chandraker, and CV Jawahar.  Universal semi-supervised semantic segmentation.  In Proceedings of the IEEE International Conference on Computer Vision, pages 5259–5270, 2019.
  * [21] Xin Lai, Zhuotao Tian, Li Jiang, Shu Liu, Hengshuang Zhao, Liwei Wang, and Jiaya Jia.  Semi-supervised semantic segmentation with directional context-aware consistency.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1205–1214, 2021.
  * [22] Dong-Hyun Lee.  Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks.  In Workshop on challenges in representation learning, ICML, volume 3, 2013.
  * [23] Weizhe Liu, David Ferstl, Samuel Schulter, Lukas Zebedin, Pascal Fua, and Christian Leistner.  Domain adaptation for semantic segmentation via patch-wise contrastive learning.  arXiv preprint arXiv:2104.11056, 2021.
  * [24] Andrew L Maas, Awni Y Hannun, and Andrew Y Ng.  Rectifier nonlinearities improve neural network acoustic models.  In Proceedings icml, volume 30, page 3, 2013.
  * [25] Geoffrey J McLachlan.  Iterative reclassification procedure for constructing an asymptotically optimal rule of allocation in discriminant analysis.  Journal of the American Statistical Association, 70(350):365–369, 1975.
  * [26] Robert Mendel, Luis Antonio de Souza, David Rauber, João Paulo Papa, and Christoph Palm.  Semi-supervised segmentation based on error-correcting supervision.  In European Conference on Computer Vision, pages 141–157. Springer, 2020.
  * [27] Sudhanshu Mittal, Maxim Tatarchenko, and Thomas Brox.  Semi-supervised semantic segmentation with high-and low-level consistency.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019\.
  * [28] Vinod Nair and Geoffrey E Hinton.  Rectified linear units improve restricted boltzmann machines.  In ICML, 2010.
  * [29] Viktor Olsson, Wilhelm Tranheden, Juliano Pinto, and Lennart Svensson.  Classmix: Segmentation-based data augmentation for semi-supervised learning.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2021.
  * [30] Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun.  Playing for data: Ground truth from computer games.  In European conference on computer vision, pages 102–118. Springer, 2016.
  * [31] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
  * [32] Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen.  Regularization with stochastic transformations and perturbations for deep semi-supervised learning.  In Advances in neural information processing systems, pages 1163–1171, 2016.
  * [33] H Scudder.  Probability of error of some adaptive pattern-recognition machines.  IEEE Transactions on Information Theory, 11(3):363–371, 1965.
  * [34] Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel.  Fixmatch: Simplifying semi-supervised learning with consistency and confidence.  arXiv preprint arXiv:2001.07685, 2020.
  * [35] Shuyang Sun, Liang Chen, Gregory Slabaugh, and Philip Torr.  Learning to sample the most useful training patches from images.  arXiv preprint arXiv:2011.12097, 2020.
  * [36] Antti Tarvainen and Harri Valpola.  Weight-averaged consistency targets improve semi-supervised deep learning results.
  * [37] Antti Tarvainen and Harri Valpola.  Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.  In Advances in neural information processing systems, pages 1195–1204, 2017.
  * [38] Jesper E Van Engelen and Holger H Hoos.  A survey on semi-supervised learning.  Machine Learning, 109(2):373–440, 2020.
  * [39] Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, and Luc Van Gool.  Unsupervised semantic segmentation by contrasting object mask proposals.  arXiv preprint arXiv:2102.06191, 2021.
  * [40] Wenguan Wang, Tianfei Zhou, Fisher Yu, Jifeng Dai, Ender Konukoglu, and Luc Van Gool.  Exploring cross-image pixel contrast for semantic segmentation.  arXiv preprint arXiv:2101.11939, 2021.
  * [41] Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li.  Dense contrastive learning for self-supervised visual pre-training.  arXiv preprint arXiv:2011.09157, 2020.
  * [42] Zhonghao Wang, Yunchao Wei, Rogerio Feris, Jinjun Xiong, Wen-Mei Hwu, Thomas S Huang, and Honghui Shi.  Alleviating semantic-level shift: A semi-supervised domain adaptation method for semantic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 936–937, 2020.
  * [43] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin.  Unsupervised feature learning via non-parametric instance discrimination.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3733–3742, 2018.
  * [44] Saining Xie, Jiatao Gu, Demi Guo, Charles R Qi, Leonidas Guibas, and Or Litany.  Pointcontrast: Unsupervised pre-training for 3d point cloud understanding.  In European Conference on Computer Vision, pages 574–591. Springer, 2020.
  * [45] Zhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao, Stephen Lin, and Han Hu.  Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning.  arXiv preprint arXiv:2011.10043, 2020.
  * [46] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.  Cutmix: Regularization strategy to train strong classifiers with localizable features.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6023–6032, 2019.
  * [47] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny.  Barlow twins: Self-supervised learning via redundancy reduction.  International Conference on Machine Learning, ICML, 2021.
  * [48] Pan Zhang, Bo Zhang, Ting Zhang, Dong Chen, Yong Wang, and Fang Wen.  Prototypical pseudo label denoising and target structure learning for domain adaptive semantic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12414–12424, 2021.


# Topological nonlocal operations on toroidal flux qubits

Adel Ali<EMAIL_ADDRESS>Department of Physics and Astronomy, Texas A&M
University, College Station, TX, 77843 USA  Alexey Belyanin<EMAIL_ADDRESS>Department of Physics and Astronomy, Texas A&M University, College Station,
TX, 77843 USA

###### Abstract

We propose a conceptual model of a toroidal flux qubit, which consists of a
quantized toroidal magnetic flux coupled to a charged particle on a quantum
ring through field-free interaction. Scaling the system to two or more flux
qubits results in emergent field-free coupling between them. We show that the
topological and nonlocal aspects of this system can have profound applications
in quantum information. We illustrate it with examples of nonlocal operations
on these flux qubits which are protected from environmental noise, including
creating entanglement and “teleporting” excitation energy between the flux
qubits.

Introduction: The quest for scalable quantum computing is believed to be
contingent on reducing decoherence and errors Siddiqi (2021). Topological
quantum computing is an attempt to do so by encoding the quantum information
in a many-body system where the local errors cannot destroy the encoded
information. Despite the mathematical beauty of this idea, finding materials
that realize non-abelian anyons has not been unambiguously confirmed Stern and
Lindner (2013). Furthermore, storing and processing quantum information
requires two seemingly contradictory processes: complete isolation of the
system from the environment, and precise control of the system’s degrees of
freedom. One example of a well isolated qubit candidate is a toroidal flux
qubit or fluxon, which is defined as a volume of enclosed magnetic flux lines
whose outer surface is topologically equivalent to a torus. As in the case of
normal flux qubits, the persistent toroidal current generating this magnetic
flux can be in a superposition Orlando  _et al._ (1999). Fluxons can be
isolated from the environmental noise such as flux noise, charge noise, 1/f
noise, external electromagnetic fields, etc. Zagoskin  _et al._ (2015);
Paladino  _et al._ (2014). This raises the question: how to do operations on
its quantum state if it is isolated from the environment? We propose to solve
this problem by coupling the fluxon with a charged particle living in a ring
that encloses the flux lines of the fluxon, or the quantum ring (QR) Seo  _et
al._ (2021); Vinasco  _et al._ (2019); Hansen  _et al._ (2001); Dauber  _et
al._ (2017); Fuhrer  _et al._ (2001). The fluxon-electron interaction is
mediated by the coupling of the electron with the gauge field of the fluxon,
i.e., essentially the field-quantized version of the Aharonov-Bohm effect. Any
noise charge must close a loop around the fluxon to get coupled to it. The
environmental noise charge fluctuations do not affect the system unless they
make coherently a trajectory that encloses the flux with a non-zero winding
number.

The Aharonov-Bohm (AB) phase has been studied in a context where the gauge
vector potential of a classical magnetic field affects nonlocally a charged
particle’s phase for scattered states and the energies of the bound states
Peshkin (1989). However, this field has not been quantized. We propose and
study the quantized flux scenario. This means that the magnetic flux itself
can be in a superposition of different quantum states.

The AB effect has two characteristic features: its topological nature which
means that it is protected from deformations of the flux lines and the charged
particle path, and its nonlocal nature which means that the flux can affect
the phase of a distant particle. These two features are of great significance
for fundamental physics and, as we show below, may lead to unique quantum
information applications. In this Letter, we analyze the quantized flux
version of the AB system and show that the joint quantum state of the particle
and fluxon can be manipulated by the external electromagnetic (EM) fields
acting only on the electron degrees of freedom; hence we can deterministically
control the fluxon state. Moreover, we show that for an electron coupled to
two fluxons that are separated in space and isolated from each other and the
environment, entanglement can be created between them, and excitation energy
can ”teleport” from one to the other. This reasoning is scalable to a chain of
coupled fluxons where it simulates interesting spin chain systems with
nonlocal coupling.

The proposed system is conceptually simple enough for a semi-analytic
treatment and at the same time, it presents an example of synergy of several
fundamental physics phenomena and models, such as the AB effect, flux qubits,
spin chains, and tunable Ising models, as well as their application to quantum
information.

AB phase topological protection: Consider a QR that surrounds a volume with a
magnetic flux trapped inside, as sketched in Fig. 1a. The eigenvalues and
eigenvectors of the system are invariant under deformations of the charge path
(given that this deformation is approximately length preserving), deformations
of the flux lines, or simultaneous deformations of the charge path and the
closed flux lines Heras (2022). Indeed, in the semiclassical Bohr–Sommerfeld
quantization condition,

$\hbar\oint d\mathbf{r}\cdot\mathbf{k}-e\oint
d\mathbf{r}\cdot\mathbf{A}=2\pi\hbar n$

the second term is clearly invariant under the deformation of the closed flux
lines. For length preserving deformations the first term is also invariant.
Hence, the allowed momenta and energies are invariant and these systems have
topological protection. That may address the design and fabrication challenges
such as uniformity and reproducibility across qubits, disorder, and
fabrication defects.

Figure 1: (a) Schematic of a fluxon coupled to an electron on a quantum ring.
The magnetic field is confined inside the toroid and the electron on the ring
is coupled to the vector potential only. (b) two fluxons coupled nonlocally
through the electron on a quantum ring. (c) A chain of coupled fluxons.

Fluxon-electron coupling: The Hamiltonian for a massive charged particle in a
field-free region outside of an infinite solenoid is given by

$\hat{H}=\frac{1}{2m_{e}}(\hat{p}_{\varphi}-e\hat{A}_{\varphi})^{2};\,A_{\varphi}=\frac{\Phi}{2\pi
r}$

and $\varphi$ is an angular coordinate along the QR which changes from 0 to
$2\pi$. We quantize the enclosed flux $\Phi$ by demanding that
$\Phi=\hat{n}\varphi_{0}$ where $\hat{n}$ is a number operator, and
$\varphi_{0}=h/2e$ is the flux quantum.

For all examples below we will limit the fluxon excitations to the two lowest
states with $n$ equal to 0 or 1 in order to have a two-state flux qubit, which
is the simplest and most studied case for applications. In the “physical”
basis $|n=0\rangle$, $|n=1\rangle$ the fluxon Hamiltonian is
$\Delta\sigma_{x}$ where $\Delta$ is the transition energy and $\sigma_{x}$ is
a Pauli matrix. Note that it is often written as $\Delta\sigma_{z}$, after
performing the Hadamard transformation Zagoskin  _et al._ (2015). Furthermore,
we will start with the $1D$ motion of a particle on a ring. The case of a
finite-width 2D annulus geometry is qualitatively the same, unless we want to
utilize the energy levels related to transverse quantization which happens at
higher energy scale. The resulting fluxon-electron Hamiltonian is

$\hat{H}=\frac{1}{2m_{e}}\left(\hat{p}_{\varphi}-\frac{e\varphi_{0}\hat{n}}{2\pi
r}\right)^{2}+\Delta\sigma_{x}$ (1)

We take the basis of the Hilbert space to be
$\left|\psi_{mn}\rangle=\right|m\rangle\otimes|n\rangle$ where

$\left\langle\varphi\mid m\right\rangle=\frac{e^{-im\varphi}}{\sqrt{2\pi
r}}\text{, and }\hat{n}|n\rangle=n|n\rangle.$

Using the ansatz
$\left|\psi_{m}\right\rangle=|m\rangle\otimes(\alpha|0\rangle+\beta|1\rangle)$
to solve the time-independent Schroedinger equation
$\hat{H}|\psi_{m}\rangle=E_{m}|\psi_{m}\rangle$, we obtain

$\displaystyle\hat{H}\left|\psi_{m}\right\rangle=\frac{\hbar^{2}}{2m_{e}r^{2}}\left[|m\rangle\otimes(\alpha
m^{2}|0\rangle)+((m-1)^{2}\beta|1\rangle)\right]$
$\displaystyle+\Delta[|m\rangle\otimes\alpha|1\rangle+|m\rangle\otimes\beta|0\rangle]|\psi_{m}\rangle.$

Setting the energy scale $a=\frac{\hbar^{2}}{2m_{e}r^{2}}\equiv 1$, so that
$\Delta$ is now in units of $a$ and simplifying, we arrive at

$\hat{H}\left|\psi_{m}\right\rangle=|m\rangle\otimes\left[\left(\alpha
m^{2}+\Delta\beta\right)|0\rangle+\left(\beta(m-1)^{2}+\Delta\alpha\right)|1\rangle\right]$
$=E_{m}[|m\rangle\otimes(\alpha|0\rangle+\beta|1\rangle)].$

This can be written as

${\left[\begin{array}[]{cc}m^{2}&\Delta\\\
\Delta&(m-1)^{2}\end{array}\right]\left[\begin{array}[]{l}\alpha\\\
\beta\end{array}\right]=E_{m}\left[\begin{array}[]{l}\alpha\\\
\beta\end{array}\right]}$ (2)

Solving this eigenvalue problem gives

$E_{m}^{\pm}=\frac{1}{2}\left(1-2m+2m^{2}\pm\sqrt{(1-2m)^{2}+4\Delta^{2}}\right).$
(3)

The ansatz is a product state of $m$ and $n$ states, so the stationary states
of the joint system have the fluxon part given by the eigenstates of the
effective Hamiltonian matrix on the left-hand side of Eq. (2). This matrix can
be written as $h_{0}I+\vec{h}.\vec{\sigma}$ where
$\vec{\sigma}=(\sigma_{x},\sigma_{y},\sigma_{z})$,
$\vec{h}=(\Delta,0,m-\frac{1}{2})$, and $h_{0}=m^{2}-m+1$. In spin-1/2
terminology, $\Delta$ is the $x$-component of the effective magnetic field,
$h_{3}$ is its $z$-component, and $h_{0}$ is a total energy shift. So changing
the quantum number $m$ (for example by the EM field) will result in changing
the effective Hamiltonian of the fluxon.

The eigenvectors describing the fluxon part of the eigenstates, namely
$\alpha|0\rangle+\beta|1\rangle$, are shown as arrows on the Bloch sphere in
Fig. 2(a), (c), and (d) for different values of $\Delta$, whereas the
eigenenergies from Eq. (3) are shown in Fig. 2(b). If $\Delta\ll 1$, the
eigenvectors are concentrated around the poles of the Bloch sphere as in Fig.
2(d) because the z-component will dominate. As $\Delta$ increases, the states
initially condensed on the poles will start to spread out as shown in Fig.
2(a) for $\Delta=5$ and (c) for $\Delta\gg 1$. In particular, the states with
$m$ of the same order as $\Delta$ will move towards the x-axis of the Bloch
sphere. One notices from Fig. 2(a) and (b) that even when $m=0$, i.e., the
electron has zero angular momentum as is the case for state $|g_{1}\rangle$,
the value of $h_{3}=-1/2$ is still non-zero and the electron still affects the
state of the fluxon. This is analogous to the vacuum effects in the
Jaynes–Cummings model.

Figure 2: (a) Plot of the eigenstates of the fluxon for $\Delta=5$ on a
section of a Bloch sphere, where each arrow represents a state that has a
different $m$ number. The red arrows on the right and blue arrows on the left
correspond, respectively, to the lower and the upper energy bands shown in
part (b). The horizontal arrow shows an example of an electric-dipole allowed
transition. (b) The energy spectrum of the states in (a) with the transition
from part (a) shown with an arrow. (c) Same as part (a) but for $\Delta\gg 1$.
(d) Same as part (a) but for $\Delta\ll 1$.

Operations : Can we use the gauge-field nonlocal coupling between the electron
and the fluxon to do quantum operations on the fluxon(s)? The fluxons are
completely shielded from any EM field and we have access only to the electron
on the QR degrees of freedom. The question is significant for fundamental
physics, e.g., within the debate on the observable effects of the gauge field
potential. This nonlocal coupling can be used to transfer energy and
information between the electron and the fluxon, and mediate interaction
between fluxons. It also has important implications for quantum information
technologies which rely on the fact that a charged particle that encloses the
fluxon can interact nonlocally with the fluxon while the fluxon is isolated
from virtually everything else in the environment.

Consider as an example the transitions between the electron $m$-states caused
by an EM field and their effect on the fluxon state. The external EM field
will couple only to the electron moments. In the electric dipole
approximation, the selection rules are $\Delta m=\pm 1$. A simple operation on
the fluxon would be to excite the joint system to a higher state, as shown in
Fig. 2(b). This will result in rotating the fluxon state on the Bloch sphere
as indicated in Fig. 2(a). Note that each eigenstate of the system is a
product state, so after the excitation, even if the ring were somehow “cut”,
the x-basis measurement of the fluxon would not change, as the fluxon state
would rotate around the x-axis. Since the fluxon is well isolated from the
environment, it can be used to store quantum states.

Another set of operations to manipulate the state of the fluxon is to add a
potential to the electron on the ring, for example, by applying lateral
electric field(s). Consider the lower two $m$ states only, so that the
Hamiltonian can be mapped to a two-qubit Ising model:
$\hat{H}_{I}=(-\sigma^{(e)}_{z}\sigma^{(f)}_{z}+I)+\Delta\sigma_{x}^{(f)}+g_{1}(t)\sigma_{x}^{(e)}+g_{2}(t)\sigma_{y}^{(e)}$.
Here superscripts $(e)$ and $(f)$ denote the operators acting only on electron
and fluxon degrees of freedom, respectively. Then geometric phase gates can be
performed on the fluxon Amniat-Talab and Rangani Jahromi (2013) by
adiabatically changing $g_{1,2}(t)$.

Creating entanglement between the electron state and the fluxon has
interesting implications in the context of nonlocal gauge effects. Indeed,
consider an initial state of the system to be a product state of the electron
and the fluxon, so that there is no correlation between the measurements done
on both. To illustrate the idea of nonlocal effects with the same simple
model, we take the same Hamiltonian as in Eq. (1) but with an arbitrary
coupling of the electron to an external field:

$\hat{H}=\frac{1}{2m_{e}}\left(\hat{p}_{\varphi}^{(e)}-\frac{e\varphi_{0}\hat{n}^{(f)}}{2\pi
r}\right)^{2}+\Delta\sigma_{x}^{(f)}+\hat{H_{e}},$ (4)

where $H_{e}=g(t)\sigma_{x}^{(e)}$. The operator $\hat{H_{e}}$ is acting only
on the electron subspace and can describe, e.g., the interaction of the
electron dipole moment with an EM field. In the basis of $|m,n\rangle$ for the
lowest energy states of the uncoupled system,
$\\{|00\rangle,|01\rangle,|10\rangle,|11\rangle\\}$, the matrix elements of
the Hamiltonian (4) are

$H=\left[\begin{array}[]{cccc}0&\Delta&g&0\\\ \Delta&1&0&g\\\ g&0&1&\Delta\\\
0&g&\Delta&0\end{array}\right]$ (5)

According to our scenario, at $t\leq 0$ we have $g=0$ and the eigenstates of
the Hamiltonian (5) are product states of the $|m\rangle$ and $|n\rangle$
basis states. At $t=0$ the interaction term $H_{e}$, which models the dipole
coupling of the electron with an external EM field, is turned on. Hence,
driving the electron will result in inducing the coupling coefficient $g$ in
(5). Accordingly, the time evolution will result in changing the reduced
density matrix of the fluxon. One can easily check numerically that the
expectation value of the energy and von Neumann entanglement entropy of the
fluxon will increase due to the coupling. This implies that the exchange of
information and energy between the two subsystems can happen nonlocally
mediated by the gauge field potential only.

Nonlocal coupling and operations on fluxons: The system can be scaled up to
include more complicated operations. Consider a setup that involves two of
such fluxons, as in Fig. 1(b). The Hamiltonian is now

$\hat{H}=\left(\hat{p}_{\varphi}-\frac{e\varphi_{0}\hat{n}_{1}}{2\pi}+\frac{e\varphi_{0}}{2\pi}\hat{n}_{2}\right)^{2}+\Delta\sigma_{x}^{(f1)}+\Delta\sigma_{x}^{(f2)}.$
(6)

Here $(f1)$ and $(f2)$ denote operators acting on the degrees of freedom of
fluxon 1 and 2. We assumed the configuration in which the two fluxons have
opposite directions; hence the opposite signs in parentheses in Eq. (6). For
the eigenstates of the system, we use the ansatz

$\left|\psi_{m}\right\rangle=|m\rangle\otimes\left(a_{1}|00\rangle+a_{2}|01\rangle+a_{3}|10\rangle+a_{4}|11\rangle\right),$

and the effective Hamiltonian becomes

$\left[\begin{array}[]{cccc}m^{2}&\Delta&\Delta&0\\\
\Delta&(m-1)^{2}&0&\Delta\\\ \Delta&0&(m-1)^{2}&\Delta\\\
0&\Delta&\Delta&m^{2}\end{array}\right]$

Its eigenenergies are $E_{m}^{(1)}=(m-1)^{2},E_{m}^{(2)}=m^{2}$
$E_{m}^{(3,4)}=\frac{1}{2}\left(1-2m+2m^{2}\pm\sqrt{(1-2m)^{2}+16\Delta^{2}}\right).$

Figure 3: The eigenenergies of the two-fluxon system plotted as a function of
$m$ for $\Delta=3$. The black arrows represent electric dipole-allowed
transitions from the ground state which is a product state for the fluxons
$|n_{1}\rangle\otimes|n_{2}\rangle$, to excited states in the second and third
bands which are entangled Bell states.

They are plotted in Fig. 3. The salient feature of the system is that the
fluxon parts of the corresponding eigenstates are entangled states of the
fluxons. For example,
$|\Phi^{-}\rangle=\frac{|00\rangle-|11\rangle}{\sqrt{2}}$ is the fluxon part
of the eigenstate of the entire $E_{m}^{(1)}$ band, and
$|\Psi^{-}\rangle=\frac{|01\rangle-|10\rangle}{\sqrt{2}}$ is the same for the
$E_{m}^{(2)}$ band. All the states of these bands are still orthogonal because
they have different quantum numbers $m$. These are maximally entangled Bell
states of the fluxons. Therefore, a simple operation of exciting the system
from the ground state $E^{(4)}_{m}$, which is a product state, to any state in
the $E_{m}^{(1)}$ or $E_{m}^{(2)}$ bands as shown in Fig. 3 will nonlocally
create a maximally entangled state between the two fluxons. This is a
remarkable and potentially very useful feature of the system.

The effective Hamiltonian given by the matrix above can be written as

$\hat{H}_{m}=h_{0}I+\vec{h}.\vec{\sigma}^{(f1)}+\vec{g}.\vec{\sigma}^{(f2)}+J_{m}\sigma^{(f1)}_{z}\sigma^{(f2)}_{z}$

where the nonzero coefficients are

$h_{0}=m^{2},\;h_{1}=g_{1}=\Delta,\;h_{3}=g_{3}=m-1/2,\;J_{m}=2m-1.$

The coupling coefficient $J_{m}$ between the two initially non-interacting
fluxons emerged due to the coupling with the electron and describes electron-
induced fluxon-fluxon interaction. Since $J_{m}$ is a function of $m$, it can
be tuned and changed between antiferromagnetic for $m>0$ and ferromagnetic for
$m<0$. Moreover, the coupling coefficient is topological, meaning that it does
not change by deforming the QR or changing the distance between the two
fluxons as long as the electron path encircles both fluxons. This is yet
another unusual property of this system as opposed to the usual situation when
the interaction strength falls off with the distance between the interacting
parts of the system.

Figure 4: Exchange of information and energy between the two fluxons. (a) for
$\Delta=1$, the probability for measuring $|010\rangle$ is plotted as function
of time normalized by $\hbar/a$. Around $t=6$ the excitation is completely
transferred from fluxon 2 to fluxon 1. (b) The von Neumann entanglement
entropy of the fluxon as a function of time normalized by $\hbar/a$.

Another interesting feature of the coupled QR-fluxon models is that the energy
can be transferred nonlocally between the two fluxons. To illustrate that,
consider the initial state in the form
$|\psi(0)\rangle=|m=0\rangle\otimes|n_{1}=0\rangle\otimes|n_{2}=1\rangle$. The
time evolution of this state will keep $m$ fixed and results in rotation of
the fluxons part of the wavefunction by the effective Hamiltonian above.
Accordingly, after some time $\tau$, the state vector becomes
$|\psi(\tau)\rangle=|0\rangle\otimes|1\rangle\otimes|0\rangle$. Fig. 4(a)
shows the probability of finding the system with $n_{1}=1$. This means that
the flux (energy) initially in fluxon 2 can “teleport” to fluxon 1. This can
be done with any value of $m$, in which case the coupling coefficient could be
larger and $\tau$ would be smaller. However, the evolution starting from the
state with $m=0$, i.e., the state with zero magnetic moment is still most
peculiar as it rules out any effect of the magnetic field produced by the
electron.

The models considered here can be straightforwardly scaled up or extended to
include more operations. For example, more than one electron can be injected
into a QR, which will lead to the occupation of several $m$-states and excited
energy bands in Fig. 3. Furthermore, a chain of the fluxons can be created
where each two consecutive fluxons are coupled to each other, as shown in Fig.
1(c). Following the same procedure as in the case of two fluxons, the
effective Hamiltonian of the fluxons on the chain can be written as

$\hat{H}_{m}=\sum_{i}J_{m}\sigma^{(i)}_{z}\sigma^{(i+1)}_{z}+\Delta\sigma_{x}^{(i)}+h_{m}\sigma_{z}^{(i)}+h_{0}I.$

This is a transverse field Ising Hamiltonian with tunable parameters. Hence,
this system can be used as a quantum simulator Georgescu  _et al._ (2014). The
two protocols discussed above can be applied for the chain setup. First, one
can ”teleport” energy by starting with only one fluxon excited, when the time
evolution results in the excitation hopping between different sites of the
chain. Second, entanglement of the fluxons on the chain can be created by
entangling the fluxons on the chain pairwise consecutively by changing the
$m$-state of an electron in the corresponding QR.

Finally, all qualitative features of the models will be preserved if one
replaces a nonrelativistic 1D electron with any other type of the QR. For
example, employing a carbon nanotube or a graphene ring will change the shape
of the energy dispersion in Fig. 2(b) and 3 while all operations remain the
same. The scale of the energy parameter $a$ defining the separation between
the eigenstates will be of course affected. Depending on the radius and the
type of the ring, one can expect the transition energies from the microwave to
the mid-infrared range.

————————-

## References

  * Siddiqi (2021) I. Siddiqi, Nature Reviews Materials 6, 875–891 (2021).
  * Stern and Lindner (2013) A. Stern and N. H. Lindner, Science 339, 1179 (2013).
  * Orlando  _et al._ (1999) T. P. Orlando, J. E. Mooij, L. Tian, C. H. van der Wal, L. S. Levitov, S. Lloyd, and J. J. Mazo, Physical Review B 60, 15398 (1999).
  * Zagoskin  _et al._ (2015) A. M. Zagoskin, A. Chipouline, E. Il’ichev, J. R. Johansson, and F. Nori, Scientific Reports 5 (2015), 10.1038/srep16934.
  * Paladino  _et al._ (2014) E. Paladino, Y. Galperin, G. Falci, and B. Altshuler, Reviews of Modern Physics 86, 361 (2014).
  * Seo  _et al._ (2021) M. Seo, H. Jung, and Y. Chung, Applied Science and Convergence Technology 30, 62 (2021).
  * Vinasco  _et al._ (2019) J. A. Vinasco, A. Radu, E. Niculescu, M. E. Mora-Ramos, E. Feddi, V. Tulupenko, R. L. Restrepo, E. Kasapoglu, A. L. Morales, and C. A. Duque, Scientific Reports 9 (2019), 10.1038/s41598-018-38114-0.
  * Hansen  _et al._ (2001) A. E. Hansen, A. Kristensen, S. Pedersen, C. B. Sørensen, and P. E. Lindelof, Phys. Rev. B 64, 045327 (2001).
  * Dauber  _et al._ (2017) J. Dauber, M. Oellers, F. Venn, A. Epping, K. Watanabe, T. Taniguchi, F. Hassler, and C. Stampfer, Physical Review B 96 (2017), 10.1103/physrevb.96.205407.
  * Fuhrer  _et al._ (2001) A. Fuhrer, S. Lüscher, T. Ihn, T. Heinzel, K. Ensslin, W. Wegscheider, and M. Bichler, Nature 413, 822–825 (2001).
  * Peshkin (1989) M. Peshkin, “The aharonov-bohm effect part one: Theory,” in  _The Aharonov-Bohm Effect_, edited by M. Peshkin and A. Tonomura (Springer Berlin Heidelberg, Berlin, Heidelberg, 1989) pp. 1–34.
  * Heras (2022) R. Heras, The European Physical Journal Plus 137 (2022), 10.1140/epjp/s13360-022-02832-2.
  * Amniat-Talab and Rangani Jahromi (2013) M. Amniat-Talab and H. Rangani Jahromi, Royal Society of London Proceedings Series A 469, 20743 (2013).
  * Georgescu  _et al._ (2014) I. M. Georgescu, S. Ashhab, and F. Nori, Rev. Mod. Phys. 86, 153 (2014).