11institutetext: Gordon Center for Medical Imaging, Department of Radiology,
Massachusetts General Hospital and Harvard Medical School, Boston, MA,
0211422institutetext: Division of Radiation Biophysics, Department of
radiation Oncology, Massachusetts General Hospital and Harvard Medical School,
Boston, MA, 0211433institutetext: Department of Electrical and Computer
Engineering, University of Southern California, Los Angeles, CA, 90007

# ACT: Semi-supervised Domain-adaptive Medical Image Segmentation with
Asymmetric Co-Training

Xiaofeng Liu 11 Fangxu Xing 11 Nadya Shusharina 22 Ruth Lim 11 C.-C. Jay Kuo
33 Georges El Fakhri 11 Jonghye Woo 11

###### Abstract

Unsupervised domain adaptation (UDA) has been vastly explored to alleviate
domain shifts between source and target domains, by applying a well-performed
model in an unlabeled target domain via supervision of a labeled source
domain. Recent literature, however, has indicated that the performance is
still far from satisfactory in the presence of significant domain shifts.
Nonetheless, delineating a few target samples is usually manageable and
particularly worthwhile, due to the substantial performance gain. Inspired by
this, we aim to develop semi-supervised domain adaptation (SSDA) for medical
image segmentation, which is largely underexplored. We, thus, propose to
exploit both labeled source and target domain data, in addition to unlabeled
target data in a unified manner. Specifically, we present a novel asymmetric
co-training (ACT) framework to integrate these subsets and avoid the
domination of the source domain data. Following a divide-and-conquer strategy,
we explicitly decouple the label supervisions in SSDA into two asymmetric sub-
tasks, including semi-supervised learning (SSL) and UDA, and leverage
different knowledge from two segmentors to take into account the distinction
between the source and target label supervisions. The knowledge learned in the
two modules is then adaptively integrated with ACT, by iteratively teaching
each other, based on the confidence-aware pseudo-label. In addition, pseudo
label noise is well-controlled with an exponential MixUp decay scheme for
smooth propagation. Experiments on cross-modality brain tumor MRI segmentation
tasks using the BraTS18 database showed, even with limited labeled target
samples, ACT yielded marked improvements over UDA and state-of-the-art SSDA
methods and approached an “upper bound” of supervised joint training.

##  1 Introduction

Accurate delineation of lesions or anatomical structures is a vital step for
clinical diagnosis, intervention, and treatment planning [24]. While recently
flourished deep learning methods excel at segmenting those structures, deep
learning-based segmentors cannot generalize well in a heterogeneous domain,
e.g., different clinical centers, scanner vendors, or imaging modalities [20,
16, 14, 4]. To alleviate this issue, unsupervised domain adaptation (UDA) has
been actively developed, by applying a well-performed model in an unlabeled
target domain via supervision of a labeled source domain [5, 15, 18, 19]. Due
to diverse target domains, however, the performance of UDA is far from
satisfactory [31, 9, 17]. Instead, labeling a small set of target domain data
is usually more feasible [25]. As such, semi-supervised domain adaptation
(SSDA) has shown great potential as a solution to domain shifts, as it can
utilize both labeled source and target data, in addition to unlabeled target
data. To date, while several SSDA classification methods have been proposed
[8, 29, 23, 13], based on discriminative class boundaries, they cannot be
directly applied to segmentation, since segmentation involves complex and
dense pixel-wise predictions.

Recently, while a few works [26, 6, 10] have been proposed to extend SSDA for
segmentation on natural images, to our knowledge, no SSDA for medical image
segmentation has yet been explored. For example, a depth estimation for
natural images is used as an auxiliary task as in [10], but that approach
cannot be applied to medical imaging data, e.g., MRI, as they do not have
perspective depth maps. Wang et al. [26] simply added supervision from labeled
target samples to conventional adversarial UDA. Chen et al. [6] averaged
labeled source and target domain images at both region and sample levels to
mitigate the domain gap. However, source domain supervision can easily
dominate the training, when we directly combine the labeled source data with
the target data [23]. In other words, the extra small amount of labeled target
data has not been effectively utilized, because the volume of labeled source
data is much larger than labeled target data, and there is significant
divergence across domains [23].

To mitigate the aforementioned limitations, we propose a practical asymmetric
co-training (ACT) framework to take each subset of data in SSDA in a unified
and balanced manner. In order to prevent a segmentor, jointly trained by both
domains, from being dominated by the source data only, we adopt a divide-and-
conquer strategy to decouple the label supervisions for the two asymmetric
segmentors, which share the same objective of carrying out a decent
segmentation performance for the unlabeled data. By “asymmetric,” we mean that
the two segmentors are assigned different roles to utilize the labeled data in
either source or target domain, thereby providing a complementary view for the
unlabeled data. That is, the first segmentor learns on the labeled source
domain data and unlabeled target domain data as a conventional UDA task, while
the other segmentor learns on the labeled and unlabeled target domain data as
a semi-supervised learning (SSL) task. To integrate these two asymmetric
branches, we extend the idea of co-training [3, 1, 22], which is one of the
most established multi-view learning methods. Instead of modeling two views on
the same set of data with different feature extractors or adversarial sample
generation in conventional co-training [3, 1, 22], our two cross-domain views
are explicitly provided by the segmentors with the correlated and
complementary UDA and SSL tasks. Specifically, we construct the pseudo label
of the unlabeled target sample based on the pixel-wise confident predictions
of the other segmentor. Then, the segmentors are trained on the pseudo labeled
data iteratively with an exponential MixUp decay (EMD) scheme for smooth
propagation. Finally, the target segmentor carries out the target domain
segmentation.

The contributions of this work can be summarized as follows:

$\bullet$ We present a novel SSDA segmentation framework to exploit the
different supervisions with the correlated and complementary asymmetric UDA
and SSL sub-tasks, following a divide-and-conquer strategy. The knowledge is
then integrated with confidence-aware pseudo-label based co-training.

$\bullet$ An EMD scheme is further proposed to mitigate the noisy pseudo label
in early epochs of training for smooth propagation.

$\bullet$ To our knowledge, this is the first attempt at investigating SSDA
for medical image segmentation. Comprehensive evaluations on cross-modality
brain tumor (i.e., T2-weighted MRI to T1-weighted/T1ce/FLAIR MRI) segmentation
tasks using the BraTS18 database demonstrate superiority performance over
conventional source-relaxed/source-based UDA methods.

Figure 1: Illustration of our proposed ACT framework for SSDA cross-modality
(e.g., T2-weighted to T1-weighted MRI) image segmentation. Note that only
target domain specific segmentor $\theta$ will be used in testing.

##  2 Methodology

In our SSDA setting for segmentation, we are given a labeled source set
$\mathcal{D}^{s}=\\{(x^{s}_{i},y^{s}_{i})\\}_{i=1}^{N^{s}}$, a labeled target
set $\mathcal{D}^{lt}=\\{(x^{lt}_{i},y^{lt}_{i})\\}_{i=1}^{N^{lt}}$, and an
unlabeled target set $\mathcal{D}^{ut}=\\{(x^{ut}_{i})\\}_{i=1}^{N^{ut}}$,
where ${N^{s}}$, ${N^{lt}}$, and ${N^{ut}}$ are the number of samples for each
set, respectively. Note that the slice $x^{s}_{i},x^{lt}_{i}$, and
$x^{ut}_{i}$, and the segmentation mask labels $y_{i}^{s}$, and $y_{i}^{lt}$
have the same spatial size of $H\times W$. In addition, for each pixel
$y^{s}_{i:n}$ or $y^{lt}_{i:n}$ indexed by $n\in\mathbb{R}^{H\times W}$, the
label has $C$ classes, i.e., $y^{s}_{i:n},y^{lt}_{i:n}\in\\{1,\cdots,C\\}$.
There is a distribution divergence between source domain samples,
$\mathcal{D}^{s}$, and target domain samples, $\mathcal{D}^{lt}$ and
$\mathcal{D}^{ut}$. Usually, ${N^{lt}}$ is much smaller than ${N^{s}}$. The
learning objective is to perform well in the target domain.

###  2.1 Asymmetric Co-training for SSDA segmentation

To decouple SSDA via a divide-and-conquer strategy, we integrate
$\mathcal{D}^{ut}$ with either $\mathcal{D}^{s}$ or $\mathcal{D}^{lt}$ to form
the correlated and complementary sub-tasks of UDA and SSL. We configure a
cross-domain UDA segmentor $\phi$ and a target domain SSL segmentor $\theta$,
which share the same objective of achieving a decent segmentation performance
in $\mathcal{D}^{ut}$. The knowledge learned from the two segmentors is then
integrated with ACT. The overall framework of this work is shown in Fig. 1.

Conventional co-training has focused on two independent views of the source
and target data or generated artificial multi-views with adversarial examples,
which learns two classifiers for each of the views and teaches each other on
the unlabeled data [3, 22]. By contrast, in SSDA, without multiple views of
the data, we propose to leverage the distinct yet correlated supervision,
based on the inherent discrepancy of the labeled source and target data. We
note that the sub-tasks and datasets adopted are different for the UDA and SSL
branches. Therefore, all of the data subsets can be exploited, following well-
established UDA and SSL solutions without interfering with each other.

To achieve co-training, we adopt a simple deep pseudo labeling method [27],
which assigns the pixel-wise pseudo label $\hat{y}_{i:n}$ for $x^{ut}_{i:n}$.
Though UDA and SSL can be achieved by different advanced algorithms, deep
pseudo labeling can be applied to either UDA [32] or SSL [27]. Therefore, we
can apply the same algorithm to the two sub-tasks, thereby greatly simplifying
our overall framework. We note that while a few methods [28] can be applied to
either SSL or UDA like pseudo labeling, they have not been jointly adopted in
the context of SSDA.

Specifically, we assign the pseudo label for each pixel $x^{ut}_{i:n}$ in
$\mathcal{D}^{ut}$ with the prediction of either $\phi$ or $\theta$, therefore
constructing the pseudo labeled sets $U^{\phi}$ and $U^{\theta}$ for the
training of another segmentor $\theta$ and $\phi$, respectively:

$\displaystyle
U^{\phi}=\\{(x^{ut}_{i:n},\hat{y}^{\phi}_{i:n}=\operatorname*{arg\,max}_{c}p(c|x^{ut}_{i:n};\phi));\text{
if }\max_{c}p(c|x^{ut}_{i:n};\phi)>\epsilon\\},$ (1) $\displaystyle
U^{\theta}=\\{(x^{ut}_{i:n},\hat{y}^{\theta}_{i:n}=\operatorname*{arg\,max}_{c}p(c|x^{ut}_{i:n};\theta));\text{
if }\max_{c}p(c|x^{ut}_{i:n};\theta)>\epsilon\\},$ (2)

where $p(c|x^{ut}_{i:n};\theta)$ and $p(c|x^{ut}_{i:n};\phi)$ are the
predicted probability of class $c\in\\{1,\cdots,C\\}$ w.r.t. $x^{ut}_{i:n}$
using $\theta$ and $\phi$, respectively. $\epsilon$ is a confidence threshold.
Note that the low softmax prediction probability indicates the low confidence
for training [32, 18]. Then, the pixels in the selected pseudo label sets are
merged with the labeled data to construct $\\{\mathcal{D}^{s},U^{\theta}\\}$
and $\\{\mathcal{D}^{lt},U^{\phi}\\}$ for the training of $\phi$ and $\theta$
with a conventional supervised segmentation loss, respectively. Therefore, the
two segmentors with asymmetrical tasks act as teacher and student of each
other to distillate the knowledge with highly confident predictions.

Input: batch size $N$, $\lambda$, $\eta$, $\epsilon$, $\mathcal{D}^{s}$,
$\mathcal{D}^{lt}$, $\mathcal{D}^{ut}$, current network parameters
$\omega_{\phi}$, $\omega_{\theta}$;

Sample
$\\{(x^{s}_{i},y^{s}_{i})\\}_{i=1}^{N},\\{(x^{lt}_{i},y^{lt}_{i})\\}_{i=1}^{N},$
and$\\{(x^{ut}_{i})\\}_{i=1}^{N}$from $\mathcal{D}^{s},\mathcal{D}^{lt},$
and$\mathcal{D}^{ut},$ respectively;

Initialize $U^{\phi}=\emptyset$, $U^{\theta}=\emptyset$;

for  _$i\leftarrow 1$ to $N$_ do

$\hat{y}^{\phi}_{i:n}=\operatorname*{arg\,max}_{c}p(c|x^{ut}_{i:n};\phi);$ and
$\hat{y}^{\theta}_{i:n}=\operatorname*{arg\,max}_{c}p(c|x^{ut}_{i:n};\theta)$

if $\max_{c}p(c|x^{ut}_{i:n};\phi)>\epsilon$: update $U^{\phi}\leftarrow
U^{\phi}\cup\\{(x^{ut}_{i:n},\hat{y}^{\theta}_{i:n})$ with Eq. (1);

if $\max_{c}p(c|x^{ut}_{i:n};\theta)>\epsilon$: update $U^{\theta}\leftarrow
U^{\theta}\cup\\{(x^{ut}_{i:n},\hat{y}^{\theta}_{i:n})$ with Eq. (2);

end for

Obtain
$\tilde{U}^{\phi}=\\{\text{EMD}({U}^{\phi}_{i},\\{(x^{lt}_{i},y^{lt}_{i})\\}_{i=1}^{N};\lambda)\\}_{i=1}^{|{U}^{\phi}|\times
N}$ with Eq. (3);

Obtain
$\tilde{U}^{\theta}=\\{\text{EMD}({U}^{\theta}_{i},\\{(x^{s}_{i},y^{s}_{i})\\}_{i=1}^{N};\lambda)\\}_{i=1}^{|{U}^{\theta}|\times
N}$ with Eq. (4);

Update
$\omega_{\phi}\leftarrow\omega_{\phi}-\eta\nabla(\mathcal{L}(\omega_{\phi},\mathcal{D}^{s})+\mathcal{L}(\omega_{\phi},\tilde{U}^{\theta}))$;
$\omega_{\theta}\leftarrow\omega_{\theta}-\eta\nabla(\mathcal{L}(\omega_{\theta},\mathcal{D}^{lt})+\mathcal{L}(\omega_{\theta},\tilde{U}^{\phi}))$;

Output : Updated network parameters $\omega_{\phi}$ and $\omega_{\theta}$.

Algorithm 1 An iteration of the ACT algorithm.

###  2.2 Pseudo-label with Exponential MixUp Decay

Initially generated pseudo labels with the two segmentors are typically noisy,
which is significantly acute in the initial epochs, thus leading to a deviated
solution with propagated errors. Numerous conventional co-training methods
relied on simple assumptions that there is no domain shift, and the
predictions of the teacher model can be reliable and be simply used as ground
truth. Due to the domain shift, however, the prediction of $\phi$ in the
target domain could be noisy and lead to an aleatoric uncertainty [7, 12, 11].
In addition, insufficient labeled target domain data can lead to an epistemic
uncertainty related to the model parameters [7, 12, 11].

To smoothly exploit the pseudo labels, we propose to adjust the contribution
of the supervision signals from both labels and pseudo labels as the training
progresses. Previously, vanilla MixUp [30] was developed for efficient data
augmentation, by combining both samples and their labels to generate new data
for training. We note that the MixUp used in SSL [2, 6] adopted a constant
sampling, and did not take the decay scheme for gradual co-training. Thus, we
propose to gradually exploit the pseudo label by mixing up $\mathcal{D}^{s}$
or $\mathcal{D}^{lt}$ with pseudo labeled $\mathcal{D}^{ut}$, and adjust their
ratio with the EMD scheme. For the selected ${U}^{\phi}$ and ${U}^{\theta}$
with the number of slices $|{U}^{\phi}|$ and $|{U}^{\theta}|$, we mix up each
pseudo labeled image with all images from $\mathcal{D}^{s}$ or
$\mathcal{D}^{lt}$ to form the mixed pseudo labeled sets $\tilde{U}^{\theta}$
and $\tilde{U}^{\phi}$. Specifically, our EMD can be formulated as:

$\displaystyle\tilde{U}^{\phi}=\\{(\tilde{x}^{lt}_{i:n}=\lambda
x^{lt}_{i:n}+(1-\lambda)x^{ut}_{i:n},\tilde{y}^{lt}_{i:n}=\lambda
y^{lt}_{i:n}+(1-\lambda)~{}~{}\hat{y}^{\theta}_{i:n})\\}_{i}^{|{U}^{\theta}|\times
N},$ (3) $\displaystyle\tilde{U}^{\theta}=\\{(\tilde{x}^{s}_{i:n}=\lambda
x^{s}_{i:n}+(1-\lambda)x^{ut}_{i:n},\tilde{y}^{s}_{i:n}=\lambda
y^{s}_{i:n}+(1-\lambda)~{}~{}\hat{y}^{\phi}_{i:n})\\}_{i}^{|{U}^{\phi}|\times
N},$ (4)

where $\lambda=\lambda^{0}\text{exp}(-I)$ is the MixUp parameter with the
exponential decay w.r.t. iteration $I$. $\lambda^{0}$ is the initial weight of
ground truth samples and labels, which is empirically set to 1. Therefore,
along with the increase over iteration $I$, we have smaller $\lambda$, which
adjusts the contribution of the ground truth label to be large at the start of
the training, while utilizing the pseudo labels at the later training epochs.
Therefore, $\tilde{U}^{\phi}$ and $\tilde{U}^{\theta}$ gradually represent the
pseudo label sets of ${U}^{\phi}$ and ${U}^{\theta}$. We note that the mixup
operates on the image level, which is indicated by $i$. The number of
generated mixed samples depends on the scale of ${U}^{\phi}$ and
${U}^{\theta}$ in each iteration and batch size $N$. With the labeled
$\mathcal{D}^{s}$, $\mathcal{D}^{lt}$, as well as the pseudo labeled sets with
EMD $\tilde{U}^{\phi}$ and $\tilde{U}^{\phi}$, we update the parameters of the
segmentors $\phi$ and $\theta$, i.e., $\omega_{\phi}$ and $\omega_{\theta}$
with SGD as:

$\displaystyle\omega_{\phi}\leftarrow\omega_{\phi}-\eta\nabla(\mathcal{L}(\omega_{\phi},\mathcal{D}^{s})+\mathcal{L}(\omega_{\phi},\tilde{U}^{\theta})),$
(5)
$\displaystyle\omega_{\theta}\leftarrow\omega_{\theta}-\eta\nabla(\mathcal{L}(\omega_{\theta},\mathcal{D}^{lt})+\mathcal{L}(\omega_{\theta},\tilde{U}^{\phi})),$
(6)

where $\eta$ indicates the learning rate, and
$\mathcal{L}(\omega_{\phi},\mathcal{D}^{s})$ denotes the learning loss on
$\mathcal{D}^{s}$ with the current segmentor $\phi$ parameterized by
$\omega_{\phi}$. The training procedure is detailed in Algorithm 1. After
training, only the target domain specific SSL segmentor $\theta$ is used for
testing.

Figure 2: Comparisons with other UDA/SSDA methods and ablation studies for the
cross-modality tumor segmentation. We show target test slices of T1, T1ce, and
FLAIR MRI from three subjects.

##  3 Experiments and Results

To demonstrate the effectiveness of our proposed SSDA method, we evaluated our
method on T2-weighted MRI to T1-weighted/T1ce/FLAIR MRI brain tumor
segmentation using the BraTS2018 database [21]. We denote our proposed method
as ACT, and used ACT-EMD for an ablation study of an EMD-based pseudo label
exploration.

Of note, the BraTS2018 database contains a total of 285 patients [21] with the
MRI scannings, including T1-weighted (T1), T1-contrast enhanced (T1ce),
T2-weighted (T2), and T2 Fluid Attenuated Inversion Recovery (FLAIR) MRI. For
the segmentation labels, each pixel belongs to one of four classes, i.e.,
enhancing tumor (EnhT), peritumoral edema (ED), necrotic and non-enhancing
tumor core (CoreT), and background. In addition, the whole tumor covers CoreT,
EnhT, and ED. We follow the conventional cross-modality UDA (i.e., T2-weighted
to T1-weighted/T1ce/FLAIR) evaluation protocols [31, 9, 17] for 8/2 splitting
for training/testing, and extend it to our SSDA task, by accessing the labels
of 1-5 target domain subjects at the adaptation training stage. All of the
data were used in a subject-independent and unpaired manner. We used SSDA:1 or
SSDA:5 to denote that one or five target domain subjects are labeled in
training.

For a fair comparison, we used the same segmentor backbone as in DSA [9] and
SSCA [17], which is based on Deeplab-ResNet50. Without loss of generality, we
simply adopted the cross-entropy loss as $\mathcal{L}$, and set the learning
rate $\eta=1\mathrm{e}{-3}$ and confidence threshold $\epsilon=0.5$. Both
$\phi$ and $\theta$ have the same network structure. For the evaluation
metrics, we adopted the widely used DSC (the higher, the better) and Hausdorff
distance (HD: the lower, the better) as in [9, 17]. The standard deviation was
reported over five runs.

Table 1: Whole tumor segmentation performance of the cross-modality UDA and
SSDA. The supervised joint training can be regarded as an “upper bound”.

|  | DICE | Score | (DSC) | [$\%$] $\uparrow$ | Hausdorff | Distance | (HD) | [mm] $\downarrow$
---|---|---|---|---|---|---|---|---|---
Method | Task | T1 | FLAIR | T1CE | Ave | T1 | FLAIR | T1CE | Ave
Source Only | No DA | 4.2 | 65.2 | 6.3 | 27.7$\pm$1.2 | 55.7 | 28.0 | 49.8 | 39.6$\pm$0.5
Target Only | SSL:5 | 43.8 | 54.6 | 47.5 | 48.6$\pm$1.7 | 31.9 | 29.6 | 35.4 | 32.3$\pm$0.8
SIFA [5] | UDA | 51.7 | 68.0 | 58.2 | 59.3$\pm$0.6 | 19.6 | 16.9 | 15.0 | 17.1$\pm$0.4
DSFN [31] | UDA | 57.3 | 78.9 | 62.2 | 66.1$\pm$0.8 | 17.5 | 13.8 | 15.5 | 15.6$\pm$0.3
DSA [9] | UDA | 57.7 | 81.8 | 62.0 | 67.2$\pm$0.7 | 14.2 | 8.6 | 13.7 | 12.2$\pm$0.4
SSCA [17] | UDA | 59.3 | 82.9 | 63.5 | 68.6$\pm$0.6 | 12.5 | 7.9 | 11.2 | 11.5$\pm$0.3
SLA [26] | SSAD:1 | 64.7 | 82.3 | 66.1 | 71.0$\pm$0.5 | 12.2 | 7.1 | 10.5 | 9.9$\pm$0.3
DLD [6] | SSAD:1 | 65.8 | 81.5 | 66.5 | 71.3$\pm$0.6 | 12.0 | 7.1 | 10.3 | 9.8$\pm$0.2
ACT | SSAD:1 | 69.7 | 84.5 | 69.7 | 74.6$\pm$0.3 | 10.5 | 5.8 | 10.0 | 8.8$\pm$0.1
ACT-EMD | SSAD:1 | 67.4 | 83.9 | 69.0 | 73.4$\pm$0.6 | 10.9 | 6.4 | 10.3 | 9.2$\pm$0.2
ACT | SSAD:5 | 71.3 | 85.0 | 70.8 | 75.7$\pm$0.5 | 10.0 | 5.2 | 9.8 | 8.3$\pm$0.1
ACT-EMD | SSAD:5 | 70.3 | 84.4 | 69.8 | 74.8$\pm$0.4 | 10.4 | 5.7 | 10.2 | 8.8$\pm$0.2
Joint Training | Supervised | 73.2 | 85.6 | 72.6 | 77.1$\pm$0.5 | 9.5 | 4.6 | 9.2 | 7.7$\pm$0.2

Table 2: Detailed comparison of Core/EnhT/ED segmentation. Results are
averaged over three tasks including T2-weighted to T1-weighted, T1CE, and
FLAIR MRI with the backbone as in [9, 17].

|  | DICE | Score (DSC) | [$\%$] $\uparrow$ | Hausdorff | Distance (HD) | [mm] $\downarrow$
---|---|---|---|---|---|---|---
Method | Task | CoreT | EnhT | ED | CoreT | EnhT | ED
Source Only | No DA | 20.6$\pm$1.0 | 39.5$\pm$0.8 | 41.3$\pm$0.9 | 54.7$\pm$0.4 | 55.2$\pm$0.6 | 42.5$\pm$0.4
Target Only | SSL:5 | 27.3$\pm$1.1 | 38.0$\pm$1.0 | 40.2.3$\pm$1.3 | 51.8$\pm$0.7 | 52.3$\pm$0.9 | 46.4$\pm$0.6
DSA [9] | UDA | 57.8$\pm$0.6 | 44.0$\pm$0.6 | 56.8$\pm$0.5 | 25.8$\pm$0.4 | 34.2$\pm$0.3 | 25.6$\pm$0.5
SSCA [17] | UDA | 58.2$\pm$0.4 | 44.5$\pm$0.5 | 60.7$\pm$0.4 | 26.4$\pm$0.2 | 32.8$\pm$0.2 | 23.4$\pm$0.3
SLA [26] | SSDA:1 | 58.9$\pm$0.6 | 48.1$\pm$0.5 | 65.4$\pm$0.4 | 24.5$\pm$0.1 | 27.6$\pm$0.3 | 20.3$\pm$0.2
DLD [6] | SSDA:1 | 60.3$\pm$0.6 | 48.2$\pm$0.5 | 66.0$\pm$0.3 | 24.2$\pm$0.2 | 27.8$\pm$0.1 | 19.7$\pm$0.2
ACT | SSDA:1 | 64.5$\pm$0.3 | 52.7$\pm$0.4 | 69.8$\pm$0.6 | 20.0$\pm$0.2 | 24.6$\pm$0.1 | 16.2$\pm$0.2
ACT | SSDA:5 | 66.9$\pm$0.3 | 54.0$\pm$0.3 | 71.2$\pm$0.5 | 18.4$\pm$0.4 | 23.7$\pm$0.2 | 15.1$\pm$0.2
Joint Training | Supervised | 70.4$\pm$0.3 | 62.5$\pm$0.2 | 75.1$\pm$0.4 | 15.8$\pm$0.2 | 22.7$\pm$0.1 | 13.0$\pm$0.2

The quantitative evaluation results of the whole tumor segmentation are
provided in Table 1. We can see that SSDA largely improved the performance
over the compared UDA methods [9, 17]. For the T2-weighted to T1-weighted MRI
transfer task, we were able to achieve more than 10% improvements over [9, 17]
with only one labeled target sample. Recent SSDA methods for natural image
segmentation [26, 6] did not take the balance between the two labeled
supervisions into consideration, easily resulting in a source domain-biased
solution in case of limited labeled target domain data, and thus did not
perform well on target domain data [23]. In addition, the depth estimation in
[10] cannot be applied to the MRI data. Thus, we reimplemented the
aforementioned methods [26, 6] with the same backbone for comparisons, which
is also the first attempt at the medical image segmentation. Our ACT
outperformed [26, 6] by a DSC of 3.3% w.r.t. the averaged whole tumor
segmentation in SSDA:1 task. The better performance of ACT over ACT-EMD
demonstrated the effectiveness of our EMD scheme for smooth adaptation with
pseudo-label. We note that we did not manage to outperform the supervised
joint training, which accesses all of the target domain labels, which can be
considered an “upper bound” of UDA and SSDA. Therefore, it is encouraging that
our ACT can approach joint training with five labeled target subjects. In
addition, the performance was stable for the setting of $\lambda$ from 1 to
10.

In Table 2, we provide the detailed comparisons for more fine-grained
segmentation w.r.t. CoreT, EnhT, and ED. The improvements were consistent with
the whole tumor segmentation. The qualitative results of three target
modalities in Fig. 2 show the superior performance of our framework, compared
with the comparison methods.

In Fig. 3(a), we analyzed the testing pixel proportion change along with the
training that has both, only one, and none of two segmentor pseudo-labels,
i.e., the maximum confidence is larger than $\epsilon$ as in Eq. (1). We can
see that the consensus of the two segmentors keeps increasing, by teaching
each other in the co-training scheme for knowledge integration. “Both” low
rates, in the beginning, indicate $\phi$ and $\theta$ may provide a different
view based on their asymmetric tasks, which can be complementary to each
other. The sensitivity studies of using a different number of labeled target
domain subjects are shown in Fig. 3(b). Our ACT was able to effectively use
$\mathcal{D}^{lt}$. In Fig. 3(c), we show that using more EMD pairs improves
the performance consistently.

Figure 3: Analysis of our ACT-based SSDA on the whole tumor segmentation task.
(a) The proportion of testing pixels that both, only one, or none of the
segmentors have high confidence on (b) the performance improvements with a
different number of labeled target domain training subjects, and (c) a
sensitivity study of changing different proportion of EMD pairs of
$|\tilde{U}^{\phi}|\times N$ and $|\tilde{U}^{\theta}|\times N$.

##  4 Conclusion

This work proposed a novel and practical SSDA framework for the segmentation
task, which has the great potential to improve a target domain generalization
with a manageable labeling effort in clinical practice. To achieve our goal,
we resorted to a divide-and-conquer strategy with two asymmetric sub-tasks to
balance between the supervisions from source and target domain labeled
samples. An EMD scheme is further developed to exploit the pseudo-label
smoothly in SSDA. Our experimental results on the cross-modality SSDA task
using the BraTS18 database demonstrated that the proposed method surpassed the
state-of-the-art UDA and SSDA methods.

## Acknowledgements

This work is supported by NIH R01DC018511, R01DE027989, and P41EB022544.

## References

  * [1] Balcan, M.F., Blum, A., Yang, K.: Co-training and expansion: Towards bridging theory and practice. Advances in neural information processing systems 17, 89–96 (2005)
  * [2] Berthelot, D., Carlini, N., Goodfellow, I., Papernot, N., Oliver, A., Raffel, C.: Mixmatch: A holistic approach to semi-supervised learning. arXiv preprint arXiv:1905.02249 (2019)
  * [3] Blum, A., Mitchell, T.: Combining labeled and unlabeled data with co-training. In: Proceedings of the eleventh annual conference on Computational learning theory. pp. 92–100 (1998)
  * [4] Che, T., Liu, X., Li, S., Ge, Y., Zhang, R., Xiong, C., Bengio, Y.: Deep verifier networks: Verification of deep discriminative models with deep generative models. In: ArXiv (2019)
  * [5] Chen, C., Dou, Q., Chen, H., Qin, J., Heng, P.A.: Synergistic image and feature adaptation: Towards cross-modality domain adaptation for medical image segmentation. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 33, pp. 865–872 (2019)
  * [6] Chen, S., Jia, X., He, J., Shi, Y., Liu, J.: Semi-supervised domain adaptation based on dual-level domain mixing for semantic segmentation. In: CVPR. pp. 11018–11027 (2021)
  * [7] Der Kiureghian, A., Ditlevsen, O.: Aleatory or epistemic? does it matter? Structural safety 31(2), 105–112 (2009)
  * [8] Donahue, J., Hoffman, J., Rodner, E., Saenko, K., Darrell, T.: Semi-supervised domain adaptation with instance constraints. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 668–675 (2013)
  * [9] Han, X., Qi, L., Yu, Q., Zhou, Z., Zheng, Y., Shi, Y., Gao, Y.: Deep symmetric adaptation network for cross-modality medical image segmentation. IEEE Transactions on Medical Imaging (2022)
  * [10] Hoyer, L., Dai, D., Wang, Q., Chen, Y., Van Gool, L.: Improving semi-supervised and domain-adaptive semantic segmentation with self-supervised depth estimation. arXiv preprint arXiv:2108.12545 (2021)
  * [11] Hu, S., Worrall, D., Knegt, S., Veeling, B., Huisman, H., Welling, M.: Supervised uncertainty quantification for segmentation with multiple annotations. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 137–145. Springer (2019)
  * [12] Kendall, A., Gal, Y.: What uncertainties do we need in bayesian deep learning for computer vision? arXiv preprint arXiv:1703.04977 (2017)
  * [13] Kim, T., Kim, C.: Attract, perturb, and explore: Learning a feature alignment network for semi-supervised domain adaptation. In: European Conference on Computer Vision. pp. 591–607. Springer (2020)
  * [14] Kong, L., Hu, B., Liu, X., Lu, J., You, J., Liu, X.: Constraining pseudo-label in self-training unsupervised domain adaptation with energy-based model. International Journal of Intelligent Systems (2022)
  * [15] Liu, X., Hu, B., Jin, L., Han, X., Xing, F., Ouyang, J., Lu, J., Fakhri, G.E., Woo, J.: Domain generalization under conditional and label shifts via variational bayesian inference. IJCAI (2021)
  * [16] Liu, X., Li, S., Ge, Y., Ye, P., You, J., Lu, J.: Recursively conditional gaussian for ordinal unsupervised domain adaptation. In: International Conference on Computer Vision (ICCV) (October 2021)
  * [17] Liu, X., Xing, F., Fakhri, G.E., Woo, J.: Self-semantic contour adaptation for cross modality brain tumor segmentation. IEEE International Symposium on Biomedical Imaging (ISBI) (2022)
  * [18] Liu, X., Xing, F., Stone, M., Zhuo, J., Reese, T., Prince, J.L., El Fakhri, G., Woo, J.: Generative self-training for cross-domain unsupervised tagged-to-cine mri synthesis. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 138–148. Springer (2021)
  * [19] Liu, X., Yoo, C., Xing, F., Kuo, C.C.J., Fakhri, G.E., Kang, J.W., Woo, J.: Unsupervised black-box model domain adaptation for brain tumor segmentation. Frontiers in Neuroscience p. 341
  * [20] Liu, X., Yoo, C., Xing, F., Oh, H., El Fakhri, G., Kang, J.W., Woo, J.: Deep unsupervised domain adaptation: A review of recent advances and perspectives. APSIPA Transactions on Signal and Information Processing (2022)
  * [21] Menze, B.H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., Kirby, J., Burren, Y., Porz, N., Slotboom, J., Wiest, R., et al.: The multimodal brain tumor image segmentation benchmark (BRATS). IEEE transactions on medical imaging 34(10), 1993–2024 (2014)
  * [22] Qiao, S., Shen, W., Zhang, Z., Wang, B., Yuille, A.: Deep co-training for semi-supervised image recognition. In: Proceedings of the european conference on computer vision (eccv). pp. 135–152 (2018)
  * [23] Saito, K., Kim, D., Sclaroff, S., Darrell, T., Saenko, K.: Semi-supervised domain adaptation via minimax entropy. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 8050–8058 (2019)
  * [24] Tajbakhsh, N., Jeyaseelan, L., Li, Q., Chiang, J.N., Wu, Z., Ding, X.: Embracing imperfect datasets: A review of deep learning solutions for medical image segmentation. Medical Image Analysis 63, 101693 (2020)
  * [25] Van Engelen, J.E., Hoos, H.H.: A survey on semi-supervised learning. Machine Learning 109(2), 373–440 (2020)
  * [26] Wang, Z., Wei, Y., Feris, R., Xiong, J., Hwu, W.M., Huang, T.S., Shi, H.: Alleviating semantic-level shift: A semi-supervised domain adaptation method for semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. pp. 936–937 (2020)
  * [27] Wei, C., Shen, K., Chen, Y., Ma, T.: Theoretical analysis of self-training with deep networks on unlabeled data. arXiv preprint arXiv:2010.03622 (2020)
  * [28] Xia, Y., Yang, D., Yu, Z., Liu, F., Cai, J., Yu, L., Zhu, Z., Xu, D., Yuille, A., Roth, H.: Uncertainty-aware multi-view co-training for semi-supervised medical image segmentation and domain adaptation. Medical Image Analysis 65, 101766 (2021)
  * [29] Yao, T., Pan, Y., Ngo, C.W., Li, H., Mei, T.: Semi-supervised domain adaptation with subspace learning for visual recognition. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. pp. 2142–2150 (2015)
  * [30] Zhang, H., Cisse, M., Dauphin, Y.N., Lopez-Paz, D.: mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 (2017)
  * [31] Zou, D., Zhu, Q., Yan, P.: Unsupervised domain adaptation with dualscheme fusion network for medical image segmentation. In: Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, International Joint Conferences on Artificial Intelligence Organization. pp. 3291–3298 (2020)
  * [32] Zou, Y., Yu, Z., Liu, X., Kumar, B., Wang, J.: Confidence regularized self-training. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 5982–5991 (2019)


# Phase-dependent dissipation and supercurrent of a graphene-superconductor
ring under microwave irradiation

Ziwei Dou1, Taro Wakamura1 , Pauli Virtanen2, Nian-Jheng Wu1,3 , Richard
Deblock1 , Sandrine Autier-Laurent1 , Kenji Watanabe4 , Takashi Taniguchi4,
Sophie Guéron1,
Hélène Bouchiat1 , and Meydi Ferrier1 1\. Université Paris-Saclay, CNRS,
Laboratoire de Physique des Solides, 91405, Orsay, France.
2\. Department of Physics and Nanoscience Center, University of Jyväskylä,
Finland
3\. Université Paris-Saclay, CNRS, Institut des Sciences Moléculaires d’Orsay,
Orsay, France
4\. National Institute for Materials Science, Tsukuba, Japan

###### Abstract

A junction with two superconductors coupled by a normal metal hosts Andreev
bound states whose energy spectrum is phase-dependent and exhibits a minigap,
resulting in a periodic supercurrent. Phase-dependent dissipation also appears
at finite frequency due to relaxation of Andreev bound states. While
dissipation and supercurrent versus phase have previously been measured near
thermal equilibrium, their behavior in nonequilibrium is still elusive. By
measuring the ac susceptibility of a graphene-superconductor junction under
microwave irradiation, we find supercurrent response deviates from adiabatic
ac Josephson effect as irradiation frequency is larger than relaxation rate.
Notably, when irradiation frequency further increases above the minigap, the
dissipation is enhanced at phase 0 where the minigap is largest and
dissipation is minimum in equilibrium. We argue that this is evidence of the
nonequilibrium distribution function which allows additional level transitions
on the same side of the minigap. These results reveal that phase-dependent
dissipation is more sensitive than supercurrent to microwave irradiation, and
suggest a new method to investigate photon-assisted physics in proximitized
superconducting system.

A junction with two superconductors coupled by a normal metal hosts Andreev
bound states (ABSs), which shuttle the Cooper pairs from one superconducting
bank to the other and whose energy spectrum depends on their phase difference
($\varphi$). The phase-dependent Andreev levels give rise to supercurrent
$I_{s}$ periodic in $\varphi$, and the measurement of $I_{s}(\varphi)$, or the
current-phase relation (CPR), has previously revealed, for example, the
singlet/doublet transition in carbon nanotube quantum dots [1, 2] and the
helical edge states in topological materials [3, 4]. If $\varphi$ acquires a
time-dependent ac component ($\delta\varphi$), generated for example by an ac
magnetic field in a ring geometry, finite-time relaxation of ABSs towards
equilibrium causes delay in the current response and consequently a counter-
intuitive dissipation appears [5]. Such dissipation involves two important
mechanisms: One is the relaxation of thermally excited ABSs via inelastic
scattering; the other is inter-level transitions induced by microwave photons
[6, 7, 8]. In the weakly driven regime where $\delta\varphi\ll\pi$, the
dissipation shed further light on the properties of the Andreev levels,
revealing, for example, protected level crossing in topological junctions [9,
10]. Strong driving power can significantly modify the distribution function
from thermal equilibrium, activating additional level transitions [11, 12].
However, this nonequilibrium state has so far mainly been investigated in
dissipationless supercurrent response, finding enhanced critical current [11,
12] or modified CPR [13, 14]. Here we present the evolution of both the CPR
and dissipation under microwave irradiation with different frequency and
power, extracted simultaneously from the ac magnetic susceptibility of a
phase-biased graphene-superconductor ring in long diffusive regime. The
Andreev spectrum is characterized by a minigap whose size relative to the
irradiation frequency plays an important role in the junction’s response [15,
16]. Graphene is chosen since it has lower density of states while maintaining
similar mean-free path compared to conventional normal metals [17]. This gives
access to similar minigap with a reduced supercurrent and the screening
effect, enabling accurate measurement over a complete phase period [6].

Figure 1: CPRs without microwave irradiation: (a) SEM images of the device.
Left: A section of the MoRe resonator. Scale bar: 100 µm. Right: zoomed-in
image of the SGS junction. Scale bar: 3 µm. Purple and magenta dashed
outlines: top and bottom BN. White dashed outline: graphene. (b) Circuit
diagram. (c) CPRs at $V_{g}$ = 2 V, and T = 12 mK, 0.9 K and 1.5 K. Inset:
$\chi^{\prime}(\Phi)$. (d) The critical current $I_{c}$ versus $T$ at $V_{g}$
= 2 V. Dashed line: fitting to diffusive model with $E_{Th}$ = 34 µeV. Inset:
The amplitudes of the first three Fourier coefficients of the CPRs $|I_{c,n}|$
(n = 1, 2, 3) versus $T$. (e) $|I_{c,n}|$ (n = 1, 2, 3) versus $V_{g}$ at $T$
= 12 mK.

The ac susceptibility is measured by coupling a graphene/superconductor ring
to a superconducting resonator. Fig. 1(a) shows the scanning electron
microscopy (SEM) images of the device. The left image shows a section of the
resonator (the meander lines) made by e-beam lithography and sputtered
molybdenum-rhenium (MoRe) on the undoped silicon substrate. The boron-
nitride/graphene/boron-nitride (BN/G/BN) stack is fabricated using exfoliated
flakes and is then connected to MoRe via side contacts [18]. The details of
the fabrication is given in [19]. The right image is a zoom of the junction
region. The junction width is $W$ = 5 µm and the length $L$ = 950 nm. The
Ti/Au top-gate covers 330 nm of the total graphene length. In Fig.1(b), the
resonator is designed to have $C_{r}$ = 180 pF and $L_{r}$ = 40 nH. The loop
at the end of the superconducting lines provides a coupling inductance $L_{c}$
= 355 pH. To maintain sufficiently high quality factor $Q$, the resonator is
coupled to the rf coaxial cables on the dilution refrigerator through a
coupling capacitor $C_{cpl}$ = 5.6 pF. At $T$ = 12 mK, the resonance frequency
$\nu_{r}$ = 60 MHz and $Q\sim$ 300\. The dc flux $\Phi$ (or the dc phase
$\varphi$ = $\varphi_{1}$ \- $\varphi_{2}$ = 2$\pi\Phi$/$\Phi_{0}$) is set by
the dc magnetic field through the area defined by the ring, and is added to
the ac flux $\delta\Phi$ inducing the ac current $\delta i_{ac}$. The ac
susceptibility is thus defined as $\chi=\delta i_{ac}/\delta\Phi$ [6]. The rf
power is heavily attenuated so that $\delta\Phi\ll\Phi_{0}$ and $\chi$ does
not depend on the rf power. On resonance, the real (dissipationless) and
imaginary (dissipative) part $\chi^{\prime}$ and $\chi^{\prime\prime}$ are
linked to $\nu_{r}$ and $Q$ via
$\chi^{\prime}=-2(L_{r}/L_{c}^{2})(\delta\nu_{r}/\nu_{r})$ and
$\chi^{\prime\prime}=(L_{r}/L_{c}^{2})\delta\left(1/Q\right)$[6]. As $\Phi$ is
swept, $\delta\nu_{r}$ and $\delta(1/Q)$ are simultaneously measured using a
phase-locked feedback loop which maintains the resonator on resonance [6, 20,
21]. At sufficiently low frequency 60 MHz where
$\chi^{\prime\prime}/\chi^{\prime}\ll 1$ (justified in [19]),
$\chi^{\prime}(\Phi)\approx\partial
I_{s}/\partial\Phi=(2\pi/\Phi_{0})(\partial I_{s}/\partial\varphi)$ where
$I_{s}$ is the supercurrent [6]. Thus, integrating the measured
$\chi^{\prime}(\Phi)$ yields $I_{s}(\varphi)$ of the junction.

We first explore the junction without microwave irradiation. Fig. 1(c)
displays the CPRs $I_{s}(\Phi)$ at $T$ = 12 mK, 0.9 K and 1.5 K
[$\chi^{{}^{\prime}}(\Phi)$ in inset]. From the CPRs we extract the critical
current $I_{c}$ as well as its first three Fourier coefficients $I_{c,n}$ (n =
1,2,3), where $I_{s}(\varphi)\approx
I_{c,1}\sin(\varphi)+I_{c,2}\sin(2\varphi)+I_{c,3}\sin(3\varphi)$. $I_{c}(T)$
and $|I_{c,n}|$ are plotted in Fig.1(d). Note $I_{c,2}$ is negative and
$|I_{c,2}|=-I_{c,2}$. At T = 12 mK, $|I_{c,2}|$ and $|I_{c,3}|$ have non-
negligible values, consistent with the skewed CPR in Fig. 1(c) while at $T$ =
1.5 K only $|I_{c,1}|$ is dominant, meaning the CPR is sinusoidal. Assuming
long diffusive junction model at low temperature, we can fit $I_{c}(T)$ by
$I_{c}=(7.7E_{Th}/eR)\left[1-1.3\exp\left(-7.7E_{Th}/3.2k_{B}T\right)\right]$,
where Thouless energy $E_{Th}$ and normal resistance $R$ are two fitting
parameters [22]. The fitting yields $R$ = 5.6 k and $E_{Th}$ = 34 µeV
(equivalently 400 mK or 8 GHz). The superconducting gap of MoRe is $\Delta$ =
1.8$k_{B}T_{c}$ where $T_{c}\approx 6K$, thus $\Delta/E_{Th}\approx$ 20\.
Since $E_{Th}=\hbar v_{F}l_{e}/2L^{2}$ [22], the mean-free path is
$l_{e}\approx$ 100 nm. The minigap $E_{g}=2\times 3.1E_{Th}=$ 206 µeV (or 46
GHz) [23] is higher than the irradiation frequency accessible in later
experiment. However, the above estimation assumes perfect interface and
strongly diffusive transport. By numeric simulation [19], we show that
$E_{g}=2\times 3.1E_{Th}$ overestimates the minigap in junction with imperfect
contacts and weak disorder, which is plausible in BN-encapsulated graphene
sample. The actual $E_{g}$ may thus fall within the energy range probed in the
experiment. In [19], we also discuss the possibility of describing the data
without irradiation with a ballistic model and find similar levels of
agreement . However, the data with irradiation agree better with the diffusive
model.

Fig.1(e) shows $|I_{c,n}|$ versus $V_{g}$ at $T$ = 12 mK. At $V_{g}$ = 2 V,
the supercurrent is almost saturated at 44 nA, smaller than in previous
studies [24, 25, 26] possibly because of insufficiently filtered radiation
from environment and the higher normal resistance of graphene in the un-gated
regions. Meanwhile, $|I_{c,1}|\sim 5|I_{c,2}|$ at $V_{g}$ = 2 V, similar to a
uniformly gated sample [24], meaning the transparency between the MoRe and
graphene is still enough to preserve $I_{c,2}$. For negative $V_{g}$,
$|I_{c,1}|$ is much reduced and $|I_{c,2}|$, $|I_{c,3}|$ become negligible,
consistent with the formation of pn junctions [24, 26]. Throughout the paper
we focus on the high electron-doping regime with $V_{g}$ = 2 V for the highest
susceptibility signal.

Figure 2: Effects of microwave irradiation on CPR Fourier coefficients: (a)
CPRs with irradiation $\nu$ = 2 GHz. The irradiation power is converted to the
normalized value $s=\alpha eV_{pk}/h\nu$ in parenthesis (see text). The sign
reversal of $I_{s}$ is observed at $s\sim$ 1.2 with halved periodicity. (b)
CPRs with $\nu$ = 19 GHz. The sign reversal of $I_{s}$ is also seen at $s\sim$
1.2, but without halved periodicity. (c, d) $I_{c,1}(s)$ and $I_{c,2}(s)$. All
data are taken at $V_{g}$ = 2 V and $T$ = 12 mK. (e, f) Calculated
$I_{c,1}(s)$ and $I_{c,2}(s)$ from Usadel equations with finite relaxation
rate $\gamma=1.2E_{Th}$. $k_{B}T=0.004\gamma$. $\Delta/E_{Th}=50$. Bessel
functions $I_{c,1}(0)J(s)$ and $I_{c,2}(0)J(2s)$ are plotted for (c, e) and
(d, f) respectively (dashed lines). $J$ is the zero-th order Bessel function
of the first kind. $I_{c,1}(0)$ = 44 nA, and $I_{c,2}(0)$ = -9 nA are the non-
irradiated values from Fig. 1.

After exploring the junction without irradiation and finding the CPRs agree
with previous experiments [24, 26, 27], we now present the data with
irradiation. In Fig. 2(a, b), the CPRs are measured at two irradiation
frequencies $\nu$ = 2 GHz and $\nu$ = 19 GHz, respectively. The power noted in
the figure is converted to the normalized power $s=\alpha eV_{pk}/h\nu$
($V_{pk}$ is the peak voltage at the source and $\alpha$ includes the
attenuation factor from source to junction). In both figures, a sign reversal
of the supercurrent occurs as the irradiation power increases. However, for
$\nu$ = 2 GHz, a strong second harmonic (halved periodicity) is observed
during the reversal while almost no phase-dependence is seen for $\nu$ = 19
GHz in a similar situation. In Figs. 2(c, d) we plot the power dependence of
the first two Fourier coefficients of the CPRs for $\nu$ between 2 GHz and 39
GHz. At low irradiation frequency, the junction shows the adiabatic ac
Josephson effect [28] where ABSs follow instantaneously the oscillating
$\delta\varphi$ and $I_{c,n}$ shows the Bessel function dependence. The
experimental $\alpha$ is hard to calibrate accurately, and for each
$I_{c,1}(s)$, $\alpha$ is chosen such that its first zero coincides with $s$ =
1.2 (the first zero of the Bessel function). The same $\alpha$ is then used
for $I_{c,2}(s)$ of the respective frequency. The 2 GHz data agrees well with
the Bessel function, indicating the microwave drive is adiabatic [29]. It also
shows that the electronic temperature is not significantly heated by
irradiation. For higher frequencies, the agreement is less satisfactory. In
particular, $I_{c,2}$ at $s=1.2$ decreases for higher $\nu$, which is
consistent with the disappearance of halved periodicity in the CPR for $\nu$ =
19 GHz. Using time-dependent Usadel equations incorporating a finite inelastic
scattering rate $\gamma$ [29], we calculate $I_{c,1}(s)$ and $I_{c,2}(s)$ at
low temperature and $\nu$ lower and higher than $\gamma$. The results are
plotted in Figs. 2(e, f). At $\nu=0.2\gamma$, $I_{c,1}(s)$ and $I_{c,2}(s)$
follow the Bessel function as expected, while for $\nu>\gamma$ they show
deviation in qualitative agreement to Figs. 2(c,d). Comparing the theory
($1.2\gamma$ and $2.5\gamma$ curves) with the experiment (5 GHz and 19 GHz
curves), the experimental $\gamma$ can be estimated as between 5/1.2 GHz = 4
GHz and 19/2.5 GHz = 8 GHz. Thus, $h\gamma\lesssim E_{Th}$, reasonable for SNS
junctions [6].

Figure 3: Effects of microwave irradiation on dissipative part of the
susceptibility: (a) $\chi^{\prime\prime}(\Phi)$ without irradiation at $T$ =
12 mK, 0.9 K, and 1.5 K, taken simultaneously as $\chi^{\prime}(\Phi)$ leading
to the CPRs in Fig. 1(c). (b) $\chi^{\prime\prime}(\Phi)$ taken simultaneously
as the CPRs in Fig. 2(a). The irradiation frequency $\nu$ = 2 GHz. The
irradiation power is converted to the normalized value $s$ as in Fig. 2. The
black dashed line is the $\chi^{\prime\prime}(\Phi)$ without irradiation.
$\chi^{\prime\prime}(\Phi)$ shows no significant $\Phi$ dependence under
irradiation. (c) $\chi^{\prime\prime}(\Phi)$ taken simultaneously as the CPRs
in Fig. 2(b). $\nu$ = 19 GHz. $\chi^{\prime\prime}(\Phi)$ changes from
2$\pi$-periodic with no irradiation to $\pi$-periodic under higher irradiation
power. (d) Left: $\delta\chi^{\prime\prime}$ normalized by un-irradiated value
($s$ = 0). Right: $I_{s}$ in Fig. 2(b) normalized by critical current at $s$ =
0. $V_{g}$ = 2 V and $T$ = 12 mK for (b, c, d). The $\chi^{\prime\prime}$
curves are shifted vertically for clarity.

The finite $\nu_{r}/\gamma$ gives rise to a nonzero dissipation
$\chi^{\prime\prime}$ [6]. Fig. 3(a) shows $\chi^{\prime\prime}(\Phi)$ taken
simultaneously as the CPRs in Fig. 1(c) without irradiation.
$\chi^{\prime\prime}(\Phi)$ peaks at $0.5\Phi_{0}$ and its height decreases
with temperature. This higher dissipation is a result of the minigap closing
which allows more excitation-relaxation events between Andreev states [6]. At
low temperature, $\delta\chi^{\prime\prime}/\delta\chi^{\prime}\sim 20$ [here
$\delta\chi=\chi(0.5\Phi_{0})-\chi(0)$] is indeed similar to $\nu_{r}/\gamma$
estimated independently from CPRs, as discussed in [19]. Figs. 3(b, c) show
$\chi^{\prime\prime}(\Phi)$ under irradiation taken simultaneously as the CPRs
in Figs. 2(a, b). For both $\nu$ = 2 GHz and 19 GHz, the small irradiation
power flattens $\chi^{\prime\prime}(\Phi)$. More data in [19] shows a gradual
diminution in $\delta\chi^{\prime\prime}$ for smaller power. This is compared
with the CPR, as illustrated in Fig. 3(d): $\delta\chi^{\prime\prime}(s=0.2)$
is decreased by 80% of the un-irradiated $\delta\chi^{\prime\prime}(s=0)$
whereas the CPR is almost unaffected, demonstrating the much higher power
sensitivity of $\chi^{\prime\prime}$ than CPR. At high irradiation power, the
dissipation response is drastically different for $\nu$ = 19 GHz: Instead of
the flat $\chi^{\prime\prime}(\Phi)$ in Fig. 3(c), an additional lobe appears
in Fig. 3(d) around flux $0$ and the peak at $0.5\Phi_{0}$ becomes pronounced
again, which turns $\chi^{\prime\prime}(\Phi)$ into a function quasi-periodic
in $\Phi_{0}/2$. To our knowledge, this emergence of an enhanced dissipation
at flux 0 under high irradiation frequency and power has never been reported
and requires an explanation.

Figure 4: Calculation of $\chi^{\prime\prime}(\Phi)$ with nonequilibrium
distribution function: (a) Andreev spectrum of a long diffusive system
displaying a minigap $E_{g}\sim 0.2\Delta$. $\Delta$ is the superconducting
pairing potential. Side plot: distribution function change $\delta f(E)$ due
to irradiation (see text). (b) The interband term $|J_{1,-1}|^{2}$ (red) and
the intraband term $|J_{23,44}|^{2}$ (blue). The corresponding transitions are
marked by the arrows in the same colors in (a). (c)
$\chi^{\prime\prime}_{0}(\varphi)$ calculated with $f_{FD}(E)$ (dashed line)
and $\chi^{\prime\prime}_{\pm h\nu}(\varphi)$ calculated with $f_{FD}(E\pm
h\nu)$ (dashed-dotted line). (d) Total $\chi^{\prime\prime}(\Phi)$. $\sigma$
stands for the irradiation power ($\sigma$ = 0 being no irradiation). Curves
are offset for clarity. Similar to experiment, $k_{B}T=h\nu_{r}=0.01\Delta\ll
E_{g}$, relaxation rate $h\gamma=0.15\Delta\sim 0.5E_{g}$ and irradiation
frequency $h\nu=0.5\Delta\sim 2E_{g}$. System size: $W\times L=50\times 30$
sites.

While $\chi^{\prime\prime}(\Phi)$ with nonzero $\nu_{r}$ can be calculated by
similar Usadel equation approach [8] to that in Fig. 2 for the CPRs in the dc
limit, it is not straightforward to include both the low-frequency ac flux and
the high-frequency irradiation. In order to explain Fig. 3(c), we thus turn to
the Kubo formalism, which has well described the $\chi^{\prime\prime}(\Phi)$
of diffusive junctions in previous experiments without irradiation [30, 6,
31]. In our junction, the rf frequency used for susceptibility measurement
$\nu_{r}$ is much smaller than the inelastic scattering rate $\gamma$, thus
$\chi^{\prime\prime}$ can be approximated by [19]:

$\chi^{\prime\prime}\approx-\operatorname{Im}\left[\displaystyle\sum_{n,m\neq
n}|J_{nm}|^{2}\frac{f_{n}-f_{m}}{\epsilon_{n}-\epsilon_{m}}\frac{ih\nu_{r}}{i(\epsilon_{n}-\epsilon_{m}-h\nu_{r})+h\gamma}\right]$
(1)

where $J_{nm}=(e\hbar/m_{e}i)\langle n|\nabla|m\rangle$ is the off-diagonal
element of the current operator, $\epsilon_{n}$ is the n-th Andreev level. The
effect of the microwave irradiation is included phenomenologically in the
distribution function:

$f_{n}=(1-\sigma)f_{FD}(\epsilon_{n})+\sigma/2[f_{FD}(\epsilon_{n}+h\nu)+f_{FD}(\epsilon_{n}-h\nu)]$
(2)

where $\nu$ is the irradiation frequency and $\sigma$ stands for the
irradiation power. Without irradiation ($\sigma$ = 0), $f_{n}$ follows the
Fermi-Dirac distribution
$f_{FD}(\epsilon_{n})=1/[1+\exp(\epsilon_{n}/k_{B}T)]$. With the irradiation,
the change in distribution function from the equilibrium one $\delta
f(\epsilon_{n})=f_{n}-f_{FD}$ shown in Fig. 4(a) qualitatively reproduces the
results based on the time-dependent Green’s function approach near $\varphi$ =
0 [29, 32, 33]: The Andreev state occupation is depleted (enhanced) within
$h\nu$ below (above) the minigap by irradiation photons. The total
$\chi^{\prime\prime}$ can thus be rewritten as
$\chi^{\prime\prime}=(1-\sigma)\chi^{\prime\prime}_{0}+\sigma/2[\chi^{\prime\prime}_{+h\nu}+\chi^{\prime\prime}_{-h\nu}]$,
where $\chi^{\prime\prime}_{0}$ and $\chi^{\prime\prime}_{\pm h\nu}$ are
calculated with Eq.1 using $f_{FD}(E)$ and $f_{FD}(E\pm h\nu)$ respectively.
We numerically calculate these terms in a diffusive SNS junction using the
tight-binding method [34, 31]. See details in [19]. The Andreev spectrum near
$E$ = 0 is plotted in Fig. 4(a) showing a minigap. Fig. 4(b) displays two
examples $|J_{1,-1}|^{2}$ and $|J_{23,44}|^{2}$ whose corresponding
$(f_{n}-f_{m})/(\epsilon_{n}-\epsilon_{m})$ are nonzero. They represent
“interband” transition [red arrow] and “intraband” transition [blue arrow],
respectively. As $\sigma$ goes from 0 to 0.8, $\chi^{\prime\prime}(\varphi)$
evolves from a 2$\pi$-periodic function to a quasi $\pi$-periodic function,
which is the key feature in Fig. 3(c). By comparing Figs. 4(c) and (d), the
rising lobe at phase 0 is attributed to the nonequilibrium terms
$\chi^{\prime\prime}_{\pm h\nu}$ and can be understood as follows: At low
temperature, the equilibrium $f_{FD}$ is close to a step function, thus only
interband terms $|J_{n,-n}|^{2}$ contribute to Eq.(6). Since $|J_{n,-n}|^{2}$
generally peaks at $\pi$ (e.g. $|J_{1,-1}|^{2}$) [31],
$\chi^{\prime\prime}_{0}(\pi)$ is also high. Meanwhile, under irradiation with
$h\nu>E_{g}$, the distribution function is modified and states near the
minigap are partially occupied, enabling more intraband transitions. The
intraband terms $|J_{nm}|^{2}$, which in general have higher magnitudes around
phase 0 (see $|J_{23,44}|^{2}$), produces an enhanced
$\chi^{\prime\prime}_{\pm h\nu}(0)$. The same simulation with $h\nu<E_{g}$
only shows the $2\pi$-periodic $\chi^{\prime\prime}$. The flat
$\chi^{\prime\prime}(\Phi)$ at small power [$s$ = 0.2 in Fig. 3(c)] is not
captured by our simple model. However, this may be due to the detailed form of
the distribution function deviating from Eq.2.

In conclusion, we measured the ac susceptibility of a phase-biased
graphene/superconductor junction in response to the irradiation of microwave
photons. For low irradiation frequency, the power dependence of the CPR
Fourier coefficients follows the adiabatic response, while the phase
dependance of the dissipation almost disappears at small irradiation power,
demonstrating its higher power sensitivity than supercurrent. At higher
irradiation frequency, the CPR responses deviate from adiabaticity and agree
with the quasiclassical theory including finite relaxation rate. More
remarkable effects manifest in dissipation as the irradiation frequency
further increases above the minigap: The dissipation is enhanced around phase
0 which results in a $\pi$-periodic phase dependence. We argue that this new
phenomenon happens as the system is driven strongly out of equilibrium. The
phase-dependent intraband transitions, which are highly suppressed in
equilibrium at low temperature, thus become possible with a nonequilibrium
distribution function induced by microwave irradiation. The measurement
exemplified here may lead to a more detailed understanding of other
nonequilibrium physics in proximitized superconducting system [35, 36, 37]. It
also offers insight to the development of novel superconducting quantum
devices operating in microwave field [38, 39, 40].

###### Acknowledgements.

We acknowledge insightful discussion with T. T. Heikkilä (University of
Jyväskylä, Finland), M. Aprili, and J. Estève (Université Paris-Saclay,
France). We also acknowledge financial support from ERC 833350, Japan Society
for the Promotion of Science (2017-684), Labex PALM, and ANR JETS
(ANR-16-CE30-0029-01).

## References

  * van Dam  _et al._ [2006] J. A. van Dam, Y. V. Nazarov, E. P. A. M. Bakkers, S. De Franceschi, and L. P. Kouwenhoven, Nature 442, 667 (2006).
  * Delagrange  _et al._ [2016] R. Delagrange, R. Weil, A. Kasumov, M. Ferrier, H. Bouchiat, and R. Deblock, Phys. Rev. B 93, 195437 (2016).
  * Murani  _et al._ [2017] A. Murani, A. Kasumov, S. Sengupta, Y. A. Kasumov, V. T. Volkov, I. I. Khodos, F. Brisset, R. Delagrange, A. Chepelianskii, R. Deblock, H. Bouchiat, and S. Guéron, Nat. Commun. 8, 15941 (2017).
  * Li  _et al._ [2019] C. Li, B. de Ronde, J. de Boer, J. Ridderbos, F. Zwanenburg, Y. Huang, A. Golubov, and A. Brinkman, Phys. Rev. Lett. 123, 026802 (2019).
  * Averin and Imam [1996] D. Averin and H. T. Imam, Phys. Rev. Lett. 76, 3814 (1996).
  * Dassonneville  _et al._ [2018] B. Dassonneville, A. Murani, M. Ferrier, S. Guéron, and H. Bouchiat, Phys. Rev. B 97, 184505 (2018).
  * Dassonneville  _et al._ [2013] B. Dassonneville, M. Ferrier, S. Guéron, and H. Bouchiat, Phys. Rev. Lett. 110, 217001 (2013).
  * Virtanen  _et al._ [2011] P. Virtanen, F. S. Bergeret, J. C. Cuevas, and T. T. Heikkilä, Phys. Rev. B 83, 144514 (2011).
  * Fu and Kane [2009] L. Fu and C. L. Kane, Phys. Rev. B 79, 161408 (2009).
  * Murani  _et al._ [2019] A. Murani, B. Dassonneville, A. Kasumov, J. Basset, M. Ferrier, R. Deblock, S. Guéron, and H. Bouchiat, Phys. Rev. Lett. 122, 076802 (2019).
  * Zaikin [1988] A. D. Zaikin,  (Nova Science Publishers, Inc, United States, 1988).
  * Klapwijk and de Visser [2020] T. M. Klapwijk and P. J. de Visser, Annals of Physics 417, 168104 (2020).
  * Basset  _et al._ [2019] J. Basset, M. Kuzmanović, P. Virtanen, T. T. Heikkilä, J. Estève, J. Gabelli, C. Strunk, and M. Aprili, Phys. Rev. Research 1, 032009 (2019).
  * Fuechsle  _et al._ [2009] M. Fuechsle, J. Bentner, D. A. Ryndyk, M. Reinwald, W. Wegscheider, and C. Strunk, Phys. Rev. Lett. 102, 127001 (2009).
  * Likharev [1979] K. K. Likharev, Rev. Mod. Phys. 51, 101 (1979).
  * Zhou  _et al._ [1998] F. Zhou, P. Charlat, B. Spivak, and B. Pannetier, Journal of Low Temperature Physics 110, 841 (1998).
  * Li  _et al._ [2016] C. Li, S. Guéron, A. Chepelianskii, and H. Bouchiat, Phys. Rev. B 94, 115405 (2016).
  * Wang  _et al._ [2013] L. Wang, I. Meric, P. Y. Huang, Q. Gao, Y. Gao, H. Tran, T. Taniguchi, K. Watanabe, L. M. Campos, D. A. Muller, J. Guo, P. Kim, J. Hone, K. L. Shepard, and C. R. Dean, Science 342, 614 (2013).
  * [19] Supplementary materials.
  * Chiodi  _et al._ [2011] F. Chiodi, M. Ferrier, K. Tikhonov, P. Virtanen, T. T. Heikkilä, M. Feigelman, S. Guéron, and H. Bouchiat, Sci. Rep. 1, 3 (2011).
  * Reulet  _et al._ [1995] B. Reulet, H. Bouchiat, and D. Mailly (Springer Netherlands, Dordrecht, 1995) pp. 341–357.
  * Dubos  _et al._ [2001] P. Dubos, H. Courtois, B. Pannetier, F. K. Wilhelm, A. D. Zaikin, and G. Schön, Phys. Rev. B 63, 064502 (2001).
  * Ivanov  _et al._ [2002] D. A. Ivanov, R. von Roten, and G. Blatter, Phys. Rev. B 66, 052507 (2002).
  * Nanda  _et al._ [2017] G. Nanda, J. L. Aguilera-Servin, P. Rakyta, A. Kormányos, R. Kleiner, D. Koelle, K. Watanabe, T. Taniguchi, L. M. K. Vandersypen, and S. Goswami, Nano Lett. 17, 3396 (2017).
  * Ke  _et al._ [2016] C. T. Ke, I. V. Borzenets, A. W. Draelos, F. Amet, Y. Bomze, G. Jones, M. Craciun, S. Russo, M. Yamamoto, S. Tarucha, and G. Finkelstein, Nano Lett. 16, 4788 (2016).
  * Borzenets  _et al._ [2016] I. V. Borzenets, F. Amet, C. T. Ke, A. W. Draelos, M. T. Wei, A. Seredinski, K. Watanabe, T. Taniguchi, Y. Bomze, M. Yamamoto, S. Tarucha, and G. Finkelstein, Phys. Rev. Lett. 117, 237002 (2016).
  * Bretheau  _et al._ [2017] L. Bretheau, J. I.-J. Wang, R. Pisoni, K. Watanabe, T. Taniguchi, and P. Jarillo-Herrero, Nat. Phys. 13, 756 (2017).
  * Tinkham [2004] M. Tinkham, Dover Books on Physics Series (Dover Publications, 2004).
  * Virtanen  _et al._ [2010] P. Virtanen, T. T. Heikkilä, F. S. Bergeret, and J. C. Cuevas, Phys. Rev. Lett. 104, 247003 (2010).
  * Trivedi and Browne [1988] N. Trivedi and D. A. Browne, Phys. Rev. B 38, 9581 (1988).
  * Ferrier  _et al._ [2013] M. Ferrier, B. Dassonneville, S. Guéron, and H. Bouchiat, Phys. Rev. B 88, 174505 (2013).
  * Gunsenheimer and Zaikin [1998] U. Gunsenheimer and A. D. Zaikin, Europhysics Letters (EPL) 41, 195 (1998).
  * Eschrig [2000] M. Eschrig, Phys. Rev. B 61, 9061 (2000).
  * Groth  _et al._ [2014] C. W. Groth, M. Wimmer, A. R. Akhmerov, and X. Waintal, New Journal of Physics 16, 063065 (2014).
  * Watfa  _et al._ [2020] D. Watfa, R. Delagrange, A. Kadlecová, M. Ferrier, A. Kasumov, H. Bouchiat, and R. Deblock, arXiv.2009.09740 (2020).
  * Deacon  _et al._ [2017] R. S. Deacon, J. Wiedenmann, E. Bocquillon, F. Domínguez, T. M. Klapwijk, P. Leubner, C. Brüne, E. M. Hankiewicz, S. Tarucha, K. Ishibashi, H. Buhmann, and L. W. Molenkamp, Phys. Rev. X 7, 021011 (2017).
  * Nakamura  _et al._ [2010] S. Nakamura, Y. Yamauchi, M. Hashisaka, K. Chida, K. Kobayashi, T. Ono, R. Leturcq, K. Ensslin, K. Saito, Y. Utsumi, and A. C. Gossard, Phys. Rev. Lett. 104, 080602 (2010).
  * Schmidt  _et al._ [2018] F. E. Schmidt, M. D. Jenkins, K. Watanabe, T. Taniguchi, and G. A. Steele, Nature Communications 9, 4069 (2018).
  * Lee  _et al._ [2020] G.-H. Lee, D. K. Efetov, W. Jung, L. Ranzani, E. D. Walsh, T. A. Ohki, T. Taniguchi, K. Watanabe, P. Kim, D. Englund, and K. C. Fong, Nature 586, 42 (2020).
  * Kokkoniemi  _et al._ [2020] R. Kokkoniemi, J.-P. Girard, D. Hazra, A. Laitinen, J. Govenius, R. E. Lake, I. Sallinen, V. Vesterinen, M. Partanen, J. Y. Tan, K. W. Chan, K. Y. Tan, P. Hakonen, and M. Möttönen, Nature 586, 47 (2020).


# Darts-Conformer: Towards Efficient Gradient-Based Neural Architecture Search
For End-to-End ASR

###### Abstract

Neural architecture search (NAS) has been successfully applied to tasks like
image classification and language modeling for finding efficient high-
performance network architectures. In ASR field especially end-to-end ASR, the
related research is still in its infancy. In this work, we focus on applying
NAS on the most popular manually designed model: Conformer, and propose an
efficient ASR model searching method that benefits from the natural advantage
of differentiable architecture search (Darts) in reducing computational
overheads. We fuse _Darts mutator_ and Conformer blocks to form a complete
search space, within which a modified architecture called Darts-Conformer cell
is found automatically. The entire searching process on AISHELL-1 dataset
costs only 0.7 GPU days. Replacing the Conformer encoder by stacking searched
architecture, we get an end-to-end ASR model (named as Darts-Conformner) that
outperforms the Conformer baseline by 4.7% relatively on the open-source
AISHELL-1 dataset. Besides, we verify the transferability of the architecture
searched on a small dataset to a larger 2k-hour dataset.

Index Terms—  speech recognition, neural architecture search, end-to-end ASR

##  1 Introduction

As a main stream of automatic machine learning (AutoML), neural architecture
search (NAS) aims at automating the manual process of architecture design [1].
In recent years, as the state-of-the-art results on tasks like image
classification and language modeling have been achieved [2, 3, 4, 5], NAS is
attracting attention increasingly.

Basically, NAS can be classified into three categories: Reinforcement learning
(RL) based [6] searching, evolution algorithm (EA) based [2] searching and
gradient based [7] searching. The former two methods generate a finite number
of discrete neural networks with controller RNN or sample a net within a
neural network population and train these models from scratch, which raises a
relatively high demand for computational overheads (2000 GPU days of RL [3] on
CIFAR-10 and 3150 GPU days of evolution [4] on ImageNet). On contrast,
differentiable architecture search (Darts) [7] is based on the continuous
relaxation of the architecture representation, allowing efficient search of
the architecture using gradient descent. It eliminates the need to train and
compare discrete neural networks. Parameters of architecture determination and
basic operation are optimized alternately within a single supernet.

The recent years have seen the rapid development of end-to-end (E2E) models
applied in automatic speech recognition (ASR) field, including connection
temporal classification (CTC) [8], recurrent neural network transducer (RNN-T)
[9, 10] and attention based encoder decoder (AED) [11, 12, 13], etc.
Benefiting from the strong ability of multi-headed-attention (MHA) in global
context modeling, transformer [11] is competitive among these models in non-
streaming tasks, hence kinds of variants come out [14, 15, 16] and become the
mainstream of ASR research.

Despite of the excellent performance of AED models, NAS has not been broadly
explored in this field for replacing artificial architectures to obtain
further improvement, which is the main contribution of our work. In the work
of Conformer [16], the researchers conducted several comparative experiments
and ablation experiments around the blocks to find the best architecture.
However, such process can be done automatically with Darts and a complete
search space. Inspired by the cell made up of nodes in Darts, we consider the
four modules (two feed-forward modules, one convolution module and one MHA
module) in Conformer block as nodes, and they can form cell with high degree
of freedom. Moreover, the candidate operation within these nodes can be chosen
simultaneously. In this paper, we introduce our method to search encoder
architecture of end-to-end ASR automatically, which we named as Darts-
Conformer.

The following of the paper is organized as follows. In Section 2, we give a
brief review of NAS and Darts algorithms as well as how to combine them with
ASR models. Our methods of Darts architecture search for Conformer, including
the search space and searching strategies, are presented in Section 3. In
Section 4, we discuss the experiments results and analyze the searching
process to show the effectiveness of Darts-Conformer. We conclude our work
with our findings and future works in Section 5.

##  2 Related Work and Background

###  2.1 NAS in speech field

Kinds of neural architecture search have been applied in speech field in
recent years, especially in speech recognition [17, 18, 19, 20] and keyword
spotting [21, 22, 23]. Evolution algorithm based and gradient based methods
are explored in these works.

Focusing on ASR, there are several valuable explorations. Hu et al. [17] used
several improved Darts based methods to find efficient context offset and
bottleneck dimension for acoustic model. Zheng et al. [18] proposed straight-
through gradient search beyond SNAS [24] and ProxylessNAS [25] and trained
acoustic model with CRF-CTC [26], but the search space was still limited to
receptive field and dimension of TDNN-F only. Kim et al. [19] proposed a
relatively complex search space: evolved Transformer, split the encoder and
decoder of Transformer into left and right branch and iterated the population
with progressive dynamic hurdles (PDH) [4]. The final network is smaller and
faster to train, however, the searching progress requires massive computing
resources. He et al. [27] applied Darts on convolution networks for ASR
encoder and yielded considerable improvements over DFSMN baseline. Despite
these previous attempts, there still remains space for improvement in search
space complexity and search efficiency, and few of these works is applied on
novel end-to-end ASR frameworks like Conformer.

###  2.2 Preliminary Darts

Darts [7] proposes a gradient-based architecture searching method for both
convolution and recurrent neural networks. For the former case, several normal
cells and reduction cells are stacked to build up the final network. A _cell_
is a directed acyclic (DAG) graph consisting of an sequence of _nodes_. Each
node $x^{(i)}$ is a latent representation and each directed edge $(i,j)$ is
associated with some operation $o^{(i,j)}$ that transforms $x^{(i)}$.

In such case we have the output of each node based on all of its predecessors:

$x^{(j)}=\sum_{i<j}o^{(i,j)}(x^{i}).$ (1)

Darts innovatively relaxes the neural architecture search space and proposes a
bi-level optimization method. Let
$\mathcal{O}=\\{o^{(i,j)}_{1},o^{(i,j)}_{2},\cdots,o^{(i,j)}_{M}\\}$ be the
set of $M$ candidate operations between $x^{(i)}$ and $x^{(j)}$. The output of
this edge is mixed with a softmax over all operations:

$\overline{o}^{(i,j)}(x)=\sum_{o\in{\mathcal{O}}}\frac{exp(\alpha_{o}^{(i,j)})}{\sum_{o^{\prime}\in{\mathcal{O}}}{exp(\alpha_{o^{\prime}}^{(i,j)})}}o(x).$
(2)

Denote the training loss and validation loss with $\mathcal{L}_{train}$ and
$\mathcal{L}_{val}$. The goal of architecture search is to find a set of
$\alpha$ that minimizes the $\mathcal{L}_{val}(w^{*},\alpha^{*})$, where
$w^{*}$ represents the model weights which minimize the training loss
$w^{*}=argmin_{w}\mathcal{L}_{train}(w,\alpha^{*})$. Such optimization problem
with both lower-level and upper-level variables implies a bi-level
optimization. Darts approximates $w^{*}(\alpha)$ with a single training step
of adapting $w$ on current $\alpha$. Thus we have the approximate architecture
gradient:

$\nabla_{\alpha}\mathcal{L}_{val}(w-\xi\nabla_{w}\mathcal{L}_{train}(w,\alpha),\alpha),$
(3)

applying chain rule to Equation (3) yeilds

$\nabla_{\alpha}\mathcal{L}_{val}(w^{{}^{\prime}},\alpha)-\xi\nabla^{2}_{\alpha,w}\mathcal{L}_{train}(w,\alpha)\nabla_{w^{{}^{\prime}}}\mathcal{L}_{val}(w^{{}^{\prime}},\alpha).$
(4)

The method in practice is to descend architecture gradient in Equation (4) and
the model gradient $\nabla_{w}\mathcal{L}_{train}(w,\alpha)$ alternately. The
case of $\xi=0$ is referred as _first-order approximation_ and the gradient
formulation with $\xi>0$ is referred as _second-order approximation_.

###  2.3 Conformer

As aforementioned, many works have been done to strengthen the local context
modeling ability of speech Transformer, which is proved to be effective.
Conformer [16] achieves state-of-the-art results on LibriSpeech, outperforming
the previous best published Transformer Transducer by 15% relative improvement
on the test-other dataset.

The core methods they proposed are replacing the original feed-forward layer
in the Transformer block into two half-step feed-forward layers which is
inspired by Macaron-Net [28], and using a convolution module which contains a
gating mechanism after multi-headed self-attention.

##  3 Methods

###  3.1 Search Space

A NAS search space is supposed to contain enough reasonable sub-networks. In
Darts case, however, it means that the supernet represented by a DAG has a
reasonable computing structure and the nodes in it are endowed with enough
degree of freedom to mutate. Inspired by Conformer and the CNN cell
architecture in Darts, we propose the Darts-Conformer search space, which
maintains the backbone components of Conformer encoder and allows gradient-
based mutator to adopt NAS within. The core idea of Darts-Conformer cell
(denoted as DC-cell) is fusing the operation in Conformer blocks into Darts
cell.

####  3.1.1 Basic Darts Cell

Considering a simple cell with four nodes shown in Figure 1, each node has all
of its predecessor nodes’ output as its candidate input. The yellow edge
refers to the set of candidate operations while the black edge refers to the
combination of zero edge and skip connection edge (for cutting off the path
and generating residual connection respectively). In searching stage, weights
of zero edge actually suppress an edge more or less and decide which path to
choose. Meanwhile, main operations and skip connection competes with each
other.

Fig. 1: Basic Darts cell architecture.

####  3.1.2 Darts-Conformer Cell

Fig. 2: The Darts-Conformer architecture (right). Notice that operations lie
on the connection lines of DAG. Each single line in DC-cell represents an Op
mutator (left). The unselected paths (searched on AISHELL-1) are represented
by dotted lines.

Intuitively, Darts-Conformer cell (DC-cell) is constructed by fusing Darts
cell into the Conformer-based encoder model which allow the preliminary blocks
of Conformer to choose inputs and the containing operation, also to generate
residual connections. As shown in Figure 2, a DC-cell contains four nodes in
sequence: Macaron feed forward (MAC), multi-headed attention (MHA),
convolution network (CNN) and a feed forward layer (FFC) at last. Their
connection order is similar to Conformer, however, the data flow is relatively
complex.

Table 1: Input candidates and n_chosen of Nodes in DC-Cell. Note that both $\rm{Node}_{0}$ and $\rm{Node}_{1}$ represent input nodes, because $\rm{Node}_{MHA}$ may discard the output of $\rm{Node}_{MAC}$. Index | Name | Input candidate | n_chosen
---|---|---|---
0,1 | $\rm{Node}_{input}$ | - | -
2 | $\rm{Node}_{MAC}$ | {0,1} | 1
3 | $\rm{Node}_{MHA}$ | {0,1,2} | 2
4 | $\rm{Node}_{CNN}$ | {1,2,3} | 2
5 | $\rm{Node}_{FFC}$ | {4} | 1

Each line in DC-Cell (except the ones towards output) represents of an
_operation candidate_ which contains a set of operations to choose from and
their weights $\alpha$. The input candidates of nodes are listed in Table 1.
The n_chosen in the table represents the number of inputs to choose from, and
for nodes with multiple inputs, they will sum up the inputs. For instance,
$\rm{Node}_{CNN}$ is the fourth node and it will choose 2 from outputs of
$\rm{Node}_{MAC}$, $\rm{Node}_{MHA}$ and $\rm{Node}_{input}$.

As for candidate operations, we focus on the kernel size of CNN block (choose
from {15, 23, 31}) and the number of heads in the self-attention. Besides, we
try to find out that if macaron feed forward will be kept by Darts.

###  3.2 Searching Strategy

Different from applying convolution networks on image classification tasks,
there is a big variation of the speech data over $T$-dimension and E2E ASR
models usually contain massive parameters, which tends to exacerbates the
_collapse issue_ in the early stage of searching [29]. Denoting the
architecture-determining parameters as $\alpha$ and parameters of basic
operation as $w$, before optimizing $\alpha$ and $w$ alternatively within a
single step, we freeze $\alpha$ for $e_{w}$ epochs to ensure the operations to
pass the initial convergence stage fairly (as shown in Algorithm 1). In
practice, we use first-order approximation and adam optimizer in searching.

Algorithm 1 Searching and training routine of Dart-Conformer.

1:

2:Form the supernet $S$ with the candidate operation and alternative inputs.

3:Freeze the mutator parameters $\alpha$ for $e_{w}$ epochs and update $w$
with CTC loss.

4:Use first-order approximation to update $\alpha$ and $w$ alternatively with
CTC loss.

5:return the found mutator parameters $\alpha^{*}$.

6:

7:Fix the architecture of a single cell $S^{{}^{\prime}}$ with $\alpha^{*}$
(abort the unselected operations and connections from $S$).

8:Stack the cell into multiple-layer encoder to compose an E2E ASR model $M$;

9:return trained $M$ as the final model and decode;

###  3.3 Strengths of Our Work

Comparing to the previous studies about NAS in ASR aforementioned, our method
has advantages in the following aspects:

  * 1)

The proposed search space absorbs both advantages of Darts cell in
architecture selection and the strengths of Conformer in local and global
context modeling.

  * 2)

The search process of gradient based methods is significantly shorter than
those of RL and evolution based methods.

  * 3)

We conduct Darts search on encoder architecture with CTC loss and frame-level
CE loss rather than the entire encoder-decoder model which eliminates the
potential impact from the source attention and the decoder.

##  4 Experiments

We follow the route below for NAS experiments on ASR task: (i) search on a
small dataset; (ii) stack a single searched architecture to make up an
encoder; (iii) validate the stacked encoder on the same dataset; (iv) evaluate
the transferability of the stacked encoder on an unseen and larger dataset.

###  4.1 Dataset

We search on the open AISHELL-1 data [30] to find the best DC-cell
architecture. This data contains 176 hours of Mandarin speech collected from
400 speakers, and 10 hours develop set. The selected Darts-Conformer is
retrained on the same data to exploit the ASR performance of the selected DC-
cell. A bigger dataset that contains 2,000 hours of Mandarin speech collected
from multi-domain by Sogou (denoted as Sogou-2k) is used to verify the
transferability of Darts-Conformer. We split 30 hours of speech data from
Sogou-2k as development set for model training. Three types of test set are
used for recognition performance comparison, which consist of 18,822 clean
read speech utterances, 11,249 noisy read utterances and 20,000 utterances of
far field spontaneous speech. We denote them as Clean, Noisy, and Far-field
respectively.

Eighty-dimension of FBank features with 3-dimension pitch features are
extracted for AISHELL-1 task and 71-dimension FBank features are extracted for
Sogou-2K. During ASR training stage of AISHELL-1, a 3-fold speed perturbation
($\times{0.9}$, $\times{1.0}$, $\times{1.1}$) is applied.

###  4.2 Architecture Searching

We implement the supernet based on the NNI toolkit [31]. In the searching
stage, we use feed forward layers with 512-dimension hidden units. We use two
separate adam optimizers for $\alpha$ and $\theta$, learning rate is $0.0002$
and $0.0003$ respectively. We use the mixture of CTC loss (0.7) and frame-
level CE loss (0.3) for gradient descent in Darts. And weights $\alpha$ is
frozen for 3 epochs. The maximum epoch is 30 and there are 1.6k steps in a
epoch under the batch size of 48. An entire searching experiment under the
setup above with a single Titan Xp GPU costs 16 hours.

Fig. 3: Operation weights $\alpha$ of $\rm{Node}_{CNN}$ while searching. p1,
p2, p3 are corresponding to input candidate [1,2,3] in Table 1.

Figure 3 shows how weights $\alpha$ in $\rm{Node}_{CNN}$ change in the
training process. The curves of zero operation (orange) indicate that the
input from $\rm{Node}_{input}$ is inhibited mostly. Thus a extra CNN operation
using the raw input is added to preliminary Conformer. And it’s clear that
Darts tends to choose CNN with smaller kernels for Conformer, which is
consistent with the previous results of some artificial network. As for
$\rm{Node}_{MHA}$, the recommended number of heads in self-attention is 4 for
both. In our series of experiments, we find that self-attention blocks are
abandoned early in Darts search, which means that the input candidate
determined by zero operation in $\rm{Node}_{MHA}$ might be stochastic, but the
operation candidate mutator works well. Under such circumstance, Darts chooses
the outputs from $\rm{Node}_{input}$ and $\rm{Node}_{CNN}$ as input, which
means an extra MHA operation to the input without fed into the convolution
layer (comparing to Conformer).

Fig. 4: Seached Darts-Conformer model architecture. The kernel size of
convolution modules is both 15 and the number of attention modules head is
both 4. Dotted boxes refer to adding operations searched from Darts comparing
to preliminary Conformer.

###  4.3 ASR Model Training Setup

We conduct baseline Conformer experiments with ESPnet toolkit [32]. The
baseline model contains a 6-layer Conformer encoder and a 3-layer self-
attention decoder, and the attention in the model is of 256-dimension with 4
heads. The kernel size of the convolution layer is 31. The encoder-decoder
model is trained jointly with CTC loss (joint-weight=0.3) for 65 epochs using
adam optimizer and Noam learning rate strategy [11]. In the inference stage,
we set CTC weight to 0.3 with beam size of 5. No language model is used in
decoding.

For evaluating the performance of Darts-Conformer, we firstly fix both the
chosen edges in the DAG and the operations in the mutator, and then stack the
single cell 6 times to make up a 6-layer encoder. The training strategies and
setups are exactly same as the baseline experiments.

###  4.4 Results on AISHELL-1

We then build up Darts-Conformer model by stacking several DC-cells in encoder
and several self-attention layers in decoder. The results of the baseline
model and proposed Darts-Conformer model in character error rate (CER) is
reported in Tabel 2. The stacked Darts-Conformer cells perform better than the
baseline Conformer 7.0% and 4.7% respectively on the development set and test
set, which proves the effectiveness of the input candidate and operation
candidate mutator of Darts. Considering that two edges selected by darts
introduce additional parameters, we increase the attention dimension in the
baseline experiment from 256 to 288 as an comparative experiment and the
parameters of the model come up to 27.67M. Under such circumstance, the
performance of Darts-Conformer is similar to baseline Conformer with less
parameters.

Table 2: Results on AISHELL-1. | #para. | CER
---|---|---
| eval. | test
Conformer Baseline | 24.17M | 5.7% | 6.4%
\+ larger adim | 27.67M | 5.3% | 6.1%
Darts-Conformer (proposed) | 26.81M | 5.3% | 6.1%
\- input candidate (ablation) | 26.81M | 5.4% | 6.2%
\- chosen operation (ablation) | 26.86M | 5.3% | 6.3%

In addition, two ablation experiments are conducted. If the additional edges
do not take the output of precursor nodes as Darts prefers and goes in the
original order (parallel to the origin layers), the performance of the model
will decrease accordingly. On the other hand, if we do not use the chosen
operations (small CNN kernels and attention heads of 4), the CERs on both sets
raise too.

###  4.5 Transferability On Sogou-2k

For evaluating both the transferability of the architecture searched from
AISHELL-1 and the performance on larger dataset, we compare the baseline model
and Darts-Conformer on Sogou-2k. The training strategies are same as we use in
AISHELL-1 experiments except maximum epoch is set to 30. We evaluate the
models on test sets and the results in shown in Table 3. It turns out that
Darts-Conformer outperforms the baseline Conformer model in the Clean and
Noisy test sets. But in the Far-field test set, they have the same
performance.

Table 3: Transferability experiment results on Sogou-2k. | test sets
---|---
| Clean | Noisy | Far-field
Conformer Baseline | 10.9% | 7.5% | 14.0%
Darts-Conformer | 10.6% | 7.2% | 14.0%

##  5 Conclusion

In this paper we propose Darts-Conformer, a gradient based neural architecture
search framework within which we can find better ASR model architectures
automatically. We fuse Darts mutator and Conformer efficiently: maintaining
the local and global modeling abilities of Conformer and making the nodes able
to be connected with each other in a high degree of freedom. Applying Darts in
the proposed search space, the searching process costs only 0.7 GPU day and
finds the preferring architecture for ASR. A series of experiments on both
AISHELL-1 and Sogou-2k prove that the automatically found architecture
outperforms the preliminary Conformer which is a strong baseline. We verify
the effectiveness of gradient-based architecture search in ASR field. We are
going deep into the gradient-based search methods in E2E ASR. And we also
attempt to design an elaborate and unified supernet where the decoder
architecture can also be searched at the same time.

##  6 Acknowledgements

The authors gratefully acknowledge the ﬁnancial support provided by the
National Key Research and Development Program of China (2020AAA0108004).

## References

  * [1] Thomas Elsken, Jan Hendrik Metzen, Frank Hutter, et al.,  “Neural architecture search: A survey.,”  J. Mach. Learn. Res., vol. 20, no. 55, pp. 1–21, 2019.
  * [2] Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc V Le, and Alexey Kurakin,  “Large-scale evolution of image classifiers,”  in International Conference on Machine Learning. PMLR, 2017, pp. 2902–2911.
  * [3] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le,  “Learning transferable architectures for scalable image recognition,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 8697–8710.
  * [4] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le,  “Regularized evolution for image classifier architecture search,”  in Proceedings of the aaai conference on artificial intelligence, 2019, vol. 33, pp. 4780–4789.
  * [5] Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar,  “Designing neural network architectures using reinforcement learning,”  arXiv preprint arXiv:1611.02167, 2016.
  * [6] Barret Zoph and Quoc V Le,  “Neural architecture search with reinforcement learning,”  arXiv preprint arXiv:1611.01578, 2016.
  * [7] Hanxiao Liu, Karen Simonyan, and Yiming Yang,  “Darts: Differentiable architecture search,”  arXiv preprint arXiv:1806.09055, 2018.
  * [8] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber,  “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,”  in Proceedings of the 23rd international conference on Machine learning, 2006, pp. 369–376.
  * [9] Kanishka Rao, Haşim Sak, and Rohit Prabhavalkar,  “Exploring architectures, data and units for streaming end-to-end speech recognition with rnn-transducer,”  in 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2017, pp. 193–199.
  * [10] Jinyu Li, Rui Zhao, Hu Hu, and Yifan Gong,  “Improving rnn transducer modeling for end-to-end speech recognition,”  in 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2019, pp. 114–121.
  * [11] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,”  arXiv preprint arXiv:1706.03762, 2017.
  * [12] W. Chan, N. Jaitly, Q. Le, and O. Vinyals,  “Listen, attend and spell: A neural network for large vocabulary conversational speech recognition,”  in 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2016.
  * [13] Suyoun Kim, Takaaki Hori, and Shinji Watanabe,  “Joint ctc-attention based end-to-end speech recognition using multi-task learning,”  in 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2017, pp. 4835–4839.
  * [14] Abdelrahman Mohamed, Dmytro Okhonko, and Luke Zettlemoyer,  “Transformers with convolutional context for asr,”  arXiv preprint arXiv:1904.11660, 2019.
  * [15] Shiliang Zhang, Zhifu Gao, Haoneng Luo, Ming Lei, Jie Gao, Zhijie Yan, and Lei Xie,  “Streaming chunk-aware multihead attention for online end-to-end speech recognition,”  arXiv preprint arXiv:2006.01712, 2020.
  * [16] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al.,  “Conformer: Convolution-augmented transformer for speech recognition,”  arXiv preprint arXiv:2005.08100, 2020.
  * [17] Shoukang Hu, Xurong Xie, Shansong Liu, Mengzhe Geng, Xunying Liu, and Helen Meng,  “Neural architecture search for speech recognition,”  arXiv preprint arXiv:2007.08818, 2020.
  * [18] Huahuan Zheng, Keyu An, and Zhijian Ou,  “Efficient neural architecture search for end-to-end speech recognition via straight-through gradients,”  arXiv preprint arXiv:2011.05649, 2020.
  * [19] Jihwan Kim, Jisung Wang, Sangki Kim, and Yeha Lee,  “Evolved speech-transformer: Applying neural architecture search to end-to-end automatic speech recognition,”  Proc. Interspeech 2020, pp. 1788–1792, 2020.
  * [20] Abhinav Mehrotra, Alberto Gil Ramos, Sourav Bhattacharya, Lukasz Dudziak, Ravichander Vipperla, Thomas Chau, Mohamed S Abdelfattah, Samin Ishtiaq, and Nicholas D Lane,  “Nas-bench-asr: Reproducible neural architecture search for speech recognition,”  in International Conference on Learning Representations (ICLR), 2021\.
  * [21] Tom Véniat, Olivier Schwander, and Ludovic Denoyer,  “Stochastic adaptive neural architecture search for keyword spotting,”  in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 2842–2846.
  * [22] Hanna Mazzawi, Xavi Gonzalvo, Aleks Kracun, Prashant Sridhar, Niranjan Subrahmanya, Ignacio Lopez-Moreno, Hyun-Jin Park, and Patrick Violette,  “Improving keyword spotting and language identification via neural architecture search at scale.,”  in INTERSPEECH, 2019, pp. 1278–1282.
  * [23] Bo Zhang, WenFeng Li, Qingyuan Li, Weiji Zhuang, Xiangxiang Chu, and Yujun Wang,  “Autokws: Keyword spotting with differentiable architecture search,”  arXiv preprint arXiv:2009.03658, 2020.
  * [24] Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin,  “Snas: stochastic neural architecture search,”  arXiv preprint arXiv:1812.09926, 2018.
  * [25] Han Cai, Ligeng Zhu, and Song Han,  “Proxylessnas: Direct neural architecture search on target task and hardware,”  arXiv preprint arXiv:1812.00332, 2018.
  * [26] Hongyu Xiang and Zhijian Ou,  “Crf-based single-stage acoustic modeling with ctc topology,”  in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 5676–5680.
  * [27] Liqiang He, Dan Su, and Dong Yu,  “Learned transferable architectures can surpass hand-designed architectures for large scale speech recognition,”  ArXiv, vol. abs/2008.11589, 2020.
  * [28] Yiping Lu, Zhuohan Li, Di He, Zhiqing Sun, Bin Dong, Tao Qin, Liwei Wang, and Tie-Yan Liu,  “Understanding and improving transformer from a multi-particle dynamic system point of view,”  arXiv preprint arXiv:1906.02762, 2019.
  * [29] Hanwen Liang, Shifeng Zhang, Jiacheng Sun, Xingqiu He, Weiran Huang, Kechen Zhuang, and Zhenguo Li,  “Darts+: Improved differentiable architecture search with early stopping,”  arXiv preprint arXiv:1909.06035, 2019.
  * [30] Hui Bu, Jiayu Du, Xingyu Na, Bengu Wu, and Hao Zheng,  “Aishell-1: An open-source mandarin speech corpus and a speech recognition baseline,”  in 2017 20th Conference of the Oriental Chapter of the International Coordinating Committee on Speech Databases and Speech I/O Systems and Assessment (O-COCOSDA). IEEE, 2017, pp. 1–5.
  * [31] Microsoft,  “Neural network intelligence (NNI),” http://nni.readthedocs.io.
  * [32] Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, Adithya Renduchintala, and Tsubasa Ochiai,  “ESPnet: End-to-end speech processing toolkit,”  in Proceedings of Interspeech, 2018, pp. 2207–2211.