# Dissipation of moving vortices in thin films

V.G. Kogan  Ames Laboratory-DOE, Ames, Iowa 50011  N. Nakagawa  Iowa State
University, Ames, IA 50011, USA

###### Abstract

Moving vortices in thin superconducting films are considered within the time-
dependent London description. The dissipation due to out-of-core normal
excitations for two vortices moving together turns out to have a minimum for
the separation vector $\bm{a}$ parallel to the velocity and equal to
$a_{m}\approx 2.2\,\Lambda$, where $\Lambda$ is the Pearl length. The minimum
entropy production suggests that moving vortices should have a tendency to
form chains along the velocity with a period of the order $a_{m}$.

##  I Introduction

Problems of vortex dynamics in superconductors have recently come back to the
community attention because new and more accurate experimental techniques
become available. Vortex velocities well above the speed of sound are now
attainable along with new methods of measuring field distributions [1, 2].

Moving vortices, pushed by the Lorentz force due to applied transport current,
dissipate energy replenished by the current source. In this situation, the
heat transfer should be taken into account [2], just to mention one of the
complications. One of the facts attracting attention is that moving vortices
tend to form chains extended along the velocity. The chains have periods
$a>>\xi$, the vortex core size, so that the linear London approach may provide
useful insights notwithstanding the London inability to treat the vortex core
physics.

In this work we study the dissipation $\cal W$ due to out-of-core
quasiparticles in thin films and find that for a pair of vortices ${\cal
W}(\bm{a})$ has a minimum at a finite separation $\bm{a}$ oriented along the
pair velocity $\bm{v}$. The value of this separation is $a_{m}\approx
2.2\,\Lambda$ with the Pearl length $\Lambda=2\lambda^{2}/d$ ($\lambda$ is the
penetration depth of the film material and $d$ is the film thickness).
According to the principle of minimum entropy production (or minimum
dissipation) in stationary processes [3] the system of moving vortices should
have a tendency to form chains along the velocity in which vortices sit at the
dissipation minima.

Within the general approach to slow relaxation processes one relates the time
derivative of whatever quantity is relaxing, say $\Psi$, to the variational
derivative of the free energy functional ${\cal F}(\Psi)$, see e.g. [4]:

$\displaystyle-\chi\frac{\partial{\Psi}}{\partial t}=\frac{\delta{\cal
F}}{\delta{\Psi}}\,,$ (1)

where $\chi$ is the proper relaxation time. The quantity of interest in our
case is the vortex field distribution $\bm{h}(\bm{r},t)$ away of the vortex
core where the London approach holds and the energy (magnetic+kinetic) is
${\cal F}=\int d^{2}{\bm{r}}\left(h^{2}+\lambda^{2}({\rm
curl}{\bm{h}})^{2}\right)/8\pi$ [5]:

$\displaystyle-\chi\frac{\partial{\bm{h}}}{\partial t}=\frac{\delta{\cal
F}}{\delta{\bm{h}}}\,.$ (2)

This yields

$\displaystyle-\chi\frac{\partial{\bm{h}}}{\partial
t}=\frac{1}{4\pi}(\bm{h}-\lambda^{2}\nabla^{2}\bm{h})\,,$ (3)

which reduces to the common London equation in equilibrium.

The relaxation constant $\chi$ is obtained by comparison with the time
dependent London equation [6], which at distances large relative to the core
size is obtained from the assumption that the current consists of the normal
and superconducting parts:

${\bm{J}}=\sigma{\bm{E}}-\frac{2e^{2}|\Psi|^{2}}{mc}\,\left({\bm{A}}+\frac{\phi_{0}}{2\pi}{\bm{\nabla}}\theta\right)\,,$
(4)

where $\bm{A}$ is the vector potential, $\Psi$ is the order parameter,
$\theta$ is the phase, $\phi_{0}$ is the flux quantum, ${\bm{E}}$ is the
electric field, and $\sigma$ is the conductivity associated with normal
excitations. At these distances, $|\Psi|$ is a constant and acting on Eq. (4)
by curl one obtains [6]:

${\bm{h}}-\lambda^{2}\nabla^{2}{\bm{h}}+\tau\,\frac{\partial{\bm{h}}}{\partial
t}=\phi_{0}\hat{\bm{z}}\sum_{\nu}\delta({\bm{r}}-{\bm{r}_{\nu}})\,,$ (5)

where ${\bm{r}_{\nu}}(t)$ is the position of the $\nu$-th vortex that may
depend on time $t$, $\hat{\bm{z}}$ is the direction of vortices, $\phi_{0}$ is
the flux quantum. The relaxation time

$\tau=4\pi\sigma\lambda^{2}/c^{2}\,.$ (6)

Comparing this with Eq. (3) one has $\chi=4\pi\tau$. In fact, the time-
dependent GL equations can be obtained in a similar manner [4].

##  II Thin films

Let the film of thickness $d$ be in the $xy$ plane. Integration of Eq. (5)
over the film thickness gives for the $z$ component of the field for a Pearl
vortex moving with velocity $\bm{v}$:

$\displaystyle\frac{2\pi\Lambda}{c}{\rm
curl}_{z}{\bm{g}}+h_{z}+\tau\frac{\partial h_{z}}{\partial
t}=\phi_{0}\delta(\bm{r}-\bm{v}t).$ (7)

Here, $\bm{g}$ is the sheet current density related to the tangential field
components at the upper film face by $2\pi\bm{g}/c=\hat{\bm{z}}\times\bm{h}$;
$\Lambda=2\lambda^{2}/d$ is the Pearl length. With the help of div$\bm{h}=0$
this equation is transformed to:

$\displaystyle h_{z}-\Lambda\frac{\partial h_{z}}{\partial
z}+\tau\frac{\partial h_{z}}{\partial t}=\phi_{0}\delta(\bm{r}-\bm{v}t).$ (8)

As was stressed by Pearl [7, 5], the problem of a vortex in a thin film is
reduced to that of the stray field distribution in free space subject to the
boundary condition (8) at the film surface. Since outside the film
curl$\bm{h}=\,\,\,$div$\bm{h}=0$, one can introduce a scalar potential for the
outside field:

$\displaystyle\bm{h}=\bm{\nabla}\varphi,\qquad\nabla^{2}\varphi=0\,.$ (9)

The general form of the potential satisfying Laplace equation and vanishing at
$z\to\infty$ is

$\displaystyle\varphi(\bm{r},z)=\int\frac{d^{2}\bm{k}}{4\pi^{2}}\varphi(\bm{k})e^{i\bm{k}\cdot\bm{r}-kz}\,$
(10)

that is checked by direct differentiation. Here, $\bm{k}=(k_{x},k_{y})$,
$\bm{r}=(x,y)$, and $\varphi(\bm{k})$ is the two-dimensional (2D) Fourier
transform of $\varphi(\bm{r},z=0)$. In the lower half-space one has to replace
$z\to-z$.

As is done in [6], one applies the 2D Fourier transform to Eq. (8) to obtain a
linear differential equation for $h_{z\bm{k}}(t)$, the solution of which is:

$\displaystyle
h_{z\bm{k}}=-k\varphi_{\bm{k}}=\frac{\phi_{0}e^{-i\bm{k}\cdot\bm{v}t}}{1+\Lambda
k-i\bm{k}\cdot\bm{v}\tau}\,.$ (11)

For two vortices separated by $\bm{a}$, the right-hand side of Eqs. (7) and
(8) is

$\displaystyle\phi_{0}\left[\delta(\bm{r}-\bm{v}t)+\delta(\bm{r}-\bm{a}-\bm{v}t)\right],$
(12)

so that we obtain for the field

$\displaystyle
h_{z\bm{k}}=\frac{\phi_{0}e^{-i\bm{k}\cdot\bm{v}t}(1+e^{-i\bm{k}\cdot\bm{a}})}{1+\Lambda
k-i\bm{k}\cdot\bm{v}\tau}\,.$ (13)

##  III Electric field and dissipation for slow motion

This field is found from quasi-stationary Maxwell equations
curl$\bm{E}=-\partial_{t}\bm{h}/c$ and div$\bm{E}=0$ [8, 4], which yeild in 2D
Fourier space:

$E_{x\bm{k}}=-\frac{k_{y}}{k_{x}}E_{y\bm{k}}=-\frac{ik_{y}}{ck^{2}}\,\frac{\partial
h_{z\bm{k}}}{\partial t}\,.$ (14)

For a pair of vortices separated by $\bm{a}$, we have

$\displaystyle\frac{\partial h_{z\bm{k}}}{\partial
t}=-i\frac{\phi_{0}(\bm{k}\cdot\bm{v})\,(1+e^{-i\bm{k}\cdot\bm{a}})}{1+\Lambda
k-i\bm{k}\cdot\bm{v}\tau}e^{-i\bm{k}\cdot\bm{v}t}\,.$ (15)

We are interested in motion with constant velocity $\bm{v}=v\hat{\bm{x}}$, so
that we can evaluate the fields at $t=0$, i.e. the factor
$e^{-i\bm{k}\cdot\bm{v}t}$ can be omitted. Then, Eqs. (14) and (15) yield:

$\displaystyle E_{x\bm{k}}$ $\displaystyle=$
$\displaystyle\frac{\phi_{0}v}{c}\frac{k_{y}k_{x}(1+e^{-i\bm{k}\bm{a}})}{k^{2}(1+\Lambda
k)}\,,$ $\displaystyle E_{y\bm{k}}$ $\displaystyle=$
$\displaystyle-\frac{\phi_{0}v}{c}\frac{k_{x}^{2}(1+e^{-i\bm{k}\bm{a}})}{k^{2}(1+\Lambda
k)}\,.$ (16)

Since the pre-factor here contains $v$, in linear approximation in velocity
the term $i\bm{k}\cdot\bm{v}\tau$ in denominators can be discarded for slow
motion.

The dissipation power follows:

$\displaystyle{\cal W}$ $\displaystyle=$ $\displaystyle\sigma d\int
d^{2}\bm{r}E^{2}=\sigma
d\int\frac{d^{2}\bm{k}}{4\pi^{2}}(|E_{x\bm{k}}|^{2}+|E_{y\bm{k}}|^{2})$ (17)
$\displaystyle=$ $\displaystyle\frac{\phi_{0}^{2}v^{2}\sigma
d}{2\pi^{2}c^{2}}\int\frac{d^{2}\bm{k}\,k_{x}^{2}(1+\cos\bm{k}\bm{a})}{k^{2}(1+k\Lambda)^{2}}\,.$

We now go to dimensionless $\bm{q}=\Lambda\bm{k}$:

$\displaystyle\frac{{\cal W}}{{\cal
W}_{0}}=\int\frac{d^{2}\bm{q}\,q_{x}^{2}(1+\cos\bm{q}\bm{R})}{q^{2}(1+q)^{2}}=W_{1}+W_{2}\,,$
(18)

where ${\cal W}_{0}=\phi_{0}^{2}v^{2}\sigma d/2\pi^{2}c^{2}\Lambda^{2}$ and
$\bm{R}=\bm{a}/\Lambda$. The first contribution

$\displaystyle
W_{1}=\int\frac{d^{2}\bm{q}\,q_{x}^{2}}{q^{2}(1+q)^{2}}=\pi\ln\frac{1}{e\xi}$
(19)

where the upper limit of the divergent integral over $q$ is taken as $1/\xi$
to avoid the vortex core ($\xi$ here is the dimensionless core size). The
second contribution is

$\displaystyle
W_{2}=\int\frac{d^{2}\bm{q}\,q_{x}^{2}\cos\bm{q}\bm{R}}{q^{2}(1+q)^{2}}$
$\displaystyle=\int_{0}^{\infty}\frac{dq\,q}{(1+q)^{2}}\int_{0}^{2\pi}d\phi\cos^{2}\phi\cos[qR\cos(\phi-\alpha)]\qquad$
(20)

with $\phi$ being the azimuth of $\bm{q}$ and $\alpha$ is the angle between
$\bm{R}=\bm{a}/\Lambda$ and $X$. After substitution $\beta=\phi-\alpha$, the
angular integral takes the form

$\displaystyle\int_{0}^{2\pi}d\beta\cos^{2}(\beta$ $\displaystyle+$
$\displaystyle\alpha)\cos(qR\cos\beta)$ (21) $\displaystyle=$ $\displaystyle
2\pi\left(\frac{J_{1}(qR)}{qR}-J_{2}(qR)\cos^{2}\alpha\right),\qquad$

where $J_{1,2}$ are Bessel functions of the first kind. The integration over
$q$ can be done analytically resulting in a cumbersome combination of Bessel
and Hypergeometric functions. We avoid this by doing this integration
numerically. The contours of ${\cal W}_{2}(X,Y)=\,$const are shown in Fig. 1.
Note that the contours of the total dissipation $\cal W$ = const, are in fact
the same because $W_{1}$ is a coordinate independent constant.

Figure 1: Contours of constant dissipation $W_{2}(X,Y)$ for a pair of
vortices, one at the origin and the other at $(X,Y)=(a_{x},a_{y})/\Lambda$
moving with the same velocity along the $X$ axis.

A surprising feature of this plot are the two minima at the $X$ axis situated
symmetrically relative to the origin ($X$ is along $\bm{v}$). One of these
minima is shown in Fig. 2 where the graph of $W_{2}(X,0)$ is plotted to
indicate the minimum position at $X_{m}\approx 2.2$.

Figure 2:  $W_{2}(X,0)$ vs $X$ for the velocity along the $X$ axis. $S=0.1$,
$X$ is in units of $\Lambda$.

To see a clear picture of the dissipation ${\cal W}(\bm{a})=W_{2}(\bm{a})+$
const, we also show the 3D version of the same result in Fig. 3.

Figure 3: 3D plot of $W_{2}(X,Y)$ for the velocity along the $X$ axis.
$(X,Y)=(a_{x},a_{y})/\Lambda$.

For an arbitrary velocity, one has to keep the term $ik_{x}v\tau$ in
denominators of electric field components (16). One then obtains

$\displaystyle\frac{{\cal W}}{{\cal
W}_{0}}=\int\frac{d^{2}\bm{q}\,q_{x}^{2}(1+\cos\bm{q}\bm{R})}{q^{2}[(1+q)^{2}+q_{x}^{2}S^{2}]}\,.$
(22)

The dimensionless parameter

$\displaystyle S=v\,\frac{2\pi\sigma d}{c^{2}}$ (23)

is small even for vortex velocities exceeding the speed of sound presently
attainable [1, 2] if one takes for the estimate the conductivity $\sigma$ of
normal quasi-particles as equal to the normal state conductivity.
Unfortunately, there is not much experimental information about the $T$
dependence of $\sigma$. Theoretically, this question is still debated, e.g.
Ref. [9] discusses possible strong enhancement of $\sigma$ due to inelastic
scattering.

Figure 4: The minimum position $X_{m}$ vs $S$.

We employ the Fast Fourier Transform to evaluate the integral (22). The
position $X_{m}$ of the minimum of ${\cal W}(X,0)$ for each S was obtained
from the contour plot similar to Fig. 1, which was sliced out of the 2D map
obtained from the cosine term of Eq. (22) via 2D FFT. The result is shown in
Fig. 4. Hence, for $S<0.2$, which is the domain of our interest, the minimum
is practically in the same place at $X_{m}=x_{m}/\Lambda\approx 2.2$.

##  IV Discussion

Hence, the dissipation $\cal W$ of two vortices separated by $\bm{R}=(X,Y)$
depends on the pair orientation relative to the velocity $\hat{\bm{v}}$ and on
the pair size $R$. The numerically evaluated dissipation ${\cal W}(X,Y)$ is
shown in Fig. 3. The dissipation power has a minimum if the pair is oriented
parallel to $\bm{v}$ and the vortices are separated by
$a_{m}=R_{m}\Lambda\approx 2.2\Lambda$.

The physical reason for this minimum can be traced to the magnetic structure
of a single moving vortex. It was shown in [6, 10] that the magnetic field is
depleted in front of the moving vortex and enhanced behind it due to induced
currents of normal excitations. If two vortices move so that one follows the
other and $\bm{a}\parallel\bm{v}$, in the space between them the depletion of
the second is compensated by the enhancement due to the leader. The resulting
magnetic field variation in this space is weaker than for a single vortex.
Then the electric field induced in this intervortex region
$\bm{E}\propto\partial_{t}\bm{h}\propto\bm{v}\cdot\bm{\nabla}\bm{h}$ is
suppressed along with the dissipation. Clearly this simple argument does not
work if the pair orientation differs from $\bm{a}\parallel\bm{v}$.

Moving vortices in Pb films were studied in [1]. The penetration depth of bulk
Pb is $\lambda\approx 96\,$nm and the film thickness $d=75\,$nm so that the
Pearl length $\Lambda\approx 246\,$nm. Vortices driven across the thin-film
bridge by a transport current are reported to form chains with spacing $a$
depending on the distance from the bridge edge. Since the driving current
decreases with distance $x$ from the edge, the vortex velocity depends on $x$
as well. The team [1] was able to measure both $v(x)$ and $a(x)$.

According to our model, the pair of moving vortices dissipates the least if it
is oriented along the velocity and separated by $a_{m}\approx 2.2\,\Lambda$.
One can expect the chain of vortices to have a period of the order $a_{m}$.
Taking the experimental estimate of $\Lambda$ we obtain $a_{m}\approx
540\,$nm. In the experiment [1] the chain period varies from $\approx
1500\,$nm near the bridge edge to $\approx 600\,$nm (for the set of data with
the transprot current $18.9\,$mA). Hence, the order of magnitude provided by
our model is correct. In other words, the idea that the chain period is
dictated by the minimum of dissipation agrees qualitatively with observations.

From the data [1], close to the bridge edge the chain period $a\approx
1.5\,\mu$m and the velocity $v\approx 16$ km/s, i.e. the ratio $a/v\approx
10^{-10}\,$s. On the other hand, the theoretical ratio

$\displaystyle\frac{x_{m}}{v}=2.2\,\frac{\Lambda}{v}=\frac{4\pi\sigma\lambda^{2}}{c^{2}S}\,,$
(24)

where we replaced the velocity with $S$ according to Eq. (23). Taking for
$x_{m}/v$ the experimental ratio $a/v\approx 10^{-10}\,$s and $\lambda\approx
96\,$nm, we estimate the conductivity of normal excitations
$\sigma\approx(3\times 10^{19}S)\,$s-1. With $S\sim 10^{-2}$ this gives the Pb
conductivity that again suggests a qualitative relevance of our model. We note
again that recent theories suggest a higher conductivity of the normal
excitations in superconductors than their normal conductivity [9].

## References

  * [1] L. Embon, Y. Anahory, Ž.L. Jelić, E.O.Lachman, Y. Myasoedov, M. E. Huber, G. P. Mikitik, A. V. Silhanek, M. V. Milosević, A. Gurevich, and E. Zeldov, Nat. Commun. 8, 85 (2017).
  * [2] O. V. Dobrovolskiy, D. Yu. Vodolazov, F. Porrati, R. Sachser, V. M. Bevz, M. Yu. Mikhailov, A. V. Chumak, and M. Huth, Nature Communications 11, 3291 (2020); arXiv:2002.08403.
  * [3] M. J. Klein and P. H. E. Meijer, Phys. Rev. 96, 250 (1954).
  * [4] L. P. Gor’kov and N. B. Kopnin, Usp. Fiz. Nauk 116, 413 (1975) [Sov. Phys.-Usp. 18, 496 (1976)].
  * [5] P. deGennes, Superconductivity of Metals and Alloys (Benjamin, New York, 1966).
  * [6] V. G. Kogan, Phys. Rev. B97, 094510 (2018).
  * [7] J. Pearl, Appl. Phys. Lett 5, 65 (1964).
  * [8] L. D. Landau, E. M. Lifshitz, and L. P. Pitaevskii, Elecrtodynamics of Continuous Media, 2nd ed. (Elsevier, Amsterdam,1984).
  * [9] M. Smith, A. V. Andreev, and B. Z. Spivak, Phys. Rev. B101, 134508 (2020).
  * [10] V.G. Kogan and N. Nakagawa, Condens. Matter, 4, 6 (2021). https://doi.org/10.3390/condmat6010004.


# Improving noisy student training for low-resource languages in End-to-End
ASR using CycleGAN and inter-domain losses

###### Abstract

Training a semi-supervised end-to-end speech recognition system using noisy
student training has significantly improved performance. However, this
approach requires a substantial amount of paired speech-text and unlabeled
speech, which is costly for low-resource languages. Therefore, this paper
considers a more extreme case of semi-supervised end-to-end automatic speech
recognition where there are limited paired speech-text, unlabeled speech (less
than five hours), and abundant external text. Firstly, we observe improved
performance by training the model using our previous work on semi-supervised
learning “CycleGAN and inter-domain losses” solely with external text.
Secondly, we enhance “CycleGAN and inter-domain losses” by incorporating
automatic hyperparameter tuning, calling “enhanced CycleGAN inter-domain
losses.” Thirdly, we integrate it into the noisy student training approach
pipeline for low-resource scenarios. Our experimental results, conducted on
six non-English languages from Voxforge and Common Voice, show a 20% word
error rate reduction compared to the baseline teacher model and a 10% word
error rate reduction compared to the baseline best student model, highlighting
the significant improvements achieved through our proposed method.

Keywords: speech recognition, low resource, semi-supervised training,
CycleGAN, noisy student training

Improving noisy student training for low-resource languages in End-to-End ASR
using CycleGAN and inter-domain losses

Chia-Yu Li and Ngoc Thang Vu
---
Institute for Natural Language Processing (IMS), University of Stuttgart
Pfaffenwaldring 5b, 70569 Stuttgart, Germany
<EMAIL_ADDRESS><EMAIL_ADDRESS>

Abstract content

## 1\. Introduction

|  |
---|---|---

(a) The architecture of semi-supervised E2E ASR.

(b) The identity mapping loss. Note that $b$ is the representations from
encode speech or text.

(c) The cycle-consistent inter-domain loss.

Figure 1: The framework of CycleGAN and inter-domain losses Li and Vu (2022).

Over the last decade, there has been a significant improvement in the
performance of speech and language processing technologies, with an increasing
number of systems being deployed across multiple languages and applications.
However, the majority of these efforts have been focused on a limited set of
languages. Given that there are over 6,900 languages worldwide, the biggest
challenge today is to quickly and cost-effectively transfer speech processing
systems to new languages with minimal manual effort. In the field of automatic
speech recognition (ASR), semi-supervised end-to-end (E2E) can be applied to
reduce the amount of annotated data. Two prominent approaches include
consistency-based and iterative self-training-based methods. The consistency-
based method focuses on enhancing the model by improving the representation of
input through training a separate task Tjandra et al. (2017); Hayashi et al.
(2018); Renduchintala et al. (2018); Karita et al. (2018); Hsu and Glass
(2018); Chung and Glass (2018); Chorowski et al. (2019); Hori et al. (2019);
Schneider et al. (2019); Baevski et al. (2019); Ling et al. (2020). The
iterative self-training technique utilizes augmentation to improve the overall
network performance Zavaliagkos et al. (1998); Novotney and Schwartz (1998);
Thomas et al. (2013); Parthasarathi and Strom (2019); Li et al. (2019); Kahn
et al. (2020a); Synnaeve et al. (2020); Hsu et al. (2022). Among the various
techniques, a widely recognized approach known as noisy student training (NST)
has emerged. NST is an iterative self-training method that leverages unlabeled
data to enhance accuracy, particularly in the domains of image classification
and machine translation Xie et al. (2020). Park et al. adapted and improved
NST by employing techniques such as SpecAugment Park et al. (2019a, b) and
incorporating shallow fusion with a language model (LM) into the teacher
network. Additionally, they introduced a normalized filtering score that aids
in generating enhanced transcripts for training the student network Park et
al. (2020). The results demonstrate significant performance on Librispeech
Panayotov et al. (2015) and LibriLight Kahn et al. (2020b).

Although NST is simple and effective, it depends on a substantial quantity of
paired speech-text to train a teacher model, which is used for labeling the
unlabeled speech data that the student model could train on. For low-resource
languages, the paired speech-text is expensive. There are techniques that can
be explored to address this limitation. One approach is to leverage pre-
trained models, such as wav2vec Schneider et al. (2019), where leverages
transfer learning to learn contextual representations from a large corpus of
unlabeled speech data. The model can then be fine-tuned for the target domain
using unlabeled speech data from the same target domain. However, this
approach still requires a reasonable quantity of speech data, which is still
expensive in low-resource scenario. Besides, this technique requires multi-
stage tuning processing which introduces computational cost. How to improve
inexpensively the teacher model in NST remains a key challenge especially in
language with very small data.

Our previous work “cycle-consistent generative adversarial networks (CycleGAN)
and inter-domain losses”, which is the dissimilarity between the intermediate
representations of encoded speech and its hypothesis Li and Vu (2022), was
proposed for semi-supervised E2E ASR. The architecture is shown in 1(a).
CycleGAN and inter-domain losses (CID) encourage the model to learn the common
representations from the speech and text. With the advantage of this structure
allowing speech and text input, we observe that training a model by CID with
small paired speech-text and additional external text (without additional
speech) can still improve the ASR performance. Therefore, we propose
leveraging it into the training pipeline of NST to enhance the teacher model
solely using a large amount of external text. Subsequently, the improved
teacher model generates better labels for the unlabeled speech, which the
student model can train on.

In this paper, we make several contributions in the following aspects:
Firstly, we observe that training a model by CID Li and Vu (2022) with lots of
external text significantly boosts performance (subsection 2.2); Secondly, we
enhance CID by incorporating automatic hyperparameter tuning, calling enhanced
CID (subsection 2.3); Thirdly, we improve the NST training pipeline for low-
resource scenarios by boosting the teacher model using enhanced CID
(subsection 2.4); Fourthly, we evaluate our method on six languages on the
Voxforge and Common Voice (section 3 and section 4). The results demonstrate
that our proposed approach achieves a 20% word error rate reduction (WERR)
compared to the baseline (NST) teacher model, and a 10% WERR compared to the
baseline student model for most languages. Notably, the improvement of teacher
model is accomplished without the need for additional speech data. Lastly, we
provide an analysis of the recognition output and cherry-pick hypothesis
(section 5).

For the sake of simplicity, throughout the rest of this paper, we use the term
“paired data” to refer to “paired speech-text,” the term “unpaired data” to
refer to “unpaired speech-text,” the term “CID” to refer to the “CycleGAN and
inter-domain” approach, and our proposed NST pipeline designed for low-
resource using CID is denoted as “cNST”.

## 2\. Method

Model | paired data | unpaired text | without LM | with LM
---|---|---|---|---
|  | (#lines) | WER(%) | WER(%)
Initial model ($M_{0}$) | Voxforge German (5 hrs.) | 0 | 63.6 | 63.1
CID model ($M_{1}$) | Voxforge German (5 hrs.) |  10K Goldhahn et al. (2012) | 38.6 | 36.3
| Voxforge German (5 hrs.) |  100K Goldhahn et al. (2012) | 31.2 | 29.4
| Voxforge German (5 hrs.) |  300K Goldhahn et al. (2012) | 30.8 | 29.1
Table 1: WERs on the Voxforg German test set. Note that the initial model is
trained by supervised objective in Equation 2 with five-hour Voxforg German
train data, and the CID model ($M_{1}$) is trained with same five-hour Voxforg
German train data and external text from Leipzig corpus Goldhahn et al. (2012)
via semi-supervised objective in Equation 1.

### 2.1. CycleGAN and inter-domain losses (CID)

Figure 1(a) shows the CID architecture, which is based on semi-supervised E2E
speech recognition and joint CTC-attention E2E Kim et al. (2017); Watanabe et
al. (2017); Karita et al. (2018). The encoder is $e=\hat{e}\circ f$ when the
input is speech. If the input is text, the encoder is the composition of text
embedding $g(.)$ and the share encoder $\hat{e}$. i.e., $\hat{e}\circ g$. The
model is trained by jointly CTC-attention objective on paired data
$S=\\{X,Y\\}$ and by CID on unpaired data $U=\\{X^{\prime},Y^{\prime}\\}$
simultaneously. The objective is as follows Karita et al. (2018); Li and Vu
(2022),

$\mathcal{L}=\alpha\mathcal{L}_{pair}(e,d,S)+(1-\alpha)\mathcal{L}_{unpair}(f,g,\hat{e},d,U)$
(1)

where the supervised ratio $\alpha$ is a tunable parameter.

The supervised objective is negative log likelihood of the ground-truth $y$
given the encoded speech $e(x)$ Watanabe et al. (2017):

$\displaystyle\begin{split}\mathcal{L}_{pair}(e,d,S)&=-\sum_{(x,y)\in S}\log
d(e(x))\\\ &=-\sum_{(x,y)\in
S}\log\prod_{t=1}^{|y|}\Pr(y_{t}|y_{t-1},e(x))\end{split}$ (2)

The unsupervised objective CID consists of the identity mapping loss, the
cycle-consistent inter-domain loss, and the text-to-text autoencoder loss with
tunable hyperparameter speech-to-text ratio $\beta\in[0,1]$ Li and Vu (2022),

$\displaystyle\begin{split}\mathcal{L}_{unpair}(f,g,\hat{e},d,U)&=\mathcal{L}_{idt}(f,g,\hat{e},U)\\\
&+\beta*\mathcal{L}_{cyc,dom}(f,g,\hat{e},d,U)\\\
&+(1-\beta)*\mathcal{L}_{text}(g,\hat{e},d,U)\end{split}$ (3)

The identity loss enhances the shared encoder $\hat{e}(.)$ to preserves
important features after translation. The computation of loss in 1(b) is as
follows,

$L_{idt}=\rVert\hat{e}(b)-b\rVert_{1}$ (4)

where the representation is coming from speech $b=f(x)$ or text $b=g(y)$.

The cycle-consistent inter-domain loss is the dissimilarity between the
representations of encoded speech and its hypothesis, which aims to let
networks learn common knowledge from speech and text. The illustration of loss
is shown in 1(c) and the definition is as follows,

$\begin{split}L_{cyc,dom}&=\mathcal{D}(input\\_B,cycle\\_B)\\\
&=\mathcal{D}(e(x),\hat{e}(g(d(e(x)))))\end{split}$ (5)

where $\mathcal{D}(.)$ is a distance measure of the distributions. In our
previous work, we use Maximum Mean Discrepancy (MMD) because it achieves the
best result Li and Vu (2022).

The text-to-text autoencoder loss measures a negative log-likelihood that the
encoder-decoder network can reconstruct text from unpaired text Hinton and
Salakhutdinov (2006); Karita et al. (2018), see the orange line in 1(a). The
loss is defined as follows,

$L_{text}=-\sum\log\Pr(y|\hat{e}(g(y)))$ (6)

### 2.2. CID solely with external text

In low-resource settings, acquiring paired data or speech data can be costly.
Therefore, this section focus on enhancing the model inexpensively. In our
previous work Li and Vu (2022), we train model by CID with an equal amount of
unlabeled speech and text. However, training a model by CID without additional
unlabeled speech and with only external text (i.e., $U=\\{X,Y^{\prime}\\}$)
might still gain performance improvements. To validate this hypothesis, Table
1 presents the evaluation of models on Voxforge German test set. These models
are trained by jointly CTC-attention objective on paired data $S=\\{X,Y\\}$
and by CID on speech from paired data and text from Leipzig German corpus
Goldhahn et al. (2012) $U=\\{X,Y^{\prime}\\}$ simultaneously. The results
demonstrate that CID models trained with 10K/100K/300K lines of external text
improve WERs from 63.6% to 38.6/31.2/30.8% without involving a language model.
Moreover, when evaluated with a language model, the CID model improves WERs
from 63.1% to 36.3/29.4/29.1%. These findings highlight the effectiveness of
incorporating CID with external text to enhance the performance of E2E model.
It also indicates that the CID allows text to benefit not only the language
model (LM) but also the encoder-decoder model.

Model |  supervised ratio $\alpha$ | adapted Equation 3 $\mathcal{L}_{unpair}$ | CER(%)
---|---|---|---
BaselineLi and Vu (2022) |  |  | 46.9
MIN-UNPAIR-LOSS | 0.5 | $\min_{\beta\in\\{0,0.1,0.2,...,1.0\\}}\mathcal{L}_{unpair}$ | 30.6
MAX-UNPAIR-LOSS | 0.5 | $\max_{\beta\in\\{0,0.1,0.2,...,1.0\\}}\mathcal{L}_{unpair}$ | 39.5
AVG-UNPAIR-LOSS | 0.5 | $\overline{\mathcal{L}_{unpair}}$ | 50.6
MED-UNPAIR-LOSS | 0.5 | Median($\mathcal{L}_{unpair}$) | 50.4
DECAY-MIN-UNPAIR-LOSS | decay | $\min_{\beta\in\\{0,0.1,0.2,...,1.0\\}}\mathcal{L}_{unpair}$ | 29.6
DECAY-MAX-UNPAIR-LOSS | decay | $\max_{\beta\in\\{0,0.1,0.2,...,1.0\\}}\mathcal{L}_{unpair}$ | 44.1
DECAY-AVG-UNPAIR-LOSS | decay | $\overline{\mathcal{L}_{unpair}}$ | 46.6
DECAY-MED-UNPAIR-LOSS | decay | Median($\mathcal{L}_{unpair}$) | 30.3
Table 2: This table compares the CERs on the Common Voice Finnish test set of
models with or without (1) the supervised ratio decay and (2) automatic
speech-to-text ratio tuning. We also observe the same conclusion in six
languages test sets from Common Voice and Voxforge.

### 2.3. Enhanced CID by incorporating automatic hyperparameter tuning

Although the CID model achieves a significant reduction in character error
rate (CERR) across English datasets, WSJ and Librispeech, as well as low
supervision non-English datasets (Voxforge) Li and Vu (2022), it requires
effort to tune the two hyperparameters, the supervised ratio $\alpha$ and the
speech-to-text ratio $\beta$, for each dataset. To streamline the training
pipeline, we propose using supervised ratio decay and automatic speech-to-text
ratio tuning by performing an operation on the unsupervised losses with all
the possible values for the speech-to-text ratio during the training. The
details are as follows: Firstly, we suggest that the model obtains lots of
guidance from the supervision data at the early stages of training. Therefore,
$\alpha$ starts at 0.9 for the first three epochs and gradually decays after
three epochs until the training is completed, which enables the model to
explore the unpaired data with increased flexibility. Secondly, we integrate
the speech-to-text ratio into the training process, we propose to use minimal,
maximal, average, or median operations on the unsupervised losses with $\beta$
from $0.0$ to $1.0$. Table 2 shows our proposed adapted unsupervised losses
and the corresponding CERs on the Common Voice Finnish test set. This table
reveals that the model using minimal operation outperforms the ones using
other operations and baseline. The best model is the model using the
supervised ratio decays and minimal operations on the unsupervised losses over
$\beta$. We observe the same conclusion in six languages from Common Voice and
Voxforge. Figure 2 and Figure 3 present the training loss and the accuracy of
baseline and models trained by our adapted objective in Table 2. The model
using minimal operation on unsupervised loss performs stable and improved
accuracy during the training, whereas the baseline and other models using
maximum, average, and median operations produce mismatched training loss and
validated loss, as well as fluctuating model accuracy during the training.
These figures resonated with the result from the Table 2, the model trained by
Equation 1 using supervised ratio decay and performing minimal operation on
unsupervised loss achieves the best performance.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

Figure 2: The training loss (left) and the accuracy (right) of models using
different automatic speech-to-text ratio tuning defined in Table 2.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

Figure 3: The training loss and accuracy of models using supervised ratio
decay and different automatic speech-to-text ratio tuning defined in Table 2.

$M_{0}$$M_{1}$$M_{2}$$M_{3}$$M_{4}$$M_{5}$$M_{6}$$M_{7}$$M_{8}$$50$$60$$70$$80$Model
generation (Finnish)WER(%)Baseline (NST)cNST

$M_{0}$$M_{1}$$M_{2}$$M_{3}$$M_{4}$$M_{5}$$M_{6}$$M_{7}$$30$$40$$50$$60$Model
generation (Greek)WER(%)Baseline (NST)cNST

Figure 4: WERs on the Common Voice (Finnish and Greek) test set against model
generations.

### 2.4. Noisy student training with CycleGAN and inter-domain losses (cNST)
for low-resource languages

NST for speech recognition is effective when sufficient paired data is
available. However, the paired data and unlabeled speech are often limited in
a low-resource setting. That leads to a low performance teacher model, which
generates low-quality labels for unlabeled speech; the training for the
student model can be severely affected, resulting in inefficient training.

We aim to improve the teacher model with little effort and less cost regarding
time and finances. subsection 2.2 demonstrates that the model can be improved
by CID solely with external text. Therefore, we propose to exploit the
enhanced CID in subsection 2.3 and external text to improve the teacher model.
A LM is also trained with the in-domain and external text
$\\{Y,Y^{\prime}\\}$. The NST algorithm is revised as follows,

  1. 1.

Train $M_{0}$ on $S$ using SpecAugment.

  2. 2.

Train $M_{1}$ on $S$ and $U=\\{Y^{\prime}\\}$ by enhanced CID and using
SpecAugment. Set $M=M_{1}$.

  3. 3.

Fuse $M$ with LM and measure performance.

  4. 4.

Generate labelled dataset $M(X^{\prime})$ with fused model.

  5. 5.

Mix dataset $M(X^{\prime})$ and $S$. Use mixed dataset to train new model
$M^{\prime}$ with SpecAugment.

  6. 6.

Set $M=M^{\prime}$ and go to 3.

where the initial model $M_{0}$ is trained with the paired data $S$ using
SpecAugment Park et al. (2019a), and we further re-train it at the stage 2
using the enhanced CID with external text with SpecAugment. At stage 3, the
teacher model is then fused with a LM to generate labels for the unlabeled
speech. Subsequently, the student model is iteratively trained with the paired
and newly labeled speech data by the supervised objective. We work with small
data, so it is better to utilize the available data wisely rather than
removing any of it. Therefore, we simplify the NST training recipe, making it
easily applicable to all languages by discarding the sophisticated filtering
and balancing stages in Park et al. (2020).

Model | Voxforge (WER%) | Common Voice (WER%)
---|---|---
| German | Italien | Dutch | Hungarian | Finnish | Greek
Initial Model ($M_{0}$) | 63.1 | 71.2 | 63.1 | 84.8 | 77.4 | 63.2
Baseline (NST) | 49.7 | 47.1 | 58.2 | 72.0 | 55.1 | 34.0
Proposed Method (cNST) | 27.3 | 42.0 | 56.3 | 58.6 | 48.4 | 29.4
WERR % (NST-cNST)/NST | 45.1 | 10.8 | 3.26 | 18.6 | 12.7 | 13.5
Table 3: WERs comparison between baseline best student model and our proposed
cNST best student model across corpus.

## 3\. Experimental Setup

### 3.1. Dataset

Common Voice is a massively multilingual collection of transcribed speech,
which is also recorded by user on Mozilla website, and recently it reaches 100
languages Ardila et al. (2020). We conducted experiments on a subset of
European languages which has limited data: Hungarian, Finnish and Greek.
Additionally, we ensured that there were no overlapping sentences or speakers
between the train, development and test set. The data size of
train/development/test sets are in an 80:10:10 ratio and the test set contains
at least two hours speech. The train set is further split to five hours paired
data and the remaining portion (around three hours to five hours) is dedicated
to the unlabeled speech. Voxforge consists of user submitted audio clips using
their own microphone Voxforge.org  and has eight European languages. Each
language has limited size of data, ranging from approximately eight to twenty
hours. In this paper, we evaluate our proposed method on German, Italian and
Dutch languages. The train set is further divide into five hours paired data,
while the remaining portion is dedicated to the unlabeled speech $X^{\prime}$.
The Leipzig corpus, which consists of annual collections of documents from
various sources such as wikis, news, and the web Goldhahn et al. (2012), is
used as external text $Y^{\prime}$ in the experiment.

### 3.2. Network architecture

The semi-supervised E2E model using CycleGAN-inter-domain losses is
implemented under Espnet1 Watanabe et al. (2018) and Li and Vu (2022). The
model consists of three layers of Vgg Simonyan and Zisserman (2015)
bidirectional long short-term memory with projection (Vggblstmp) encoder and
attention based decoder, which is one layer long short-term memory (LSTM) with
320 units. The text embedding $g(.)$ encodes the labels over
$\\{Y,Y^{\prime}\\}$ to an one-hot vector and process it by one layer
bidirectional long short-term memory (BLSTM). Byte pair encoding (BPE) Gage
(1994); Sennrich et al. (2016) is used for some languages, some have better
performance without using BPE. The input acoustic feature is 80-bin log-Mel
filterbank with three pitch coefficients. For decoding, we use a beam search
algorithm with beam size of 20. Our training recipe and
code111https://github.com/chiayuli/Improved-NST-for-low-resource-language.git

Models |  | Hypothesis
---|---|---
Ground-Truth |  | es ist sehr beständig gegen witterungseinflüsse und insektenbefall
Initial Model |  | es ist sehr BESTÄNDIGEN ***** WEITEREN SPÄTEREN SECKER
Baseline(NST) |  | es ist sehr BESTÄNDIGEN ***** WEITEREN EINFLÜSSE *** *************
CID |  | es ist sehr BESTÄNDE gegen WEITERUNGSFLÜSSE und IN SEKTEN BEFALL
cNST |  | es ist sehr BESTÄNDE gegen WEITERUNGSEINFLÜSSE und INSEKTEN BEFALL
Ground-Truth |  | der anspruch ist von der Frau auf den Mann Übergegangen
Initial Model |  | der SPRUCH ist *** *** **** *** *** VOLLKOMMEN REGELT
Baseline(NST) |  | der anspruch ist *** *** **** *** *** **** FREI
CID |  | ER EINE SPRUCH ist von der frau auf DIE LANDEN Übergegangen
cNST |  | der anspruch ist von der frau auf DIE LANDEN Übergegangen
Ground-Truth |  | der Traffic des ersten anbieters wird zum zweiten anbieter weitergeleitet
Initial Model |  | der ******* *** ****** DRITTES SPÄTER NETZwerK KANN NETZwerK GELEITET
Baseline(NST) |  | der TRITTE IST ALS anbieters **** *** ZWEI LIETER GELEITET
CID |  |  der TRÄFT IST ES ANBIETS werT ZU zweiten anbieter WEITER GELEITET
cNST |  |  der TRÄFT IST ES anbieters wird ZU zweiten anbieter WEITER GELEITET
Table 4: The hypothesis of all the models on the unlabeled speech from Voxforge German. Note that the words in uppercase are incorrect compared to the ground-truth and the words in yellow means insertion. Models | WER(%) | INS | DEL | SUB
---|---|---|---|---
Initial Model | 63.1 | 1.8 | 20.6 | 40.7
Baseline | 49.7 | 1.0 | 21.0 | 27.9
CID | 29.4 | 3.3 | 4.0 | 22.0
cNST | 27.3 | 3.2 | 3.6 | 20.5
Table 5: The WER, insertion, deletion, and substitution at word level on the
Voxforge German test set. Note that all the results are with the same LM.

## 4\. Result

### 4.1. WERs against model generation

Figure 4 shows WERs on the Common Voice (Finnish and Greek) test sets against
model generations . We trained the models using our proposed algorithm cNST in
subsection 2.4 and evaluated the teacher model and all the student models at
different stages. Based on the observed trend in model performance, it is
evident that the red line (cNST) demonstrates a steeper progression compared
to the blue line (NST) from $M_{0}$ to $M_{1}$. This suggests that the
enhanced CID plays a crucial role in accelerating the iterative training
process and achieving better results compared to the baseline for all the
model generations. Besides, red and blue lines fluctuate over the generations,
which might be because the models are over-fitting on the train set, but it
does not hurt the subsequent student model performance.

### 4.2. cNST effectiveness across corpus

Table 3 presents the performance of our proposed method, cNST, across various
corpora. We examine the baseline best student model and our proposed cNST best
student model on Voxforge German, Italien, Dutch and Common Voice Hungarian,
Finnish Greek datasets. The result shows that cNST outperforms the baseline by
achieving at least $10$% WERR for most languages. Moreover, when the initial
model performs poorly (above 70% WER), our proposed cNST successfully reduces
the WERs to 40$\sim$50%, indicating the effectiveness of our proposed method.

## 5\. Analysis

### 5.1. Recognition output

We want to gain insights and the reasons for the improvements brought about by
enhanced CID. Table Table 5 presents the WER, insertion, deletion, and
substitution on the test set of Voxforge German. The initial model experiences
a high number of deletion errors, which are propagated to the subsequent
student models in the baseline (NST). However, with enhanced CID, the deletion
errors decrease from 20.6 to 4.0. On the other hand, there is a side-effect as
the insertion errors increase from 1.8 to 3.3. Overall, the subsequent student
model of our proposed cNST achieve the best WER and better substitution and
deletion.

### 5.2. Cherry-pick hypothesis

Some cherry-pick examples in Table 4 demonstrate that the initial model and
baseline experience high deletion errors. However, the baseline exhibits a
further worsening of these errors as the student model undergoes iterative
training using labels that contain such errors. This observation resonates
with the findings presented in Table 5. The enhanced CID model and our
proposed cNST successfully reduce deletion errors. However, there is still
room for improvement in terms of substitution and insertion errors.
Interestingly, In the last example, if we combine both insertion words “WEITER
GELEITET” to “WEITERGELEITET”, it aligns with the correct word in the
reference. The issue with insertions can be attributed to inaccurate word
boundary predictions from our proposed models.

## 6\. Conclusion

We enhance the CID by incorporating automatic hyperparameter tuning and
propose an improved noisy student training that leverages the enhanced CID for
low-resource languages. The enhanced CID accelerates the iterative self-
training process by sorely utilizing external text. The results demonstrate
the effectiveness of our proposed method cNST across six non-English languages
from two datasets, surpassing the baseline by 10% WER.

## References

  * Ardila et al. (2020) Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M. Tyers, and Gregor Weber. 2020.  Common Voice: A Massively-Multilingual Speech Corpus.  In _Proc. of LREC_.
  * Baevski et al. (2019) Alexei Baevski, Steffen Schneider, and Michael Auli. 2019.  Vq-wav2vec: Self-supervised learning of discrete speech representations.  _arXiv:1910.05453_.
  * Chorowski et al. (2019) Jan Chorowski, Ron J. Weiss, Samy Bengio, and Aäron van den Oord. 2019.  Unsupervised speech representation learning using WaveNet autoencoders.  In _Proc. of IEEE TASLP_.
  * Chung and Glass (2018) Yu-An Chung and James R. Glass. 2018.  Speech2vec: A sequenceto-sequence framework for learning word embeddings from speech.  In _Proc. of Interspeech_.
  * Gage (1994) Philip Gage. 1994.  A new algorithm for data compression.  _The C Users Journal_.
  * Goldhahn et al. (2012) Dirk Goldhahn, Thomas Eckart, and Uwe Quasthoff. 2012.  Building Large Monolingual Dictionaries at the Leipzig Corpora Collection: From 100 to 200 Languages.  In _Proc. of LREC_.
  * Hayashi et al. (2018) Tomoki Hayashi, Shinji Watanabe, Yu Zhang, Tomoki Toda, Takaaki Hori, Ramon Astudillo, and Kazuya Takeda. 2018.  Back-Translation-Style Data Augmentation for End-to-End ASR.  In _Proc. of SLT_.
  * Hinton and Salakhutdinov (2006) G. E. Hinton and R. R. Salakhutdinov. 2006.  Reducing the dimensionality of data with neural networks.  In _Science_ , volume 313, page 504–507.
  * Hori et al. (2019) Takaaki Hori, Ramón Fernandez Astudillo, Tomoki Hayashi, Yu Zhang, Shinji Watanabe, and Jonathan Le Roux. 2019.  Cycle-consistency training for end-to-end speech recognition.  In _Proc. of ICASSP_.
  * Hsu and Glass (2018) Wei-Ning Hsu and James R. Glass. 2018.  Extracting domain invariant features by unsupervised learning for robust automatic speech recognition.  In _Proc. of ICASSP_.
  * Hsu et al. (2022) Wei-Ning Hsu, Ann Lee, Gabriel Synnaeve, and Awni Y. Hannun. 2022.  Self-supervised speech recognition via local prior matching.  _arXiv:2002.10336_.
  * Kahn et al. (2020a) Jacob Kahn, Ann Lee, and Awni Y. Hannun. 2020a.  Self-training for End-to-End speech recognition.  In _Proc. of ICASSP_.
  * Kahn et al. (2020b) Jacob Kahn, Morgane Rivière, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu, Pierre-Emmanuel Mazaré, Julien Karadayi, Vitaliy Liptchinsky, Ronan Collobert, Christian Fuegen, Tatiana Likhomanenko, Gabriel Synnaeve, Armand Joulin, Abdelrahman Mohamed, and Emmanuel Dupoux. 2020b.  Libri-Light: A Benchmark for ASR with Limited or No Supervision.  In _Proc. of ICASSP_.
  * Karita et al. (2018) Shigeki Karita, Shinji Watanabe, Tomoharu Iwata, Marc Delcroix, Atsunori Ogawa, and Tomohiro Nakatani. 2018.  Semi-Supervised End-to-End Speech Recognition.  In _Proc. of Interspeech_.
  * Kim et al. (2017) Suyoun Kim, Takaaki Hori, and Shinji Watanabe. 2017.  Joint CTC-attention based end-to-end speech recognition using multi-task learning.  In _Proc. of ICASSP_.
  * Li et al. (2019) Bo Li, Tara N. Sainath, Ruoming Pang, and Zelin Wu. 2019.  Semi-supervised training for End-to-End models via weak distillation.  In _Proc. of ICASSP_.
  * Li and Vu (2022) Chia-Yu Li and Thang Vu. 2022.  Improving Semi-supervised End-to-end Automatic Speech Recognition using CycleGAN and Inter-domain Losses.  In _Proc. of SLT_.
  * Ling et al. (2020) Shaoshi Ling, Yuzong Liu, Julian Salazar, and Katrin Kirchhoff. 2020.  Deep contextualized acoustic representations for semi-supervised speech recognition.  In _Proc. of ICASSP_.
  * Novotney and Schwartz (1998) Scott Novotney and Richard Schwartz. 1998.  Analysis of low-resource acoustic model self-training.  In _Proc. of BNTUW_.
  * Panayotov et al. (2015) Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015.  Librispeech: An ASR corpus based on public domain audio books.  In _Proc. of ICASSP_.
  * Park et al. (2019a) Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D. Cubuk, and Quoc V. Le. 2019a.  SpecAugment: A simple data augmentation method for automatic speech recognition.  In _Proc. of Interspeech_.
  * Park et al. (2019b) Daniel S. Park, Yu Zhang, Chung-Cheng Chiu, Youzheng Chen, Bo Li, William Chan, Quoc V. Le, and Yonghui Wu. 2019b.  SpecAugment on large scale datasets.  _arXiv:1912.05533_.
  * Park et al. (2020) Daniel S. Park, Yu Zhang, Ye Jia, Wei Han, Chung-Cheng Chiu, Bo Li, Yonghui Wu, and Quoc V. Le. 2020.  Improved Noisy Student Training for Automatic Speech Recognition.  In _Proc. of Interspeech_.
  * Parthasarathi and Strom (2019) Sree Hari Krishnan Parthasarathi and Nikko Strom. 2019.  Lessons from building acoustic models with a million hours of speech.  In _Proc. of ICASSP_.
  * Renduchintala et al. (2018) Adithya Renduchintala, Shuoyang Ding, Matthew Wiesner, and Shinji Watanabe. 2018\.  Multi-modal data augmentation for End-to-End ASR.  In _Proc. of Interspeech_.
  * Schneider et al. (2019) Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. 2019.  Wav2vec: Unsupervised pre-training for speech recognition.  In _Proc. of Interspeech_.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.  In _Proc. of ACL_.
  * Simonyan and Zisserman (2015) Karen Simonyan and Andrew Zisserman. 2015.  Very deep convolutional networks for large-scale image recognition.  In _Proc. of ICLR_.
  * Synnaeve et al. (2020) Gabriel Synnaeve, Qiantong Xu, Jacob Kahn, Edouard Grave, Tatiana Likhomanenko, Vineel Pratap, Anuroop Sriram, Vitaliy Liptchinsky, and Ronan Collobert. 2020\.  End-to-end ASR: from supervised to semi-supervised learning with modern architectures.  In _Proc. of ICML_.
  * Thomas et al. (2013) Samuel Thomas, Michael L. Seltzer, Kenneth Church, and Hynek Hermansky. 2013.  Deep neural network features and semi-supervised training for low resource speech recognition.  In _Proc. of ICASSP_.
  * Tjandra et al. (2017) Andros Tjandra, Sakriani Sakti, and Satoshi Nakamura. 2017.  Listening while speaking: Speech chain by deep learning.  In _Proc. of ASRU_.
  * (32) Voxforge.org.  Free speech recognition: voxforge.org.  http://www.voxforge.org/.  Accessed 06/25/2014.
  * Watanabe et al. (2018) Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, Adithya Renduchintala, and Tsubasa Ochiai. 2018.  ESPnet: End-to-end speech processing toolkit.  In _Proc. of Interspeech_.
  * Watanabe et al. (2017) Shinji Watanabe, Takaaki Hori, Suyoun Kim, John R. Hershey, and Tomoki Hayashi. 2017\.  Hybrid CTC/attention architecture for End-to-End speech recognition.  _IEEE Journal of Selected Topics in Signal Processing_.
  * Xie et al. (2020) Qizhe Xie, Eduard H. Hovy, Minh-Thang Luong, and Quoc V. Le. 2020.  Self-training with noisy student improves imagenet classification.  In _Proc. of CVPR_.
  * Zavaliagkos et al. (1998) George Zavaliagkos, Man-Hung Siu, Thomas Colthurst, and Jayadev Billa. 1998.  Using untranscribed training data to improve performance.  In _Proc. of the 5th International Conference on Spoken Language Processing (ICSLP)_.