gbsn

# Nonreciprocal single-photon band structure

Jiangshan Tang  College of Engineering and Applied Sciences, National
Laboratory of Solid State Microstructures, and Collaborative Innovation Center
of Advanced Microstructures, Nanjing University, Nanjing 210023, China  School
of Physics, Nanjing University, Nanjing 210023, China  Wei Nie  RIKEN Quantum
Computing Center, RIKEN Cluster for Pioneering Research, Wako-shi, Saitama
351-0198, Japan  Lei Tang  College of Engineering and Applied Sciences,
National Laboratory of Solid State Microstructures, and Collaborative
Innovation Center of Advanced Microstructures, Nanjing University, Nanjing
210023, China  Mingyuan Chen  College of Engineering and Applied Sciences,
National Laboratory of Solid State Microstructures, and Collaborative
Innovation Center of Advanced Microstructures, Nanjing University, Nanjing
210023, China  Xin Su  College of Engineering and Applied Sciences, National
Laboratory of Solid State Microstructures, and Collaborative Innovation Center
of Advanced Microstructures, Nanjing University, Nanjing 210023, China  School
of Electronic Science and Engineering, Nanjing University, Nanjing 210023,
China  Yanqing Lu  College of Engineering and Applied Sciences, National
Laboratory of Solid State Microstructures, and Collaborative Innovation Center
of Advanced Microstructures, Nanjing University, Nanjing 210023, China  School
of Physics, Nanjing University, Nanjing 210023, China  Franco Nori  RIKEN
Quantum Computing Center, RIKEN Cluster for Pioneering Research, Wako-shi,
Saitama 351-0198, Japan  Physics Department, The University of Michigan, Ann
Arbor, Michigan 48109-1040, USA  Keyu Xia<EMAIL_ADDRESS>College of
Engineering and Applied Sciences, National Laboratory of Solid State
Microstructures, and Collaborative Innovation Center of Advanced
Microstructures, Nanjing University, Nanjing 210023, China  School of Physics,
Nanjing University, Nanjing 210023, China  Jiangsu Key Laboratory of
Artificial Functional Materials, Nanjing University, Nanjing 210023, China

###### Abstract

We study single-photon band structure in a one-dimensional (1D) coupled-
resonator optical waveguide (CROW) which chirally couples to an array of two-
level quantum emitters (QEs). The chiral interaction between the resonator
mode and the QE can break the time-reversal symmetry without the magneto-
optical effect. As a result, a nonreciprocal single-photon edge state, band
gap and flat band appear. By using such a chiral QE-CROW system, including a
finite number of unit cells and working in the nonreciprocal band gap, we
achieve frequency-multiplex single-photon circulators with high fidelity and
low insertion loss. The chiral QE-light interaction can also protect one-way
propagation of single photons against backscattering. Our work opens a new
door for studying nonreciprocal photonic band structure and exploring its
applications in the quantum regime.

_Introduction_.—Optical nonreciprocity plays an indispensable role in many
important applications including quantum-information processing [1], invisible
sensing [2, 3], noise-free information processing [4] and unconventional
lasing [5, 6]. Conventional nonreciprocal optical devices breaking the time-
reversal symmetry rely on the weak magneto-optical effect and are difficult to
be integrated on a chip due to the constraint of a strong magnetic field.

Various theoretical schemes and experimental methods have been reported to
circumvent the constraint of magnetic fields by using optical nonlinearity [7,
8, 9, 10, 11, 12, 13, 14], optomechanical resonators [15, 16, 17], spinning
resonators [18, 19], tempo-spatial modulation of the medium [20, 21, 22] or
chiral quantum optics systems [23, 24, 25, 26, 27, 28, 29, 30, 31]. In
contrast to classical methods, a chiral quantum optics system can realize
optical nonreciprocity in the quantum regime [24, 25, 26, 10, 9, 11, 28] and
can perform chiral quantum information processing [32, 33, 34, 35].

It is of great interest to study photon behavior under the chiral interaction
between quantum emitters (QEs) and a coupled resonator array. The chiral
interaction of some QEs with few resonators has been extensively studied and
promises many important applications [36, 37]. Although the coupled-resonator
optical waveguide (CROW) has been widely investigated [38, 39, 40, 41, 42],
the optical chirality in the CROW is barely addressed. In analogy to the
electronic band structure in condensed matter, a periodic photonic structure
exhibits nontrivial band structures and allows defect-immune photon transport
[43, 44, 45, 46, 47]. Nevertheless, such topological optical system can break
time-reversal symmetry and is appealing for backscattering-immune optical
isolation only in the presence of an external/synthetic magnetic field. Thus,
it is highly desirable to achieve one-way photonic band structure such as
nonreciprocal single-photon edge states (SPESs) and band gap without magnetic
fields, in particular, in the _quantum_ regime.

By exploring the chiral interaction between an array of QEs and a 1D CROW,
here we show that _nonreciprocal single-photon band structure, including SPES,
band gap and flat band, can be achieved without a magnetic field._
Interestingly, we can conduct _frequency-multiplex single-photon circulators_
with such a chiral QE-CROW system working in the nonreciprocal band gap. Our
circulator is _immune to backscattering_.

Figure 1: (a) Schematic of a chiral QE-CROW system containing $N$ unit cells
and the input and output waveguides. The arrows represent the circulating
direction of the whispering-gallery mode for an input to port 1 or 3 (forward
input). The photon incident to ports 2 and 4 (backward input) excites the
oppositely propagating resonator modes. The QEs with $\sigma^{+}$-polarized
optical transition are periodically coupled to the clockwise resonator mode of
the sublattice A in each unit cell, see dashed box. (b) The L-type trimer
chain in the forward-input case. (c) The dimer chain for the backward-input
case.

_System and model_.—The schematic of the chiral QE-CROW system is depicted in
Fig. 1. In the CROW, the evanescent field of each microring resonator is
almost perfectly circularly polarized and thus possesses nearly unitary
optical chirality [24, 25, 31], i.e. possessing spin-momentum locking [48, 49,
50, 51]. In each unit cell, the adjacent resonators are separated by $\Lambda$
and coherently coupled via evanescent fields. Each resonator supports two
degenerate optical whispering-gallery modes circulating along either the
clockwise (CW) or counterclockwise (CCW) direction. We divide the resonators
into A and B sublattice groups, respectively denoted as $\text{R}_{\text{A}}$
and $\text{R}_{\text{B}}$. In each unit cell, a two-level QE with frequency
$\omega_{q}$ is only embedded in the A-sublattice resonator
$\text{R}_{\text{A}}$, but decouples from the B-sublattice resonator
$\text{R}_{\text{B}}$. Within a cell, the CW (CCW) mode of
$\text{R}_{\text{A}}$ couples to the CCW (CW) mode of $\text{R}_{\text{B}}$
with strength $J_{1}$, see the green box in Fig. 1(a). The $j$th resonator
$\text{R}_{\text{A}}$ couples to the $(j+1)$th resonator $\text{R}_{\text{B}}$
in the next cell with strength $J_{2}$. The oppositely circulating whispering-
gallery modes in the sublattices A and B form $N$ unit cells with a lattice
constant of $2\Lambda$ [39, 52]. We assume that the resonators, with resonance
frequency $\Omega$ and internal dissipation rate $\gamma_{\text{in}}$, are
identical.

Hereafter, we consider only a single excitation in the QE-CROW system. The QEs
can be precisely positioned atoms [25, 53, 40], quantum dots [37, 54, 55], or
nanopillars covered by monolayers [56, 57]. By optically initializing the QEs
in specific spin ground states or shifting the transition energy with a
circularly-polarized field via the optical Stark effect, we can induce a
chiral transition in the QE. Without loss of generality, we assume that the QE
transition and the evanescent field of the CW mode of $\text{R}_{\text{A}}$
are both $\sigma_{+}-$polarized. In this arrangement, we create a chiral
interaction between the QE and the CW mode of $\text{R}_{\text{A}}$ without
using a magnetic field [24, 23, 25, 26, 29, 31, 30]. However, the QE decouples
from the CCW mode [24, 25, 26, 31].

When a single photon is incident into port 1 or 3, namely the forward-input
case, the CW modes in the $\text{R}_{\text{A}}$ resonators and the CCW ones in
the $\text{R}_{\text{B}}$ resonators are driven. We refer to this excitation
as the $\text{CW}_{A}-\text{CCW}_{B}$ supermode ( opposite to the
$\text{CCW}_{A}-\text{CW}_{B}$ supermode). In this case, the QEs are coupled
to $\text{R}_{\text{A}}$ in each unit cell. The QE-CROW system is equivalent
to an L-type trimer chain with each cell containing three sublattices, see
Fig. 1(b). We take $\omega_{q}=\Omega$ and can write the Hamiltonian of the
system in the rotating frame as:

$\begin{split}\hat{H}_{\text{CW}_{A}-\text{CCW}_{B}}&=\sum_{j}^{N}\left(g\hat{a}^{{\dagger}}_{j,\circlearrowright}\hat{\sigma}_{j}+J_{1}\hat{a}^{{\dagger}}_{j,\circlearrowright}\hat{b}_{j,\circlearrowleft}+\text{H.c.}\right)\;\\\
&+\sum_{j}^{N-1}\left(J_{2}\hat{a}^{{\dagger}}_{j+1,\circlearrowright}\hat{b}_{j,\circlearrowleft}+\text{H.c.}\right)\;,\end{split}$
(1)

where $\hat{a}_{j,\circlearrowright}$ ($\hat{b}_{j,\circlearrowleft}$) is the
annihilation operator of the CW (CCW) mode of the $\text{R}_{\text{A}}$
($\text{R}_{\text{B}}$) resonator in the $j$th unit cell, and
$\hat{\sigma}_{j}$ denotes the lowering operator of the $j$th two-level QE.

Now we study the band structure of an infinite 1D-CROW model in a single-
excitation subspace. We assume a periodic boundary condition along the 1D-CROW
chain and apply the Fourier transform,
$\zeta_{j}=\left(1/\sqrt{N}\sum_{k}e^{ikj}\zeta_{k}\right)$ with
$\zeta_{j}=\left(\hat{a}_{j,\circlearrowright},\hat{\sigma}_{j},\hat{b}_{j,\circlearrowleft}\right)^{T}$,
to Eq. (1), thus,
$\hat{H}_{\text{CW}_{A}-\text{CCW}_{B}}=\sum_{k}\zeta_{k}^{{\dagger}}\hat{H}_{\text{CW}_{A}-\text{CCW}_{B}}(k)\zeta_{k}$.
Then, the Hamiltonian takes the following form in wavevector space

$\hat{H}_{\text{CW}_{A}-\text{CCW}_{B}}\left(k\right)=\left(\begin{array}[]{ccc}0&g&J_{1}+J_{2}e^{-ik}\\\
g&0&0\\\ J_{1}+J_{2}e^{ik}&0&0\end{array}\right)\;.$ (2)

In comparison, if a single photon incidents to port 2 or 4 corresponding to
the backward-input case, then the $\text{CCW}_{A}-\text{CW}_{B}$ supermode is
driven in each unit cell and the QEs decouple from all resonators. Our system
reduces to a dimer chain, corresponding to a standard Su-Schrieffer-Heeger
(SSH) model [58, 59], see Fig. 1(c). The Hamiltonian then becomes

$\hat{H}_{\text{CCW}_{A}-\text{CW}_{B}}\left(k\right)=\left(\begin{array}[]{cc}0&J_{1}+J_{2}e^{-ik}\\\
J_{1}+J_{2}e^{ik}&0\end{array}\right)\;.$ (3)

The band structures and energy spectra of the system in the
$\text{CW}_{A}-\text{CCW}_{B}$ and $\text{CCW}_{A}-\text{CW}_{B}$ supermodes
can be found by solving the eigenvalues and eigenstates of Eq. (2) and Eq.
(3), respectively. In the case related to the $\text{CW}_{A}-\text{CCW}_{B}$
supermode, the QEs interact with the $\text{R}_{\text{A}}$ resonators. This
interaction causes a phase and amplitude modulation $e^{i\varphi}$ to the
$\text{R}_{\text{A}}$ CW mode [60, 61]. Here, $\varphi$ is a complex number.
In contrast, the QE-resonator interaction is absent for the
$\text{CCW}_{A}-\text{CW}_{B}$ supermode. Thus, the extra modulation
disappears, i.e. $\varphi=0$. The dispersion relations of the
$\text{CW}_{A}-\text{CCW}_{B}$ supermode consists of three dispersive bands.
It is essentially different from the $\text{CCW}_{A}-\text{CW}_{B}$ supermode,
which only contains two bands. Therefore, the single-photon band structures
are nonreciprocal in two counter-propagating cases. This nonreciprocal single-
photon dispersion occurs because the chiral QE-light coupling breaks the time-
reversal symmetry of the system.

A finite 1D-CROW chain with a large number of unit cells can approximately
exhibit the property of an infinite chain [39]. Next, we discuss a finite
chiral 1D CROW containing $N$ unit cells ($N\geq 1$) and evaluate the single-
photon transmission. Two optical waveguides are side-coupled to the terminal
resonators as input and output channels. We utilize the transfer matrix method
to investigate the transmission properties of the system [41, 60]. We consider
the forward case. The notations for the field components $\\{a\\}$ and
$\\{b\\}$ are shown in Fig. 1(a). By successively multiplying the transfer
matrices of all the coupling relations, we obtain the transport relation

$\left(\begin{array}[]{l}a_{N+1}\\\
b_{N+1}\end{array}\right)=M\left(\begin{array}[]{l}a_{0}\\\
b_{0}\end{array}\right)\equiv\left(\begin{array}[]{cc}M_{11}&M_{12}\\\
M_{21}&M_{22}\end{array}\right)\left(\begin{array}[]{c}a_{0}\\\
b_{0}\end{array}\right)\;,$ (4)

$M=M_{\text{out}}M_{\text{p,B}}M_{\text{c,1}}M_{\text{p,A}}(M_{\text{c,2}}M_{\text{p,B}}M_{\text{c,1}}M_{\text{p,A}})^{N-1}M_{\text{in}}$,
where

$\displaystyle M_{\text{p,A}}=\left(\begin{array}[]{cc}\alpha
e^{-i\theta}&0\\\ 0&\alpha e^{i(\theta+\varphi)}\end{array}\right),\quad
M_{\text{p,B}}=\left(\begin{array}[]{cc}\alpha e^{-i\theta}&0\\\ 0&\alpha
e^{i\theta}\end{array}\right),$ (5) $\displaystyle
M_{\text{c,1}}=\frac{1}{\kappa_{1}}\left(\begin{array}[]{cc}1&-t_{1}\\\
t_{1}^{*}&-1\end{array}\right),\quad M_{\text{c,
2}}=\frac{1}{\kappa_{2}}\left(\begin{array}[]{cc}1&-t_{2}\\\
t_{2}^{*}&-1\end{array}\right),$ $\displaystyle
M_{\text{in}}=\frac{1}{\kappa_{\text{in}}}\left(\begin{array}[]{cc}-t_{\text{in}}&1\\\
-1&t_{\text{in}}^{*}\end{array}\right),\quad
M_{\text{out}}=\frac{1}{\kappa_{\text{out}}}\left(\begin{array}[]{cc}1&-t_{\text{out}}\\\
t_{\text{out }}^{*}&-1\end{array}\right).$

We define $\\{t_{\text{in}},t_{1},t_{2},t_{\text{out}}\\}$ and
$\\{\kappa_{\text{in}},\kappa_{1},\kappa_{2},\kappa_{\text{out}}\\}$ as the
transmission and coupling coefficients of the intrasite and intersite
resonators, where $t_{i}~{}(i=\text{in},1,2,\text{out})$ is a real number,
$\kappa_{i}$ is an imaginary number, and $|t_{i}|^{2}+|\kappa_{i}|^{2}=1$
[41]. Above, $\theta$ indicates the phase change and $\alpha$ is the loss
coefficient of propagating single photons. This can be calculated as
$\alpha\approx 1-2\gamma_{\text{in}}/\mathcal{F}$ from the intrinsic loss rate
$\gamma_{\text{in}}$ of the resonator [61], where $\mathcal{F}$ is the free
spectral range of the resonator considered here. The ratio
$\kappa_{\text{in}}/\mathcal{F}$ is negligible for our high-quality
resonators. Thus, hereafter we can take $\alpha\approx 1$.

For simplicity, we assume $\kappa_{\text{in}}=\kappa_{\text{out}}$
($t_{\text{in}}=t_{\text{out}}$). The external losses of the edge resonators
$A_{1}$ and $B_{\text{N}}$ are
$\gamma_{\text{ex}}=\gamma_{\text{ex,1}}=\gamma_{\text{ex,N}}=-\text{ln}\left(|t_{\text{in}}|\right)\times\mathcal{F}$.
Considering a single-photon entering port 1 ($a_{N+1}=0$), we obtain the
transmission matrix elements between the input and output ports:

$\displaystyle T_{12}$
$\displaystyle=\left|\frac{b_{0}}{a_{0}}\right|^{2}=\left|\frac{M_{11}}{M_{12}}\right|^{2}\;,$
(6a) $\displaystyle T_{14}$
$\displaystyle=\left|\frac{b_{N+1}}{a_{0}}\right|^{2}=\left|M_{21}-\frac{M_{11}M_{22}}{M_{12}}\right|^{2}\;,$
(6b)

where $T_{mn}$ is the transmission from port $m$ to port $n$, with
$m,n=1,2,3,4$. Because the input to port 3 excites the
$\text{CW}_{A}-\text{CCW}_{B}$ supermode circulating along the same direction,
we have $T_{12}=T_{34}$ and $T_{14}=T_{32}$. When the
$\text{CCW}_{A}-\text{CW}_{B}$ supermode is excited in the backward case, the
QEs decouple from the resonators and we have $\varphi=0$. Therefore, we have
$T_{mn}\neq T_{nm}$ for $m\neq n$, indicating the occurrence of optical
nonreciprocity in our chiral QE-CROW system.

Below, we consider an ideal scenario in which the QE dissipation rate $\gamma$
is negligible compared to the coupling strength $g$. For simplicity, we set
$\gamma=0$.

Figure 2: Single-photon band structures of the chiral QE-CROW system in the
case of $J_{1}<J_{2}$. (a) Band structure when the
$\text{CCW}_{A}-\text{CW}_{B}$ supermode is excited in the absence of the QEs,
i.e., $g/\Omega=0$. (b) Band structure when the $\text{CW}_{A}-\text{CCW}_{B}$
supermode, coupling to the QEs with $g/\Omega=10^{-4}$, is driven. (c) Band
structure versus the coupling strength $g$ when the
$\text{CW}_{A}-\text{CCW}_{B}$ supermode is driven. Other parameters are
$J_{1}/\Omega=3\times 10^{-4}$, $J_{2}=2J_{1}$.
Figure 3: Nonreciprocal energy spectra (a, b) and probability distribution
(c-e). (a) and (c) for the $\text{CCW}_{A}-\text{CW}_{B}$ supermode without
coupling to the QEs, $g=0$. (b), (d), and (e) for the
$\text{CW}_{A}-\text{CCW}_{B}$ supermode with coupling to the QEs,
$g/\Omega=10^{-4}$. Other parameters are $J_{1}/\Omega=3\times 10^{-4}$,
$J_{2}=2J_{1}$.

_Nonreciprocal single-photon flat band and edge states_.—We first discuss the
case $J_{1}<J_{2}$. As seen from the dispersion relations in Fig. 2, the QE-
CROW system exhibits a nonreciprocal single-photon band structure. Typically,
two dispersive bands of the SSH model appear in the backward case, as shown in
Fig. 2(a) [62] for $J_{1}/\Omega=3\times 10^{-4}$ and $J_{2}=2J_{1}$. The
curves with opposite slopes imply two opposite directions of single-photon
propagation, corresponding to the input to port 2 or 4 in Fig. 1,
respectively. In the forward case, the QEs interact with the
$\text{R}_{\text{A}}$ resonator. A flat band appears at $\omega=\Omega$ due to
the coupling to the QEs, dividing the original band gap into two parts, see
Fig. 2(b). Thus, our QE-CROW system allows _slow light and delay of single-
photon pulses_ [63, 64, 65]. As the QE-resonator coupling strength increases,
the two band gaps become wider, see Fig. 2(c). Interestingly, these two band
gaps possess very peculiar SPESs. These SPESs are very different from the
backward case.

Figure 3 shows energy spectra of a finite unit cell with $N=20$ and the
probability distribution corresponding to the SPESs in the two oppositely
circulating supermodes of the system. As for the
$\text{CCW}_{A}-\text{CW}_{B}$ supermode without coupling to the QEs, there
are $2N$ eigenvalues and two degenerate zero-energy SPESs falling in the bulk
band gap [see the purple asterisks in Fig. 3(a)]. The wave functions of two
edge states localize at the left or right boundary and exponentially decay
from the boundaries [see Fig. 3(c)].

In stark contrast, when the system is excited in the
$\text{CW}_{A}-\text{CCW}_{B}$ supermode, the coupling to the QEs results in
$N$ more eigenvalues. Due to the coupling with the QEs, the left SPES becomes
doublet, forming two superstates of the $\text{R}_{A}$ set and the QEs with an
energy splitting proportional to the coupling strength $g$ [66, 67, 60], see
the dashed black curves in Fig. 2(c). As an example, the energy spectrum of
the QE-CROW system with $N=20$ and $g/\Omega=10^{-4}$ is shown in Fig. 3(c).
The left SPESs have eigenenergies at $\omega-\Omega=\pm g$ (see the blue
asterisks). In Fig. 3(d), the probability of the A sublattice includes the
contribution of the QE. Unlike the SPESs of the $\text{CCW}_{A}-\text{CW}_{B}$
supermode, the wave functions of these SPESs only localize on the left edge of
the system with an exponentially decaying probability distribution at the
sites of both the A-sublattice resonator and the QE. The right zero-energy
SPES still exists, [see the red asterisk in Fig. 3(b)], because the
$\text{R}_{\text{B}}$ resonators decouple from the QE. The wave function of
this right SPES localizes at the right boundary [see Fig. 3(e)]. The
nonreciprocal SPESs is the unique property of our chiral QE-CROW system.

For small $N$, the left and right SPESs overlap in the backward-input case
because of their exponentially decaying distribution. This wavefunction
overlap at zero-energy frequency results in a SPES tunneling. In contrast, in
the forward case, this SPES-overlap induced tunneling is greatly suppressed
because the left and right SPESs are nondegenerate in energy. Therefore, we
can realize a single-photon circulator via the nonreciprocal SPESs with few
unit cells [60].

Figure 4: Single-photon band structure (a, b) and transmission (c, d) with
$N=10$ for $J_{1}=J_{2}$. In (a, c) the $\text{CCW}_{A}-\text{CW}_{B}$
supermode is excited, and $g=0$. In (b, d) the $\text{CW}_{A}-\text{CCW}_{B}$
supermode is driven, and $g/\Omega=3\times 10^{-4}$. Other parameters are
$J_{1}/\Omega=3\times 10^{-4}$, $J_{2}=J_{1}$.

_Nonreciprocal single-photon band gap_.—The case $J_{1}>J_{2}$ opens a band
gap but exhibits trivial physical properties. We are not interested in it
here. Now we discuss the case $J_{1}=J_{2}$. Figures 4(a,b) show a
nonreciprocal single-photon band gap for oppositely propagating photons. For
the backward input, the two bands are closed and thus form continuous
conduction bands. In comparison, as in the case of $J_{1}<J_{2}$, the
$\text{CW}_{A}-\text{CCW}_{B}$ supermode also exhibits two band gaps,
separated by a flat band due to the interaction with QEs, see the yellow
region in Fig. 4(b). The band gap increases with the QE-resonator coupling
strength, permitting _a wider nonreciprocal bandwidth_. This nonreciprocal
single-photon band gap allows to conduct _single-photon circulators_. The same
results can be derived by using the transfer matrix method [60].

The band structures of our system characterize the transmission of single
photons. When we consider the case with finite unit cells in the 1D CROW, the
conduction band region turns into transmission peaks. Using Eqs. (6), we
calculate the transmission spectrum of our QE-CROW system with $N=10$ in the
two opposite-propagation cases. For single photons input to port 2, see Fig.
4(c), the excited $\text{CCW}_{\text{A}}-\text{CW}_{\text{B}}$ supermode
decouples from the QEs. This system is equivalent to the conventional CROW
without QEs. There are $2N$ peaks in the transmission $T_{23}$ and dips in
$T_{21}$. In contrast, when the single-photon wave packet inputs to port 1,
the transmission $T_{14}$ is zero, because the photon transport is forbidden
in the band gap, and thus $T_{12}=1$, see Fig. 4(d). Clearly, we obtain
_strong nonreciprocity in the transmission_ at the frequencies corresponding
to the peaks of $T_{23}$, where $T_{23}\approx 1$, or the dips of $T_{21}$,
where $T_{21}\approx 0$. For $g/\Omega=3\times 10^{-4}$, we have eight
nonreciprocal frequency windows at
$\left(\omega-\Omega\right)/\Omega\approx\\{\pm 30,\pm 22,\pm 13,\pm
4.5\\}\times 10^{-5}$, see blue circles. Due to the symmetry of the CROW, we
obtain $T_{12}=T_{34}$ and $T_{23}=T_{41}$. Thus, we can choose any one of
eight frequencies in the band-gap region to achieve $T_{23}=1$ ($T_{21}=0$).
In this case, we can attain _a frequency-multiplex single-photon circulator_
with a circling photon transport direction $1\rightarrow 2\rightarrow
3\rightarrow 4\rightarrow 1$ at frequencies marked by circles in Figs. 4(d).

Note that this single-photon circulator is robust against backscattering (see
the Supplemental Material [60]). The chiral coupling between the QE and the
vacuum field of $\text{R}_{\text{A}}$ eigenmodes induces an extra phase shift
$\text{Re}[\varphi]$ for the CW mode, shifting its resonance frequency and
leading to non-degenerate with the CCW mode. Thus, the opposite CW mode
excited by a backscattering of a propagating CCW mode is forbidden in the
nonreciprocal band gap, see Figs. 4 (a,b). Our chiral QE-CROW system promises
a new type of backscattering-immune optical device.

_Implementation._ —The required 1D CROW can be made with silicon oxynitride
[68], silicon on insulator [69, 70] or lithium niobium oxynitride [71]. We
consider a QE-CROW system consisting of $N=10$ resonators with a radius of
$r=40~{}\mathrm{\SIUnitSymbolMicro}\mathrm{m}$ and $n_{\text{eff}}=2$,
yielding $\mathcal{F}/2\pi=0.6~{}\mathrm{T}\mathrm{Hz}$. The external
dissipations of resonators can be calculated as
$\gamma_{\text{ex}}/2\pi=19.4~{}\mathrm{G}\mathrm{Hz}$ [setting
$\gamma_{\text{in}}=0.02\gamma_{\text{ex}}$, thus
$\gamma_{\text{tol}}/2\pi=(\gamma_{\text{ex}}+\gamma_{\text{in}})/2\pi=19.8~{}\mathrm{G}\mathrm{Hz}$]
[61]. For the coupling coefficient $\kappa_{1}=\kappa_{2}=0.1i$, the coupling
strength between resonators can be deduced as
$J_{1}/\gamma_{\text{tol}}=J_{2}/\gamma_{\text{tol}}=3$. We take
$\Omega/2\pi=195~{}\mathrm{T}\mathrm{Hz}$. Regarding QEs, we choose
$\lambda=1.538~{}\mathrm{\SIUnitSymbolMicro}\mathrm{m}$ [72, 73, 74] so that
$\omega_{q}=\Omega$ and a dipole moment of $|\textbf{d}|=20~{}\text{Debye}$
[75], yielding $\gamma=2\pi\times 5.48~{}\mathrm{M}\mathrm{Hz}$, and
$g/\gamma_{\text{tol}}=0.77$ for a resonator mode volume of
$V_{\text{m}}=2.0~{}\mathrm{\SIUnitSymbolMicro}\mathrm{m}^{3}$ [76, 31]. The
chiral QE-light interaction can be optically induced as aforementioned. The
fidelity and the average photon survival probability of the circulator are
calculated as $\\{0.99,0.98\\}$ at two optimal frequency windows
$(\omega-\Omega)/\Omega\approx\pm 4.5\times 10^{-5}$ [25, 10]. Moreover, we
achieve a nonreciprocal transmission bandwidth of $\sim 2\pi\times
2.4~{}\mathrm{G}\mathrm{Hz}$ and an average insertion loss of
$~{}1.12~{}\mathrm{d}\mathrm{B}$ for a circulator fidelity larger than $0.95$.
Thus, the nonreciprocal SPES and high-performance single-photon circulator can
be achieved even in the weak coupling regime $g<\gamma_{\text{tol}}$. In stark
contrast, a conventional chiral quantum optics system using a single resonator
can achieve the single-photon diode and circulator only in the strong coupling
regime [24, 25, 26, 31]. Thus, our proposal greatly relaxes its experimental
challenge.

_Conclusion._ —We have shown that nonreciprocal single-photon band structures
can be achieved in a chiral QE-CROW system. The chiral QE-CROW system with a
nonreciprocal band gap can construct frequency-multiplex single-photon
circulators. Our work extends photonic band structures of periodic photonic
structures to exhibit remarkable magnetic-free quantum nonreciprocity, likely
beyond condensed matter.

K.X. and Y.L thank Prof. Haiqing Lin for his stimulating and constructive
discussion. J.T. thanks Chaohua Wu for helpful discussions. Y.L. and K.X. are
supported by the National Key R&D Program of China (Grants No. 2019YFA0308700,
No. 2017YFA0303703, and No. 2017YFA0303701), the National Natural Science
Foundation of China (Grants No. 11874212 and No. 11890704), the Fundamental
Research Funds for the Central Universities (Grant No. 021314380095), the
Program for Innovative Talents and Entrepreneurs in Jiangsu, and the Excellent
Research Program of Nanjing University (Grant No. ZYJH002). W.N is supported
by the National Natural Science Foundation of China (Grants No. 11690031).
F.N. is supported by the Nippon Telegraph and Telephone Corporation (NTT)
Research, the Japan Science and Technology Agency (JST) [via the Quantum Leap
Flagship Program (Q-LEAP), the Moonshot R&D Grant Number JPMJMS2061, and the
Centers of Research Excellence in Science and Technology (CREST) Grant No.
JPMJCR1676], the Japan Society for the Promotion of Science (JSPS) [via the
Grants-in-Aid for Scientific Research (KAKENHI) Grant No. JP20H00134 and the
JSPS-RFBR Grant No. JPJSBP120194828], the Army Research Office (ARO) (Grant
No. W911NF-18-1-0358), the Asian Office of Aerospace Research and Development
(AOARD) (via Grant No. FA2386-20-1-4069), and the Foundational Questions
Institute Fund (FQXi) via Grant No. FQXi-IAF19-06. We thank the High
Performance Computing Center of Nanjing University for allowing the numerical
calculations on its blade cluster system.

## References

  * Kimble [2008] H. J. Kimble, The quantum internet, Nature (London) 453, 1023 (2008).
  * Lin  _et al._ [2011] Z. Lin, H. Ramezani, T. Eichelkraut, T. Kottos, H. Cao, and D. N. Christodoulides, Unidirectional invisibility induced by $\mathcal{P}\mathcal{T}$-symmetric periodic structures, Phys. Rev. Lett. 106, 213901 (2011).
  * Özdemir  _et al._ [2019] Ş. K. Özdemir, S. Rotter, F. Nori, and L. Yang, Parity-time symmetry and exceptional points in photonics, Nat. Mater. 18, 783 (2019).
  * Jalas  _et al._ [2013] D. Jalas, A. Petrov, M. Eich, W. Freude, S. Fan, Z. Yu, R. Baets, M. Popović, A. Melloni, J. D. Joannopoulos, M. Vanwolleghem, C. R. Doerr, and H. Renner, What is—and what is not—an optical isolator, Nat. Photonics 7, 579 (2013).
  * Bahari  _et al._ [2017] B. Bahari, A. Ndao, F. Vallini, A. El Amili, Y. Fainman, and B. Kanté, Nonreciprocal lasing in topological cavities of arbitrary geometries, Science 358, 636 (2017).
  * Jiang  _et al._ [2018] Y. Jiang, S. Maayani, T. Carmon, F. Nori, and H. Jing, Nonreciprocal phonon laser, Phys. Rev. Appl. 10, 064037 (2018).
  * Peng  _et al._ [2014] B. Peng, Ş. K. Özdemir, F. Lei, F. Monifi, M. Gianfreda, G. L. Long, S. Fan, F. Nori, C. M. Bender, and L. Yang, Parity-time-symmetric whispering-gallery microcavities, Nat. Phys. 10, 394 (2014).
  * Cao  _et al._ [2017] Q.-T. Cao, H. Wang, C.-H. Dong, H. Jing, R.-S. Liu, X. Chen, L. Ge, Q. Gong, and Y.-F. Xiao, Experimental demonstration of spontaneous chirality in a nonlinear microresonator, Phys. Rev. Lett. 118, 033901 (2017).
  * Zhang  _et al._ [2018] S. Zhang, Y. Hu, G. Lin, Y. Niu, K. Xia, J. Gong, and S. Gong, Thermal-motion-induced non-reciprocal quantum optical system, Nat. Photonics 12, 744 (2018).
  * Xia  _et al._ [2018] K. Xia, F. Nori, and M. Xiao, Cavity-free optical isolators and circulators using a chiral cross-Kerr nonlinearity, Phys. Rev. Lett. 121, 203602 (2018).
  * Yang  _et al._ [2019] P. Yang, X. Xia, H. He, S. Li, X. Han, P. Zhang, G. Li, P. Zhang, J. Xu, Y. Yang, and T. Zhang, Realization of nonlinear optical nonreciprocity on a few-photon level based on atoms strongly coupled to an asymmetric cavity, Phys. Rev. Lett. 123, 233604 (2019).
  * Li  _et al._ [2020] E.-Z. Li, D.-S. Ding, Y.-C. Yu, M.-X. Dong, L. Zeng, W.-H. Zhang, Y.-H. Ye, H.-Z. Wu, Z.-H. Zhu, W. Gao, G.-C. Guo, and B.-S. Shi, Experimental demonstration of cavity-free optical isolators and optical circulators, Phys. Rev. Res. 2, 033517 (2020).
  * Tang  _et al._ [2021a] L. Tang, J. Tang, H. Wu, J. Zhang, M. Xiao, and K. Xia, Broad-intensity-range optical nonreciprocity based on feedback-induced Kerr nonlinearity, Photonics Res. 9, 1218 (2021a).
  * Tang  _et al._ [2021b] L. Tang, J. Tang, M. Chen, F. Nori, M. Xiao, and K. Xia, Quantum squeezing induced optical nonreciprocity, arXiv:2110.05016 [quant-ph] (2021b).
  * Hafezi and Rabl [2012] M. Hafezi and P. Rabl, Optomechanically induced non-reciprocity in microring resonators, Opt. Express 20, 7672 (2012).
  * Shen  _et al._ [2016] Z. Shen, Y.-L. Zhang, Y. Chen, C.-L. Zou, Y.-F. Xiao, X.-B. Zou, F.-W. Sun, G.-C. Guo, and C.-H. Dong, Experimental realization of optomechanically induced non-reciprocity, Nat. Photonics 10, 657 (2016).
  * Ruesink  _et al._ [2016] F. Ruesink, M.-A. Miri, A. Alù, and E. Verhagen, Nonreciprocity and magnetic-free isolation based on optomechanical interactions, Nat. Commun. 7, 13662 (2016).
  * Maayani  _et al._ [2018] S. Maayani, R. Dahan, Y. Kligerman, E. Moses, A. U. Hassan, H. Jing, F. Nori, D. N. Christodoulides, and T. Carmon, Flying couplers above spinning resonators generate irreversible refraction, Nature (London) 558, 569 (2018).
  * Huang  _et al._ [2018] R. Huang, A. Miranowicz, J.-Q. Liao, F. Nori, and H. Jing, Nonreciprocal photon blockade, Phys. Rev. Lett. 121, 153601 (2018).
  * Wang  _et al._ [2013] D.-W. Wang, H.-T. Zhou, M.-J. Guo, J.-X. Zhang, J. Evers, and S.-Y. Zhu, Optical diode made from a moving photonic crystal, Phys. Rev. Lett. 110, 093901 (2013).
  * Horsley  _et al._ [2013] S. A. R. Horsley, J.-H. Wu, M. Artoni, and G. C. La Rocca, Optical nonreciprocity of cold atom Bragg mirrors in motion, Phys. Rev. Lett. 110, 223602 (2013).
  * Sounas and Alù [2017] D. L. Sounas and A. Alù, Non-reciprocal photonics based on time modulation, Nat. Photonics 11, 774 (2017).
  * Sayrin  _et al._ [2015] C. Sayrin, C. Junge, R. Mitsch, B. Albrecht, D. O’Shea, P. Schneeweiss, J. Volz, and A. Rauschenbeutel, Nanophotonic optical isolator controlled by the internal state of cold atoms, Phys. Rev. X 5, 041036 (2015).
  * Xia  _et al._ [2014] K. Xia, G. Lu, G. Lin, Y. Cheng, Y. Niu, S. Gong, and J. Twamley, Reversible nonmagnetic single-photon isolation using unbalanced quantum coupling, Phys. Rev. A 90, 043802 (2014).
  * Scheucher  _et al._ [2016] M. Scheucher, A. Hilico, E. Will, J. Volz, and A. Rauschenbeutel, Quantum optical circulator controlled by a single chirally coupled atom, Science 354, 1577 (2016).
  * Lodahl  _et al._ [2017] P. Lodahl, S. Mahmoodian, S. Stobbe, A. Rauschenbeutel, P. Schneeweiss, J. Volz, H. Pichler, and P. Zoller, Chiral quantum optics, Nature (London) 541, 473 (2017).
  * Mehranad  _et al._ [2020] M. J. Mehranad, A. P. Foster, R. Dost, E. Clarke, P. K. Patil, A. M. Fox, M. S. Skolnick, and L. R. Wilson, Chiral topological photonics with an embedded quantum emitter, Optica 7, 1690 (2020).
  * Dong  _et al._ [2021] M.-X. Dong, K. Xia, W.-H. Zhang, Y.-C. Yu, Y.-H. Ye, E.-Z. Li, L. Zeng, D.-S. Ding, B.-S. Shi, G.-C. Guo, and F. Nori, All-optical reversible single-photon isolation at room temperature, Sci. Adv. 7, eabe8924 (2021).
  * Hu  _et al._ [2021] X.-X. Hu, Z.-B. Wang, P. Zhang, G.-J. Chen, Y.-L. Zhang, G. Li, X.-B. Zou, T. Zhang, H. X. Tang, C.-H. Dong, G.-C. Guo, and C.-L. Zou, Noiseless photonic non-reciprocity via optically-induced magnetization, Nat. Commun. 12, 2389 (2021).
  * Pucher  _et al._ [2021] S. Pucher, C. Liedl, S. Jin, A. Rauschenbeutel, and P. Schneeweiss, Atomic spin-controlled non-reciprocal raman amplification of fibre-guided light, arXiv preprint arXiv:2107.07272 (2021).
  * Tang  _et al._ [2019] L. Tang, J. Tang, W. Zhang, G. Lu, H. Zhang, Y. Zhang, K. Xia, and M. Xiao, On-chip chiral single-photon interface: Isolation and unidirectional emission, Phys. Rev. A 99, 043833 (2019).
  * Ramos  _et al._ [2014] T. Ramos, H. Pichler, A. J. Daley, and P. Zoller, Quantum spin dimers from chiral dissipation in cold-atom chains, Phys Rev Lett 113, 237203 (2014).
  * Söllner  _et al._ [2015] I. Söllner, S. Mahmoodian, S. L. Hansen, L. Midolo, A. Javadi, G. Kiršanskė, T. Pregnolato, H. El-Ella, E. H. Lee, J. D. Song, S. Stobbe, and P. Lodahl, Deterministic photon-emitter coupling in chiral photonic circuits, Nat. Nanotechnol. 10, 775 (2015).
  * Li  _et al._ [2021a] T. Li, Z. Gao, and K. Xia, Nonlinear-dissipation-induced nonreciprocal exceptional points, Opt. Express 29, 17613 (2021a).
  * Tang  _et al._ [2021c] J. Tang, L. Tang, H. Wu, Y. Wu, H. Sun, H. Zhang, T. Li, Y. Lu, M. Xiao, and K. Xia, Towards on-demand heralded single-photon sources via photon blockade, Phys. Rev. Appl. 15, 064020 (2021c).
  * Roy  _et al._ [2017] D. Roy, C. M. Wilson, and O. Firstenberg, Colloquium: Strongly interacting photons in one-dimensional continuum, Rev. Mod. Phys. 89, 021001 (2017).
  * Chang  _et al._ [2018] D. E. Chang, J. S. Douglas, A. González-Tudela, C.-L. Hung, and H. J. Kimble, Colloquium: Quantum matter built from nanoscopic lattices of atoms and photons, Rev. Mod. Phys. 90, 031002 (2018).
  * Yariv  _et al._ [1999] A. Yariv, Y. Xu, R. K. Lee, and A. Scherer, Coupled-resonator optical waveguide: a proposal and analysis, Opt. Lett. 24, 711 (1999).
  * Poon  _et al._ [2004a] J. K. S. Poon, J. Scheuer, S. Mookherjea, G. T. Paloczi, Y. Huang, and A. Yariv, Matrix analysis of microring coupled-resonator optical waveguides, Opt. Express 12, 90 (2004a).
  * Perczel  _et al._ [2020] J. Perczel, J. Borregaard, D. E. Chang, S. F. Yelin, and M. D. Lukin, Topological quantum optics using atomlike emitter arrays coupled to photonic crystals, Phys. Rev. Lett. 124, 083603 (2020).
  * Ao  _et al._ [2020] Y. Ao, X. Hu, Y. You, C. Lu, Y. Fu, X. Wang, and Q. Gong, Topological phase transition in the non-Hermitian coupled resonator array, Phys. Rev. Lett. 125, 013902 (2020).
  * Li  _et al._ [2021b] J. Li, B. Gao, C. Zhu, J. Xu, and Y. Yang, Nonreciprocal photonic composited Su-Schrieffer-Heeger chain, Appl. Phys. Lett. 119, 141108 (2021b).
  * Fang  _et al._ [2011] K. Fang, Z. Yu, and S. Fan, Microscopic theory of photonic one-way edge mode, Phys. Rev. B 84, 075477 (2011).
  * Lu  _et al._ [2014] L. Lu, J. D. Joannopoulos, and M. Soljačió, Topological photonics, Nat. Photonics 8, 821 (2014).
  * Khanikaev and Shvets [2017] A. B. Khanikaev and G. Shvets, Two-dimensional topological photonics, Nat. Photonics 11, 763 (2017).
  * Ozawa  _et al._ [2019] T. Ozawa, H. M. Price, A. Amo, N. Goldman, M. Hafezi, L. Lu, M. C. Rechtsman, D. Schuster, J. Simon, O. Zilberberg, and I. Carusotto, Topological photonics, Rev. Mod. Phys. 91, 015006 (2019).
  * Xi  _et al._ [2021] X. Xi, J. Ma, S. Wan, C.-H. Dong, and X. Sun, Observation of chiral edge states in gapped nanomechanical graphene, Sci. Adv. 7, eabe1398 (2021).
  * Bliokh and Nori [2011] K. Y. Bliokh and F. Nori, Characterizing optical chirality, Phys. Rev. A 83, 021803 (2011).
  * Bliokh  _et al._ [2014] K. Y. Bliokh, A. Y. Bekshaev, and F. Nori, Extraordinary momentum and spin in evanescent waves, Nat. Commun. 5, 3300 (2014).
  * Bliokh  _et al._ [2015] K. Y. Bliokh, D. Smirnova, and F. Nori, Quantum spin Hall effect of light, Science 348, 1448 (2015).
  * Alpeggiani  _et al._ [2018] F. Alpeggiani, K. Y. Bliokh, F. Nori, and L. Kuipers, Electromagnetic helicity in complex media, Phys. Rev. Lett. 120, 243605 (2018).
  * Poon  _et al._ [2004b] J. K. S. Poon, J. Scheuer, Y. Xu, and A. Yariv, Designing coupled-resonator optical waveguide delay lines, J. Opt. Soc. Am. B 21, 1665 (2004b).
  * Thompson  _et al._ [2013] J. D. Thompson, T. G. Tiecke, N. P. de Leon, J. Feist, A. V. Akimov, M. Gullans, A. S. Zibrov, V. Vuletić, and M. D. Lukin, Coupling a single trapped atom to a nanoscale optical cavity, Science 340, 1202 (2013).
  * Barik  _et al._ [2018] S. Barik, A. Karasahin, C. Flower, T. Cai, H. Miyake, W. DeGottardi, M. Hafezi, and E. Waks, A topological quantum optics interface, Science 359, 666 (2018).
  * Yang  _et al._ [2020] J. Yang, C. Qian, X. Xie, K. Peng, S. Wu, F. Song, S. Sun, J. Dang, Y. Yu, S. Shi, J. He, M. J. Steer, I. G. Thayne, B.-B. Li, F. Bo, Y.-F. Xiao, Z. Zuo, K. Jin, C. Gu, and X. Xu, Diabolical points in coupled active cavities with quantum emitters, Light-Sci. Appl. 9, 6 (2020).
  * Branny  _et al._ [2017] A. Branny, S. Kumar, R. Proux, and B. D. Gerardot, Deterministic strain-induced arrays of quantum emitters in a two-dimensional semiconductor, Nat. Commun. 8, 15053 (2017).
  * Palacios-Berraquero  _et al._ [2017] C. Palacios-Berraquero, D. M. Kara, A. R. P. Montblanch, M. Barbone, P. Latawiec, D. Yoon, A. K. Ott, M. Loncar, A. C. Ferrari, and M. Atatüre, Large-scale quantum-emitter arrays in atomically thin semiconductors, Nat. Commun. 8, 15093 (2017).
  * Su  _et al._ [1979] W. P. Su, J. R. Schrieffer, and A. J. Heeger, Solitons in polyacetylene, Phys. Rev. Lett. 42, 1698 (1979).
  * Jiao  _et al._ [2021] Z.-Q. Jiao, S. Longhi, X.-W. Wang, J. Gao, W.-H. Zhou, Y. Wang, Y.-X. Fu, L. Wang, R.-J. Ren, L.-F. Qiao, and X.-M. Jin, Experimentally detecting quantized Zak phases without chiral symmetry in photonic lattices, Phys. Rev. Lett. 127, 147401 (2021).
  * [60] See Supplemental Material for the detailed derivation .
  * Tang  _et al._ [2021d] J. Tang, L. Tang, and K. Xia, Single-photon transport in a whispering-gallery mode microresonator directionally coupled with a two-level quantum emitter, arXiv preprint arXiv:2110.09375 (2021d).
  * Asbóth  _et al._ [2016] J. K. Asbóth, L. Oroszlány, and A. Pályi, A short course on topological insulators, Lecture notes in physics 919, 166 (2016).
  * Yanik  _et al._ [2004] M. F. Yanik, W. Suh, Z. Wang, and S. Fan, Stopping light in a waveguide with an all-optical analog of electromagnetically induced transparency, Phys. Rev. Lett. 93, 233903 (2004).
  * Wang  _et al._ [2020a] P. Wang, Y. Zheng, X. Chen, C. Huang, Y. V. Kartashov, L. Torner, V. V. Konotop, and F. Ye, Localization and delocalization of light in photonic moiré lattices, Nature (London) 577, 42 (2020a).
  * He  _et al._ [2021] Y. He, R. Mao, H. Cai, J.-X. Zhang, Y. Li, L. Yuan, S.-Y. Zhu, and D.-W. Wang, Flat-band localization in Creutz superradiance lattices, Phys. Rev. Lett. 126, 103601 (2021).
  * Peng  _et al._ [2017] Y. Peng, Y. Bao, and F. von Oppen, Boundary Green functions of topological insulators and superconductors, Phys. Rev. B 95, 235143 (2017).
  * Martinez Alvarez and Coutinho-Filho [2019] V. M. Martinez Alvarez and M. D. Coutinho-Filho, Edge states in trimer lattices, Phys. Rev. A 99, 013833 (2019).
  * Andrea Melloni and Martinelli [2008] C. F. Andrea Melloni, Francesco Morichetti and M. Martinelli, Continuously tunable 1 byte delay in coupled-resonator optical waveguides, Opt. Lett. 33, 2389 (2008).
  * Hafezi  _et al._ [2013] M. Hafezi, S. Mittal, J. Fan, A. Migdall, and J. M. Taylor, Imaging topological edge states in silicon photonics, Nat. Photonics 7, 1001 (2013).
  * Wang  _et al._ [2015] J. Wang, Z. Yao, and A. W. Poon, Silicon-nitride-based integrated optofluidic biochemical sensors using a coupled-resonator optical waveguide, Front. Mater. 2, 10.3389/fmats.2015.00034 (2015).
  * Wang  _et al._ [2020b] M. Wang, N. Yao, R. Wu, Z. Fang, S. Lv, J. Zhang, J. Lin, W. Fang, and Y. Cheng, Strong nonlinear optics in on-chip coupled lithium niobate microdisk photonic molecules, New J. Phys. 22, 073030 (2020b).
  * Kors  _et al._ [2018] A. Kors, J. P. Reithmaier, and M. Benyoucef, Telecom wavelength single quantum dots with very small excitonic fine-structure splitting, Appl. Phys. Lett. 112, 172102 (2018).
  * Haffouz  _et al._ [2018] S. Haffouz, K. D. Zeuner, D. Dalacu, P. J. Poole, J. Lapointe, D. Poitras, K. Mnaymneh, X. Wu, M. Couillard, M. Korkusinski, E. Schöll, K. D. Jöns, V. Zwiller, and R. L. Williams, Bright single inasp quantum dots at telecom wavelengths in position-controlled inp nanowires: The role of the photonic waveguide, Nano Lett. 18, 3047 (2018).
  * Arakawa and Holmes [2020] Y. Arakawa and M. J. Holmes, Progress in quantum-dot single photon sources for quantum information technologies: A broad spectrum overview, Appl. Phys. Rev. 7, 021309 (2020).
  * Htoon  _et al._ [2002] H. Htoon, T. Takagahara, D. Kulik, O. Baklenov, A. L. Holmes, and C. K. Shih, Interplay of rabi oscillations and quantum interference in semiconductor quantum dots, Phys. Rev. Lett. 88, 087401 (2002).
  * Xu  _et al._ [2008] Q. Xu, D. Fattal, and R. G. Beausoleil, Silicon microring resonators with 1.5-µm radius, Opt. Express 16, 4309 (2008).


# SNRi Target Training for Joint Speech Enhancement and Recognition

###### Abstract

Speech enhancement (SE) is used as a frontend in speech applications including
automatic speech recognition (ASR) and telecommunication. A difficulty in
using the SE frontend is that the appropriate noise reduction level differs
depending on applications and/or noise characteristics. In this study, we
propose “signal-to-noise ratio improvement (SNRi) target training”; the SE
frontend is trained to output a signal whose SNRi is controlled by an
auxiliary scalar input. In joint training with a backend, the target SNRi
value is estimated by an auxiliary network. By training all networks to
minimize the backend task loss, we can estimate the appropriate noise
reduction level for each noisy input in a data-driven scheme. Our experiments
showed that the SNRi target training enables control of the output SNRi. In
addition, the proposed joint training relatively reduces word error rate by
4.0% and 5.7% compared to a Conformer-based standard ASR model and
conventional SE-ASR joint training model, respectively. Furthermore, by
analyzing the predicted target SNRi, we observed the jointly trained network
automatically controls the target SNRi according to noise characteristics.
Audio demos are available in our demo page111google.github.io/df-
conformer/snri_target/.

Index Terms: Speech enhancement, signal-to-noise ratio improvement, multi-task
learning, noise robust ASR.

##  1 Introduction

Speech enhancement (SE) is the task of recovering target speech from a noisy
signal [1]. Single-channel SE is an indispensable frontend in most speech
tasks; for example, improving speech intelligibility for telecommunication [2,
3, 4], reducing noise for automatic speech recognition (ASR) [5, 6, 7, 8, 9]
systems, and estimating spatial covariance matrix for multi-channel SE [10,
11].

A difficulty in using SE frontend is that the appropriate noise reduction
level is different depending on the applications and/or noise characteristics.
For example, maximizing signal-to-noise ratio (SNR) improvement does not
necessarily lead to better ASR performance [9] and perceptual speech quality
[12]. This is likely due to the distortions introduced by non-linear
processing in single-channel SE such as time-frequency (TF) masking. While
typical single-channel SE frontends aim to perfectly remove noise, in practice
they cause artifacts in the resulting denoised speech.

One strategy to solve this problem is restricting SE performance [12, 13, 14,
15, 16, 17, 18, 19, 20]. Earlier SE studies limit noise suppression in non-
speech TF bins by flooring [12, 13], smoothing [12, 14], and scaling [15, 16]
the estimated TF mask in the short-time Fourier transform (STFT) domain. Since
recent time-domain SE [21, 22, 23, 24, 25] cannot control noise reduction
level by manipulating estimated masks, the observed and/or estimated noise
signals are added to the estimated speech signal [19, 20]. This strategy is
certainly effective, and therefore, it would be worthwhile to extend this
approach to an more interpretable form, optimized in data-driven manner.

In this paper, we address the following question: How much signal-to-noise
ratio improvement (SNRi) is required in each task for a given noisy input? To
provide interpretability to the SE frontend, we propose a new framework named
as SNRi target training. The SE frontend uses an auxiliary scalar input, which
represents the target SNRi of the output signal. Instead of optimizing the
enhancement to maximize the SNR, its goal is altered to produce an output
signal with the specified target SNRi. We call this SE model, “SNRi-Net”. A
block diagram of SNRi-Net is shown in Fig. 1. As a specific use case, we adopt
ASR as the backend task to evaluate the merit of SNRi-Net. In joint training,
the target SNRi value is estimated by an auxiliary network, which we call,
SNRi-Pred-Net. All networks are trained to minimize ASR loss in an end-to-end
manner as shown in Fig. 2. This way, we can estimate the appropriate noise
reduction level for each noisy input in a data-driven manner.

Experiments show that our SNRi target training enables control of the SNRi
more accurately than post-mixing of separate signals. For evaluating the
proposed method as the SE frontend, our ASR system was compared with a
Conformer-based standard ASR model and conventional SE-ASR joint training
model. We used noisy datasets with and without reverberation, and the proposed
method achieved the WER reduction by 12.5% (dry) and 4.0% (reverberant) from
the standard ASR model, and 1.5% (dry) and 5.7% (reverberant) from the SE-ASR
joint model, respectively. Furthermore, we observed that the predicted target
SNRi was controlled dynamically according to noise characteristics. Audio
demos are available in our demo page1.

Figure 1: Network architecture of SNRi-Net trained by SNRi target training.
$N$, $D_{e}$, and $D_{b}$ mean number of time frames, encoder/decoder basis,
and bottleneck feature, respectively. Figure 2: (a) Overview of proposed joint
training of SNRi-Net and ASR backend. (b) Network architecture of SNRi-Pred-
Net. Black solid lines show variable flow in forward-propagation, and
red/green dotted lines show gradient flow in back-propagation.

##  2 Conventional Method

Let the $T$-sample time-domain observation $\bm{x}\in\mathbb{R}^{T}$ be a
mixture of a target speech $\bm{s}$ and noise $\bm{n}$, such that,
$\bm{x}=\bm{s}+\bm{n}$. The goal of standard SE is to recover $\bm{s}$ from
$\bm{x}$. A popular strategy in supervised SE is the time-domain mask-based
method [21, 22, 23, 24, 25]. As an implementation example, [25] estimates
masks for separating speech and noise by a mask prediction network and applies
it to the representation of $\bm{x}$ encoded by an encoder. The estimated
signals $\bm{y}\in\mathbb{R}^{T\times 2}$ are then re-synthesized using a
decoder. Here, $\bm{y}_{:,1}$ and $\bm{y}_{:,2}$ are the estimates of speech
and noise, respectively. Then, a mixture consistency projection layer [26] is
applied to ensure the sum of $\bm{y}_{:,1}$ and $\bm{y}_{:,2}$ equals
$\bm{x}$:

$\displaystyle\bm{y}_{:,1}\leftarrow\bm{y}_{:,1}+\zeta\bm{e},\quad\bm{y}_{:,2}\leftarrow\bm{y}_{:,2}+(1-\zeta)\bm{e},$
(1)

where $\bm{e}=\bm{x}-(\bm{y}_{:,1}+\bm{y}_{:,2})$, and $\zeta\in[0,1]$ is a
tunable hyperparameter. The negative thresholded SNR [24] is used as the loss:

$\displaystyle\mathcal{L}^{\mbox{\tiny{SE}}}$
$\displaystyle=\alpha\mathcal{L}^{\mbox{\tiny{SNR}}}_{s,y_{:,1}}+(1-\alpha)\mathcal{L}^{\mbox{\tiny{SNR}}}_{n,y_{:,2}},$
(2) $\displaystyle\mathcal{L}^{\mbox{\tiny{SNR}}}_{a,b}$
$\displaystyle=-10\log_{10}(\lVert\bm{a}\rVert^{2}/(\lVert\bm{a}-\bm{b}\rVert^{2}+\tau\lVert\bm{a}\rVert^{2})),$
(3)

where $\lVert\cdot\rVert$ is $\ell_{2}$ norm and $\tau=10^{-3}$ is a soft
threshold that clamps the loss at 30 dB [24] and $\alpha\in[0,1]$ is a tunable
hyperparameter [24].

The SE fronend can be joined with a backend such as ASR [8, 9]. In a typical
framework, the estimated speech $\bm{y}_{:,1}$ is fed to the backend, then
both frontend and backed are jointly fine-tuned by minimizing a loss function.
In noise robust ASR tasks, the loss function $\mathcal{L}$ can be a weighted
sum of the ASR loss $\mathcal{L}^{\mbox{\tiny{ASR}}}$ and
$\mathcal{L}^{\mbox{\tiny{SE}}}$ as

$\displaystyle\mathcal{L}=\mathcal{L}^{\mbox{\tiny{ASR}}}+\gamma\mathcal{L}^{\mbox{\tiny{SE}}},$
(4)

where $\gamma\geq 0$ is a tunable hyperparameter.

In practice, $\bm{y}_{:,1}$ includes artifacts in speech. Such distortions
cause degradation of the backend tasks such as ASR. To reduce distortion,
several post-processing methods have been proposed [15, 16, 17, 18, 19, 20].
For time-domain speech separation [19] and SE [20], a possible strategy is to
add $\bm{y}_{:,2}$222In conventional studies [19, 20], $\bm{x}$ is added
instead of $\bm{y}_{:,2}$, which is the same as in Eq. (5) except for the
constant multiplication. To clarify the relationship with SNRi, we formulate
the post-mixing using $\bm{y}_{:,2}$.. Since SE estimates of clean speech
$\bm{y}_{:,1}$ and noise $\bm{y}_{:,2}$, we can control noise reduction level
as

$\displaystyle\bm{y}=\bm{y}_{:,1}+w\bm{y}_{:,2},$ (5)

where $w\in[0,1]$. By assuming that the SE module perfectly separate speech
and noise, SNRi of $\bm{y}$ can be controlled as $w=10^{-\lambda/20}$, where
$\lambda$ is a target SNRi scalar.

##  3 Proposed Method

###  3.1 SNRi target training

In contrast the conventional SE, the goal of SNRi target training is to
control SNRi of the SE output $\bm{y}_{:,1}$ according to $\lambda$. To
achieve this, $\lambda$ is also input to the mask prediction network as an
auxiliary variable. Specifically, we concatenate $\lambda$ to the encoder
output in the feature dimension as shown in Fig. 1. We call this network as
SNRi-Net. After applying a mixture consistency projection layer [26], the loss
value is calculated as the squared-error between the target SNRi, $\lambda$,
and SNRi of the output signal as:

$\displaystyle\mathcal{L}^{\mbox{\tiny{SNRi}}}$
$\displaystyle=\lvert\lambda-\operatorname{SNRi}\rvert^{2}+\beta\mathcal{L}^{\mbox{\tiny{SAR}}},$
(6) $\displaystyle\operatorname{SNRi}$
$\displaystyle=10\log_{10}\left(\frac{\lVert\bm{s}\rVert^{2}}{\lVert\bm{y}_{:,1}\\!-\\!\bm{s}\rVert^{2}}\right)-10\log_{10}\left(\frac{\lVert\bm{s}\rVert^{2}}{\lVert\bm{n}\rVert^{2}}\right),$
(7) $\displaystyle\mathcal{L}^{\mbox{\tiny{SAR}}}$
$\displaystyle=-10\log_{10}(\lVert\bm{s}\rVert^{2}/(\lVert\bm{e}_{\mbox{\tiny{artif}}}\rVert^{2}+\tau\lVert\bm{s}\rVert^{2})),$
(8)

where $\beta\in[0,1]$ is a weight parameter and
$\mathcal{L}^{\mbox{\tiny{SAR}}}$ is the negative thresholded source-to-
artifact ratio (SAR) which reduces artifacts in the output signal based on the
SAR [27]. $\bm{e}_{\mbox{\tiny{artif}}}$ are the artifacts in the output
signal which can be obtained by decomposing the residual noise
$\bm{y}_{:,1}-\bm{s}=\bm{e}_{\mbox{\tiny{interf}}}+\bm{e}_{\mbox{\tiny{artif}}}$
where $\bm{e}_{\mbox{\tiny{interf}}}$ is the orthogonal projection of the
residual noise onto the subspace spanned by both $\bm{s}$ and $\bm{n}$ [27].

###  3.2 Joint training of SNRi-Net and ASR backend

SNRi-Net can also be jointly trained with a backend task, like ASR. The loss
function for joint-training then becomes:

$\displaystyle\mathcal{L}=\mathcal{L}^{\mbox{\tiny{ASR}}}+\eta\mathcal{L}^{\mbox{\tiny{SNRi}}},$
(9)

where, $\eta\geq 0$ is a tunable hyperparameter. The problem of the joint
network is that the appropriate target SNRi $\lambda$ is unknown for each task
and/or input. To address this, we propose using an auxiliary network called as
SNRi-Pred-Net, which predicts a $\lambda$ as $\hat{\lambda}$ that minimizes
$\mathcal{L}^{\mbox{\tiny{ASR}}}$. Fig. 2 (a) shows the overview of the
proposed joint training framework. The architecture of SNRi-Pred-Net used in
this study is shown in Fig. 2 (b).

Log-mel filterbank is used as inputs by SNRi-Pred-Net. The input features are
first passed to a dense layer for computing bottleneck features, which are
then processed by Conformer blocks [28]. We apply average-pooling across time
to the the output of the Conformer to obtain a single, vector representation
for the inputs. A second dense layer followed by a sigmoid function coverts
the vector to a scalar, which is the predicted target SNRi. A scaling
operation is also used to restrict the predicted target SNRi to the range
$[\lambda_{\min},\lambda_{\max}]$. Here, $\lambda_{\min}$ and $\lambda_{\max}$
are hyperparameters that represents the minimum and maximum values of target
SNRi $\hat{\lambda}$.

To enhance the noisy signal, $\bm{x}$ and $\hat{\lambda}$ are passed to SNRi-
Net, and its speech output $\bm{y}_{:,1}$ is passed to the ASR backend. We
train the joint network by minimizing Eq. (9). That is, the ASR model, SNRi-
Net and SNRi-Pred-Net are all trained to minimize ASR loss,
$\mathcal{L}^{\mbox{\tiny ASR}}$. Importantly, only SNRi-Net uses the SE loss,
$\mathcal{L}^{\mbox{\tiny{SNRi}}}$; we stop the gradient of
$\mathcal{L}^{\mbox{\tiny{SNRi}}}$ from optimizing SNRi-Pred-Net. Since the
predicted $\lambda$ is the prediction target for SNRi-Net, if the SE loss is
back-propagated to SNRi-Pred-Net, this network will be optimized to output
$\lambda$ that makes the task of SNRi-Net easy, but sub-optimal for ASR.

##  4 Experiment

###  4.1 Experimental settings

Dataset: We used the same training dataset as [29] with and without
reverberation. The dataset without reverberation is called “Dry” and with
reverberation as “Reverb”. This dataset includes 4,249 hours of clean speech
which consists of 281k utterances from the LibriSpeech [30] training set and
1,916k utterances from an internal dataset. The noisy utterances were
generated using a room simulator [31], with SNR from -10 dB to 30 dB. Noise is
sampled from internally collected noise snippets that simulate conditions like
cafe, kitchen, and cars, and freely available noise sources from Getty [32]
and YouTube Audio Library [33]. For Reverb, the room configurations have
reverberation times (RT60) ranging from 0 ms to 900 ms. For Dry, RT60 is
always 0 ms. We generated multiple copies of the data under different mixing
conditions in order to model enough combinations of clean speech, background
noise, and room-configuration. The training dataset includes 39.6M noisy
samples (55,027 hours) in total.

For the validation and test datasets, we used utterances from the LibriSpeech
[30] dev and test sets. The noisy utterances were generated using the same
manner as the training dataset with SNR from -5 dB to 20 dB, but with noise
sources that are held out from the training dataset. We generated “Dry” and
“Reverb” dev and test sets using the same T60 condition as the training set.
The test dataset consists of 2,620 utterances and the average SNR of the input
signals was 7.5 dB.

Models and hyper-parameters: As SNRi-Net, we used the same architecture of DF-
Confromer-8 [25] except for concatenating the target SNRi. All hyper-
parameters were the same as that used in [25]; the number of Conformer layers
was 8, $D_{e}=256$, $D_{b}=216$, and the window and hop sizes of filterbanks
were 2.5 ms and 1.25 m. Other hyper-parameters were decided based on the WER
on the validation dataset as $\beta=0.01$, $\lambda_{\min}=0.0$ and
$\lambda_{\max}=20.0$.

We use the large size Conformer Transducer ASR backend referred as
Conformer(L) [28] that has 17 Conformer blocks of 512-dim 8-head dot-product
attention, 512-dim 32-frame kernel 1d convolution, 2048-dim feedforward
module, and a 640-dim long short-term memory (LSTM) decoder. The input feature
was 80-dim filterbank features computed from a 25 ms window with a stride of
10 ms. We use SpecAugment [34] with mask parameter ($F=27$), and 10 time-masks
with maximum time-mask ratio ($p_{S}=0.05$). All transcriptions were tokenized
with a word piece model with a 1,024 vocabulary built from LibriSpeech 960h.
The ASR loss $\mathcal{L}^{\mbox{\tiny{ASR}}}$ was the recurrent neural
network transducer (RNN-T) loss [35, 36].

SNRi-Pred-Net had 2-Conformer-blocks where each block has 128-dim 6-head dot-
product attention, 128-dim 5-frame kernel 1d convolution, and 512-dim
feedforward module. The parameter of mel-filterbank was the same as the ASR
backend.

SNRi-Net and ASR backend were pre-trained for 200k steps individually; SNRi-
Net was pre-trained to minimize $\mathcal{L}^{\mbox{\tiny{SNRi}}}$ with the
training dataset, and the ASR backend was pre-trained to minimize
$\mathcal{L}^{\mbox{\tiny{ASR}}}$ using both clean and noisy speech in the
training dataset. While pre-training of SNRi-Net, $\hat{\lambda}$ was randomly
drawn from the uniform distribution
$\mathcal{U}(\lambda_{\min},\lambda_{\max})$.

The joint network was fine-tuned for an additional 100k steps to minimize Eq.
(9) where $\eta=0.01$ was determined based on WER on the validation dataset.
In the fine-tune stage, we skipped SNRi-Net and SNRi-Pred-Net with a
probability of 5% as a multi-condition learning strategy [8]. In addition, we
used a random $\hat{\lambda}$ drawn from
$\mathcal{U}(\lambda_{\min},\lambda_{\max})$ instead of the predicted one with
25% probability. All training used the Adam optimizer [37] with the same
setting as [25] except for using 1/10 learning rate in the fine-tuning stage,
and 128 Google TPUv3 cores with a global batch size of 512.

###  4.2 Evaluation for control accuracy of output SNRi

We compared SNRi target training with post-mixing based SNRi control in Eq.
(5). We used DF-Conformer-8 [25] with hyper-parameters set to the same values
as [25], i.e. $\alpha=0.8$ and $\zeta=0.5$. We tested both methods with two
input SNR conditions; SNR of all test samples were adjusted to -5 dB and 5 dB.
The target SNRi were 3 dB, 6 dB, 9 dB, and 12 dB.

Figure 3 shows the experimental results and audio examples of SNRi-Net are
available in our demo page1. SNRi-Net trained by SNRi target training achieved
significantly better control accuracy of output SNRi than supervised SE on
both Dry and Reverb test sets. Although the post-mixing-based control assumes
that the output signals are perfectly separated, the outputs usually contains
separation errors. Such separation errors affects the output SNRi of the post-
mixed signal, resulting in the under-separation problem; the output SNRi is
always lower than the target SNRi. SNRi target training has succeeded to avoid
the under-separation problem by directly controlling output SNRi in 3 dB and 6
dB target SNRi conditions.

Whereas in 9 dB and 12 dB target SNRi conditions, output SNRi of both methods
are significantly lower than target SNRi, even though SNRi-Net was still
better than the supervised method. It might be due to the performance upper-
bound of the base SE network. It is necessary to accurately distinguish speech
and noise to output high SNR signals, and the accuracy should correlate on the
separation performance of the SE network. Therefore, to achieve higher control
performance in high target SNRi conditions, it is necessary to improve the
base SE network performance.

Figure 3: Results of experiment on controlling SNRi of output signal. Top and
bottom figures are results on Dry and Reverb test sets, respectively. Solid
lines and colored are show mean and 99 % confidence intervals, respectively.
Legends SE and SNRi mean SE+post-mixing and SNRi-Net, respectively.

###  4.3 Evaluation as ASR frontend

Table 1: Word error rate on test dataset and number of trainable parameters for each model. Network | #Params | WER [%] ($\downarrow$)
---|---|---
Dry | Reverb
Conformer(L) | 118.8M | 10.5 | 14.4
Conformer(L)+ | 132.3M | 10.7 | 14.9
SE+Conformer(L) | 127.4M | 9.3 | 14.7
SNRi+Conformer(L) | 128.3M | 9.2 | 13.8
Figure 4: Log-mel spectrogram examples of input and output of joint trained
model. Input SNR of first two column figures are -5.0 dB, and later two
figures are 5.0 dB. In the same input SNR figures, left figures are input
signal, and right ones are output signal. Noise type of top, middle, and
bottom figures are environmental noise, white noise, and 4 kHz sine wave.

We compared the ASR performance of the proposed joint-network on WER metrics
with two standard ASR models and an SE-ASR joint model. The first model is
Conformer(L) [28], and the second model is Conformer(L)+ which has 19
Conformer blocks and whose model size is roughly the same as the proposed
methodṪhese models give us the baseline WER of using a large ASR models
without SE frontend. These models were trained for 300k steps from scratch.
The third model jointly trains the supervised SE frontend and Conformer(L) ASR
model. This model gives us WER of the joint training strategy of an SE
frontend and a large ASR backend. The network architecture of the SE frontend
was DF-Conformer-8 used in Sec, 4.2. $\gamma=0.25$ was determined based on WER
on the validation dataset. In addition, we skipped the SE frontend with 50%
probability in the fine-tuning stage as a multi-condition learning strategy
[8]. In this experiment, we excluded Eq. (5)-based joint models [19, 20] for
fair comparison because these methods did not fine-tune the joint model.

Table 1 shows WER on the test datasets. The proposed method achieved the best
WER performance on both datasets; it reduced WER by 12.5% and 1.5% compared to
Conformer(L) and SE+Conformer(L) for Dry, and 4.0% and 5.7% Reverb,
respectively. In addition, SE-Conformer(L) increased WER on Reverb compared to
Conformer(L), whereas SNRi-Conformer(L) improved WER. From these results, the
proposed method consistently improved the ASR accuracy by comparing to merely
increasing the ASR backend model size and/or jointly training with a
conventional SE frontend.

Although the performance gains in terms of ASR quality from the
SE+Conformer(L) method are limited the gains a consistent across conditions.
Moreover, the model provides a novel way of controlling the SNRi, thereby
providing new insights on how to model SE in a larger system. Figure 4 shows
log-mel spectrogram examples of the input and output of the proposed joint
model. These examples indicate that the predicted SNRi is affected by two
factors; input SNR and noise type. By comparing input SNR at -5 dB and 5 dB
cases, the predicted SNRi tends to be high when the input SNR is low. This is
an intuitive result due to the fact that the larger noise makes ASR more
difficult. The other aspect is the noise type; in the case of noise types that
only affects a certain frequency, such as a tonal noise, the predicted SNRi
tends to be low even if the input SNR is low. As can be seen from the bottom
spectrograms, if the majority of the frequency bands are clean, the speech
characteristics can be analyzed even if the noise is large. These results show
that the proposed method also provides interpretable insights for future
development of single-channel SE frontends, e.g. tuning $w$ in Eq. (5) using a
small scale dataset without joint training [20].

##  5 Conclusion

We proposed “SNRi target training”; the SE frontend uses an auxiliary scalar
input which represents target SNRi, and enhances the input so that SNRi of the
output signal is close to the desired target value. In joint training with the
ASR backend, the target SNRi value was also estimated to minimize the ASR
loss. Experiments showed that SNRi-Net controls the SNRi more accurately than
post-mixing of separated signals, and our joint ASR system achieved the best
WER on noisy ASR datasets. Furthermore, by analyzing the output of the joint
model, we observed the model automatically controls the target SNRi according
to noise characteristics.

The limitation of the proposed method is the requirement of clean speech, in
contrast to several conventional joint training strategies that only need
noisy speech and its transcription [8, 19, 38]. Future work will include fine-
tuning only SNRi-Pred-Net [19] and/or incorporating an unsupervised SE
training [24].

## References

  * [1] D. Wang and J. Chen, “Supervised speech separation based on deep learning: An overview,” _IEEE/ACM Trans. Audio Speech Lang. Process._ , vol. 26, no. 10, p. 1702–1726, 2018.
  * [2] B. C. J. Moore, “Speech processing for the hearing-impaired: successes, failures, and implications for speech mechanisms,” _Speech Commun._ , 2003\.
  * [3] D. L. Wang, “Time-frequency masking for speech separation and its potential for hearing aid design,” trends in amplification,” _Trends Amplif._ , 2008\.
  * [4] C. K. A. Reddy, H. Dubey, V. Gopal, R. Cutler, S. Braun, H. Gamper, R. Aichner, and S. Srinivasan, “ICASSP 2021 deep noise suppression challenge,” in _Proc. Int. Conf. on Acoust., Speech, and Signal Process. (ICASSP)_ , 2021.
  * [5] M. Fujimoto. and Y. Ariki, “Robust speech recognition in additive and channel noise environments using GMM and EM algorithm,” in _Proc. Int. Conf. on Acoust., Speech, and Signal Process. (ICASSP)_ , 2004.
  * [6] F. Weninger, H. Erdogan, S. Watanabe, E. Vincent, J. Le Roux, J. R. Hershey, and B. Schuller, “Speech enhancement with lstm recurrent neural networks and its application to noise-robust ASR,” in _Proc. Int. Conf. Latent Var. Anal. Signal Sep. (LVA/ICA)_ , 2015.
  * [7] H. Erdogan, J. R. Hershey, S. Watanabe, and J. Le Roux, “Phase-sensitive and recognition-boosted speech separation using deep recurrent neural networks,” in _Proc. Int. Conf. on Acoust., Speech, and Signal Process. (ICASSP)_ , 2015.
  * [8] K. Kinoshita, T. Ochiai, M. Delcroix, and T. Nakatani, “Improving noise robust automatic speech recognition with single-channel time-domain enhancement network,” in _Proc. Int. Conf. on Acoust., Speech, and Signal Process. (ICASSP)_ , 2020.
  * [9] C. Li, J. Shi, W. Zhang, A. S. Subramanian, X. Chang, N. Kamo, M. Hira, T. Hayashi, C. Boeddeker, Z. Chen, and S. Watanabe, “ESPnet-SE: end-to-end speech enhancement and separation toolkit designed for ASR integration,” in _Proc. IEEE Spok. Lang. Technol. Workshops (SLT)_ , 2021.
  * [10] H. Erdogan, J. R. Hershey, S. Watanabe, M. I. Mandel, and J. Le Roux, “Improved MVDR beamforming using single-channel mask prediction networks,” in _Proc. Interspeech_ , 2016.
  * [11] T. Higuchi, K. Kinoshita, N. Ito, S. Karita, and T. Nakatani, “Frame-by-frame closed-form update for mask-based adaptive MVDR beamforming,” in _Proc. Int. Conf. on Acoust., Speech, and Signal Process. (ICASSP)_ , 2018.
  * [12] Y. Koizumi, K. Niwa, Y. Hioka, K. Kobayashi, and Y. Haneda, “DNN-based source enhancement to increase objective sound quality assessment score,” _IEEE/ACM Trans. Audio Speech Lang. Process._ , vol. 26, no. 10, pp. 1780–1792, 2018.
  * [13] I. Cohen, “Optimal speech enhancement under signal presence uncertainty using log-spectral amplitude estimator,” _IEEE Signal Process. Lett._ , 2002.
  * [14] E. Vincent, “An experimental evaluation of wiener filter smoothing techniques applied to under-determined audio source separation,” in _Proc. Int. Conf. Latent Var. Anal. Signal Sep. (LVA/ICA)_ , 2010.
  * [15] A. Narayanan and D. Wang, “Joint noise adaptive training for robust automatic speech recognition,” in _Proc. Int. Conf. on Acoust., Speech, and Signal Process. (ICASSP)_ , 2014.
  * [16] Q. Wang, K. A. Lee, T. Koshinaka, K. Okabe, and H. Yamamoto, “Task-aware warping factors in mask-based speech enhancement,” in _Proc. Eur. Signal Process. Conf. (EUSIPCO)_ , 2021.
  * [17] T. Menne, R. Schlüter, and H. Ney, “Investigation into joint optimization of single channel speech enhancement and acoustic modeling for robust ASR,” in _Proc. Int. Conf. on Acoust., Speech, and Signal Process. (ICASSP)_ , 2019\.
  * [18] A. Pandey, C. Liu, Y. Wang, and Y. Saraf, “Dual application of speech enhancement for automatic speech recognition,” in _IEEE Spok. Lang. Technol. Workshop (SLT)_ , 2021.
  * [19] H. Sato, T. Ochiai, M. Delcroix, K. Kinoshita, N. Kamo, and T. Moriya, “Learning to Enhance or Not: Neural network-based switching of enhanced and observed signals for overlapping speech recognition,” in _Proc. Int. Conf. on Acoust., Speech, and Signal Process. (ICASSP)_ , 2022\.
  * [20] K. Iwamoto, T. Ochiai, M. Delcroix, R. Ikeshita, H. Sato, S. Araki, and S. Katagiri, “How Bad Are Artifacts?: Analyzing the impact of speech enhancement errors on ASR,” _arXiv:2201.06685_ , 2022.
  * [21] Y. Luo and N. Mesgarani, “Conv-TasNet: Surpassing ideal time–frequency magnitude masking for speech separation,” _IEEE/ACM Trans. Audio Speech Lang. Process._ , vol. 27, no. 8, pp. 1256–1266, 2019.
  * [22] I. Kavalerov, S. Wisdom, H. Erdogan, B. Patton, K. Wilson, J. Le Roux, and J. R. Hershey, “Universal sound separation,” in _Proc. IEEE Workshop Appl. Signal Process. Audio Acoust. (WASPAA)_ , 2019.
  * [23] M. Pariente, S. Cornell, A. Deleforge, and E. Vincent, “Filterbank design for end-to-end speech separation,” in _Proc. Int. Conf. on Acoust., Speech, and Signal Process. (ICASSP)_ , 2020.
  * [24] S. Wisdom, E. Tzinis, H. Erdogan, R. Weiss, K. Wilson, and J. R. Hershey, “Unsupervised sound separation using mixture invariant training,” in _Proc. Adv. Neural Inf. Process. Syst. (NeurIPS)_ , 2020.
  * [25] Y. Koizumi, S. Karita, S. Wisdom, H. Erdogan, J. R. Hershey, L. Jones, and M. Bacchiani, “DF-Conformer: Integrated architecture of Conv-TasNet and Conformer using linear complexity self-attention for speech enhancement,” in _Proc. IEEE Workshop Appl. Signal Process. Audio Acoust. (WASPAA)_ , 2021\.
  * [26] S. Wisdom, J. R. Hershey, K. Wilson, J. Thorpe, M. Chinen, B. Patton, and R. A. Saurous, “Differentiable consistency constraints for improved deep speech enhancement,” in _Proc. Int. Conf. on Acoust., Speech, and Signal Process. (ICASSP)_ , 2020.
  * [27] J. Le Roux, S. Wisdom, H. Erdogan, and J. R. Hershey, “SDR–Half-baked or well done?” in _Proc. Int. Conf. on Acoust., Speech, and Signal Process. (ICASSP)_ , 2019.
  * [28] A. Gulati, C.-C. Chiu, J. Qin, J. Yu, N. Parmar, R. Pang, S. Wang, W. Han, Y. Wu, Y. Zhang, and Z. Zhang, “Conformer: Convolution-augmented transformer for speech recognition,” in _Proc. Interspeech_ , 2020.
  * [29] A. Narayanan, C.-C. Chiu, T. O’Malley, Q. Wang, and Y. He, “Cross-attention Conformer for context modeling in speech enhancement for ASR,” in _Proc. IEEE Autom. Speech Recognit. Underst. Workshop (ASRU)_ , 2021.
  * [30] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: An ASR corpus based on public domain audio books,” in _Proc. Int. Conf. on Acoust., Speech, and Signal Process. (ICASSP)_ , 2015.
  * [31] C. Kim, A. Misra, K. Chin, T. Hughes, A. Narayanan, T. Sainath, and M. Bacchiani, “Generation of large-scale simulated utterances in virtual rooms to train deep-neural networks for far-field speech recognition in Google Home,” in _Proc. Interspeech_ , 2017.
  * [32] https://www.gettyimages.com/about-music.
  * [33] https://youtube.com/audiolibrary.
  * [34] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, “SpecAugment: A simple data augmentation method for automatic speech recognition,” 2019.
  * [35] A. Graves, “Sequence transduction with recurrent neural networks,” in _Proc. Int. Conf. Mach. Learn. (ICML)_ , 2012.
  * [36] S. Karita, Y. Kubo, M. Bacchiani, and L. Jones, “A comparative study on neural architectures and training methods for Japanese speech recognition,” in _Proc. Interspeech_ , 2021.
  * [37] D. P. Kingma and J. L. Ba, “Adam: A method for stochastic optimization,” in _Proc. Int. Conf. Learn. Represent. (ICLR)_ , 2015.
  * [38] S. E. Eskimez, X. Wang, M. Tang, H. Yang, Z. Zhu, Z. Chen, H. Wang, and T. Yoshioka, “Human listening and live captioning: Multi-task training for speech enhancement,” in _Proc. Interspeech_ , 2021.