# Massively Scalable Wavelength Diverse Integrated Photonic Linear Neuron

Matthew van Niekerk1,†, Anthony Rizzo2, Hector Rubio Rivera1, Gerald Leake3,
Daniel Coleman3, Christopher Tison4, Michael Fanto4, Keren Bergman2, and
Stefan Preble1 1 Microsystems Engineering, Rochester Institute of Technology,
Rochester, NY, USA  2 Department of Electrical Engineering, Columbia
University, New York, NY, USA  3 State University of New York Polytechnic
Institute, Albany, NY, USA  4 Information Directorate, Air Force Research
Laboratory, Rome, NY, USA<EMAIL_ADDRESS>
###### Abstract

As computing resource demands continue to escalate in the face of big data,
cloud-connectivity and the internet of things, it has become imperative to
develop new low-power, scalable architectures. Neuromorphic photonics, or
photonic neural networks, have become a feasible solution for the physical
implementation of efficient algorithms directly on-chip. This application is
primarily due to the linear nature of light and the scalability of silicon
photonics, specifically leveraging the wide-scale complementary metal-oxide-
semiconductor (CMOS) manufacturing infrastructure used to fabricate
microelectronics chips. Current neuromorphic photonic implementations stem
from two paradigms: wavelength coherent and incoherent. Here, we introduce a
novel architecture that supports coherent and incoherent operation to increase
the capability and capacity of photonic neural networks with a dramatic
reduction in footprint compared to previous demonstrations. As a proof-of-
principle, we experimentally demonstrate simple addition and subtraction
operations on a foundry-fabricated silicon photonic chip. Additionally, we
experimentally validate an on-chip network to predict the logical 2-bit gates
AND, OR, and XOR to accuracies of $96.8\%,99\%,$ and $98.5\%$, respectively.
This architecture is compatible with highly wavelength parallel sources,
enabling massively scalable photonic neural networks.

##  1 Introduction

In recent years, integrated photonics has emerged as a viable and highly
appealing candidate for neural network hardware implementation. Photonic
neural networks (PNN) hold particular promise compared to traditional
electronic implementations due to the highly favorable physical properties of
photons compared to electrons for performing linear operations; namely, (i)
the bosonic nature of light permits high degrees of parallelism in spatial
mode and wavelength, (ii) such linear transformations are naturally
implemented using standard photonic elements (i.e., beamsplitters and phase
shifters) at the speed of light, and (iii) these operations can consume orders
of magnitude less energy than their electronic counterparts [1, 2, 3, 4, 5].
In particular, silicon photonics provides an ideal platform for PNNs due to
its compatibility with the standard complementary metal-oxide-semiconductor
(CMOS) infrastructure used to fabricate microelectronic chips and the high
index contrast between silicon and silicon dioxide which permits unparalleled
device density [3]. Numerous approaches to optical neural networks have been
demonstrated in various platforms, including as spiking [6, 7], phase-change
materials [8, 9, 10], metal-sulfide microfibers [11, 12], photoelectric
multiplication [13, 10], and semiconductor optical amplifiers [14, 15].
However, two primary categories have emerged for PNNs in the silicon photonics
platform: wavelength coherent and incoherent [3].

Wavelength coherent designs rely on careful placement and tuning of Mach-
Zehnder interferometers (MZI), such that optical interference provides the
multiplication and summation operations. Carefully designed MZI mesh circuits
can implement any real-valued matrix on-chip by reconfiguring phase shifters
according to singular value decomposition [1, 16]. The coherent optical linear
neuron (COLN) architecture implements vector-vector multiplication and
summation by leveraging MZIs as inputs and weights for multiplication and
coherent interference as the summation operator [17, 18]. Wavelength
incoherent designs rely on multi-wavelength operation, often employing a
broadband photodetector to perform destructive summation of the various
signals. For example, in the “Broadcast & Weight” protocol, multiple
wavelength channels are launched as inputs along the same waveguide, which
allows for parallel weighting with micro-ring resonator weight banks [19]. A
balanced photodetector pair sums the vector-vector multiplication; however,
multiple micro-ring weight banks can operate in parallel for vector-matrix
operation, as demonstrated in [20].

In this work, we present a novel architecture that leverages both coherent and
incoherent aspects to create a massively scalable PNN specific linear
operator. This architecture requires fewer components, provides orders-of-
magnitude footprint reduction, and consumes substantially less power than
comparable designs. As an initial proof-of-principle, we experimentally
demonstrate simple addition and subtraction on-chip. Additionally, we
implement and demonstrate a neural network task on-chip, designed to perform
logic gate operations (AND, OR, and XOR). Due to the compact footprint, low
energy consumption, and scalability of the demonstrated linear neuron, these
results pave the way toward large-scale PNNs with hundreds to thousands of
neurons on a single silicon chip.

##  2 Proposed Architecture

Figure 1:  (a) The COLN architecture. The carrier signal is fanned out, where
it meets an MZM that imparts the input and then meets another MZM that impart
the magnitude of the weight, with the phase shifter controlling the sign of
the weight. Each bus is then recombined at the Fan-In stage, representing the
vector-vector multiplication of the COLN. (b) An equivalent NN diagram for
this circuit. The unshaded region represents the multiplication and summation
performed by the COLN, ignoring the activation function which is external to
this architecture. (c) A typical transfer function for a MZM. The extinction
ratio (ER) and phase can vary with design.

We begin our design derivation with inspiration from the COLN [17]. The
operating principle of the COLN employs four key stages: Fan-Out, Input,
Weighting, and Fan-In, as shown in Figure 1(a). At the Fan-Out, a continuous-
wave (CW) optical carrier signal is split onto $N$ copies. Here, the Input and
Weight stage is represented by amplitude modulators, which are implemented in
hardware as Mach-Zehnder Modulators (MZM) and impose the inputs and weights
onto the CW carrier signal. This signal is recombined from $N$ paths down to 1
path, performing summation of the $N$ input/weight products at the Fan-In
through constructive interference. Figure 1(b) demonstrates the equivalent
neural network (NN) diagram for this circuit, which performs the linear
summation of the inputs multiplied by the synaptic weights, preparing the
carrier signal for the activation stage. This optical circuit and the NN both
reduce to the general form:

$out=\sum_{n}^{N}\mathbf{w}_{n}\mathbf{x}_{n},$ (1)

where, for the optical circuit, $out$ is the electric field output, $N$ as the
number of layers in the width (or Fan-Out), $\mathbf{w}$ is the weight vector,
and $\mathbf{x}$ is the input vector. In this formulation, $\mathbf{w}$ and
$\mathbf{x}$ are represented by the electro-optic transfer function of the MZM
in each layer similar to that seen in Figure 1(c). A key feature of the COLN
circuit is that the output is in a state which, with minimal additional
passive circuitry, is fully compatible both with an activation based on
optical-electrical (OE), electrical processing, and electrical-optical (EO)
conversion [21], direct optical-electrical-optical (OEO) activation [22] or
all-optical, non-linear activation [23, 24].

In a fiber-optic based system, phase-shifters and splitters are readily
available off-the-shelf [17]. However, when attempting to scale these down for
an integrated photonic system, the size of MZM translates into a practical
barrier. Current PN diode, free-carrier MZM devices each occupy a physical
space of $>1$ mm2 in on-chip area, severely limiting the scalabilty of this
architecture. The first on-chip demonstration of the COLN in a silicon
photonic chip utilizes a $25$ mm2 chip to implement a $4$ input/weight linear
neuron, implementing the weights with thermal phase shifters, which are
dramatically smaller at the cost of update speed [25, 26].

Figure 2:  (a) The naïve WDIPLN architecture. The carrier signal is fanned
out, where it meets an MRD that imparts the input and then meets another MRD
that imparts the magnitude of the weight, with the phase shifter controlling
the sign of the weight. Each bus is then recombined at the Fan-In stage,
representing the vector-vector multiplication of the WDIPLN. (b) An equivalent
NN diagram for this circuit. The unshaded region represents the multiplication
and summation performed by the WDIPLN, ignoring the activation function which
is external to this architecture. The tabs in the upper left hand side
represent the WDIPLN’s ability to represent $M$ many-to-one networks shown
here. (c) A typical transfer function for a MRD. The ER and phase can vary
with design, however, in the “slightly undercoupled” regime we will see a
relatively flat phase response.  Figure 3:  (a) The fully formed WDIPLN
architecture. The carrier signal is fanned out, where it meets a large MRD
that imparts the input to each channel simultaneously. Each channels input
then meets a small MRD which imposes the magnitude of the weight, with the
phase shifter controlling the sign of the weight. Each bus is then recombined
at the Fan-In stage, representing the vector-matrix multiplication of the
WDIPLN. (b) An equivalent NN diagram for this circuit. The unshaded region
represents the multiplication and summation performed by the WDIPLN, ignoring
the activation function which is external to this architecture. This
architecture fully enables the linear stage of MLP layer.

Our circuit architecture, which we subsequently refer to as Wavelength Diverse
Integrated Photonic Linear Neuron (WDIPLN), begins as a COLN with naïve
replacement of MZMs with micro-resonant devices (MRD), either micro-ring
resonators or a micro-disks, as shown in Figure 2(a). With MRDs, we
immediately benefit from the decrease in footprint. Single-channel MRDs are
commonly $<100\ \mu$m2 in size [27]. In addition, careful design of the MRD
allows for a desirable adaptation of the previous electro-optic transfer
function, displayed in Figure 2(c). The MRD acts as an amplitude filter both
in the input and weight stage of the network, for each bus in the layer width.
MRDs are sensitive, but if the coupling condition is designed to be slightly
under-coupled, it is possible to mitigate some of the phase sensitivity (for
example, a depletion biased MRD would remain under-coupled during operation).
This enables the coherent summation at the Fan-In stage to remain relatively
stable for each channel.

Naïve WDIPLN introduces a feature beyond the COLN, namely wavelength
diversity. For sufficiently small radii MRD (or, conversely, sufficiently
large free-spectral-range (FSR)), we can place additional MRDs at new resonant
wavelengths (or along specific channels, similar to wavelength-division-
multiplexing (WDM) in optical communications). Recently, a similar WDM scheme
was proposed using MZMs, but the WDIPLN implementation with ring resonators
have the advantage that the wavelength multiplexing/de-multiplexing is done
intrinsically using the single device in a compact footprint [18]. We show a
schematic for the naïve WDIPLN in Figure 2(a). This enables wavelength diverse
operation of the coherent circuit at each wavelength channel, such that we
have an incoherent circuit which acts like a coherent operator at each
channel:

$out_{\lambda}=\sum_{n}^{N}\mathbf{w}_{\lambda,n}\mathbf{x}_{\lambda,n},$ (2)

where $\lambda$ indicates the wavelength channel. This design not only
leverages smaller footprint devices, but enables further footprint reduction
of our system from the limit of one optical circuit per preceptron to one
optical circuit per fully connected multi-layer perceptron (MLP) by using the
input/weight pairs at each wavelength as shown in Figure 2(b).

In the naïve case, we have simply swapped each MZM with an MRD, such that the
circuit requires $2NM$ MRDs, for layer width $N$ and wavelength channels (or
layer depth) $M$. This case is the same as using $M$ COLN circuits, which is
equivalent only in abstracted device count complexity. Thus, an actual network
with COLNs would require $M$ copies of the circuit resulting in a larger
implementation.

Architecture | COLN | COLN | WDIPLN | WDIPLN
---|---|---|---|---
(variation) | (nominal) | (w/ thermal MZI) | (naïve) | (nominal)
Input Element | MZM | MZM | Small MRD | Large MRD
Approx. Element Size (mm2) | $0.8$ | $0.8$ | $10^{-4}$ | $10^{-2}$
Weight Element | MZM | Thermal MZI | Small MRD | Small MRD
Approx. Element Size (mm2) | $0.8$ | $10^{-1}$ | $10^{-4}$ | $10^{-4}$
Scaling Rule | $2NM$ | $2NM$ | $2NM$ | $N(M+1)$
Physical Size (mm2) | Component Size Only (no routing considered)
$N=8,M=1$ | $12.8$ | $7.2$ | $1.6\times 10^{-3}$ | $8.08\times 10^{-2}$
$N=8,M=8$ | $102.4$ | $57.6$ | $1.28\times 10^{-2}$ | $8.64\times 10^{-2}$
Electrical I/O | Consider 4 Electrical I/O per element (2 EO, 2 Thermal)
$N=8,M=1$ | $64$ | $64$ | $64$ | $64$
$N=8,M=8$ | $512$ | $512$ | $512$ | $288$

Table 1: Table comparing the primary differences between the COLN and WDIPLN
architectures. We consider the nominal COLN, a COLN where weights are done
with thermal MZIs, the naïve WDIPLN and nominal WDIPLN. For physical size
calculations, we only consider the footprint of individual elements and
recognize additional routing, both optical and electrical, will be necessary
for each design. The physical size values were estimated from available
foundry PDKs in [28, 29, 30].

With naïve WDIPLN, we recognize that the input stage MRDs are redundant.
Therefore, to realize full WDIPLN we replace all $M$ input MRDs for a given
layer with a carefully designed, larger radius MRD with an FSR that matches
the channel spacing, as outlined in Figure 3(a). This adjustment lowers the
total device count to $N(M+1),$ which represents a dramatic improvement in
device count scalability in addition to the physical size. As seen in Figure
3(b), the WDIPLN circuit in this form represents the linear weighting and
summation stage of a fully connected MLP. Table 1 summarizes the key
differences between the COLN and WDIPLN, including scaling rules, physical
size and electrical I/O. We note that the physical size is calculated purely
as fundamental device area, although both the COLN and WDIPLN require
additional optical and electrical routing to connect to the outside world.
This routing footprint was omitted from the total since it is highly
implementation-specific and depends heavily on packaging constraints (i.e.,
edge coupled versus grating coupled, flip-chip bonded or wire-bonded, etc.).
However, is is clear that the extremely small device footprint of the WDIPLN
architectures leave ample room for additional circuitry, such that optical
couplers and electrical bond pads will dominate the final device area. For
example, $288$ electrical pads ($60~{}\mu$m $\times$ $60~{}\mu$m) on a
$16\times 18$ grid at $150~{}\mu$m spacing take up 6.48 mm2 in on-chip area.
If we add optical couplers ($400~{}\mu$m $\times$ $20~{}\mu$m) on the edge of
the chip at a large pitch for ease of packaging, the total on-chip area is
$<7$ mm2 for the WDIPLN of $N=8,M=8$.

##  3 Experimental Demonstration of Simple Addition

As an initial demonstration, we show the ability of WDIPLN to add and subtract
in a simplified form. This represents a fundamental building block of the
architecture. While tuning the MRDs ultimately imposes the input/weight onto
the circuit, each of the $N$ buses in the circuit are additionally
reconfigurable via a phase shifter. The phase shifter rotates the phase of a
given bus so that we are able to express the sign coefficient of the weights
as either $-1$ or $1$. Bias plays a role in many NN architectures, effectively
shifting the set point of the linear fit. Similarly, in this architecture, we
can introduce a bias to change our set point. From an optical perspective,
this allows us to validate the operation of the circuit more fully. The
summation operator occurs as optical interference, which will “add”
(constructively interfere) when the phase is the same, and “subtract”
(destructively interfere) when the phase is opposite. For an initial
demonstration, we consider the optical experiment in Figure 4(a). We designed
an WDIPLN circuit with an added bias. There are five circuit elements: the
bias ring, $R_{0}$, bias phase shifter, $PS_{0}$, top arm ring $R_{1}$, top
arm phase shifter $PS_{1}$ and bottom arm ring $R_{2}$. Each element is
electrically connected via a wirebond to an electrical fanout, which is in
turn wirebonded to a printed circuit board (PCB) and connected to a source
measure unit (SMU). The circuit is optically coupled from two single mode
fibers (SMF28) embedded in a titled fiber array to two grating couplers on
either side of the circuit. A laser and detector pair are connected to the
other ends of the respective SMF28 fibers. The MRD we employ is a carrier
injection-based PIN micro-ring resonator. We employ a PIN for proof of
principle due to the large wavelength change, however, future systems will
utilize application specific, optimized MRDs operating in carrier depletion
mode. The photonic integrated circuit (PIC) was fabricated in American
Institute for Manufacturing (AIM) Photonics’ 300 mm silicon photonics process.

We demonstrate a variety of circuit configurations as summarized in Figure
4(b). We achieve these configurations by tuning the phase shifters either to
$0$ or $\pi$ (columns), and set the rings at various resonant wavelengths
(rows). The bias path is slightly shorter than the internal WDILPN, which
means we can utilize the phase shifter and the natural wavelength dependent
interference to extract the set point behavior. Sub-figures (i, v, ix) show
three states of addition with phase shifters tuned to $0$ where the rings are
all on resonance (i), $R_{1}$ is tuned off (v), and all the rings are at
different resonances (ix). This state manifests as an effectively singular
ring resonance, two peaks and three peaks, respectively. Sub-figures (ii, vi,
x) demonstrate the corresponding states where the bias phase is at $\pi$
rather than $0$. Due to manufacturing variations, observe behavior which
deviates from the design point. In (ii), we expect no signal to pass through;
however, due to path and y-branch imbalance, the floor of the circuit is
roughly $-30$ dB. Additionally, small differences in the ring show in the
transmission. Therefore, the maximum value in (ii) is $\sim-33$ dB, where the
resonant wavelength sits, which indicates a slight imbalance but overall high
suppression of any difference in the rings. Sub-figure (vi) shows two peaks
above the minimum, as $R_{0}\neq R_{1}=R_{2}$, and (x) shows nearly three
peaks as $R_{0}\neq R_{1}\neq R_{2}$. The peaks are overlapping such that
distinguishing individual peaks is difficult. Sub-figures (iii, iv)
demonstrate a single resonance, similar to (i); however, note the maximum
power is roughly $-6$ dB due to the optical loss of the lower path. Sub-
figures (vii, viii) show detuned $R_{0}$, and $R_{1}=R_{2}$, where these rings
fully cancel out each others’ magnitudes (subtraction). Sub-figures (xi, xii)
demonstrate $R_{0}\neq R_{1}\neq R_{2}$ for the subtraction operator, which is
seen by the distinct peak difference above and below the bias line. Finally,
$R_{1},R_{2}$ flip states between (xi, xii) due to the overall phase of the
bias line.

Figure 4: (a) Experimental setup. We optically couple the laser and detector
to the photonic integrated circuit (PIC) using SMF28 fiber. The PIC consists
of a simplified WDIPLN circuit with a bias line, for a total of 3 MRDs
($R_{0},R_{1},R_{2}$) and two phase shifters
($\textrm{sign}(R_{0}),\textrm{sign}(R_{1})$. The electrical traces connect
the device terminals to the bond pads, which are wirebonded from the PIC to an
electrical fanout and then connected to a printed circuit board. The printed
circuit board is routed to an source measure unit (SMU) through a flat
conductor cable (FCC). The SMU enables electro-optic control of each device.
(b) Demonstration of addition and subtraction. The columns here represent the
tuning of the phase shifters in (a), and the rows represent the tuning of the
MRDs. The four columns show the following behavior: $+R_{0}+R_{1}+R_{2}$,
$-R_{0}+R_{1}+R_{2}$, $+R_{0}-R_{1}+R_{2}$, $-R_{0}-R_{1}+R_{2}$. The three
rows demonstrate states where $R_{0}\sim R_{1}\sim R_{2}$, $R_{0}\neq
R_{1}\sim R_{2}$, $R_{0}\neq R_{1}\neq R_{2}$. These sweeps are shown to
provide a ”big-picture” perspective of the circuit, however, we evaluate each
operation at $\lambda_{0}$.

##  4 Experimental Demonstration of Logic Gates

Figure 5: Photonic Neural Network experimental demonstration. (a) Experimental
set-up, similar to Figure 4. We optically couple the laser and detector to the
PIC. The PIC consists of a WDIPLN circuit with $N=2,M=1$ and a bias line, for
a total of 5 MRDs and 3 phase shifters. The electrical wiring goes out the
bond pads, which are wirebonded from the PIC to an electrical fanout and then
connected to a printed circuit board. The printed circuit board is routed to a
source measure unit (SMU) through a flat conductor cable (FCC). The SMU
enables electro-optic control of each device. (b) The selected architecture
for the simple neural network we implement for this task. Inputs
($x_{1},x_{2}$) are connected by a weight matrix, $\mathbf{w}$, to a hidden
layer with nodes $h_{1},h_{2}.$ From here, a simple weight vector,
$\mathbf{t}$, connects the hidden layer to the output. (c–e) The configure-
recycle process for the circuit in (a). We send in the input pairs
$[0,0],[0,1],[1,0],[1,1]$ as $x_{1},x_{2}$ for the first two stages, and
subsequently send in $h_{1},h_{2}$ in the final stage. We configure the
weights of the WDIPLN to match that in (b) step-by-step from pairs
[$w_{11},w_{21}$], [$w_{12},w_{22}$], and [$t_{1},t_{2}$] for each stage of
the network, respectively. Each configure-to-measurement cycle takes
approximately 1 second, which is dominated by the read/write speeds of the
SMUs. (f–h) Results of the experimental demonstration. The AND, OR and XOR
gate correctly predict the outputs with accuracy $96.8\%,99\%,$ and $98.5\%$,
respectively.

We designed a second circuit in order to demonstrate a learning task. This
design reflects the WDIPLN architecture for $N=2,M=1$, as seen in Figure 5(a).
This circuit places both input and weight rings along the inner buses, along
with a bias branch that also contains a ring. We wirebond out to an electrical
carrier, which we then connect to programmable SMU channels in order to
electrically control each element. We optically couple the laser and detector
through fiber to grating couplers on the input and output waveguides,
respectively.

We selected a simple network architecture that is able to reconfigure the
weights in order to represent AND, OR and XOR gates, shown in Figure 5(b). We
trained these networks on a CPU using PyTorch [31]. For the activation
function we employ the rectified linear-unit (ReLU) for all stages. We can
create a simple map from the trained model to our physical chip by
reconfiguring the chip in between the three linear stages, leading to the
three non-input nodes $h_{1},h_{2}$ and $out$. For the ring resonators, we
define a value of “0” as $V_{off}=1.4$ V and “1” as $V_{on}=1.2$ V. This
ensures we are operating in the slightly undercoupled regime for both “0” and
“1”. We note that for this demonstration, the $V_{off}$ and $V_{on}$ values
are globally set for all rings. In addition, the bias ring is set to $V_{off}$
and the bias phase shifter is set to $\pi/2$.

The process of reconfiguration and implementation of the two hidden nodes and
the output is shown in Figure 5(c–e). For the two hidden nodes, $h_{1}$ and
$h_{2}$, we configure the weights of the optical circuit, namely $|W_{1}|$ and
$|W_{2}|$, according to the pairs of $[w_{11},w_{21}]$ and $[w_{12},w_{22}]$,
respectively. In addition, if the sign of the weight is negative, we adjust
the corresponding phase shifter, namely $\textrm{sign}(W_{1})$ and
$\textrm{sign}(W_{2}),$ from $0$ to $\pi$. Once we configure the chip for
$h_{1}$, we feed the four input pairs $[0,0],[0,1],[1,0],[1,1]$ into
$X_{1},X_{2}$. For each input, we measure the output from the detector, apply
the (ReLU) activation function used in training with a “$3$ dB cutoff”, and
record the result. After the two hidden nodes are measured, we reconfigure the
circuit for the final stage, according to the final weight stage from training
$t_{1},t_{2}$. Finally, we feed the four recorded pairs of $h_{1},h_{2}$ into
the circuit as inputs, measure the output, apply the activation function and
record the result.

According to the training-to-implementation scheme described above and in
Figure 5(c–e), we demonstrate the circuit’s performance for three different
2-bit logic gates, namely AND, OR and XOR. In training, the accuracy of this
simple network reaches $\sim 100\%$. The resulting outputs of the three gates
are shown in Figure 5(f–h). The black dots are the square root (i.e.
magnitude) of the output values measured by the detector at the laser probe
wavelength, which in this experiment was globally set to $\lambda_{0}=1,526$
nm. The error bars represent the standard output deviation for a $100$ pm
window around, $\lambda_{0}\pm 50$ pm. The blue triangles show the outputs of
the circuit after activation, and the red circles show the target for the
gates. The AND, OR and XOR gates achieve predictive accuracy of $96.8\%,99\%,$
and $98.5\%$, respectively.

##  5 Conclusion

Silicon photonics holds promise for implementing large-scale PNNs directly on
chip with substantially improved performance compared to electronic
implementations. In this work, we propose and demonstrate a novel PNN
architecture, WDIPLN, which exhibits highly favorable characteristics compared
to previous literature. The WDIPLN derives from the COLN architecture detailed
in [17], enabling similar vector-vector multiplication and summation using an
optical carrier signal with the additional benefits of wavelength parallelism,
drastically reduced device footprint, and significantly lower energy
consumption. The majority of these beneficial properties come from exchanging
large, power-hungry MZMs with compact MRDs. While we can trivially exchange
the fundamental elements (MRDs for MZMs) in a one-to-one manner to enable
multi-channel operation, we show that by replacing the input MRDs with a
single, multi-channel MRD with an FSR equal to the weighting channel spacing
we can reduce the total device count significantly. As a first proof-of-
principle, we experimentally demonstrated simple addition and subtraction
between rings in a WDIPLN circuit. We then show a WDIPLN configured with $N=2$
and $M=1$, which we configure and recycle to perform 2-bit logic gates (AND,
OR and XOR). We observe the implementation accuracy for each gate to be
$96.8\%,99\%,$ and $98.5\%$, respectively. Furthermore, the natural wavelength
parallelism of the proposed WDIPLN architecture can be exploited using chip-
based Kerr frequency comb sources [32] for massive scaling in the frequency
domain, similar to recent demonstrations in the silicon photonics platform for
high bandwidth data communications [33]. This demonstration opens new
opportunities in massively parallel silicon photonic PNNs and paves the way to
large-scale systems in the thousand-neuron regime on a single chip with
minimal energy consumption.

This work was supported by Air Force Research Laboratory (FA8750-21-2-0004 and
FA8650-21-2-1000) and the National Science Foundation (Award #1810282). The
views and conclusions contained herein are those of the authors and should not
be interpreted as necessarily representing the official policies or
endorsements, either expressed or implied, of the United States Air force, the
Air Force Research Laboratory or the U.S. Government. The U.S. Government is
authorized to reproduce and distribute reprints for Governmental purposes
notwithstanding any copyright notation thereon. M.v.N is funded by NSF award
DMR-1747426. The authors thank AIM Photonics for chip fabrication.

##  Appendix A Operating Conditions and Variation of MRDs

The mechanism of tuning for MRDs here is assumed to be electro-optic (where
thermo-optic is an accepted simplification) and well defined by free-carrier
plasma dispersion [34]. The transfer function from the electrical to optical
domain is defined as

$E_{\textrm{all-pass-
ring}}(\lambda;V)=e^{i(\pi+\theta(\lambda,V))}\frac{a(\lambda,V)-re^{-i\theta(\lambda,V)}}{1-ra(\lambda,V)e^{i\theta(\lambda,V)}},$
(3)

where $a$ the voltage (V) dependent absorption (or loss) in the cavity lumped
with intrinsic cavity loss, $\theta$ is the wavelength- and voltage-dependent
phase shift, $r$ is the self-coupling coefficient (with a corresponding cross
coupling coefficient, $\kappa$, such that $\kappa^{2}+r^{2}=1$). Equation 3
depicts the coupled effect of tuning, which both changes the loss and the
relative cavity size (i.e. phase shift).

In this work, we employ carrier injection-based ring resonators through the
well-known free carrier plasma dispersion effect in silicon. These rings
demonstrate a large spectral tuning range, as seen in Figure 6(a). However,
this shift comes at the cost of change of coupling conditions due to the
strongly voltage-dependent round-trip loss. Changing the coupling conditions
from over coupled to critically coupled to under coupled has a large impact on
the phase shift associated the ring. It is precisely this phase that we aim to
leverage in this architecture. Therefore, we operate the rings in the
slightly-to-moderately under coupled regime in this experiment, which
corresponds to voltages $V>1.1~{}V$, as seen in Figure 6(b).

Figure 6:  (a) The spectral behavior of the ring modulator used in our
demonstration. As the voltage increases, the resonant wavelength blue-shifts
to a lower wavelength. In addition, we see that the ring traverses the initial
condition (over-coupled), to critical coupled, finally to under-coupled. This
ring was selected due to the large tuning range for initial demonstration. (b)
Optical and electrical characterization of the ring. A single wavelength,
$\lambda-\lambda_{0}=0$ from (a), is plotted to show the change in optical
transmission as a function of applied bias. From an electrical perspective, we
see that the operating regime for this diode is indeed in forward-bias, with a
maximum power for full tunability of approximately $12$ mW.  Figure 7: Full
implementation sequence for the hidden layers of the OR gate. (a–d) The
circuit response when configured to the trained weights of hidden node 1 for
the incoming inputs $[0,0],[0,1],[1,0],$ and $[1,1]$. (e) Circuit output for
Hidden Node 1 at the probe wavelength, with a range of $\pm 100$ pm. (a–d)
Show the circuit response when configured to the trained weights of Hidden
Node 2 for the incoming inputs $[0,0],[0,1],[1,0],$ and $[1,1]$. (e) Circuit
output for Hidden Node 2 at the probe wavelength, with a range of $\pm 100$
pm.

Additionally, this demonstration did not employ thermal tuners to shift the
rings coarsely. This leaves the demonstration susceptible to fabrication
variations. For large-scale demonstrations, these thermal phase shifters can
be implemented with isolated heaters with high efficiency [35, 36] in concert
with reverse-biased MRDs for low-power electro-optic configuration [27, 37].

##  Appendix B Hidden Layer Demonstration Results

The hidden layers are not explicitly necessary in order to run the network
since the final stage weights are known. However, they are important in
demonstrating the full generality of the WDIPLN architecture, including the
ability to reconfigure. In Figure 7, we show an in-depth look at the procedure
of validating the hidden nodes, $h_{1}$ and $h_{2}$. First, we configure the
circuit to measure $h_{1},$ as shown in Figure 5(c). Figure 7(a–d) show the
optical circuit response in the Hidden 1 configuration for the four logic
inputs, with (e) showing the outputs at the probe wavelength. Figure 7(f–i)
shows the optical circuit response in the Hidden 2 configuration for the four
logic inputs, with (j) showing the outputs at the probe wavelength.

## References

## References

  * [1] Shen Y, Harris N C, Skirlo S, Prabhu M, Baehr-Jones T, Hochberg M, Sun X, Zhao S, Larochelle H, Englund D et al. 2017 Nature Photonics 11 441–446
  * [2] Totović A R, Dabos G, Passalis N, Tefas A and Pleros N 2020 IEEE Journal of Selected Topics in Quantum Electronics 26 1–15
  * [3] Shastri B J, Tait A N, Ferreira de Lima T, Pernice W H, Bhaskaran H, Wright C D and Prucnal P R 2021 Nature Photonics 15 102–114
  * [4] Nahmias M A, De Lima T F, Tait A N, Peng H T, Shastri B J and Prucnal P R 2019 IEEE Journal of Selected Topics in Quantum Electronics 26 1–18
  * [5] Cheng Q, Kwon J, Glick M, Bahadori M, Carloni L P and Bergman K 2020 Proceedings of the IEEE 108 1261–1282
  * [6] Nahmias M A, Shastri B J, Tait A N and Prucnal P R 2013 IEEE journal of selected topics in quantum electronics 19 1–12
  * [7] Shastri B J, Tait A N, Nahmias M A and Prucnal P R 2014 arXiv preprint arXiv:1407.2917
  * [8] Cheng Z, Ríos C, Pernice W H, Wright C D and Bhaskaran H 2017 Science advances 3 e1700160
  * [9] Wu C, Yu H, Lee S, Peng R, Takeuchi I and Li M 2021 Nature communications 12 1–8
  * [10] Feldmann J, Youngblood N, Karpov M, Gehring H, Li X, Stappers M, Le Gallo M, Fu X, Lukashchuk A, Raja A S et al. 2021 Nature 589 52–58
  * [11] Gholipour B, Bastock P, Craig C, Khan K, Hewak D and Soci C 2015 Advanced Optical Materials 3 635–641
  * [12] Sun J, Oh S, Choi Y, Seo S, Oh M J, Lee M, Lee W B, Yoo P J, Cho J H and Park J H 2018 Advanced Functional Materials 28 1804397
  * [13] Hamerly R, Bernstein L, Sludds A, Soljačić M and Englund D 2019 Physical Review X 9 021032
  * [14] Shi B, Calabretta N and Stabile R 2019 IEEE Journal of Selected Topics in Quantum Electronics 26 1–11
  * [15] Fouskidis D E, Zoiros K E and Hatziefremidis A 2020 IEEE Journal of Selected Topics in Quantum Electronics 27 1–15
  * [16] Zhang H, Gu M, Jiang X, Thompson J, Cai H, Paesani S, Santagati R, Laing A, Zhang Y, Yung M et al. 2021 Nature Communications 12 1–11
  * [17] Mourgias-Alexandris G, Totović A, Tsakyridis A, Passalis N, Vyrsokinos K, Tefas A and Pleros N 2019 Journal of Lightwave Technology 38 811–819
  * [18] Totovic A, Giamougiannis G, Tsakyridis A, Lazovsky D and Pleros N 2022 Scientific Reports 12 1–13
  * [19] Tait A N, Nahmias M A, Shastri B J and Prucnal P R 2014 Journal of Lightwave Technology 32 3427–3439
  * [20] Tait A N, De Lima T F, Zhou E, Wu A X, Nahmias M A, Shastri B J and Prucnal P R 2017 Scientific reports 7 1–10
  * [21] Williamson I A, Hughes T W, Minkov M, Bartlett B, Pai S and Fan S 2019 IEEE Journal of Selected Topics in Quantum Electronics 26 1–12
  * [22] Tait A N, De Lima T F, Nahmias M A, Miller H B, Peng H T, Shastri B J and Prucnal P R 2019 Physical Review Applied 11 064043
  * [23] Jha A, Huang C and Prucnal P R 2020 Optics Letters 45 4819–4822
  * [24] Mourgias-Alexandris G, Tsakyridis A, Passalis N, Tefas A, Vyrsokinos K and Pleros N 2019 Optics express 27 9620–9630
  * [25] Giamougiannis G, Tsakyridis A, Mourgias-Alexandris G, Moralis-Pegios M, Totovic A, Dabos G, Passalis N, Kirtas M, Bamiedakis N, Tefas A et al. 2021 Silicon-integrated coherent neurons with 32gmac/sec/axon compute line-rates using eam-based input and weighting cells 2021 European Conference on Optical Communication (ECOC) (IEEE) pp 1–4
  * [26] Mourgias-Alexandris G, Totovic A, Passalis N, Dabos G, Tefas A and Pleros N 2020 Neuromorphic computing through photonic integrated circuits Smart Photonic and Optoelectronic Integrated Circuits XXII vol 11284 (International Society for Optics and Photonics) p 1128403
  * [27] Timurdogan E, Sorace-Agaskar C M, Sun J, Shah Hosseini E, Biberman A and Watts M R 2014 Nature communications 5 1–11
  * [28] Fahrenkopf N M, McDonough C, Leake G L, Su Z, Timurdogan E and Coolbaugh D D 2019 IEEE Journal of Selected Topics in Quantum Electronics 25 1–6
  * [29] Giewont K, Nummy K, Anderson F A, Ayala J, Barwicz T, Bian Y, Dezfulian K K, Gill D M, Houghton T, Hu S, Peng B, Rakowski M, Rauch S, Rosenberg J C, Sahin A, Stobert I and Stricker A 2019 IEEE Journal of Selected Topics in Quantum Electronics 25 1–11
  * [30] Littlejohns C G, Rowe D J, Du H, Li K, Zhang W, Cao W, Dominguez Bucio T, Yan X, Banakar M, Tran D et al. 2020 Applied Sciences 10 8201
  * [31] Paszke A, Gross S, Massa F, Lerer A et al. 2019 Pytorch: An imperative style, high-performance deep learning library Advances in Neural Information Processing Systems 32 ed Wallach H, Larochelle H, Beygelzimer A, d'Alché-Buc F, Fox E and Garnett R (Curran Associates, Inc.) pp 8024–8035
  * [32] Gaeta A L, Lipson M and Kippenberg T J 2019 Nature Photonics 13 158–169
  * [33] Rizzo A et al. 2021 arXiv:2109.10297
  * [34] Soref R and Bennett B 1987 IEEE journal of quantum electronics 23 123–129
  * [35] Coenen D, Oprins H, Ban Y, Ferraro F, Pantouvaki M, Van Campenhout J and De Wolf I 2022 Journal of Lightwave Technology
  * [36] van Niekerk M, Deenadalayan V, Rizzo A et al. 2022 Wafer-scale-compatible substrate undercut for ultra-efficient soi thermal phase shifters Submitted and Accepted to CLEO: Science and Innovations (Optical Society of America)
  * [37] van Niekerk M, Cheng C et al. 2022 High extinction micro-ring modulator Submitted and Accepted to CLEO: Science and Innovations (Optical Society of America)


Shuo Liu1,2, Leda Sarı1, Chunyang Wu1, Gil Keren1, Yuan Shangguan1,
Jay Mahadeokar1, Ozlem Kalinli1

# Towards Selection of Text-to-speech Data to Augment ASR Training

###### Abstract

This paper presents a method for selecting appropriate synthetic speech
samples from a given large text-to-speech (TTS) dataset as supplementary
training data for an automatic speech recognition (ASR) model. We trained a
neural network, which can be optimised using cross-entropy loss or Arcface
loss, to measure the similarity of a synthetic data to real speech. We found
that incorporating synthetic samples with considerable dissimilarity to real
speech, owing in part to lexical differences, into ASR training is crucial for
boosting recognition performance. Experimental results on Librispeech test
sets indicate that, in order to maintain the same speech recognition accuracy
as when using all TTS data, our proposed solution can reduce the size of the
TTS data down below its $30\,\%$, which is superior to several baseline
methods.

Index Terms: Data augmentation, text-to-speech, automatic speech recognition,
synthetic speech, Transducer

††footnotetext:
Work was done when Shuo was interning at Meta AI.

##  1 Introduction

Speech synthesis, or Text-to-Speech (TTS), is a fast developing technology
that generates speech from given text. It has reached the stage that it can
create audio that closely resembles natural human voice, allowing for the use
of synthetic data to improve the training of Automatic Speech Recognition
(ASR) models [1, 2, 3]. Ideally, creating more synthetic data for ASR training
has always the potential to improve the accuracy of speech recognition.
However, as the quantity of the training data grows, more training time and
computational resources are demanded. On the other hand, since TTS speech data
are often produced with limited alternative constrains in terms of, for
instance, speakers, speech speed and pitch, the resultant synthetic dataset
may include a substantial amount of redundancy. Hence, it should be possible
to minimise the data size by only choosing more typical and representative
samples in a given large TTS dataset to achieve a more cost-efficient ASR
training.

Recent research [4, 5] have demonstrated some data selection strategies that
are effective for this data reduction purpose. In [4], it is suggested to
exploit rejection sampling to accept or reject synthetic samples such that the
distribution of the selected synthetic data is close to that of the real
speech. The distribution is represented in five dimensions depending on the
outputs of a pre-trained ASR model, such as Xent loss, CTC loss, and token
lengths, etc. Thus, the chosen synthetic data should own high similarity to
actual speech. In [5], the objective is to choose data from a general pool
that reflects a domain of interest, in order to build a domain-optimised ASR
model. For this purpose, two language models are separately trained on the
data from the target domain and the general domain using discrete speech
tokens as input. A domain relevance score is computed based on the output
probabilities from the target and general domain language models. The general
domain samples scored highest are selected for training the domain specific
ASR model. Data selection is also considered essential for training natural
language processing (NLP) models efficiently [6, 7]. To empower an NLP model
with more lexical information, however, it has been advised to choose the data
of low certainty for training so that extra linguistic knowledge may be
included.

When selecting TTS data for ASR training, a question remains unanswered is
whether we should choose the data that are similar to or distinct from the
original real speech. From the acoustic perspective, we may want to select the
samples of high similarity to ensure signal quality. However, these data may
contain much less additional information and hence can only marginally improve
an ASR model; On the other hand, a TTS sample that differs significantly from
the original real speech may indicate the synthetic audio is contaminated with
noise or artefacts produced during the synthesis process, which can be
detrimental to the ASR training. To choose the appropriate TTS data for ASR
training, our methodology compromises the similarity and discrepancy of the
synthetic and the original real speech. In particular, we use a basic
recurrent neural network, a Gated Recurrent Unit (GRU), to generate speech
embedding. Then, we compare the embeddings of the real and synthetic speech
samples and use two scoring methods to reflect their agreement. For both
scoring methods, when selecting the TTS data inside a certain scoring range,
we can lower the needed TTS data size while preserving the same level of ASR
performance as using all available TTS data.

Figure 1: System overview

.

##  2 Methodology

The system overview used in this work is given in Figure 1. We generate
synthetic speech files from additional text resources using a TTS model. A
scoring model is then trained to select some of these audio files that
facilitate ASR training. These files are then added to the original real
dataset to train an ASR model. For the test phase, we use only real test data.

To develop the scoring model, we train a two-layer Gated Recurrent Unit (GRU)
model to extract speech representation from an input audio file. The output of
the last time-step is fed into a fully-connected (FC) layer, which is then
optimised using binary cross-entropy (BCE) or Arcface loss [8]. When using the
BCE loss, the FC output is linearly projected to the two classes, real or
synthetic speech, through an additional FC layer. Arcface loss adds a margin
parameter to the angle between the normalised speech representation and the
weights of the FC layer that is implemented in the loss function. The
insertion of the margin tends to increase the separability between speech
representations of distinct classes.

Real speech is regarded as class 1 and synthetic speech as class 0 in the
training of the GRU model using BCE loss. After training, we calculate the
Softmax value of the output logit to score an input audio, and the score of
0.5 can be seen as the threshold for distinguishing between a real or
synthetic speech. If the score exceeds $0.5$, the input is recognised as a
real data, otherwise it is classified as a TTS data. Hence, when a TTS
sample's score is closer to $1$, it is more likely to be identified as real
data, indicating its higher similarity to real speech. However, as mentioned
earlier, a dissimilar TTS sample may provide extra information needed to
enhance ASR training. Therefore, it is necessary, however challenging, to find
a proper scoring range for selecting the appropriate TTS data.

A more straightforward approach to quantify the similarity between real and
synthetic speech is to compute the cosine similarity of their speech
representations. In an attempt to improve the representation separability,
Arcface loss is applied to the GRU model's optimisation. Based on the real
samples in training set, we next calculate the average embedding for real
speech, named as average real embedding. To score a TTS sample, the cosine
similarity between its embedding and the average real embedding is computed.
Again, the TTS samples scored with low similarity to real speech may signal
the presence of new information that has the chance to contribute to the ASR
training. Notably, since our proposed method exploits only a simple neural
network, it allows the rapid scoring of synthetic speech samples compared to
previous work that typically relies on a pre-trained ASR model, particularly
when the size of the involved TTS dataset is enormous.

##  3 Experiments

###  3.1 Dataset

Our scoring models are trained based on the LibriSpeech corpus [9]. It
consists of around 960 hours of read, clean speech derived from over 8 000
public domain audiobooks and has its own train, development, and test splits.
We apply the TTS model described in Section 3.2 to the Librispeech
transcriptions to generate the synthetic audio files. During training, the
performance of the scoring models are monitored on the Librispeech development
set without touching the test set. As additional text resources to generate
the synthetic speech files for ASR training, we employ $10\,\%$ of data from
the language model (LM) corpus provided in Librispeech corpus, yielding a
total of 4.11 million utterances.

###  3.2 TTS & ASR Models

The TTS model has a multi-stage framework. It begins with a rule-based
Grapheme-to-Phoneme (G2P) Conversion module that transforms the input text
into various levels of linguistic representations. These features are consumed
by a prosody model [10] and a spectrum model [11] to generate spectrum
features. The prosody model consists of multiple Transformer layers [12], with
each layer contains a multi-head attention module and a feed-forward layer. In
addition, residual connection and layer normalisation are performed to each
transformer layer. Finally, a neural vocoder using WaveRNN achitecture [13] is
used to transform the features into audio waveform.

The ASR model used in this work processes a log Mel-spectrogram with $80$ Mel
bands, the spectrogram is created by taking the Short-time Fourier transfrom
(STFT) of a speech signal with a window size of $25$ms and hop size of $10$ms.
SpecAugment is applied to strengthen the robustness of the ASR model [14]. The
model has the output vocabulary of $5000$ sentence pieces [15] estimated over
the $960$-hours Librispeech training set [9] (LS 960h). The Mel-spectrogram is
linearly projected to $128$ channels using an encoder, and every four time
steps are concatenated along the feature dimensionality to squeeze the total
frames. Then a stack of $20$ efficient memory transformer (Emformer [16])
layers are used to capture the temporal context. The Emformer output is then
fed into a recurrent neural transducer (RNN-T). Using a 3-layer LSTM model
[17], the Emformer predictor generates embeddings of the size of $512$ based
on all the previous predicted symbols. The outputs of the Emformer encoder and
predictor are combined and projected to a probability distribution over a
vocabulary.

Table 1: Testing results, WER [$\%$], of different data selection approaches on Librispeech test set. # add. utter indicates the number of TTS samples selected to be added to Librispeech 960h (LS) training data. The proportion of utterances selected from the synthetic samples generated based on LM text database are additionally given, where ``high'' and ``low'' denote the TTS samples are selected from those with highest or lowest scores, respectively. The binary classifiers are trained using Cross-entropy (Xent) or Arcface loss. Method | # add. utter. [M] | dev-clean | test-clean | dev-other | test-other
---|---|---|---|---|---
LS | $-$ | $3.2$ | $3.5$ | $9.2$ | $8.7$
LS + all synthetic speech |  $4.11$ ($100\,\%$) | $2.67$ | $2.97$ | $8.39$ | $8.05$
Random |  $1.23$ ($30\,\%$) | $2.87$ | $3.21$ | $8.63$ | $8.63$
Confidence score |  $1.23$ ($30\,\%$ high) | $3.10$ | $3.24$ | $8.90$ | $8.65$
$1.23$ ($30\,\%$ low) | $2.77$ | $3.05$ | $8.46$ | $8.28$
ULM accuracy |  $1.50$ ($36\,\%$ high) | $2.72$ | $2.97$ | $8.63$ | $8.22$
$1.23$ ($30\,\%$ low) | $2.91$ | $3.13$ | $8.58$ | $8.31$
ULM perplexity |  $1.50$ ($36\,\%$ high) | $2.87$ | $2.99$ | $8.96$ | $8.35$
$1.23$ ($30\,\%$ low) | $2.76$ | $3.01$ | $8.39$ | $8.20$
Binary classifier (Xent) |  $1.10$ ($27\,\%$) | $2.78$ | $2.96$ | $8.42$ | $8.16$
Binary classifier (Arcface) |  $1.23$ ($30\,\%$) | $2.81$ | $2.97$ | $8.46$ | $8.09$

###  3.3 Settings

For a fair comparison of different data selection methods, the training
parameters for all the ASR models developed throughout this work are
consistent. The models are optimised by minimising the transducer loss using
an Adam optimiser with an initial learning rate of $0.001$. The total number
of training epochs is $200$, and beginning with the $60$ epoch, the learning
rate annealing is applied at a shrink rate of $0.96$ per epoch. The maximum
number of tokens in a training batch is capped to $40000$ with the number of
utterances not exceeding $1000$. The ASR models are trained using $32$ A100
GPU units.

Both layers of our GRU scoring model have hidden units of $256$. The
dimensionality of the subsequent fully-connected layer is set to $64$ when the
model is trained with BCE loss. Its optimisation employs a batch size of $64$
and Adam optimiser with a fixed learning rate of $0.0001$. When using Arcface
loss for training, the scale and margin parameters are set to $10$ and $0.5$,
respectively. A lower learning rate of $0.00001$ is applied to ensure stable
convergence.

###  3.4 Baseline Methods

As the primary comparison result, we first evaluate the effect of adding all
available TTS data to the original Librispeech training set as the new
training data. In addition, we compare our data selection approach with three
baseline methods: random selection, a method depending on confidence score,
and an acoustic unit language model-based scoring method. Confidence score
represents the joiner score of the word-pieces making up the words in an
utterance. We consider the selection of TTS samples with the highest and
lowest confidence scores for ASR training. We compare our methods with these
baseline methods on the test sets of Librispeech in terms of word error rate
(WER).

The scoring method using an acoustic unit language model (ULM) is inspired by
[5]. The acoustic units are obtained by a Hidden-Unit BERT (HuBERT) model [18]
which is a self-supervised speech model. To emulate a written language having
a finite vocabulary of discrete units, a HuBERT model produces the discrete
speech representation by directly processing a speech waveform. The pseudo
labels for HuBERT training are created by performing unsupervised clustering,
such as K-means, on the MFCCs of the input signal. Hence, the learnt discrete
speech representation is optimised to be close to the centroid of its
belonging cluster, and far from other cluster centroids. It has been shown
that fine-tuning a HuBERT model for ASR can reach state-of-the-art results
[19]. In this work, we deploy a HuBERT model trained using K-means clustering
method with $K=100$. A 13-layer neural LM is trained to predict the next
HuBERT unit for a given sequence of input HuBERT units. Please note that this
LM is not a conventional text-based LM but one which is trained on discrete
speech units and hence, it has the ability to model the acoustic properties.
In order to obtain a score per utterance, we used either the average next unit
prediction accuracy or the perplexity.

Table 2: Informativeness Comparison in three scoring range using our binary classifier trained with cross-entropy or Arcface loss. Method | Scoring range | # add. utter. [M] | dev-clean | test-clean | dev-other | test-other
---|---|---|---|---|---|---
| $Score>0.5$ | $0.77$ ($18\,\%$) | $3.06$ | $3.20$ | $8.95$ | $8.58$
Binary classifier | $0.2<Score<0.5$ | $2.85$ | $3.11$ | $8.67$ | $8.17$
(Xent) | $Score<0.2$ | $1.10$ ($27\,\%$) | $2.80$ | $3.13$ | $8.48$ | $8.42$
| $0.2<Score<0.5$ | $2.78$ | $2.96$ | $8.42$ | $8.16$
| $Similarity>0.8$ | $1.22$ ($30\,\%$) | $2.85$ | $3.04$ | $8.60$ | $8.34$
Binary classifier | $0.2<Similarity<0.8$ | $2.81$ | $2.97$ | $8.46$ | $8.09$
(Arcface) | $Similarity<0.2$ | $0.77$ ($18\,\%$) | $2.96$ | $3.25$ | $9.03$ | $8.45$
| $0.2<Similarity<0.8$ | $2.91$ | $3.00$ | $8.39$ | $8.19$
Table 3: Number of words existing in Librispeech evaluation set, but not in training set, in terms of different scoring range from binary classifier trained with cross-entropy loss. Number of TTS sentences that contain these words. Scoring range | # add. | Words |  | Utterances
---|---|---|---|---
utter. [M] | dev-clean | test-clean | dev-other | test-other |  | dev-clean | test-clean | dev-other | test-other
$Score>0.5$ |  $0.77$ ($18\,\%$) | $48$ | $46$ | $49$ | $66$ |  | $290$ | $322$ | $234$ | $378$
$0.2<Score<0.5$ |  $2.24$ ($55\,\%$) | $157$ | $151$ | $156$ | $182$ |  | $1073$ | $1191$ | $927$ | $1285$
$Score<0.2$ |  $1.10$ ($27\,\%$) | $133$ | $108$ | $118$ | $149$ |  | $674$ | $711$ | $514$ | $824$
Table 4: Testing results, WER [$\%$], of classification scoring methods on Librispeech test set, after removing the LM TTS utterances containing unseen words. Method | Unseen | dev-clean | test-clean | dev-other | test-other
---|---|---|---|---|---
words
Bin. cls. | ✗ | $2.90$ | $3.09$ | $8.74$ | $8.30$
(Xent) | ✔ | $2.78$ | $2.96$ | $8.42$ | $8.16$
Bin. cls | ✗ | $2.88$ | $3.07$ | $8.65$ | $8.20$
(Arcface) | ✔ | $2.81$ | $2.97$ | $8.46$ | $8.09$
Table 5: Testing results, WER [$\%$], by using the TTS data selected by both ULM and our data selection approaches. Method | # add. | dev-clean | test-clean | dev-other | test-other
---|---|---|---|---|---
utter. [M]
ULM accuracy | $0.72$ | $2.90$ | $3.02$ | $8.81$ | $8.23$
& Bin. cls. (Xent) |  ($18\,\%$)
ULM accuracy | $0.71$ | $2.89$ | $3.05$ | $8.83$ | $8.26$
& Bin. cls. (Arcface) |  ($17\,\%$)
LM accuracy | $0.65$ | $2.89$ | $3.06$ | $8.67$ | $8.28$
& Bin. cls. (Xent) | ($16\,\%$)
& Bin. cls. (Arcface)

###  3.5 Results

The ASR model trained using the original Librispeech training set achieves a
WER of $3.5\,\%$ on the test-clean split and $8.7\,\%$ on the test-other split
(Table 1). Adding all of our generated TTS data, which consists of $4.11$
million synthetic speech files, as additional training material decreases the
WER results on the evaluation sets to $2.97\,\%$ and $8.05\,\%$, respectively.
We assess our scoring methods for selecting a subset of the TTS audio files
for ASR training while aiming to keep the same effectiveness of WER reduction.
With the allowance of $2\,\%$ rising rate, we expect the resulting WER on
test-clean to be lower than $3.03\,\%$, and on the test-other to be lower than
$8.24\,\%$

As seen in Table 1, the data selection methods based on ULM can produce
superior results than the other two baseline methods, namely random selection
and confidence score-based scoring. Using the average next unit prediction
perplexity as the criterion for data selection, $30\,\%$ of the synthetic
audio files with the lowest scores yields a WER of $3.01\,\%$ on the test-
clean set and $8.20\,\%$ on the test-other set. To attain the same level of
performance, however, we need to incorportate $1.5$ million TTS samples with
the highest accuracy scores when choosing the data based on the average
prediction accuracy. In particular, the trained ASR can reach the same WER on
the test-clean set as using all of the TTS data available, and a WER of $8.22$
on the test-other split.

Instead of choosing the highest or lowest scored samples, as was the case with
baseline methods, we discovered that our binary classification-based scoring
approaches perform best when employing the TTS data within a medium scoring
range. In other words, the synthetic audio files that can be differentiated
from real speech samples are more beneficial for improving the ASR
performance. After training our GRU scoring model using BCE loss, we acquire
an unweighted average recall of $92\,\%$ classification accuracy on the
Librispeech validation set. In particular, the recall score reaches
approximately $100\,\%$ for real speech (class 1), and $83\,\%$ for synthetic
speech (class 0), demonstrating that almost all the real speech samples can be
identified accurately while some of the TTS samples are incorrectly recognised
as real speech. The scoring model's prediction output are then used to score
the TTS samples created from LM text resources. The WER can be reduced to
$2.96\,\%$ and $8.16\,\%$ for the test-clean and test-other sets,
respectively, by choosing only $27\,\%$ of the synthetic audio files rated
between 0.2 and 0.5. Similar to this, when using Arcface loss to train the GRU
model, we can successfully select TTS data that yields an even better result
on the test-other set, a WER of $8.09\,\%$, by selecting $30\,\%$ of all TTS
audio files which have similarity scores within the range of 0.2 to 0.8.

##  4 Discussion

To support our recommended scoring ranges for TTS data selection, i.e., score
between $0.2$ and $0.5$ for the GRU scoring model trained with BCE loss,
similarity between $0.2$ to $0.8$ similarity for the scoring model trained
with Arcface loss, we compare their effectiveness to the data scored outside
the range. Specifically, for the first scoring model, we randomly select
synthetic samples from those scored between $0.2$ and $0.5$ as the same size
of the samples scored greater than 0.5, and compare their effectiveness in ASR
improvement. Similar comparison has been made for the TTS samples with scores
under $0.2$. The random selection was performed $5$ times for each comparison,
and the averaged outcomes, shown in Table 2, reveal that the TTS samples from
our suggested scoring range are more effective in ASR improvement compared to
the top- and bottom-scoring samples. However, the samples scored below $0.2$
are also not very helpful for ASR, as shown in Table 2. This lower-score
boundary has been established experimentally; however, its determination can
be challenging in practice. Thus, it is suggested to simply choose the
highest-scoring TTS samples right below $0.5$. Similarly, for the second
scoring model, the TTS samples with too high or too low similarity to real
speech are not as competitive as the samples with similarity between $0.2$ and
$0.8$ in terms of ASR improvement. Overall, we are able to optimize the ASR
performance using TTS samples while excluding samples that are too similar or
too dissimilar to real speech.

The results are partly due to the inclusion of out-of-vocabulary words in LM
texts like Table 2, supporting the idea that we should select TTS data with
enough uncertainty as additional ASR training material.

These results are partially attributable to the incorporation of some extra
unseen words given by LM text resources, and as a result, the selected data
provide more additional information to benefit ASR training. In Table 3, we
list the number of out-of-vocabulary words that are included in the evaluation
splits of the Librispeech dataset and added to the new training set due to LM
texts. An utterance containing these unseen words may have a lower similarity
to the original training samples, but including them in ASR training can help
to improve speech recognition accuracy. If the selected utterances containing
these unseen words are filtered out for other utterances without them in the
same scoring range, the results provided in Table 4 show that the speech
recognition performance for both of our scoring methods declines to some
extent.

In an effort to further reduce the required TTS data size, we additionally
investigate the combination of our classification-based scoring method with
the ULM scoring method . Only the data chosen by both approaches is used to
train ASR models, and the best-performing combination is given in Table 5.
Combining ULM accuracy and a binary classifier (Xent) retains the WER results
of $3.02$ and $8.23$ on the test-clean and test-other sets while only
requiring $0.72$ million of TTS samples, or about $18\,\%$ of the total
generated TTS data. However, the WER rise on the development set raises the
question of the validity of this ASR model in practical use.

##  5 Conclusions

In this paper, we presented a solution for selecting useful samples from a
given large TTS dataset to augment the ASR training data. Our method uses a
simple neural network to produce agreement between real and synthetic speech.
Experimental results indicated that the ASR benefits from choosing synthetic
data that are of sufficient additional information or uncertainty, resulting
in better speech recognition performance compared to other existing methods
that consider the maximum or minimum amount of agreement.

## References

  * [1] A. Rosenberg, Y. Zhang, B. Ramabhadran, Y. Jia, P. Moreno, Y. Wu, and Z. Wu, ``Speech recognition with augmented synthesized speech,'' in _IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_ , Sentosa,Singapore, 2019, pp. 996–1002.
  * [2] G. Wang, A. Rosenberg, Z. Chen, Y. Zhang, B. Ramabhadran, Y. Wu, and P. Moreno, ``Improving speech recognition using consistent predictions on synthesized speech,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , Barcelona,Spain, 2020, pp. 7029–7033.
  * [3] S. Ueno, M. Mimura, S. Sakai, and T. Kawahara, ``Data augmentation for asr using tts via a discrete representation,'' in _IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_ , Cartagena,Colombia, 2021, pp. 68–75.
  * [4] T. Hu, M. Armandpour, A. Shrivastava, J. R. Chang, H. Koppula, and O. Tuzel, ``Synt++: Utilizing imperfect synthetic data to improve speech recognition,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , Singapore,Singapore, 2022, pp. 7682–7686.
  * [5] Z. Lu, Y. Wang, Y. Zhang, W. Han, Z. Chen, and P. Haghani, ``Unsupervised data selection via discrete speech representation for ASR,'' _arXiv preprint arXiv:2204.01981_ , 2022.
  * [6] B. Settles and M. Craven, ``An analysis of active learning strategies for sequence labeling tasks,'' in _Conference on Empirical Methods in Natural Language Processing_ , 2008, pp. 1070–1079.
  * [7] R. Hazra, P. Dutta, S. Gupta, M. A. Qaathir, and A. Dukkipati, ``Active2 learning: Actively reducing redundancies in active learning methods for sequence tagging and machine translation,'' _arXiv preprint arXiv:1911.00234_ , 2019.
  * [8] J. Deng, J. Guo, N. Xue, and S. Zafeiriou, ``Arcface: Additive angular margin loss for deep face recognition,'' in _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , Long Beach, CA, USA, 2019, pp. 4685–4694.
  * [9] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, ``Librispeech: an asr corpus based on public domain audio books,'' in _2015 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 2015, pp. 5206–5210.
  * [10] Q. He, Z. Xiu, T. Koehler, and J. Wu, ``Multi-rate attention architecture for fast streamable text-to-speech spectrum modeling,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , Toronto,Canada, 2021, pp. 5689–5693.
  * [11] C. Wu, Z. Xiu, Y. Shi, O. Kalinli, C. Fuegen, T. Koehler, and Q. He, ``Transformer-Based Acoustic Modeling for Streaming Speech Synthesis,'' in _Proc. Interspeech 2021_ , Shanghai,China, 2021, pp. 146–150.
  * [12] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, ``Attention is all you need,'' Long Beach, CA, USA, 2017.
  * [13] N. Kalchbrenner, E. Elsen, K. Simonyan, S. Noury, N. Casagrande, E. Lockhart, F. Stimberg, A. van den Oord, S. Dieleman, and K. Kavukcuoglu, ``Efficient neural audio synthesis,'' in _International Conference on Machine Learning (ICML)_ , 2018, pp. 2410–2419.
  * [14] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, ``Specaugment: A simple data augmentation method for automatic speech recognition,'' _arXiv preprint arXiv:1904.08779_ , 2019.
  * [15] T. Kudo and J. Richardson, ``Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing,'' _arXiv preprint arXiv:1808.06226_ , 2018.
  * [16] Y. Shi, Y. Wang, C. Wu, C.-F. Yeh, J. Chan, F. Zhang, D. Le, and M. Seltzer, ``Emformer: Efficient memory transformer based acoustic model for low latency streaming speech recognition,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2021, pp. 6783–6787.
  * [17] S. Hochreiter and J. Schmidhuber, ``Long short-term memory,'' _Neural computation_ , vol. 9, no. 8, pp. 1735–1780, 1997.
  * [18] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, ``Hubert: Self-supervised speech representation learning by masked prediction of hidden units,'' _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 3451–3460, 2021.
  * [19] S. wen Yang, P.-H. Chi, Y.-S. Chuang, C.-I. J. Lai, K. Lakhotia, Y. Y. Lin, A. T. Liu, J. Shi, X. Chang, G.-T. Lin, T.-H. Huang, W.-C. Tseng, K. tik Lee, D.-R. Liu, Z. Huang, S. Dong, S.-W. Li, S. Watanabe, A. Mohamed, and H. yi Lee, ``SUPERB: Speech processing universal performance benchmark,'' in _Proc. Interspeech_ , Brno,Czech Republic, 2021, pp. 1194–1198.