# Extracting the Speed of Light from Matrix Product States

Alexander A. Eberharter  Institut für Theoretische Physik, Universität
Innsbruck, A-6020 Innsbruck, Austria  Laurens Vanderstraeten  Department of
Physics and Astronomy, University of Ghent, Belgium  Center for Nonlinear
Phenomena and Complex Systems, Université Libre de Bruxelles, Brussels,
Belgium  Frank Verstraete  Department of Physics and Astronomy, University of
Ghent, Belgium  Department of Applied Mathematics and Theoretical Physics,
University of Cambridge, Cambridge, United Kingdom  Andreas M. Läuchli
Laboratory for Theoretical and Computational Physics, Paul Scherrer Institute,
5232 Villigen, Switzerland  Institute of Physics, École Polytechnique Fédérale
de Lausanne (EPFL), 1015 Lausanne, Switzerland

###### Abstract

We provide evidence that the spectrum of the local effective Hamiltonian and
the transfer operator in infinite-system matrix product state simulations are
identical up to a global rescaling factor, i.e. the speed of light of the
system, when the underlying system is described by a 1+1 dimensional CFT. We
provide arguments for this correspondence based on a path integral point of
view. This observation turns out to yield very precise estimates for the speed
of light in practice, confirming exact results to high precision where
available, but also allowing us to finally determine the speed of light of the
non-integrable, critical $SU(2)$ Heisenberg chains with half-integer spin
$S>1/2$ with unprecedented accuracy. We also show that the same technology
applied to doped Hubbard ladders provides highly accurate velocities for a
range of dopings. Combined with measurements of compressibilities we present
new results for the Luttinger liquid parameter in the Luther-Emery regime of
doped Hubbard ladders, outperforming earlier approaches based on the fitting
of real-space correlation functions.

## Introduction —

Relativistic invariance is a property that quantum many body phases or
critical points often exhibit in the low-energy or long-wavelength limit. The
speed of light (or speed of sound) translates between momentum and energies
$E(k)=v|k|$ for massless systems ($\hbar\equiv 1$). This family of systems
encompasses Goldstone phases corresponding to the symmetry breaking of
continuous symmetries, and quantum critical points in low dimensions with a
dynamical critical exponent $z=1$, in particular conformal field theories
(CFTs).

The speed of light enters many important quantities in quantum many body
systems, such as the Casimir effect of the ground state energy density [1, 2],
the low-temperature behaviour of the specific heat [1, 2], formulas for the
critical exponents of Luttinger liquids [3], or the prefactor in the finite-
size energy spectrum of conformal field theories, to name a few. In the past
an accurate numerical extraction of the velocity of a non-integrable quantum
many body system was rather tedious and often not very precise. For example,
before the discovery of the relation between the entanglement entropy and the
central charge, the latter was estimated using an analysis of the Casimir
correction of the ground state energy density. In that formula the speed of
light also enters, and estimates of the velocity were notoriously difficult,
thus hampering also the precision of the extracted central charge. In recent
years, using quantum Monte-Carlo [4, 5] or tensor-network methods [6, 7],
dispersion relations can be computed with some precision, but reaching high
accuracy on the velocity for strongly correlated models remains challenging.

On a different line of research, the study of 1+1 dimensional CFTs using
entanglement measures has become a very active field. Over time it has been
demonstrated or observed that in many places glimpses of the CFT appear. The
entanglement entropy of a subsystem is governed by the central charge $c$ [8]
and the entanglement spectrum is described by the spectrum of an appropriate
boundary CFT [9, 10]. This spectrum can also be observed in the corner
transfer matrix of two-dimensional classical statistical mechanics models
[11].

The critical ground-state properties of one-dimensional lattice models can be
simulated accurately using the formalism of (infinite) matrix product states
(MPS). Although the finite bond dimension of the MPS always induces a cutoff
on the entanglement in a critical model, the critical data can be extracted
from a finite-entanglement scaling that is ruled by the properties of the
underlying CFT. For example, the scaling of the entanglement entropy was shown
to be solely determined by the central charge [12, 13, 14, 15]. In this
finite-entanglement scaling, the MPS transfer matrix seems to play an
important role, as it encodes the structure of all effective length scales
[16, 17] and correlation functions, including the structure of the low-lying
excitations in the system [18]. On the other hand, for finite-size systems the
local effective Hamiltonian was shown to be identical to the spectrum of the
finite-size spectrum [19], which yields a very powerful method for extracting
critical data [20]. Nonetheless, the detailed structure of both the transfer
matrix and the effective Hamiltonian in the regime of finite-entanglement
scaling are not understood.

In this paper we now provide strong evidence that these two spectra are in
fact identical up to a global rescaling factor, which we identify as the speed
of light of the system when the underlying system is described by a 1+1
dimensional CFT. Apart from its importance as a conceptual result, this
finding offers a long-sought and very precise approach to measure the speed of
light in MPS simulations.

Figure 1: Tensor network diagrams for the effective Hamiltonian (a) and the
MPS transfer matrix (b) for an infinite MPS consisting of a two-site unit cell
formed by the tensors $A$ and $B$. $h_{0}$ and $h_{1}$ are the tensors of the
MPO which represents the Hamiltonian, whereas $L$ and $R$ are the left and
right boundary tensors of the MPS. (c) The MPS path-integral representation
with temporal and spatial transfer matrices (for a one-site unit cell); the
effective Hamiltonian and MPS transfer matrix appear as truncated versions of
these two transfer matrices.

## Infinite system matrix product states —

In the following we study one-dimensional translation invariant lattice
systems with Hamiltonian $H$, which we assume can be presented by a matrix
product operator (MPO) [21]. For illustration, we consider the situation that
the ground state of $H$ can be approximated by an infinite MPS with a two-site
unit cell ($l_{\mathrm{UC}}=2)$, with $A$ and $B$ the two MPS matrices.
Following standard MPS notation and algorithms [22, 23, 24, 25, 26] we study
the spectrum of the effective Hamiltonian $\tilde{H}$, as shown in Fig. 1(a).
The effective Hamiltonian $\tilde{H}$ is a Hermitian matrix of linear
dimension $D^{2}d^{2}$, with $D$ the MPS bond dimension and $d$ the local
dimension of the physical sites 111Note that we could also consider the
effective Hamiltonian for the single site or zero-site updates [26], for which
the dimension would be $D^{2}d$ or $D^{2}$.. While in a standard DMRG-like
update step only the ground state energy and corresponding eigenvector of
$\tilde{H}$ are needed for a subsequent update of the MPS tensors, here we
calculate a certain number of excited energies as well. We call these energies
$E_{i}$, with $E_{0}$ being the ground state energy. In the presence of
symmetries we can calculate excited energies in all relevant symmetry sectors.

In the MPS framework the transfer matrix $\tilde{T}$ of the MPS across its
unit cell (see Fig. 1(b)) is a very powerful object, as it allows to measure
the correlation length(s) and also gives access to the real space structure of
correlations [18]. The transfer operator $\tilde{T}$ is a square matrix of
linear dimension $D^{2}$. The largest eigenvalue of $\tilde{T}$ for a
normalized infinite MPS is one, whereas all other eigenvalues of $\tilde{T}$
have absolute values $|\epsilon_{i}|<1,\ i>0$. The correlations lengths are
then obtained as $\xi_{i}=-l_{\mathrm{UC}}/\ln|\epsilon_{i}|$, where we
express the correlation lengths in lattice spacings of $H$.

Our central observation is that when the Hamiltonian $H$ is tuned to a
parameter set where it is governed by a 1+1 dimensional CFT at low energies,
then

$(E_{i}-E_{0})=v/\xi_{i},\quad i>0,$ (1)

with $v$ the effective speed of light. We show examples of this observation
below and its practical power to determine the speed of light, but want to
highlight here that this property is emergent, and so far only expected to be
valid in a situation described by a CFT with a single velocity. We expect this
correspondence to be valid in the regime of finite-entanglement scaling, where
the spacings in the spectrum of both operators are due to the finite bond
dimension in the MPS approximation; for a critical model, both spectra become
continuous in the limit of infinite bond dimension. We also want to stress
that this novel way to determine the speed of light only requires to calculate
an additional set of excited state energies of the effective Hamiltonian
$\tilde{H}$ in an existing infinite MPS implementation. We suggest to first
calculate a large number of eigenvalues of both $\tilde{H}$ and $\tilde{T}$ to
check for the spectral correspondence in a new situation, but then in
principle the subleading energy and transfer matrix eigenvalue is enough to
determine the speed of light $v$ to high accuracy.

## Path-integral Viewpoint —

At first sight, it seems surprising that these two operators should have the
same low-energy spectrum. The first clue is the observation that the MPS
transfer matrix contains crucial information about the low-energy spectrum of
the Hamiltonian for which the MPS was optimized. In Ref. 18 this curious
feature was understood through the path-integral representation: The ground
state $\ket{\psi_{0}}$ of a given Hamiltonian $H$ is obtained by applying an
infinite stack of MPO-representations of $e^{-H\tau}$ (with $\tau$ a small
time step) onto a random initial state, so that $\braket{\psi_{0}}{\psi_{0}}$
can be represented as the contraction of an infinite two-dimensional tensor
network [28], see Fig. 1(c). In this tensor network, we can identify the
spatial transfer matrix $T_{x}$ as an infinite column and the temporal
transfer matrix $T_{\tau}$ as an infinite row. When the low-energy sector of
the Hamiltonian is described by a relativistic field theory, we expect that
the low-energy spectra of both transfer matrices are equivalent, up to global
factor that is the effective velocity in the system [18, 28]. When
approximating the ground state of $H$ as an MPS, we are effectively truncating
both transfer matrices [29, 30].

We can understand the effect of this truncation by first considering a fully
isotropic two-dimensional tensor network, corresponding to the partition
function of an isotropic classical stat-mech model at criticality. The
transfer matrices in the horizontal and vertical directions are now the same
(with effective velocity equal to one). In the case that the MPO tensors are
invariant under reflection and conjugation, the transfer matrix is hermitian
and we can find an MPS of the row-to-row transfer matrix through a variational
principle [31, 32]. In this case, the variational optimality condition for the
MPS tensors is equivalent to a set of fixed-point equations [33] that are
manifestly rotation invariant, which directly implies that the truncated
transfer matrices in the horizontal and the vertical directions are the same
operators, possibly up to a gauge transformation [33] that leaves the spectrum
invariant. The fact that the truncation is isotropic in this case is also
apparent in the isotropic version of the corner-transfer matrix
renormalization group [34, 35].

For anisotropic critical partition functions the exact transfer matrices are
still expected to yield the same low-energy spectrum, now with a non-trivial
effective velocity. Now the MPS fixed-point equations are no longer rotation
invariant, but the truncation is still performed in an isotropic fashion, as
it corresponds to the truncation of the singular values that reside in a
corner tensor [33]. This suggests that the effect of the truncation is still
isotropic, at least in the low-energy sector, and that we can expect the
transfer matrices in both directions to have the same low-energy spectra up to
an effective velocity. Finally, we can expect that this observation continues
to hold in the (anisotropic) Hamiltonian limit, where time becomes continuous
[28] and the logarithm of the spectrum of the truncated temporal transfer
matrix becomes the spectrum of the effective Hamiltonian as defined above.

Figure 2:  Spectra of the effective Hamiltonian and transfer matrix of the
infinite ground state MPS with bond dimension $D=128,256$ and $512$ for (a),
(b) and (c) respectively. Both spectra differ only by a single global factor
$\nu_{\textrm{(exact)}}$ which we take from exactly known velocities. The
common vertical axis is scaled using the effective Hamiltonian’s energy gap
$\Delta$, such that the first non-zero eigenvalue is set to one.

## Applications to Spin Chains —

In Fig. 2 we present our central observation, the compelling correspondence
between the low energy spectrum of the effective Hamiltonian $\tilde{H}$ as
circles and the inverse correlation lengths derived from the transfer matrix
eigenvalues as crosses, where we use the exactly known velocities
$v_{\mathrm{(exact)}}$ for a parameter-free match of the spectra. In panel
(a1) and (a2) we show data for the critical transverse-field Ising model,
without (a1) and with (a2) imposing $\mathbb{Z}_{2}$ symmetry in the MPS
tensors. In panel (b) we show $\mathbb{Z}_{3}$ symmetry resolved spectra for
the critical three state quantum Potts chain. Finally in (c) we display the
$S^{z}_{\mathrm{tot}}$ resolved spectra for the $S=1/2$ antiferromagnetic
Heisenberg chain. All the spectra show an almost perfect correspondence
between energy and transfer matrix spectrum. The range of matching eigenvalues
depends on the bond dimension $D$ of the MPS: For larger bond dimension a
larger range of eigenvalues match before deviations become visible.

We can now use the first subleading eigenvalue in each spectrum type to
directly determine the velocity from the numerical data
$v_{\mathrm{MPS}}=(E_{i}-E_{0})\times\xi_{i}$ and display the results in Tab.
1 222The symbol $J$ is used for the nearest neighbor coupling strength of the
Potts and Heisenberg models. The determined velocity for the critical Ising
model is about three orders of magnitude more accurate than previous numerical
approaches [19, 37], and about two orders of magnitude more accurate for the
$Q=3$ and $Q=4$ Potts models [37].

$Q$ | 2 | 3 | 4
---|---|---|---
$v$ (exact) | $2$ |  | $2.598076$ | $\ \approx\scriptstyle\,3\sqrt{3}/2$ | $3.141593$ | $\ \ \ \approx\pi$
$v\ (D)$ | $1.999975$ | (128) | $2.59799$ | (512) | $3.14141$ | (1024)
$v\ (D)$ | $1.999904$ | (64) | $2.59785$ | (256) | $3.14116$ | (512)

Spin $S$ | 1/2 | 3/2 | 5/2 | 7/2
---|---|---|---|---
$v$ (exact) | $1.570796$ | $\ \approx\pi/2$ | - | - | -
$v\ (D=1024)$ | $1.57059$ | ($D=512$) | $3.4392$ | $5.4037$ | $7.386$
$v\ (D=512)$ | $1.57035$ | ($D=256$) | $3.4379$ | $5.4014$ | $7.377$

Table 1:  Comparison of numerical results of the velocities for the two
highest bond dimensions $D$ used and the exact values if available [38, 39].
(Top) $Q$-state Potts model, (bottom) SU(2) AF model. Here $J=1$ is used.
Figure 3:  The velocity obtained by our method as
$\nu=-(E_{1}-E_{0})/\log(|\epsilon_{1}|)$ plotted against the inverse of the
correlation length $\xi^{-1}=-\log(|\epsilon_{1}|)$ for several values of bond
dimension $D$. (a) Critical Potts models for $Q=2$ (Ising)$,3,4$. (b) Critical
$SU(2)$ AF Heisenberg chains. Light dashed lines denote exact results from
integrability. The bolder dashed line in (c) is taken from Ref. [40].

As a nontrivial application we now want to check a long-standing analytical
prediction [41] and its recent refinement [40] for the speed of light (spin
wave velocity) of the half-integer spin-$S$ Heisenberg chains, which flow to
the $SU(2)_{1}$ Wess-Zumino-Novikov-Witten CFT in the infrared with central
charge $c=1$ [42]. The speed of light is not universal and is not accurately
known beyond the $S=1/2$ case [43], but from a mapping onto the non-linear
$\sigma$ model one expects the velocity to scale as $v(S)\approx
J(2S+|(2/\pi-1)|)$ for large $S$ [40]. In Fig. 3(b,c) we make use of our new
method to extract the velocities for several $S$. In panel (b) we display the
velocities on a common scale and one can see how the velocities grow linearly
with $S$. We plot the velocity as a function of the correlation lengths in the
MPS, which are induced by the different bond dimensions. For large bond
dimensions the velocities converge to a constant value, see also Tab. 1. In
Fig. 3(c) we plot $(v_{\mathrm{MPS}}(S)-2SJ)/J$, so that we can check that
indeed the velocities for larger $S$ and correspondingly large bond dimension
approach the positive offset $|(2/\pi-1)|\approx 0.36338$.

## Application to Luther-Emery Liquids —

Next we test our method for systems that fall within the Luther-Emery
universality class, characterized by gapped spin excitations and a single
gapless charge mode. Obtaining the infrared properties of Luther-Emery liquids
has become very relevant in the context of MPS-based simulations of two-
dimensional $t{-}J$ and Hubbard models [44, 45, 46, 47, 48, 49]. Getting
accurate estimates of critical exponents is crucial in these studies, but
requires the fitting of the correlation functions on long cylinders, the use
of large bond dimensions and tedious extrapolation procedures. Here we show
that using infinite MPS and our velocity scaling yields a more efficient
method for obtaining accurate exponents.

We will consider Hubbard models [50] with nearest-neighbour hopping (with
parameter $t$) and an on-site interaction strength $U$. We also introduce a
chemical potential $\mu$ in order to consider incommensurate electron fillings
between $n=0$ (zero filling) and $n=2$ (complete filling). Consequently we
cannot exploit charge conservation, so all simulations are performed with
$\mathrm{SU}(2)$ spin-rotation symmetry and fermion parity in the MPS
representation.

We start with the simple Hubbard chain with an attractive interaction, $U<0$.
The ground state exhibits a spin gap and critical charge correlations [50],
and falls within the Luther-Emery class. We perform ground state optimizations
at different bond dimensions, and extract the effective velocity through the
transfer matrix and effective Hamiltonian gaps. In Fig. 4 we plot the velocity
as a function of the inverse correlation length. We observe a quick
convergence to the exact result from the Bethe ansatz [50].

As a more challenging case we take the (non integrable) Hubbard ladder. We
work in the case of isotropic hopping and repulsive interactions $U/t=8$ and
tune the chemical potential in the weakly-doped regime. In this regime,
analytical results in the weak coupling limit [51] suggest that there is one
gapless charge mode and a gap in the spin sector, which was confirmed by early
DMRG calculations [52, 53]. The infrared properties are determined by the
Luttinger parameter $K$ 333We use the convention for $K$ from Ref. [58], which
differs by a factor of two from the one in, e.g., Ref. [57].. It is known [51,
55, 56] that in the limit of half filling ($n\to 1$) we find $K\to 1$, but the
numerical determination for the doped case is very challenging [57]. Using
Luttinger liquid theory [3], we can circumvent the fitting of correlation
functions by obtaining $K$ as the product of the velocity $v$ and the
compressibility (with an additional constant that depends on the convention)
[58], where the latter can be simply determined from the filling as a function
of chemical potential.

Our numerical results are presented in Fig. 4. We find that the extracted
velocities are converged already at reasonably small bond dimensions. In the
inset, we show the Luttinger parameter and compare to estimates from fitting
correlation functions using finite size DMRG simulations [57]. The latter
yields estimates that depend quite strongly on the procedure for extrapolating
the correlation functions to infinite MPS bond dimension, so the agreement
seems reasonable.

Figure 4: Numerical results for the velocity of (a) the Hubbard chain with
attractive interaction $U/t=-2$ and (b) the Hubbard ladder with repulsive
interaction $U/t=8$. For the chain we set $\mu/t=-1.5$ resulting in a filling
of $n\approx 0.78$ and repeat the MPS ground state calculations for multiple
values of the bond dimension in the range $D=23$ to $1321$. Here we show the
effective velocity plotted against inverse correlation length and compare our
data with the exact Bethe-ansatz result. For the ladder we plot the velocity
against the filling $n$ for three values of the bond dimension. In the inset
we show the Luttinger parameter extracted from the velocity and the
compressibility for the largest bond dimension, and compare to the result in
Ref. 57 and the limiting value at half filling.

## Outlook —

In this work we have introduced the intriguing equivalence between the low-
energy spectra of the transfer matrix and effective Hamiltonian in the
infinite MPS simulation of critical 1+1 dimensional quantum chains, and we
have illustrated the power of this equivalence for estimating the effective
velocity in these systems. Our observation opens the door to the
investigations of other systems of current interest, such as the nature of
phase transitions in Rydberg blockaded spin chains [59, 60], where extended
floating phases with $c=1$ and finite velocity can terminate in phase
transitions with $z=2$ as the velocity $v\rightarrow 0$. A more challenging
scenario to explore are effective Hamiltonian and transfer matrix spectra in
systems with two or more different velocities, for example a doped Hubbard
chain with $c=2$, but different charge and spin velocities [50]. Finally,
similarly powerful tensor network methods for 2+1 dimensional CFTs would be
welcome, as Quantum Monte Carlo methods - which work well for some classes of
systems [4, 5] \- can face difficulties in situations where the finite size
spectrum is affected by interaction effects [61, 62, 63]. Conceptually, this
work will allow for a better understanding of the structure of the MPS
transfer matrix spectrum, which remains a fundamental open question within the
program of finite-entanglement scaling with tensor networks.

## Acknowledgements —

The authors would like to thank N. Chepiga, T. Giamarchi, S. Gozel, J.
Haegeman, R. Huang, L. Tagliacozzo and B. Vanhecke for inspiring discussions
and J. De Nardis and F. Essler for sharing useful Bethe-ansatz data. AEE and
AML acknowledge support by the Austrian Science Fund (FWF) through project
I-4548. LV is supported by the FRS-FNRS (Belgium). This work was also
supported by a grant from FWO, no. G0E1820N. Some of the simulations have been
performed using the TenPy package [24].

## References

  * Blöte  _et al._ [1986] H. W. J. Blöte, J. L. Cardy, and M. P. Nightingale, Conformal invariance, the central charge, and universal finite-size amplitudes at criticality, Phys. Rev. Lett. 56, 742 (1986).
  * Affleck [1986] I. Affleck, Universal term in the free energy at a critical point and the conformal anomaly, Phys. Rev. Lett. 56, 746 (1986).
  * Giamarchi [2003] T. Giamarchi,  _Quantum physics in one dimension_ (Clarendon Press, 2003).
  * Sen  _et al._ [2015] A. Sen, H. Suwa, and A. W. Sandvik, Velocity of excitations in ordered, disordered, and critical antiferromagnets, Phys. Rev. B 92, 195145 (2015).
  * Tan and Jiang [2017] D.-R. Tan and F.-J. Jiang, Universal scaling of Néel temperature, staggered magnetization density, and spin-wave velocity of three-dimensional disordered and clean quantum antiferromagnets, Phys. Rev. B 95, 054435 (2017).
  * Haegeman  _et al._ [2012] J. Haegeman, B. Pirvu, D. J. Weir, J. I. Cirac, T. J. Osborne, H. Verschelde, and F. Verstraete, Variational matrix product ansatz for dispersion relations, Phys. Rev. B 85, 100408 (2012).
  * Vanderstraeten  _et al._ [2019a] L. Vanderstraeten, J. Haegeman, and F. Verstraete, Simulating excitation spectra with projected entangled-pair states, Phys. Rev. B 99, 165121 (2019a).
  * Calabrese and Cardy [2004] P. Calabrese and J. Cardy, Entanglement entropy and quantum field theory, Journal of Statistical Mechanics: Theory and Experiment 2004, P06002 (2004).
  * Läuchli [2013] A. M. Läuchli, Operator content of real-space entanglement spectra at conformal critical points (2013), arXiv:1303.0741 .
  * Cardy and Tonni [2016] J. Cardy and E. Tonni, Entanglement Hamiltonians in two-dimensional conformal field theory, Journal of Statistical Mechanics: Theory and Experiment 2016, 123103 (2016).
  * Baxter [1982] R. J. Baxter,  _Exactly solved models in statistical mechanics_ (Academic Press, 1982).
  * Nishino  _et al._ [1996] T. Nishino, K. Okunishi, and M. Kikuchi, Numerical renormalization group at criticality, Physics Letters A 213, 69 (1996).
  * Tagliacozzo  _et al._ [2008] L. Tagliacozzo, T. R. de Oliveira, S. Iblisdir, and J. I. Latorre, Scaling of entanglement support for matrix product states, Phys. Rev. B 78, 024410 (2008).
  * Pollmann  _et al._ [2009] F. Pollmann, S. Mukerjee, A. M. Turner, and J. E. Moore, Theory of Finite-Entanglement Scaling at One-Dimensional Quantum Critical Points, Phys. Rev. Lett. 102, 255701 (2009).
  * Pirvu  _et al._ [2012] B. Pirvu, G. Vidal, F. Verstraete, and L. Tagliacozzo, Matrix product states for critical spin chains: Finite-size versus finite-entanglement scaling, Phys. Rev. B 86, 075117 (2012).
  * Rams  _et al._ [2018] M. M. Rams, P. Czarnik, and L. Cincio, Precise Extrapolation of the Correlation Function Asymptotics in Uniform Tensor Network States with Application to the Bose-Hubbard and XXZ Models, Phys. Rev. X 8, 041033 (2018).
  * Vanhecke  _et al._ [2019] B. Vanhecke, J. Haegeman, K. Van Acoleyen, L. Vanderstraeten, and F. Verstraete, Scaling Hypothesis for Matrix Product States, Phys. Rev. Lett. 123, 250604 (2019).
  * Zauner  _et al._ [2015] V. Zauner, D. Draxler, L. Vanderstraeten, M. Degroote, J. Haegeman, M. M. Rams, V. Stojevic, N. Schuch, and F. Verstraete, Transfer matrices and excitations with matrix product states, New Journal of Physics 17, 053002 (2015).
  * Chepiga and Mila [2017] N. Chepiga and F. Mila, Excitation spectrum and density matrix renormalization group iterations, Phys. Rev. B 96, 054425 (2017).
  * Chepiga  _et al._ [2020] N. Chepiga, I. Affleck, and F. Mila, Floating, critical, and dimerized phases in a frustrated spin-$\frac{3}{2}$ chain, Phys. Rev. B 101, 174407 (2020).
  * Pirvu  _et al._ [2010] B. Pirvu, V. Murg, J. I. Cirac, and F. Verstraete, Matrix product operator representations, New Journal of Physics 12, 025012 (2010).
  * McCulloch [2008] I. P. McCulloch, Infinite size density matrix renormalization group, revisited (2008), arXiv:0804.2509 .
  * Kjäll  _et al._ [2013] J. A. Kjäll, M. P. Zaletel, R. S. K. Mong, J. H. Bardarson, and F. Pollmann, Phase diagram of the anisotropic spin-2 XXZ model: Infinite-system density matrix renormalization group study, Phys. Rev. B 87, 235106 (2013).
  * Hauschild and Pollmann [2018] J. Hauschild and F. Pollmann, Efficient numerical simulations with Tensor Networks: Tensor Network Python (TeNPy), SciPost Phys. Lect. Notes , 5 (2018).
  * Zauner-Stauber  _et al._ [2018] V. Zauner-Stauber, L. Vanderstraeten, M. T. Fishman, F. Verstraete, and J. Haegeman, Variational optimization algorithms for uniform matrix product states, Phys. Rev. B 97, 045145 (2018).
  * Vanderstraeten  _et al._ [2019b] L. Vanderstraeten, J. Haegeman, and F. Verstraete, Tangent-space methods for uniform matrix product states, SciPost Phys. Lect. Notes , 7 (2019b).
  * Note [1] Note that we could also consider the effective Hamiltonian for the single site or zero-site updates [26], for which the dimension would be $D^{2}d$ or $D^{2}$.
  * Tirrito  _et al._ [2018] E. Tirrito, N. J. Robinson, M. Lewenstein, S.-J. Ran, and L. Tagliacozzo, Characterizing the quantum field theory vacuum using temporal Matrix Product states (2018), arXiv:1810.08050 .
  * Rams  _et al._ [2015] M. M. Rams, V. Zauner, M. Bal, J. Haegeman, and F. Verstraete, Truncating an exact matrix product state for the XY model: Transfer matrix and its renormalization, Phys. Rev. B 92, 235150 (2015).
  * Bal  _et al._ [2016] M. Bal, M. M. Rams, V. Zauner, J. Haegeman, and F. Verstraete, Matrix product state renormalization, Phys. Rev. B 94, 205122 (2016).
  * Fishman  _et al._ [2018] M. T. Fishman, L. Vanderstraeten, V. Zauner-Stauber, J. Haegeman, and F. Verstraete, Faster methods for contracting infinite two-dimensional tensor networks, Phys. Rev. B 98, 235148 (2018).
  * Vanderstraeten  _et al._ [2022] L. Vanderstraeten, L. Burgelman, B. Ponsioen, M. Van Damme, B. Vanhecke, P. Corboz, J. Haegeman, and F. Verstraete, Variational methods for contracting projected entangled-pair states, Phys. Rev. B 105, 195140 (2022).
  * Haegeman and Verstraete [2017] J. Haegeman and F. Verstraete, Diagonalizing Transfer Matrices and Matrix Product Operators: A Medley of Exact and Computational Methods, Annual Review of Condensed Matter Physics 8, 355 (2017).
  * Baxter [1978] R. J. Baxter, Variational approximations for square lattice models in statistical mechanics, Journal of Statistical Physics 19, 461 (1978).
  * Nishino and Okunishi [1996] T. Nishino and K. Okunishi, Corner transfer matrix renormalization group method, Journal of the Physical Society of Japan 65, 891 (1996).
  * Note [2] The symbol $J$ is used for the nearest neighbor coupling strength of the Potts and Heisenberg models.
  * Wu and Car [2020] Y. Wu and R. Car, Continuous-time Monte Carlo renormalization group, Phys. Rev. B 102, 014456 (2020).
  * Alcaraz  _et al._ [1988] F. C. Alcaraz, M. N. Barber, and M. T. Batchelor, Conformal invariance, the XXZ chain and the operator content of two-dimensional critical systems, Annals of Physics 182, 280 (1988).
  * des Cloizeaux and Pearson [1962] J. des Cloizeaux and J. J. Pearson, Spin-Wave Spectrum of the Antiferromagnetic Linear Chain, Phys. Rev. 128, 2131 (1962).
  * Gozel  _et al._ [2019] S. Gozel, F. Mila, and I. Affleck, Asymptotic Freedom and Large Spin Antiferromagnetic Chains, Phys. Rev. Lett. 123, 037202 (2019).
  * Haldane [1983] F. D. M. Haldane, Nonlinear Field Theory of Large-Spin Heisenberg Antiferromagnets: Semiclassically Quantized Solitons of the One-Dimensional Easy-Axis Néel State, Phys. Rev. Lett. 50, 1153 (1983).
  * Affleck and Haldane [1987] I. Affleck and F. D. M. Haldane, Critical theory of quantum spin chains, Phys. Rev. B 36, 5291 (1987).
  * Hallberg  _et al._ [1996] K. Hallberg, X. Q. G. Wang, P. Horsch, and A. Moreo, Critical Behavior of the $\mathit{S}\phantom{\rule{0.0pt}{0.0pt}}=\phantom{\rule{0.0pt}{0.0pt}}3/2$ Antiferromagnetic Heisenberg Chain, Phys. Rev. Lett. 76, 4955 (1996).
  * Jiang  _et al._ [2018] H.-C. Jiang, Z.-Y. Weng, and S. A. Kivelson, Superconductivity in the doped $\mathit{t}-\mathit{J}$ model: Results for four-leg cylinders, Phys. Rev. B 98, 140505 (2018).
  * Jiang  _et al._ [2020a] H.-C. Jiang, S. Chen, and Z.-Y. Weng, Critical role of the sign structure in the doped Mott insulator: Luther-Emery versus Fermi-liquid-like state in quasi-one-dimensional ladders, Phys. Rev. B 102, 104512 (2020a).
  * Jiang  _et al._ [2020b] Y.-F. Jiang, J. Zaanen, T. P. Devereaux, and H.-C. Jiang, Ground state phase diagram of the doped hubbard model on the four-leg cylinder, Phys. Rev. Res. 2, 033073 (2020b).
  * Jiang  _et al._ [2021] S. Jiang, D. J. Scalapino, and S. R. White, Ground-state phase diagram of the $t{-}J$ model, Proceedings of the National Academy of Sciences 118, e2109978118 (2021).
  * Gong  _et al._ [2021] S. Gong, W. Zhu, and D. N. Sheng, Robust $d$-Wave Superconductivity in the Square-Lattice $t\text{$-$}J$ Model, Phys. Rev. Lett. 127, 097003 (2021).
  * He  _et al._ [2021] Y. He, D. Pekker, and R. S. K. Mong, One-dimensional repulsive hubbard model with mass imbalance: Orders and filling anomaly, Phys. Rev. B 104, 195126 (2021).
  * Essler  _et al._ [2005] F. H. L. Essler, H. Frahm, F. Göhmann, A. Klümper, and V. E. Korepin,  _The one-dimensional Hubbard model_ (Cambridge University Press, 2005).
  * Balents and Fisher [1996] L. Balents and M. P. A. Fisher, Weak-coupling phase diagram of the two-chain hubbard model, Phys. Rev. B 53, 12133 (1996).
  * Noack  _et al._ [1994] R. M. Noack, S. R. White, and D. J. Scalapino, Correlations in a Two-Chain Hubbard Model, Phys. Rev. Lett. 73, 882 (1994).
  * Noack  _et al._ [1996] R. M. Noack, S. R. White, and D. J. Scalapino, The ground state of the two-leg Hubbard ladder a density-matrix renormalization group study, Physica C: Superconductivity 270, 281 (1996).
  * Note [3] We use the convention for $K$ from Ref. [58], which differs by a factor of two from the one in, e.g., Ref. [57].
  * Schulz [1999] H. J. Schulz, Metal-insulator transition in the two-chain model of correlated fermions, Phys. Rev. B 59, R2471 (1999).
  * Siller  _et al._ [2001] T. Siller, M. Troyer, T. M. Rice, and S. R. White, Bosonic model of hole pairs, Phys. Rev. B 63, 195106 (2001).
  * Dolfi  _et al._ [2015] M. Dolfi, B. Bauer, S. Keller, and M. Troyer, Pair correlations in doped Hubbard ladders, Phys. Rev. B 92, 195139 (2015).
  * White  _et al._ [2002] S. R. White, I. Affleck, and D. J. Scalapino, Friedel oscillations and charge density waves in chains and ladders, Phys. Rev. B 65, 165122 (2002).
  * Rader and Läuchli [2019] M. Rader and A. M. Läuchli, Floating Phases in One-Dimensional Rydberg Ising Chains (2019), arXiv:1908.02068 .
  * Chepiga and Mila [2019] N. Chepiga and F. Mila, Floating Phase versus Chiral Transition in a 1D Hard-Boson Model, Phys. Rev. Lett. 122, 017205 (2019).
  * Tang  _et al._ [2018] H.-K. Tang, J. N. Leaw, J. N. B. Rodrigues, I. F. Herbut, P. Sengupta, F. F. Assaad, and S. Adam, The role of electron-electron interactions in two-dimensional Dirac fermions, Science 361, 570 (2018).
  * Hesselmann  _et al._ [2019] S. Hesselmann, T. C. Lang, M. Schuler, S. Wessel, and A. M. Läuchli, Comment on “The role of electron-electron interactions in two-dimensional Dirac fermions”, Science 366, eaav6869 (2019).
  * Schuler  _et al._ [2021] M. Schuler, S. Hesselmann, S. Whitsitt, T. C. Lang, S. Wessel, and A. M. Läuchli, Torus spectroscopy of the Gross-Neveu-Yukawa quantum field theory: Free Dirac versus chiral Ising fixed point, Phys. Rev. B 103, 125128 (2021).


# PaddleSpeech: An Easy-to-Use All-in-One Speech Toolkit

Hui Zhang1, Tian Yuan1, Junkun Chen3, Xintong Li2, Renjie Zheng2,
Yuxin Huang1, Xiaojie Chen1, Enlei Gong1, Zeyu Chen1, Xiaoguang Hu1,
Dianhai Yu1, Yanjun Ma1, Liang Huang2,3
1Baidu Inc., Beijing, China
2Baidu Research, Sunnyvale, CA, USA
3Oregon State University, Corvallis, OR, USA
{zhanghui41, yuantian01, xintongli, renjiezheng,
huangyuxin, chenxiaojie06, gongenlei, chenzeyu01}baidu.com

###### Abstract

PaddleSpeech is an open-source all-in-one speech toolkit. It aims at
facilitating the development and research of speech processing technologies by
providing an easy-to-use command-line interface and a simple code structure.
This paper describes the design philosophy and core architecture of
PaddleSpeech to support several essential speech-to-text and text-to-speech
tasks. PaddleSpeech achieves competitive or state-of-the-art performance on
various speech datasets and implements the most popular methods. It also
provides recipes and pretrained models to quickly reproduce the experimental
results in this paper. PaddleSpeech is publicly avaiable at
https://github.com/PaddlePaddle/PaddleSpeech.111Demo video:
https://paddlespeech.readthedocs.io/en/latest/demo_video.html

Task | Description | Techniques | Datasets
---|---|---|---
Sound Classification | Label sound class |  Finetuned PANN Kong et al. (2020b) |  ESC-50 dataset Piczak (2015)
Speech Recognition |  Transcribe speech to text |  Deepspeech2 Amodei et al. (2016) Conformer Zhang et al. (2020) Transformer Zhang et al. (2020) |  Librispeech (Panayotov et al., 2015) AISHELL-1 (Bu et al., 2017)
Punctuation Restoration |  Post-add punctuation to transcribed text |  Finetuned ERNIE (Sun et al., 2019) |  IWSLT2012-zh (Federico et al., 2012)
Speech Translation |  Translate speech to text |  Transformer (Vaswani et al., 2017) |  MuST-C Di Gangi et al. (2019)
Text To Speech |  Synthesis speech from text |  Acoustic Model Tacotron 2 (Shen et al., 2018) Transformer TTS (Li et al., 2019) SpeedySpeech (Vainer and Dušek, 2020) FastPitch (Łańcucki, 2021) FastSpeech 2 (Ren et al., 2020) Vocoder WaveFlowPing et al. (2020) Parallel WaveGAN (Yamamoto et al., 2020) MelGAN (Kumar et al., 2019) Style MelGAN (Mustafa et al., 2021) Multi Band MelGAN (Yang et al., 2021) HiFi GAN (Kong et al., 2020a) |  CSMS (DataBaker) AISHELL-3 (Shi et al., 2020) LJSpeech (Ito and Johnson, 2017) VCTK (Yamagishi et al., 2019)
Table 1: List of speech tasks and corpora that are currently supported by
PaddleSpeech.

##  1 Introduction

Speech processing technology enables humans to directly communicate with
computers, which is an essential part of enormous applications such as smart
home devices Hoy (2018), autonomous driving, and simultaneous translation
Zheng et al. (2020). Open-source toolkits boost the development of speech
processing technology by lowering the barrier of application and research in
this area (Young et al., 2002; Lee et al., 2001; Huggins-Daines et al., 2006;
Rybach et al., 2011; Povey et al., 2011; Watanabe et al., 2018; Han et al.,
2019; Wang et al., 2020; Ravanelli et al., 2021; Zhao et al., 2021).

However, the current prevailing speech processing toolkits presume that their
users are experienced practitioners or researchers, so beginners might feel
baffled when developing their exciting applications. For example, to prototype
new speech applications with Kaldi (Povey et al., 2011), the users have to be
comfortable reading and revising the provided recipes written in Bash, Perl,
and Python scripts and be proficient at C++ to hack its implementation. The
more recent toolkits, such as Fairseq S2T (Wang et al., 2020) and NeurST (Zhao
et al., 2021), become more flexible by building on general-purpose deep
learning libraries. But their complicated code styles also make it time-
consuming to learn and hard to migrate from one to another. So, we have
developed PaddleSpeech, providing a command-line interface and portable
functions to make the development of speech-related applications accessible to
everyone.

Notably, the Chinese community has many developers eager to contribute to the
community. However, nearly all deep learning libraries, such as Pytorch
(Paszke et al., 2019) and Tensorflow (Abadi et al., 2016), target the English
community mainly, so it significantly increases the difficulty for Chinese
developers. PaddlePaddle, as the only fully-functioning open-source deep
learning platform targeting both the English and Chinese community, has
accumulated more than 500k commits, 476k models, and is used by 157k
enterprises. So, we expect PaddleSpeech, developed with PaddlePaddle can
remove the barriers between the English and Chinese communities to boost the
development of speech technologies and applications.

Developing speech applications for the industry is not the same scenario as
conducting research in academia. The research papers mainly focus on
developing novel models to perform better on specific datasets. However, a
clean dataset usually does not exist when applying a speech product. So,
PaddleSpeech provides on-the-fly preprocessing for the raw audios to make
PaddleSpeech directly usable in product-oriented applications. Notably, some
preprocessing methods are exclusive in PaddleSpeech, such as rule-based
Chinese text-to-speech frontend, which can significantly benefit the
performance of synthesized speech.

Performance is the cornerstone of all applications. PaddleSpeech achieves
state-of-the-art or competitive performers on various commonly used
benchmarks, as shown in Table 1.

Our main contributions in this paper are two-folds.

  * •

We introduce how we designed PaddleSpeech and what features it supports.

  * •

We provide the implementation and reproducible experimental details that
result in state-of-the-art or competitive performance on various tasks.

Fundamental PlatformsCommon ModulesModels & Updaters &
DatasetsRecipesPaddlePaddleOther LibrariesKaldi, Sclite, Sox,
OpenblasPaddleAudio paddleaudio.features.Spectrogram
---
paddleaudio.features.LogMelSpectrogram
Utils utils/build_vocab.py
---
utils/parse_options.sh
PaddleSpeech cli/asr/infer.py
---
t2s/exps/fastspeech2/train.py
t2s/models/fastspeech2/fastspeech2.py
t2s/models/fastspeech2/fastspeech2_updater.py
Datasets dataset/aishell/aishell.py
---
dataset/librispeech/librispeech.py
Examples/Demos examples/aishell/run.sh
---
demos/metaverse/run.sh

Figure 1: Software architecture of PaddleSpeech.

##  2 Design of PaddleSpeech

Figure 1 shows the software architecture of PaddleSpeech. As an easy-to-use
speech processing toolkit, PaddleSpeech provides many complete recipes to
perform various speech-related tasks and demo usage of the command line
interface. Getting familiar with the top level should be enough for building
speech-related applications.

The second level faces researchers in speech and language processing. The
design philosophy of PaddleSpeech is model-centric to simplify the learning
and development of speech processing methods. For a specific method, all
computations of a specific model are included in two files under
PaddleSpeech/<task>/models/<model>.222<task> includes s2t and t2s which stands
for speech-to-text and text-to-speech respectively.

PaddleSpeech has implemented most of the commonly used and well-performing
models. A model architecture is implemented in a standalone file named by the
method. Its corresponding training step and evaluation step are implemented in
another updater file. Generally, reading or hacking these two files is enough
to understand or design a model. More advanced hacking on more fine data
processing or more complicated training/evaluation loop is also available at
PaddleSpeech/<task>/exps/<model>. The original datasets can be obtained by
scripts in corresponding dataset/<dataset>/. PaddleSpeech supports distributed
multi-GPU training with good efficiency.

The standard modules, such as audio and text feature transformation and
utility scripts, are implemented as libraries in the third level. The backend
of PaddleSpeech is mainly PaddlePaddle with some functions from third-party
libraries as shown in the fourth level. PaddleSpeech provides multiple ways to
extract multiple types of speech features from raw audios using PaddleAudio
and Kaldi, such as spectrogram and filterbanks, which can be varied according
to the needs of the tasks.

##  3 Experiments

In this section, we compare the performance of models in PaddleSpeech with
other popular implementations in five speech-related tasks, including sound
classification, speech recognition, punctuation, speech translation, and
speech synthesizing. The toolkit can reach SOTA on most tasks. All experiments
in this section include details on data preparation, evaluation metrics, and
implementation to enhance
reproducibility.333https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples

###  3.1 Sound Classification

Sound Classification is a task to recognize particular sounds, including
speech commands (Warden, 2018), environment sounds Piczak (2015), identifying
musical instruments Engel et al. (2017), finding birdsongs Stowell et al.
(2018), emotion recognition Xu et al. (2019) and speaker verification Liu et
al. (2018).

#### Datasets

In this section, we analyze the performance of PaddleSpeech in Sound
Classification on ESC-50 dataset (Piczak, 2015). The ESC-50 dataset is a
labeled collection of 2000 environmental 5-second audio recordings consisting
of 50 sound events, such as "Dog", "Cat", "Breathing" and "Fireworks", with 40
recordings per event.

#### Data Preprocessing

First, we resample all audio recordings to 32 kHz, and convert them to
monophonic to be consistent with the PANNs trained on AudioSet (Kong et al.,
2020b). And then, we transform the recordings into log mel spectrograms by
applying short-time Fourier transform on the waveforms with a Hamming window
of size 1024 and a hop size of 320 samples. This configuration leads to 100
frames per second. Following Kong et al. (2019), we apply 64 mel filter banks
to calculate the log mel spectrogram.

#### Implementation

PANNs Kong et al. (2020b) is one of the pre-trained CNN models for audio-
related tasks, which is characterized in terms of being trained with the
AudioSet Gemmeke et al. (2017). PANNs are helpful for tasks where only a
limited number of training clips are provided. In this case, we fine-tune all
parameters of a PANN for the environment sounds classification task. All
parameters are initialized from the PANN, except the final fully-connected
layer which is randomly initialized. Specifically, we implement CNNs with 6,
10 and 14 layers, respectively Kong et al. (2020b).

#### Results

We report 5-fold cross validation accuracy values on ESC-50 dataset. As shown
in Table 2, PANNs-CNN14 achieves 0.9500 5-fold cross validation accuracy that
is comparable to the current state-of-the-art method Gong et al. (2021).

Model | Accuracy
---|---
AST-P (Gong et al., 2021) | $95.6\pm 0.4$
PANNs-CNN14 | 95.00
PANNs-CNN10 | 89.75
PANNs-CNN6 | 88.25
Table 2: 5-fold cross validation accuracy of ESC-50.

Data | Model | Streaming | Test Data | Language Model | CER | WER
---|---|---|---|---|---|---
Aishell | WeNet Conformer†∗ Yao et al. (2021) | ✓ |  |  | 5.45 | -
WeNet Conformer† Yao et al. (2021) |  |  |  | 4.61 | -
WeNet Transformer† (Yao et al., 2021) |  |  |  | 5.30 | -
ESPnet Conformer† (Inaguma et al., 2020) |  |  |  | 5.10 | -
ESPnet Transformer† (Inaguma et al., 2020) |  |  |  | 6.70 | -
| SpeechBrain Transformer† (Ravanelli et al., 2021) |  |  |  | 5.58 | -
| Deepspeech 2 | ✓ |  | char 5-gram | 6.66 | -
| Deepspeech 2 |  |  | char 5-gram | 6.40 | -
| Transformer |  |  |  | 5.23 | -
| Conformer∗ | ✓ |  |  | 5.44 | -
| Conformer |  |  |  | 4.64 | -
Librispeech | WeNet Conformer† Yao et al. (2021) |  | test-clean |  | - | 2.85
SpeechBrain Transformer† (Ravanelli et al., 2021) |  | test-clean | TransformerLM | - | 2.46
ESPnet Transformer† (Inaguma et al., 2020) |  | test-clean | TransformerLM | - | 2.60
Deepspeech 2 |  | test-clean | word 5-gram | - | 7.25
Conformer |  | test-clean |  | - | 3.37
Transformer |  | test-clean | TransformerLM | - | 2.40

  * †

denotes the results are reported in their public repositories.

  * ∗

denotes the results are streaming with chunk size 16.

Table 3: WER/CER on Aishell, Librispeech for ASR Tasks.

###  3.2 Automatic Speech Recognition

Automatic Speech Recognition (ASR) is a task to transcribe the audio content
to text in the same language.

#### Datasets

We conduct the ASR experiments on two major datasets including
Librispeech444http://www.openslr.org/12/ (Panayotov et al., 2015) and
Aishell-1555http://www.aishelltech.com/kysjcp (Bu et al., 2017). Librispeech
contains 1000 hours speech data. The whole dataset is divided into 3 training
sets (100h clean, 360h clean, 500h other), 2 validation sets (clean, other),
and 2 test sets (clean, other). Aishell contains 178 hours speech data. 400
speakers from different accent areas in China participate in the recording.
The dataset is divided into the training set (340 speakers), validation set,
(40 speakers) and test set (20 speakers).

#### Data Preprocessing

Deepspeech 2 takes character-level vocabularies for both English and Mandarin
tasks. For other models, we use character-level vocabulary for Mandarin. And
English text is preprocessed with SentencePiece Kudo and Richardson (2018).
Both two kinds of datasets are added four additional characters, which are
<’>, <space>, <blank> and <eos>. For cepstral mean and variance normalization
(CMVN), a subset of or full of the training set is selected and be used to
compute the feature mean and standard error. For feature extraction, we have
several methods implemented, such as linear spectrogram, filterbank, and mfcc.
Currently, the Deepspeech 2 model uses linear spectrogram or filterbank, but
Transformer and Conformer models use filterbank. For a fair comparison, we
take additional 3 dimensional pitch features into Transformer to be consistent
with ESPnet.

#### Implementation

We implement both streaming and non-streaming Deepspeech 2 (Amodei et al.,
2016). The non-streaming model has 2 convolution layers and 3 LSTM layers. The
streaming model has 2 convolution layers and 5 LSTM layers. The Conformer and
Transformer models are implemented following Zhang et al. (2020) with 12
encoder layers and 6 decoder layers.

#### Results

We report word error rate (WER) and character error rate (CER) for Librispeech
(English) and Aishell (Mandarin) speech recognition, respectively. As shown in
Table 3, Conformer and Transformer are better than Deepspeech 2. Our best
models achieve comparable performance on both datasets compared with related
works.

Frameworks | De | Es | Fr | It | Nl | Pt | Ro | Ru
---|---|---|---|---|---|---|---|---
ESPnet-ST Inaguma et al. (2020) | 22.9 | 28.0 | 32.8 | 23.8 | 27.4 | 28.0 | 21.9 | 15.8
fairseq-ST Wang et al. (2020) | 22.7 | 27.2 | 32.9 | 22.7 | 27.3 | 28.1 | 21.9 | 15.3
NeurST Zhao et al. (2021) | 22.8 | 27.4 | 33.3 | 22.9 | 27.2 | 28.7 | 22.2 | 15.1
PaddleSpeech | 23.0 | 27.4 | 32.9 | 22.9 | 26.7 | 28.8 | 22.2 | 15.4
Table 4: Case-sensitive detokenized BLEU scores on MuST-C tst-COMMON.

###  3.3 Punctuation Restoration

Punctuation restoration is a post-processing problem for ASR systems. It is
crucial to improve the readability of the transcribed text for the human
reader and facilitate down-streaming NLP tasks.

#### Datasets

We conduct experiments on IWSLT2012-zh666https://hltc.cs.ust.hk/iwslt/
dataset, which contains 150k Chinese sentences with punctuation. We select
comma, period, and question marks as restore targets in this task, so we
replace other punctuation with these three marks before training a model. We
split the data into training, validation and testing sets with 147k, 2k, and
1k samples, respectively.

#### Implementation

We formulate the problem of punctuation restoration as a sequence labeling
task with four target classes including EMPTY, COMMA, PERIOD, and QUESTION
(Nagy et al., 2021b). ERNIE (Sun et al., 2019), as a pretrained language
model, achieves new state-of-the-art results on five Chinese natural language
processing tasks, including natural language inference, semantic similarity,
named entity recognition, sentiment analysis, and question answering. So, we
finetune an ERNIE model for this task. More specifically, all parameters are
initialized from the ERNIE pretrained model, except the final shared fully-
connected layer, which is randomly initialized.

#### Results

We report F1-score values on IWSLT2012-zh dataset. As shown in Table 5, our
ERNIELinear model achieves 0.6331 overall F1-score, which is comparable with
the previous work Nagy et al. (2021a).

model | COMMA | PERIOD | QUESTION | Overall
---|---|---|---|---
BERTLinear† | 0.4646 | 0.4227 | 0.7400 | 0.5424
BERTBiLSTM† | 0.5190 | 0.5707 | 0.8095 | 0.6330
ERNIELinear | 0.5142 | 0.5447 | 0.8406 | 0.6331

  * †

denotes the results come from our reproduced models.

Table 5: F1-score values on IWSLT2012-zh dataset.

###  3.4 Speech Translation

Speech translation, where translating speech in a source language to text in
another language, is beneficial in human communications.

#### Datasets

In this section, we analyze the performance of speech-to-text translation with
PaddleSpeech on MuST-C dataset (Di Gangi et al., 2019) with 8 different
language translation pairs, which take the English speech as the source input.

#### Implementation

We process the raw audios with Kaldi (Povey et al., 2011) and extract
80-dimensional log-mel filterbanks stacked with 3-dimensional pitch feature
using a 25ms window size and a 10ms step size. Text is firstly tokenized with
Moses tokenizer777https://github.com/moses-smt/mosesdecoder and then processed
by SentencePiece Kudo and Richardson (2018) with a joint vocabulary whose size
is 8K for each language pair. We employ Transformer (Vaswani et al., 2017) as
the base architecture for the speech translation experiments. In detail, the
Transformer model has 12 encoder layers that follow 2 layers of 2D convolution
with kernel size of 3 and stride size of 2, and 6 decoder layers. Each layer
contains 4 attention heads with a size of 256. The encoder is initialized from
a pretrained ASR model.

#### Results

We report detokenized case-sensitive
BLEU888https://github.com/mjpost/sacrebleu. As shown in Table 4, PaddleSpeech
can achieve competitive results compared with other frameworks.

| Module | Result |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---|---|---
PaddleSpeech |  | jīn tiān | shì |  |  | zuì dī | wēn dù | shì |  |
Text | 今天 | 是 | 2020/10/29 | ， | 最低 | 温度 | 是 | -3°C | 。
| today | is |  |  | lowest | temperature | is |  |
|  |  | èr líng èr líng nían shí yùe èr shí jǐu rì |  |  |  |  | líng xià sān dù |
TN | 今天 | 是 | 二 零 二 零 年 十 月 二 十 九 日 | ， | 最低 | 温度 | 是 | 零下 三 度 | 。
|  |  | 2 0 0 2 year 10 month 29 day |  |  |  |  | negative three degree |
WS | 今天 / | 是 / | 二零二零年 / 十月 / 二十九日 | ， | / 最低 | 温度 / | 是 / | 零下 / 三度 | 。
G2P | jin1 tian1 | shi4 | er4 ling2 er4 ling2 nian2 shi4 yue4 er4 shi2 jiu3 ri4 |  | zui4 di1 | wen1 du4 | shi4 | ling2 xia4 san1 du4 |
ESPnet | jin1 tian1 | shi4 | 2020/10/29 |  | zui4 di1 | wen1 du4 | shi4 | -3°C |

Table 6: An example of the text preprocessing pipeline for Mandarin TTS of
PaddleSpeech and ESPnet. TN stands for the text normalization module, WS
stands for the word segmentation module, G2P stands for the grapheme-to-
phoneme module. The text normalization module for mandarin of ESPnet is not
able to correctly handle dates (2020/10/29) and temperatures (-3°C).

###  3.5 Text-To-Speech

A Text-To-Speech (TTS) system converts given language text into speech.
PaddleSpeech’s TTS pipeline includes three steps. We first convert the
original text into the characters/phonemes through the text frontend module.
Then, through an Acoustic model, we convert the characters or phonemes into
acoustic features, such as mel spectrogram, Finally, we generate waveform from
the acoustic features through a Vocoder. In PaddleSpeech, the text frontend is
a rule-based model inspired by expert knowledge. The Acoustic models and
Vocoders are trainable.

#### Datasets

In PaddleSpeech, we mainly focus on Mandarin and English speech synthesis. We
have benchmarks on CSMSC999https://www.data-baker.com/open_source.html,
AISHELL-3101010http://www.aishelltech.com/aishell_3,
LJSpeech111111https://keithito.com/LJ-Speech-Dataset/,
VCTK121212https://datashare.ed.ac.uk/handle/10283/3443. Due to the limit of
space, we only list the experimental results on CSMSC, which includes 12 hours
speech audio corresponding to 10k sentences.

#### Text Frontend

A text frontend module is used to extract linguistic features, characters and
phonemes from given text. It mainly includes: Text Segmentation, Text
Normalization (TN), Word Segmentation (WS), Part-of-Speech Tagging, Prosody
Prediction and Grapheme-to-Phoneme (G2P) (see Table 6).

For Mandarin, our G2P system consists of a polyphone module, which uses
pypinyin and g2pM, and a tone sandhi module which uses rules based on chinese
word segmentations. To the best of our knowledge, our Mandarin text frontend
system is the most complete one compared with other publicly released works.

#### Data Preprocessing

PaddleSpeech TTS uses the following modules for data
preprocessing131313https://github.com/PaddlePaddle/PaddleSpeech/blob/develop/examples/csmsc/tts3/local/preprocess.sh:
First, we use Montreal-Forced-Aligner to get the duration for corresponding
phonemes. Second, we extract mel spectrograms as the features (additional
pitch and energy features for Fastspeech 2). Last, we conduct the statistical
normalization for each feature.

#### Acoustic Model

Acoustic models can be mainly classified into autoregressive and non-
autoregressive models. The decoding of the autoregressive model relies on
previous predictions at each step, which leads to longer inference time but
relatively better quality. While the non-autoregressive model generates the
outputs in parallel, so the inference speed is faster, but the quality of
generated result is relatively poor.

As shown in Table 1, PaddleSpeech has implemented the following commonly used
autoregressive acoustic models: Tacotron 2 and Transformer TTS, and non-
autoregressive acoustic models: SpeedySpeech, FastPitch and FastSpeech 2.

#### Vocoder

As shown in Table 1, PaddleSpeech has implemented the following vocoders:
WaveFlow, Parallel WaveGAN, MelGAN, Style MelGAN, Multi Band MelGAN, and HiFi
GAN.

#### Implementation

The PadddleSpeech TTS implementation of FastSpeech 2 adopts some improvement
from FastPitch and uses MFA to obtain the forced alignment (the original
FastSpeech paper uses Tacotron 2). Notably, the speech feature parameters of
the acoustic model and the vocoder of one TTS pipeline should be the same.
Detailed settings can be found in the sample config
file141414https://github.com/PaddlePaddle/PaddleSpeech/blob/develop/examples/csmsc/tts2/conf/default.yaml
on CSMSC dataset.

| Acoustic Model | Vocoder |  MOS $\uparrow$
---|---|---|---
ESPnet | Fastspeech 2 | PWGAN | 2.55 $\pm$ 0.19
| Tacotron 2 | PWGAN | 3.69 $\pm$ 0.11
| Speedyspeech | PWGAN | 3.79 $\pm$ 0.09
| Fastspeech 2 | PWGAN | 4.25 $\pm$ 0.09
PaddleSpeech | Fastspeech 2 | Style MelGAN | 4.32 $\pm$ 0.10
| Fastspeech 2 | MB MelGAN | 4.43 $\pm$ 0.09
| Fastspeech 2 | HiFi GAN | 4.72 $\pm$ 0.08

Table 7: The MOS evaluation with 95% confidence intervals for TTS models
trained using CSMSC dataset. PWGAN stands for Parallel WaveGan, MB MelGAN
stands for Multi-Band MelGAN.

#### Results

We report the mean opinion score (MOS) for naturalness evaluation in Table 7.
We use the crowdMOS toolkit Ribeiro et al. (2011), where 14 Mandarin samples
(see Appendix A) from these 7 different models were presented to 14 workers on
Mechanical Turk. As shown in Table 7, PaddleSpeech can largely outperform
ESPnet on Mandarin TTS. The main reason is that PaddleSpeech TTS has a better
text frontend as shown in Table 6. Compared with other models, Fastspeech 2
with HiFi GAN can achieve the best results.

##  4 Conclusion

This paper introduces PaddleSpeech, an open-source, easy-to-use, all-in-one
speech processing toolkit. We illustrated the main design philosophy behind
this toolkit to conduct development and research on various speech-related
tasks accessible. A number of reproducible experiments and comparisons show
that PaddleSpeech achieves state-of-the-art or competitive performance with
the most popular models on standard benchmarks.

##  5 Acknowledgment

We sincerely thank the anonymous reviewers for their valuable comments and
suggestions. This work was supported by the National Key Research and
Development Project of China (2020AAA0103503).

## References

  * Abadi et al. (2016) Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. 2016\.  Tensorflow: A system for large-scale machine learning.  In _12th $\\{$USENIX$\\}$ symposium on operating systems design and implementation ($\\{$OSDI$\\}$ 16)_, pages 265–283.
  * Amodei et al. (2016) Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al. 2016.  Deep speech 2: End-to-end speech recognition in english and mandarin.  In _International conference on machine learning_ , pages 173–182. PMLR.
  * Bu et al. (2017) Hui Bu, Jiayu Du, Xingyu Na, Bengu Wu, and Hao Zheng. 2017.  Aishell-1: An open-source mandarin speech corpus and a speech recognition baseline.  In _2017 20th Conference of the Oriental Chapter of the International Coordinating Committee on Speech Databases and Speech I/O Systems and Assessment (O-COCOSDA)_ , pages 1–5. IEEE.
  * Di Gangi et al. (2019) Mattia A Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2019.  Must-c: a multilingual speech translation corpus.  In _2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2012–2017. Association for Computational Linguistics.
  * Engel et al. (2017) Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Mohammad Norouzi, Douglas Eck, and Karen Simonyan. 2017.  Neural audio synthesis of musical notes with wavenet autoencoders.  In _ICML_.
  * Federico et al. (2012) Marcello Federico, Mauro Cettolo, Luisa Bentivogli, Paul Michael, and Stüker Sebastian. 2012.  Overview of the iwslt 2012 evaluation campaign.  In _IWSLT-International Workshop on Spoken Language Translation_ , pages 12–33.
  * Gemmeke et al. (2017) Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter. 2017.  Audio set: An ontology and human-labeled dataset for audio events.  _2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 776–780.
  * Gong et al. (2021) Yuan Gong, Yu-An Chung, and James R. Glass. 2021.  Ast: Audio spectrogram transformer.  _ArXiv_ , abs/2104.01778.
  * Han et al. (2019) Kun Han, Junwen Chen, Hui Zhang, Haiyang Xu, Yiping Peng, Yun Wang, Ning Ding, Hui Deng, Yonghu Gao, Tingwei Guo, Yi Zhang, Yahao He, Baochang Ma, Yulong Zhou, Kangli Zhang, Chao Liu, Ying Lyu, Chenxi Wang, Cheng Gong, Yunbo Wang, Wei Zou, Hui Song, and Xiangang Li. 2019\.  DELTA: A DEep learning based Language Technology plAtform.  _arXiv e-prints_.
  * Hoy (2018) Matthew B Hoy. 2018.  Alexa, siri, cortana, and more: an introduction to voice assistants.  _Medical reference services quarterly_ , 37(1):81–88.
  * Huggins-Daines et al. (2006) David Huggins-Daines, Mohit Kumar, Arthur Chan, Alan W Black, Mosur Ravishankar, and Alexander I Rudnicky. 2006.  Pocketsphinx: A free, real-time continuous speech recognition system for hand-held devices.  In _2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings_ , volume 1, pages I–I. IEEE.
  * Inaguma et al. (2020) Hirofumi Inaguma, Shun Kiyono, Kevin Duh, Shigeki Karita, Nelson Yalta, Tomoki Hayashi, and Shinji Watanabe. 2020.  Espnet-st: All-in-one speech translation toolkit.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 302–311.
  * Ito and Johnson (2017) Keith Ito and Linda Johnson. 2017.  The lj speech dataset.  https://keithito.com/LJ-Speech-Dataset/.
  * Kong et al. (2020a) Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. 2020a.  Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis.  _arXiv preprint arXiv:2010.05646_.
  * Kong et al. (2020b) Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark D. Plumbley. 2020b.  Panns: Large-scale pretrained audio neural networks for audio pattern recognition.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 28:2880–2894.
  * Kong et al. (2019) Qiuqiang Kong, Yin Cao, Turab Iqbal, Yong Xu, Wenwu Wang, and Mark D. Plumbley. 2019\.  Cross-task learning for audio tagging, sound event detection and spatial localization: Dcase 2019 baseline systems.  _ArXiv_ , abs/1904.05635.
  * Kudo and Richardson (2018) Taku Kudo and John Richardson. 2018.  Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 66–71.
  * Kumar et al. (2019) Kundan Kumar, Rithesh Kumar, Thibault de Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brébisson, Yoshua Bengio, and Aaron Courville. 2019.  Melgan: Generative adversarial networks for conditional waveform synthesis.  _arXiv preprint arXiv:1910.06711_.
  * Łańcucki (2021) Adrian Łańcucki. 2021.  Fastpitch: Parallel text-to-speech with pitch prediction.  In _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 6588–6592. IEEE.
  * Lee et al. (2001) Akinobu Lee, Tatsuya Kawahara, and Kiyohiro Shikano. 2001.  Julius—an open source real-time large vocabulary recognition engine.  _EUROSPEECH2001: the 7th European Conference on Speech Communication and Technology_.
  * Li et al. (2019) Naihan Li, Shujie Liu, Yanqing Liu, Sheng Zhao, and Ming Liu. 2019.  Neural speech synthesis with transformer network.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pages 6706–6713.
  * Liu et al. (2018) Bin Liu, Shuai Nie, Yaping Zhang, Shan Liang, and Wenju Liu. 2018.  Deep segment attentive embedding for duration robust speaker verification.
  * Mustafa et al. (2021) Ahmed Mustafa, Nicola Pia, and Guillaume Fuchs. 2021.  Stylemelgan: An efficient high-fidelity adversarial vocoder with temporal adaptive normalization.  In _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 6034–6038. IEEE.
  * Nagy et al. (2021a) Attila Nagy, Bence Bial, and Judit Ács. 2021a.  Automatic punctuation restoration with bert models.  _arXiv preprint arXiv:2101.07343_.
  * Nagy et al. (2021b) Attila Matyas Nagy, Bence Bial, and Judit Ács. 2021b.  Automatic punctuation restoration with bert models.  _ArXiv_ , abs/2101.07343.
  * Panayotov et al. (2015) Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015.  Librispeech: an asr corpus based on public domain audio books.  In _2015 IEEE international conference on acoustics, speech and signal processing (ICASSP)_ , pages 5206–5210. IEEE.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019\.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in neural information processing systems_ , 32:8026–8037.
  * Piczak (2015) Karol J. Piczak. 2015.  Esc: Dataset for environmental sound classification.  _Proceedings of the 23rd ACM international conference on Multimedia_.
  * Ping et al. (2020) Wei Ping, Kainan Peng, Kexin Zhao, and Zhao Song. 2020.  Waveflow: A compact flow-based model for raw audio.  In _International Conference on Machine Learning_ , pages 7706–7716. PMLR.
  * Povey et al. (2011) Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, et al. 2011.  The kaldi speech recognition toolkit.  In _IEEE 2011 workshop on automatic speech recognition and understanding_ , CONF. IEEE Signal Processing Society.
  * Ravanelli et al. (2021) Mirco Ravanelli, Titouan Parcollet, Peter Plantinga, Aku Rouhe, Samuele Cornell, Loren Lugosch, Cem Subakan, Nauman Dawalatabad, Abdelwahab Heba, Jianyuan Zhong, et al. 2021.  Speechbrain: A general-purpose speech toolkit.  _arXiv preprint arXiv:2106.04624_.
  * Ren et al. (2020) Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. 2020\.  Fastspeech 2: Fast and high-quality end-to-end text to speech.  _arXiv preprint arXiv:2006.04558_.
  * Ribeiro et al. (2011) Flávio Ribeiro, Dinei Florêncio, Cha Zhang, and Michael Seltzer. 2011.  Crowdmos: An approach for crowdsourcing mean opinion score studies.  In _2011 IEEE international conference on acoustics, speech and signal processing (ICASSP)_ , pages 2416–2419. IEEE.
  * Rybach et al. (2011) David Rybach, Stefan Hahn, Patrick Lehnen, David Nolden, Martin Sundermeyer, Zoltan Tüske, Simon Wiesler, Ralf Schlüter, and Hermann Ney. 2011.  Rasr-the rwth aachen university open source speech recognition toolkit.  In _Proc. ieee automatic speech recognition and understanding workshop_.
  * Shen et al. (2018) Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al. 2018\.  Natural tts synthesis by conditioning wavenet on mel spectrogram predictions.  In _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 4779–4783. IEEE.
  * Shi et al. (2020) Yao Shi, Hui Bu, Xin Xu, Shaoji Zhang, and Ming Li. 2020.  Aishell-3: A multi-speaker mandarin tts corpus and the baselines.  _arXiv preprint arXiv:2010.11567_.
  * Stowell et al. (2018) Dan Stowell, Yannis Stylianou, Mike Wood, Hanna Pamula, and Hervé Glotin. 2018\.  Automatic acoustic detection of birds through deep learning: the first bird audio detection challenge.  _ArXiv_ , abs/1807.05812.
  * Sun et al. (2019) Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019.  Ernie: Enhanced representation through knowledge integration.  _ArXiv_ , abs/1904.09223.
  * Vainer and Dušek (2020) Jan Vainer and Ondřej Dušek. 2020.  Speedyspeech: Efficient neural speech synthesis.  _arXiv preprint arXiv:2008.03802_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998–6008.
  * Wang et al. (2020) Changhan Wang, Yun Tang, Xutai Ma, Anne Wu, Dmytro Okhonko, and Juan Pino. 2020\.  Fairseq S2T: Fast speech-to-text modeling with fairseq.  In _Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing: System Demonstrations_ , pages 33–39, Suzhou, China. Association for Computational Linguistics.
  * Warden (2018) Pete Warden. 2018.  Speech commands: A dataset for limited-vocabulary speech recognition.  _ArXiv_ , abs/1804.03209.
  * Watanabe et al. (2018) Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, et al. 2018.  Espnet: End-to-end speech processing toolkit.  _arXiv preprint arXiv:1804.00015_.
  * Xu et al. (2019) Haiyang Xu, Hui Zhang, Kun Han, Yun Wang, Yiping Peng, and Xiangang Li. 2019.  Learning alignment for multimodal emotion recognition from speech.  _CoRR_ , abs/1909.05645.
  * Yamagishi et al. (2019) Junichi Yamagishi, Christophe Veaux, Kirsten MacDonald, et al. 2019.  Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit (version 0.92).  _University of Edinburgh. The Centre for Speech Technology Research (CSTR)_.
  * Yamamoto et al. (2020) Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim. 2020.  Parallel wavegan: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram.  In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 6199–6203. IEEE.
  * Yang et al. (2021) Geng Yang, Shan Yang, Kai Liu, Peng Fang, Wei Chen, and Lei Xie. 2021.  Multi-band melgan: Faster waveform generation for high-quality text-to-speech.  In _2021 IEEE Spoken Language Technology Workshop (SLT)_ , pages 492–498. IEEE.
  * Yao et al. (2021) Zhuoyuan Yao, Di Wu, Xiong Wang, Binbin Zhang, Fan Yu, Chao Yang, Zhendong Peng, Xiaoyu Chen, Lei Xie, and Xin Lei. 2021.  Wenet: Production oriented streaming and non-streaming end-to-end speech recognition toolkit.  _arXiv preprint arXiv:2102.01547_.
  * Young et al. (2002) Steve Young, Gunnar Evermann, Mark Gales, Thomas Hain, Dan Kershaw, Xunying Liu, Gareth Moore, Julian Odell, Dave Ollason, Dan Povey, et al. 2002.  The htk book.  _Cambridge university engineering department_ , 3(175):12.
  * Zhang et al. (2020) Binbin Zhang, Di Wu, Zhuoyuan Yao, Xiong Wang, Fan Yu, Chao Yang, Liyong Guo, Yaguang Hu, Lei Xie, and Xin Lei. 2020.  Unified streaming and non-streaming two-pass end-to-end model for speech recognition.  _arXiv preprint arXiv:2012.05481_.
  * Zhao et al. (2021) Chengqi Zhao, Mingxuan Wang, Qianqian Dong, Rong Ye, and Lei Li. 2021.  NeurST: Neural speech translation toolkit.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations_ , pages 55–62, Online. Association for Computational Linguistics.
  * Zheng et al. (2020) Renjie Zheng, Mingbo Ma, Baigong Zheng, Kaibo Liu, Jiahong Yuan, Kenneth Church, and Liang Huang. 2020.  Fluent and low-latency simultaneous speech-to-speech translation with self-adaptive training.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 3928–3937.

##  Appendix A TTS Examples

We use the following sentences as the MOS evaluation test set in Table 7.

  * •

早上好，今天是2020/10/29，最低温度是-3°C。

  * •

你好，我的编号是37249，很高兴为您服务。

  * •

我们公司有37249个人。

  * •

我出生于2005年10月8日。

  * •

我们习惯在12:30吃中午饭。

  * •

只要有超过3/4的人投票同意，你就会成为我们的新班长。

  * •

我要买一只价值999.9元的手表。

  * •

我的手机号是18544139121，欢迎来电。

  * •

明天有62%的概率降雨。

  * •

手表厂有五种好产品。

  * •

跑马场有五百匹很勇敢的千里马。

  * •

有一天，我看到了一栋楼，我顿感不妙，因为我看不清里面有没有人。

  * •

史小姐拿着小雨伞去找她的老保姆了。

  * •

不要相信这个老奶奶说的话，她一点儿也不好。