789200620051199988

11institutetext:  Sydney Institute for Astronomy (SIfA), School of Physics,
University of Sydney, NSW 2006, Australia 22institutetext: Department of
Astronomy, Yale University, P.O. Box 208101, New Haven, CT 06520-8101
33institutetext: Department of Physics and Astronomy, Aarhus University, 8000
Aarhus C, Denmark 44institutetext: LESIA, CNRS, Université Pierre et Marie
Curie, Université Denis Diderot, Observatoire de Paris, 92195 Meudon, France
55institutetext: School of Physics and Astronomy, University of Birmingham,
Edgbaston, Birmingham B15 2TT, UK 66institutetext: Laboratoire AIM, CEA/DSM-
CNRS, Université Paris 7 Diderot, IRFU/SAp, Centre de Saclay, 91191, Gif-sur-
Yvette, France 77institutetext: Space Telescope Science Institute, 3700 San
Martin Drive, Baltimore, Maryland 21218, USA 88institutetext: GEPI,
Observatoire de Paris, CNRS, Université Paris Diderot, 5 Place Jules Janssen,
92195 Meudon, France 99institutetext: High Altitude Observatory, NCAR, P.O.
Box 3000, Boulder, CO 80307, USA 1010institutetext: Harvard-Smithsonian Center
for Astrophysics, 60 Garden Street, Cambridge, MA, 02138, USA
1111institutetext: Instytut Astronomiczny Uniwersytetu Wrocławskiego, ul.
Kopernika 11, 51-622 Wrocław, Poland 1212institutetext: Institut
d’Astrophysique et de Géophysique de l’Université de Liège, 17 Allée du 6
Août, B-4000 Liège, Belgium 1313institutetext: Queen Mary University of
London, Mile End Road, London E1 4NS, UK 1414institutetext: Max Planck
Institute for Astrophysics, Karl Schwarzschild Str. 1, Garching bei München,
D-85741, Germany 1515institutetext: Vrije Universiteit Brussel, Pleinlaan 2,
B-1050 Brussels, Belgium 1616institutetext: Konkoly Observatory, H-1525
Budapest, P.O. Box 67, Hungary

later

# Solar-like oscillations in cluster stars††thanks: Data from Kepler

D. Stello Corresponding author:
11<EMAIL_ADDRESS>S. Basu 22 T. R. Bedding 11 K. Brogaard 33 H.
Bruntt 44 W. J. Chaplin 55 J. Christensen-Dalsgaard 33 P. Demarque 22 Y. P.
Elsworth 55 R. A. García 66 R. L. Gilliland 77 S. Hekker 55 D. Huber 11 C.
Karoff 55 H. Kjeldsen 33 Y. Lebreton 88 S. Mathur 99 S. Meibom 1010 J.
Molenda-Żakowicz 1111 A. Noels 1212 I. W. Roxburgh 1313 V. S. Aguirre 1414 C.
Sterken 1515 R. Szabó 1616

(25 May 2010; 24 June 2010)

###### Abstract

We present a brief overview of the history of attempts to obtain a clear
detection of solar-like oscillations in cluster stars, and discuss the results
on the first clear detection, which was made by the Kepler Asteroseismic
Science Consortium (KASC) Working Group 2.

###### keywords:

stars: fundamental parameters — stars: oscillations — stars: interiors —
techniques: photometric — open clusters and associations: individual (NGC
6819)

##  1 Introduction

Star clusters are extremely important in stellar astrophysics. Most stars form
in open clusters, many of which disperse into the diversity of field stars in
the interstellar medium. Understanding the formation and evolution of cluster
stars is therefore important for achieving a comprehensive theory of stellar
evolution. Stars in a cluster are thought to be formed coevally, from the same
interstellar cloud of gas and dust. Each cluster member is therefore expected
to have some properties in common (age, composition, distance), which
strengthens our ability to constrain our stellar models when tested against an
ensemble of cluster stars, especially for asteroseismic analyses (Gough &
Novotny 1993). Asteroseismology has the capability to probe the interior of
stars and hence help us understand the fundamental physical process that
govern stellar structure and evolution (e.g., Christensen-Dalsgaard 2002). In
particular, the detection of solar-like oscillations provide many modes, which
each carrying unique information about the stellar interior. Stars that
potentially exhibit solar-like oscillations, covering most stars that we see,
are cooler than the red edge of the classical instability strip, and have a
convection zone near the surface (necessary for the excitation of the modes).
Solar-like oscillations are reasonably well described by current theory,
giving us some confidence that we can use them as tools to understand stellar
physics, and hopefully also to learn more about the more subtle aspects of the
oscillations themselves. Combining asteroseismic analysis of solar-like
oscillations with the study of cluster stars has therefore been a long-sought
goal.

##  2 Previous attempts

Kepler is certainly not the first attempt to detect solar-like oscillations in
cluster stars. A quick (and hence incomplete) perusal of the history of
previous attempts to detect solar-like oscillations in open and globular
clusters shows that several attempts were made to detect oscillations since
the early 1990s. Among the most ambitious was that of Gilliland et al. (1993),
who used 4-m class telescopes to target the stars in the open cluster M67 at
the cluster turn-off in a multi-site campaign that lasted one week. While an
impressively low noise level was obtained, the data did not reveal the clear
detection of stellar oscillations (Figure 1). However, a red giant star that
happened to be in the field did show intriguing evidence of excess power in
the expected frequency range (Figure 2). Unfortunately, the length of the time
series did not allow individual modes to be resolved for such an evolved star
with much smaller frequency separations between modes. A clear detection
remained elusive, as oscillations could not be distinguished from the rising
background towards low frequency.

Figure 1: Amplitude spectrum (high-pass filtered) of one of the stars targeted
by Gilliland et al. (1993). The horizontal line marks the expected location of
the oscillations.  Figure 2: Amplitude spectrum of red giant star observed by
Gilliland et al. (1993). Horizontal line marks the expected location of the
oscillations.

Inspired by Gilliand’s results, Stello et al. (2007) targeted specifically the
red giants in M67 during a 6-week long multi-site campaign of 1–2m class
telescopes. Strong evidence for excess power was found in a number of stars,
but no unambiguous detection of the solar-like pattern of equally spaced modes
was claimed by the authors (Figure 3).

Figure 3: Power spectra of three red giant stars observed by Stello et al.
(2007). Black arrow marks the location of the oscillations expected from
scaling the solar value.

In parallel, several attempts to detect oscillations in globular clusters were
carried out. From the ground, Frandsen et al. (2007) aimed at the red giants
in M4, which delivered lower limits on amplitudes, indicating that the low
metallicity of M4 could have the effect of lowering the oscillation
amplitudes. Again, detection was hindered by long-term stability not being
high enough and varying data quality resulting in strong aliasing in the
weighted amplitude spectra. Slightly more successful were the efforts using
the Hubble Space Telescope by Edmonds & Gilliland (1996); Stello & Gilliland
(2009). In the former study, clear variation was found in a large number of
red giants in 47 Tuc, but the low frequency resolution provided by the 40-hour
time series did not allow the authors to establish this as solar-like
oscillations. The later study was aimed at the red giants in the extremely
metal poor NGC 6397, using archival data originally obtained to detect the
cluster’s faint white dwarf population. The far from ideal data of highly
saturated photometry of the red giants meant that only one star showed good
evidence for oscillations, with excess power at the right frequency range and
amplitude. Despite the 27-day long time series, this fell just short for an
unambiguous detection of equally spaced frequencies in this highly evolved
asymptotic giant branch star.

The main conclusion from these previous efforts is that dedicated space-based
missions are required to achieve the ultra-high precision photometry and long-
term stability in order to detect solar-like oscillations in clusters with
such accuracy that they will be useful for asteroseismic analysis.

We note that in addition to the previous marginal detections, these campaigns
resulted in firm detection of oscillations in a number of classical pulsators
that exhibit much large amplitude than solar-like oscillations (see e.g.
Bruntt 2007, and references therein).

##  3 First results from Kepler

Kepler has a unique capability to overcome the shortcomings that have limited
previous efforts aimed at stellar clusters. Both quality and quantity of the
Kepler data outshine that of early explorations by several orders of
magnitude, and it will undoubtedly be the front runner for cluster seismology
in the next 5–10 years.

As reported by Stello et al. (2010), the first month of Kepler data already
revealed clear detection of solar-like oscillations in a large sample of red
giant stars in the open cluster NGC 6819 (see also Gilliland et al. 2010).
Based on the spacecrafts so called long-cadence mode, which provides a time
averaged exposure every 29.4 minutes, detection was reported in 47 red giant
stars that range almost from the bottom to the tip of the red giant branch
(Figure 4). We saw periodicity in the light curves that span about a factor of
100, corresponding to a factor of $\sim 10$ in radius. Two sample light curves
are shown in Figure 5.

Figure 4: HR-diagram of NGC 6819. Empty symbols mark those where a detection
of solar-like oscillations was reported by Stello et al. (2010).  Figure 5:
Kepler time series for two red giants in NGC 6819. Numbers refer to the
numbering in Figure 4. Note the different time scale of the variation.
Photometry and isochrone is of Hole et al. (2009) and Marigo et al. (2008),
respectively.

Power spectra of the stars marked with numbers in Figure 4 are shown in Figure
6. Panels are sorted according to apparent magnitude (brightest at the top),
which for a cluster is indicative of luminosity. One noticeable result is that
not all stars with high membership probability from radial velocity surveys
(see Hole et al. 2009) follow the expected monotonic trend of increasing
frequency of the oscillations (and decreasing amplitude) for decreasing
luminosity. We indicate the expected frequency location with an arrow for
stars that seem to behave strangely compared to the classical scaling
relations for the amplitude and the frequency of maximum power (e.g. Kjeldsen
& Bedding 1995).

Figure 6: Power spectra of 11 stars marked in Figure 4, which are
representative for the entire sample. ‘AM’ indicates that the star is an
asteroseismic member (i.e. observation agrees with scaling relations). Dashed
lines show the measured large frequency separation. For stars where the large
separation could not be determined (no dashed lines), we localised the power
excess from the hump of power in the smoothed power spectrum (solid black
curve). The arrows indicate the expected location of the excess power for
stars where observations do not agree with expectation.

Possible explanations for this behaviour are that these “odd” stars are not
members, or that they have unusual evolution histories.

Stello et al. (2010) were further able to measure the amplitudes of the modes
using the method by Kjeldsen et al. (2009), assuming the relative amplitudes
of the modes of different spherical degree was the same as for the Sun. From
this we could test the $L/M$ scaling relation (Kjeldsen & Bedding 1995; Samadi
et al. 2007), and found that $(L/M)^{0.7}/T_{\mathrm{eff}}^{2}$ provided the
best match to the data.

For further details on what is reported here, we refer to the source paper of
Stello et al. (2010).

##  4 Future

There are four open clusters in Kepler’s field of view. They span a range in
metallicity and age, which brackets the solar values, and are therefore ideal
for testing our current models of stellar evolution (Table 1).

Table 1: Open clusters in Kepler field Cluster | Age | [Fe/H] | Mturnoff
---|---|---|---
| Gyr |  | M⊙
NGC 6866 |  $\sim$0.4 |  $\sim-$0.1 |  $\sim$1.7
NGC 6811 |  $\sim$1.0 |  $\sim-$0.07 |  $\sim$1.5
NGC 6819 |  $\sim$2.5 |  $\sim-$0.05 |  $\sim$1.3
NGC 6791 |  $\sim$8.5 |  $\sim+$0.4 |  $\sim$1.0
Values are from Grundahl et al. (2008) (NGC 6791),
Hole et al. (2009) (NGC 6819), Loktin & Matkin (1994) (NGC 6866)
and unpublished work by Meibom.

In Figure 7 we show $\log(g$) vs $T_{\mathrm{eff}}$ for a representative
sample of the stars in Kepler’s field of view together with the representative
isochrones for the four open clusters that are targets in our future
asteroseismic analyses.

Figure 7: $\log(g$) vs $T_{\mathrm{eff}}$ for stars in Kepler’s field of view.
We represent the four open clusters by suitable isochrones. The order in which
we have plotted the cluster names corresponds to their turn-off stars, with
NGC 6866 having the hottest (heaviest) turn-off stars and NGC 6791 the coolest
(lightest). The dashed line indicates the red edge of the classical
instability strip.

For NGC 6819 we expect to achieve a signal-to-noise level for the turn-off
stars that after 3.5 years of data matches what we see in the bottom panels of
Figure 6. This will provide detection in up to 100 stars ranging stellar
evolution from the main sequence F stars to the asymptotic giant branch
including M giants, as well as a number of blue stragglers. This will
potentially provide unprecedented tests of state-of-the-art stellar evolution
models.

In NGC 6791 we already see evidence for power in the red giants, and expect
firm detections for all stars on this highly populated red giant branch, with
unique potential for testing intrinsic variation among practically identical
stars.

The two younger clusters NGC 6811 and NGC 6866 are less populated but provide
the opportunity to investigate classical pulsators in great detail. NGC 6811
also contains a few He-core burning red giants.

The combination of results from all four clusters promises great prospects for
testing asteroseismic scaling relations on distinct stellar populations that
span a large range in stellar age and brackets the solar metallicity.

###### Acknowledgements.

Funding of the Discovery mission is provided by NASA’s Science Mission
Directorate. The authors thank the entire Kepler team without whom this
investigation would not have been possible. The authors also thank all funding
councils and agencies that have supported the activities for Working Group 2
of the KASC. In particular, DS would like to thank HELAS for support to attend
the HELAS IV meeting in Lanzarote.

## References

  * Bruntt (2007) Bruntt, H. 2007, Communications in Asteroseismology, 150, 326
  * Christensen-Dalsgaard (2002) Christensen-Dalsgaard, J. 2002, Reviews of Modern Physics, 74, 1073
  * Edmonds & Gilliland (1996) Edmonds, P. D., & Gilliland, R. L. 1996, ApJ, 464, L157
  * Frandsen et al. (2007) Frandsen, S., et al. 2007, A&A, 475, 991
  * Gilliland et al. (1993) Gilliland, R. L., et al. 1993, AJ, 106, 2441
  * Gilliland et al. (2010) —. 2010, PASP, 122, 131
  * Gough & Novotny (1993) Gough, D. O., & Novotny, E. 1993, in ASP Conf. Ser. 42: GONG 1992. Seismic Investigation of the Sun and Stars, ed. T. M. Brown, 355
  * Grundahl et al. (2008) Grundahl, F., Clausen, J. V., Hardis, S., & Frandsen, S. 2008, A&A, 492, 171
  * Hole et al. (2009) Hole, K. T., Geller, A. M., Mathieu, R. D., Platais, I., Meibom, S., & Latham, D. W. 2009, AJ, 138, 159
  * Kjeldsen & Bedding (1995) Kjeldsen, H., & Bedding, T. R. 1995, A&A, 293, 87
  * Kjeldsen et al. (2009) Kjeldsen, H., Bedding, T. R., & Christensen-Dalsgaard, J. 2009, in IAU Symposium, Vol. 253, IAU Symposium, 309–317
  * Loktin & Matkin (1994) Loktin, A. V., & Matkin, N. V. 1994, Astronomical and Astrophysical Transactions, 4, 153
  * Marigo et al. (2008) Marigo, P., Girardi, L., Bressan, A., Groenewegen, M. A. T., Silva, L., & Granato, G. L. 2008, A&A, 482, 883
  * Samadi et al. (2007) Samadi, R., Georgobiani, D., Trampedach, R., Goupil, M. J., Stein, R. F., & Nordlund, Å. 2007, A&A, 463, 297
  * Stello & Gilliland (2009) Stello, D., & Gilliland, R. L. 2009, ApJ, 700, 949
  * Stello et al. (2007) Stello, D., et al. 2007, MNRAS, 377, 584
  * Stello et al. (2010) —. 2010, ApJ, 713, L182


# Low-latency Incremental Text-to-Speech Synthesis with
Distilled Context Prediction Network

###### Abstract

Incremental text-to-speech (TTS) synthesis generates utterances in small
linguistic units for the sake of real-time and low-latency applications. We
previously proposed an incremental TTS method that leverages a large pre-
trained language model to take unobserved future context into account without
waiting for the subsequent segment. Although this method achieves comparable
speech quality to that of a method that waits for the future context, it
entails a huge amount of processing for sampling from the language model at
each time step. In this paper, we propose an incremental TTS method that
directly predicts the unobserved future context with a lightweight model,
instead of sampling words from the large-scale language model. We perform
knowledge distillation from a GPT2-based context prediction network into a
simple recurrent model by minimizing a teacher-student loss defined between
the context embedding vectors of those models. Experimental results show that
the proposed method requires about ten times less inference time to achieve
comparable synthetic speech quality to that of our previous method, and it can
perform incremental synthesis much faster than the average speaking speed of
human English speakers, demonstrating the availability of our method to real-
time applications.

Index Terms—  Incremental text-to-speech synthesis, end-to-end text-to-speech
synthesis, knowledge distillation, context estimation, language model

##  1 Introduction

Text-to-speech (TTS) synthesis, which artificially synthesizes speech from
text, can now generate extremely human-like and high-quality speech with the
recent development of deep learning [1, 2, 3, 4]. Most of these TTS methods
are “sentence-level TTS” methods, which take an entire input sentence into
account with an encoder-decoder model to generate the acoustic feature of the
output speech. However, some applications, such as simultaneous speech-to-
speech translation [5, 6, 7] or real-time dialog speech generation [8], need
streaming synthesis, namely “incremental TTS”, that processes small linguistic
units one by one rather than sentence-level TTS. Although incremental TTS can
immediately process the input text, its synthetic speech quality tends to be
lower than sentence-level TTS. The biggest challenge of incremental TTS is to
naturally synthesize the current segment from only the observed context
information without enough subsequent context information. Incremental TTS
that takes only the past observed information into account [9] can efficiently
synthesize the current speech segment without any waiting time, but it tends
to output significantly unnatural and discontinuous speech compared with
sentence-level TTS.  The lookahead-$k$ policy [10], which waits a word
sequence of length $k$ before generating the current segment, successfully
copes with the quality degradation, but it suffers from latency entailed by
waiting for the subsequent text segments at every time step.

We previously proposed an incremental TTS method that leverages a large pre-
trained language model to take future context into account by using only the
observed segment [11] (hereafter, “the conventional method”). This method uses
subsequent text segments generated with GPT2 [12] (hereafter, “pseudo
lookahead”) instead of waiting for the unobserved segment as in the
lookahead-$k$ policy. Although this method can improve the naturalness of
incremental TTS without waiting for the observation of the subsequent segment,
the inference time of GPT2 is critical for low-latency incremental TTS
especially when computational resources are limited. Therefore, for practical
applications, it is necessary to predict the unobserved future context from
the observed information by using only a lightweight model, which is generally
a challenging task (see Section 4.3).

In this paper, we propose a low-latency incremental TTS method that predicts
the unobserved future context with a lightweight model. This method directly
predicts the unobserved future context needed for incremental TTS based on the
linguistic knowledge of the large pre-trained language model without heavy
computation of the inference process. On the basis of the recent success of
task-specific knowledge distillation of a large-scale language model in the
field of natural language processing [13, 14, 15], the proposed method
leverages the teacher-student training to obtain a lightweight context
prediction network from the GPT2-based model. The teacher model used in this
paper is based on the incremental TTS model of our previous work [11], with
some improvements. The student model is an incremental TTS model which has a
simple recurrent network consisting of a single-layer bi-directional long-
short-term memory (BLSTM) as the distilled context prediction network.
Experimental results show that the proposed method 1) achieves the equivalent
inference speed and much higher synthetic speech quality compared to the
method without the subsequent information, 2) attains ten times faster
inference to comparable synthetic speech quality to that of the conventional
method, and 3) performs incremental synthesis much faster than the average
speaking speed of human English speakers, demonstrating the availability of
our method to real-time applications.

##  2 Related work

A number of TTS methods based on neural sequence-to-sequence models have been
proposed [16, 1, 17, 18, 2]. Our method is based on Tacotron2 [1], with some
modifications to formulate incremental TTS. Many recent streaming, incremental
TTS studies also leverage neural sequence-to-sequence models [9, 10, 19, 20,
21, 22, 23]. Yanagita et al. proposed an incremental TTS method using Tacotron
[9]. This method was the first neural sequence-to-sequence incremental TTS, in
which each input text segment is processed independently of the pre- and post-
contexts, which limits the naturalness of output speech. Ma et al. proposed
prefix-to-prefix decoding, which waits for the next $k$ words at every time
step of incremental generation [10]. Although this method takes contexts into
account, as does our method, the training process uses the entire utterance
and the inference process is not segmental, in contrast to our method. We
previously proposed an incremental TTS method with a mechanism for generating
unobserved context by using GPT2 [11], which we call “the conventional method”
in this paper. In contrast to the conventional method, the method proposed
here does not use GPT2 during inference and uses a simple distilled context
prediction network to obtain the context embedding, which significantly
reduces the processing time. To achieve highly accurate context prediction
with a lightweight model, we leverage a knowledge distillation approach [24,
25] where a smaller student model learns to mimic a larger teacher model.
There have been several studies on task-specific knowledge distillation of
large-scale language models in the field of natural language processing [13,
14, 15]. In particular, a study on distilling BERT [26] into a single-layer
BLSTM [14] showed that the student model achieved comparable performance to
the teacher model on several benchmark tasks. Our work focuses on the task of
context estimation in TTS using a similar recurrent architecture for the
student model.

Fig. 1: Proposed incremental TTS method with distilled context prediction
network. Student model can be obtained by distilling GPT2-based context
prediction model into lightweight recurrent network with teacher-student
training.

##  3 Conventional method

This section describes the conventional incremental TTS method [11] that uses
GPT2 [12] and Tacotron2 [1]’s encoder-decoder architecture, as shown on the
left side of Fig. 1. Let $\textrm{\boldmath$w$}_{n}$ be the $n$-th word of the
input sentence, where each input segment consists of $N$ words.
$\textrm{\boldmath$w$}_{1:Nt}=\textrm{\boldmath$w$}_{1},\cdots,\textrm{\boldmath$w$}_{n},\cdots,\textrm{\boldmath$w$}_{Nt}$
represents “the observed segment” and the last $N$-word sequence
$\textrm{\boldmath$w$}_{N(t-1)+1:Nt}=\textrm{\boldmath$w$}_{N(t-1)+1},\cdots,\textrm{\boldmath$w$}_{Nt}$
is “the current segment” in the time step $t$. GPT2 assumes that the
probability distribution of an $M$-word sequence $\textrm{\boldmath$w$}_{1:M}$
can be decomposed into the product of conditional probabilities, as

$\displaystyle
p(\textrm{\boldmath$w$}_{1:M})={\displaystyle\prod_{m=1}^{M}}p(\textrm{\boldmath$w$}_{m}|\textrm{\boldmath$w$}_{1:m-1}).$
(1)

We can obtain a future $L$-word sequence
$\hat{\textrm{\boldmath$w$}}_{Nt+1:Nt+L}=\hat{\textrm{\boldmath$w$}}_{Nt+1},\cdots,\hat{\textrm{\boldmath$w$}}_{Nt+L}$
by sampling from the probability distribution
$p(\textrm{\boldmath$w$}_{Nt+1:Nt+L}|\textrm{\boldmath$w$}_{1:Nt})$, where
$\hat{\textrm{\boldmath$w$}}_{Nt+1:Nt+L}$ becomes the pseudo lookahead for the
future contextual information of incremental TTS. Then, we estimate the
contextual embedding $\textrm{\boldmath$e$}_{\mathrm{pseudo}}$ from a hidden
state of the past observed segment $\textrm{\boldmath$h$}_{1:N(t-1)}$ and that
of the pseudo lookahead $\hat{\textrm{\boldmath$h$}}_{Nt+1:Nt+L}$, as

$\displaystyle\textrm{\boldmath$e$}_{\mathrm{pseudo}}$
$\displaystyle=E(\textrm{\boldmath$h$}_{1:N(t-1)},\hat{\textrm{\boldmath$h$}}_{Nt+1:Nt+L})$
(2) $\displaystyle\hat{\textrm{\boldmath$h$}}_{Nt+1:Nt+L}$
$\displaystyle=encoder(\hat{\textrm{\boldmath$w$}}_{Nt+1:Nt+L})$ (3)
$\displaystyle\textrm{\boldmath$h$}_{1:N(t-1)}$
$\displaystyle=encoder(\textrm{\boldmath$w$}_{1:N(t-1)}),$ (4)

where $E(\cdot)$ denotes a contextual embedding network, which consists of a
style token layer [27]. Then, we obtain the output mel-spectrogram
$\textrm{\boldmath$y$}_{N(t-1)+1:Nt}$ by conditioning the decoder on
$\textrm{\boldmath$e$}_{\mathrm{pseudo}}$, as

$\displaystyle\textrm{\boldmath$y$}_{N(t-1)+1:Nt}$
$\displaystyle=decoder(\textrm{\boldmath$h$}_{N(t-1)+1:Nt},\textrm{\boldmath$e$}_{\mathrm{pseudo}})$
(5) $\displaystyle\textrm{\boldmath$h$}_{N(t-1)+1:Nt}$
$\displaystyle=encoder(\textrm{\boldmath$w$}_{N(t-1)+1:Nt}).$ (6)

After that, we jointly train the encoder, decoder, and $E(\cdot)$ to minimize
an objective function $\mathcal{L}_{\mathrm{target}}$, which can be calculated
from the ground-truth mel-spectrogram and the output mel-spectrogram
$\textrm{\boldmath$y$}_{N(t-1)+1:Nt}$. $\mathcal{L}_{\mathrm{target}}$
consists of the mean squared error (MSE) loss and the stop flag prediction
loss, in the same manner as Tacotron2.

The above method enhances the quality of synthetic speech by using pseudo
lookahead without waiting for the observation of the future context. However,
the latency in synthesis significantly increases as the time step $t$
increases because of the high computational cost of GPT2’s inference.

##  4 Proposed method

This section describes the proposed incremental TTS method. Section 4.1
presents the teacher-student training to obtain the lightweight distilled
context prediction network. Section 4.2 describes the implementation details
of the teacher and student models. In Section 4.3, we discuss the proposed
method in detail based on the results of preliminary experiments.

###  4.1 Knowledge distillation of context prediction network

Fig. 1 illustrates the proposed knowledge distillation method. The context
prediction network with GPT2 is distilled into a simple recurrent network
(hereafter, “the distilled context prediction network”) on the basis of
teacher-student training. The teacher model estimates the contextual embedding
$\textrm{\boldmath$e$}_{\mathrm{pseudo}}^{(T)}$ by using GPT2 and the
contextual embedding network $E(\cdot)$, as shown in equation (2)–(4). In the
student model, the distilled context prediction network directly predicts the
contextual embedding from the observed sentence. We use a pre-trained word
embedding to obtain the distributed representation of the input text, as in
the previous study [14]. We apply a pre-trained FastText [28] to each word of
the observed segment $\textrm{\boldmath$w$}_{1:Nt}$ and obtain a word
embedding sequence $\textrm{\boldmath$v$}_{1:Nt}$. When we define the
distilled context prediction network of the student model as $G^{(S)}(\cdot)$,
the contextual embedding $\textrm{\boldmath$e$}_{\mathrm{pseudo}}^{(S)}$ is
estimated as

$\displaystyle\textrm{\boldmath$e$}_{\mathrm{pseudo}}^{(S)}=G^{(S)}(\textrm{\boldmath$v$}_{1:Nt}).$
(7)

The objective of the teacher-student training is the MSE of the two contextual
embedding vectors, which is formulated as

$\displaystyle\mathcal{L}_{\mathrm{distil}}=||\textrm{\boldmath$e$}_{\mathrm{pseudo}}^{(S)}-\textrm{\boldmath$e$}_{\mathrm{pseudo}}^{(T)}||_{2}^{2}.$
(8)

In our preliminary experiment, we also tried the cosine similarity loss for
$\mathcal{L}_{\mathrm{distil}}$, and the MSE showed slightly better results.
To avoid degrading the naturalness of output speech when distilling the
context prediction network, we also used the target loss
$\mathcal{L}_{\mathrm{target}}$, which can be calculated from the ground-truth
mel-spectrogram and the output mel-spectrogram of the student model, as
described in Section 3. The whole training objective is formulated as

$\displaystyle\mathcal{L}=(1-\lambda)\cdot\mathcal{L}_{\mathrm{target}}+\lambda\cdot\mathcal{L}_{\mathrm{distil}},$
(9)

where $\lambda$ denotes the weight of the teacher-student loss. Since the
training process aims at obtaining a network that predicts the unobserved
context from the observed segment, the student model inherits and fixes the
weights of the encoder and decoder from the teacher model and only updates the
weights of the distilled context prediction network $G^{(S)}$.

###  4.2 Implementation details

The architecture of the TTS model that estimates the mel-spectrogram from text
is the same for the teacher and student models, and it is identical to the
architecture of the conventional method. After the encoder-decoder model
outputs the mel-spectrogram from the current segment as in equation 5, the
WaveGlow Vocoder [29] synthesizes the waveform segment and the output speech
is obtained by combining it with the segments synthesized so far. As in the
study [10], the last $\delta$ frame of the mel-spectrogram of each segment is
copied and padded to the end of the mel-spectrogram before synthesizing the
waveform to alleviate the discontinuity of the concatenation. As a preliminary
experiment, we varied $\delta$ from zero to 20 and objectively compared the
synthetic speech qualities as in Section 5.2. On the basis of this evaluation,
we decided to use $\delta=1$ as it gave the lowest character error rate (CER).
Moreover, we set the number of words in each segment $N$ to two, in the same
manner as our prior work [11], since our preliminary studies found that all
the methods described in Section 5 often generate unintelligible speech with
$N=1$, as reported in Yanagita et al.’s work [9].

Fig. 2: Model architecture of distilled context prediction network. Model
consists of BLSTM layer and fully-connected layers, and it predicts contextual
embedding from word embedding sequence of observed text.

The teacher model’s contextual embedding network $E(\cdot)$ has the same
architecture as the conventional method. When sampling with GPT2, we set the
maximum sampling length to $L=5$ and stop sampling when the end of a sentence
is sampled, instead of always sampling at $L=5$ as in our prior work [11].
This operation generates a context embedding with the pseudo lookahead that is
closer to using the ground-truth lookahead.

The student model consists of a distilled context prediction network that
predicts $\textrm{\boldmath$e$}_{\mathrm{pseudo}}^{(S)}$ from the observed
segment. We use three sizes of distilled context prediction network, i.e.,
“small”, “medium”, and “large”. Fig. 2 illustrates the model architecture of
the distilled context prediction network. It consists of a single-layer BLSTM
without attention, similar to the student model used in the previous work
[14]. The number of embedding dimensions of the word embedding sequence
$\textrm{\boldmath$v$}_{1:Nt}$ is 300, and the number of dimensions of the
hidden state of the BLSTM is set to 100, 300, and 500 in the small, medium,
and large models, respectively. The initial and the last hidden states of the
BLSTM layer are combined and passed to a fully-connected layer for mapping to
200, 600, and 1000–dimensional vectors in the small, medium, and large models,
respectively. After applying a ReLU activation function, it is passed through
another fully-connected layer that maps it to a 256–dimensional vector
$\textrm{\boldmath$e$}_{\mathrm{pseudo}}^{(S)}$, which has the same number of
dimensions as the teacher model’s contextual embedding
$\textrm{\boldmath$e$}_{\mathrm{pseudo}}^{(T)}$.

Fig. 3: Average cosine similarity plots using medium distilled context
prediction network with and without GPT2. Note that results for small and
large distilled context prediction networks are almost identical to those for
medium in this graph.

###  4.3 Discussion

First, let us investigate the necessity of GPT2 for training a student model
with knowledge distillation. The proposed method trains the student model to
obtain the context embedding $\textrm{\boldmath$e$}_{\mathrm{pseudo}}^{(T)}$
that the teacher model estimates using the pseudo lookahead generated with
GPT2. On the other hand, there is an alternative method that does not use
GPT2, which trains the student model to obtain the context embedding by using
the teacher model and the ground-truth lookahead of the training data, instead
of the pseudo lookahead. We calculated the cosine similarities between the
embedding vectors of the teacher and student models at time step $t$ and
compare them with the proposed method (“Proposed”) and the method without GPT2
(“w/o GPT2”). Note that we used the objective function in eq. 9 with
$\lambda=1$ to evaluate only the accuracy of the context estimation. Fig. 3
shows the results. “Proposed” indicates the average cosine similarity of
$\textrm{\boldmath$e$}_{\mathrm{pseudo}}^{(S)}$ and
$\textrm{\boldmath$e$}_{\mathrm{pseudo}}^{(T)}$ at each time step $t$. “w/o
GPT2” is the average cosine similarity of
$\textrm{\boldmath$e$}_{\mathrm{truth}}^{(S)}$ and
$\textrm{\boldmath$e$}_{\mathrm{truth}}^{(T)}$ for each time step $t$. At high
cosine similarity values, the student model can predict the contextual
embedding of the teacher model with high accuracy. The high cosine similarity
at time step $t=0$ implies that context estimation is easier at the beginning
of a sentence; our previous study found the same trend [11]. As a result, we
can conclude that “Proposed” significantly outperforms “w/o GPT2” for all time
steps and model sizes. This indicates that the student model obtained with the
ground-truth sentences cannot be generalized to unknown test data and cannot
predict the contextual embedding of the teacher model so well. On the other
hand, the student model distilled using GPT2 can predict the contextual
embedding with high accuracy for the test data.

Now let us discuss the weight $\lambda$ in Eq. 9. In the case of $\lambda=0$,
the recurrent network simply estimates the contextual embedding for the TTS
model instead of using GPT2’s knowledge because the model uses only the target
loss. The case of $\lambda=1$ uses only the teacher-student loss
$\mathcal{L}_{\mathrm{distil}}$ without the target loss
$\mathcal{L}_{\mathrm{target}}$. In our preliminary experiment, we found that
$\lambda=0$ and $\lambda=0.5$ cases significantly deteriorated the stop flag
prediction accuracy and often led to collapsed output speech. Furthermore,
since $\mathcal{L}_{\mathrm{distil}}$ is much smaller than
$\mathcal{L}_{\mathrm{target}}$, $\lambda$ needs to be set to a value close to
1.0 in order to equalize the scale of these loss values. Therefore, in the
experimental evaluation of Section 5, we chose $\lambda=0.95$ as the case of
jointly using the teacher-student loss and the target loss, and we compared it
with the case of $\lambda=1$ case.

##  5 Experimental evaluations

We conducted experimental evaluations by comparing the following methods: (1)
Ground-truth, ground-truth audio clips included in the test data; (2) Full-
sentence, sentence-level Tacotron2 model [1]; (3) Teacher, the teacher model
described in Section 3; (4) Student, the student model obtained with the
proposed method; (5) Unicontext, the incremental TTS method used only the past
observed segment with the contextual embedding network $E(\cdot)$; (6)
Independent, the incremental TTS model that synthesizes the current speech
segment independently [9]. As described in Section 4.2 and Section 4.3, we
investigated three model sizes and $\lambda=\\{0.95,1.0\\}$ for Student. Note
that we did not include Independent in the evaluation of synthetic speech
quality because its naturalness was significantly lower than those of other
methods, as reported in our prior work [11]. Audio samples with the above
methods are available online111https://takaaki-
saeki.github.io/itts_distil_demo.

###  5.1 Experimental conditions

The dataset used for the evaluation was LJSpeech [30], which consists of
13,100 utterances (about 24 hours) by a single female English speaker. We
randomly selected 100 for the validation set and 500 utterances for the test
set, and used the rest as the training set. The sampling frequency was set to
22.05 kHz. When extracting the mel-spectrogram from each audio clip, we used a
1024-sample frame size, 256-sample hop size, a Hann window function, and an 80
channel mel-filterbank.

As pre-processing for training, we applied a sliding text window with a window
length of three and a hop size of one, as in our previous study [11]. During
inference, the number of words in the input segment
$\textrm{\boldmath$w$}_{N(t-1)+1:Nt}$ was set to two. The Kaldi-based toolkit
[31] was used to extract the ground-truth speech segments from the training
data by using forced alignment. For the evaluation, we used the published pre-
trained GPT2222https://github.com/graykode/gpt-2-Pytorch and
WaveGlow333https://github.com/NVIDIA/waveglow. When training the teacher
model, we performed GPT2-guided fine-tuning for 4000 iterations after
consistently training the TTS model and the context embedding network for
76000 iterations in the same manner as our previous study [11]. We performed
the proposed knowledge distillation by setting the learning rate and batch
size to $10^{-3}$ and 64, respectively. We trained the distilled context
prediction network for 20000 iterations by using the Adam [32] optimizer with
$\beta_{1}=0.9$, $\beta_{2}=0.999$, and $\epsilon=10^{-6}$. We used Intel(R)
Core(TM) i7-6850K @ 3.60 GHz CPUs and an NVIDIA GeForce 1080Ti GPU for
measuring of the inference time.

###  5.2 Objective evaluation of synthetic speech quality

As an objective evaluation of naturalness, we conducted an evaluation using an
automatic speech recognition model to calculate the CER and the word error
rate (WER), defined as the character- and word-level Levenshtein distance
[33], by using the entire 500 utterances of the test data. We used a joint-CTC
Transformer-based model [34] trained on Librispeech [35], which is included in
ESPnet [36]. Table 1 lists the results.

First, Unicontext, which does not use any subsequent information, showed much
higher CER and WER values than sentence-level TTS due to its unstable
generation and discontinuous output speech.  On the other hand, all of the
Student cases had significantly better scores than that of Unicontext,
demonstrating that the distilled context prediction network improves the
quality of the synthetic speech more than when using only the observed
information. Furthermore, comparing the $\lambda=1$ and $\lambda=0.95$ cases
of Student, we can see that the CER for $\lambda=1$ is about 4 % lower than
that for $\lambda=0.95$, indicating that training with only the teacher-
student loss gives better results. This result is consistent with the previous
study on the task-specific knowledge distillation of BERT for NLP tasks [14],
which reported that using only teacher-student loss achieved higher scores.
One possible reason is that, in the case of $\lambda=0.95$, the target loss
causes the over-fitting of the distilled context prediction network to the
target mel-spectrogram, instead of inheriting the context estimation ability
of Teacher model. Moreover, looking at the results for each model size in
Student with $\lambda=1$, we can see that the scores are almost constant
regardless of the model size. Although the CERs for Student methods were lower
than those of Teacher, “Student (medium, $\lambda=1$)” had the same WER score
, suggesting that the knowledge distillation learning can improve the score to
the same level.

###  5.3 Evaluation of inference time

To evaluate the processing time, we measured the cumulative processing time at
each time step. Fig. 4 shows the cumulative processing time for a time step
$t$.

In Fig. 4(a), we can see that Teacher requires a significantly large amount of
processing time for sentence generation at each time step, and it needs about
1.5 second to generate each segment. On the other hand, Student, which is
obtained with the proposed knowledge distillation method, can efficiently
generate each segment in a shorter processing time. Fig. 4(b) shows that the
processing time of Student is comparable to that of Independent and
Unicontext, which do not use the future context. The average amount of
processing of Student is even slightly smaller than that of Independent,
because Independent sometimes fails to predict the stop flag and requires a
large amount of time to finish the decoding process.

Previous studies on the reading rate of English speech have indicated that
100–120 words per minute (wpm) is optimal for an input utterance of
simultaneous interpretation [37] and that the average speech rate for reading
aloud is about 180 wpm [38]. Therefore, the conventional method, which can
synthesize output speech at around 80 wpm, is not sufficient for simultaneous
speech-to-speech translation. On the other hand, the proposed method has a
generation speed of 800 wpm, which is much faster than the average English
reading speed.

Table 1: Evaluation results on synthetic speech quality.

Method | CER | WER | MOS
---|---|---|---
Grund-truth | 5.1 % | 17.9 % | $4.16\pm 0.11$
Full-sentence | 5.5 % | 18.2 % | $3.82\pm 0.11$
Teacher | 7.8 % | 22.2 % | $3.51\pm 0.15$
Unicontext | 20.8 % | 49.4 % | $3.10\pm 0.16$
Student (medium, $\lambda=1$) | 8.4 % | 22.2 % | $3.47\pm 0.15$
Student (medium, $\lambda=0.95$) | 12.7 % | 33.8 % | $3.39\pm 0.15$
Student (small, $\lambda=1$) | 8.5 % | 23.9 % | -
Student (large, $\lambda=1$) | 8.3 % | 23.3 % | -

Fig. 4: Cumulative processing time for time step $t$. (a) compares all
incremental TTS methods, and (b) plots methods other than Teacher.

###  5.4 Subjective evaluation of synthetic speech quality

As a subjective evaluation, we conducted a mean opinion score (MOS) evaluation
test [39] on naturalness. We compared the methods described above except for
“Student (small, $\lambda=1$)” and “Student (large, $\lambda=1$)”, whose error
rates were equivalent to those of “Student (medium, $\lambda=1$)”. Forty
listeners recruited through Amazon Mechanical Turk [40] participated in the
evaluation, and each listener evaluated 30 speech samples, where we randomly
chose five samples from the output utterances of the test data for each
method.

Table 1 shows the average MOS with 95 % confidence intervals. The MOS of
“Student ($\lambda=1$)” was significantly higher than that of Unicontext, as
were the results of the objective evaluation experiment. The difference
between the scores of Teacher and Student was 0.04, indicating that the
proposed knowledge distillation method can make the quality of the lightweight
student model very close to the teacher model. However, since there was still
a quality gap between Teacher and Full-sentence, further research is needed to
improve the model structure of the teacher model.

##  6 Conclusions

We proposed a knowledge distillation method for efficiently estimating
contextual embedding while using the linguistic knowledge of a large pre-
trained language model. The proposed training method using the teacher-student
loss between the context embedding vectors distills knowledge from a
GPT2-based teacher model into a lightweight student model. Experimental
results show that the proposed method achieves ten times faster inference than
the conventional method without degrading synthetic speech quality and
performs incremental synthesis much faster than the average speaking speed of
human English speakers, demonstrating the availability of our method to real-
time applications. Our future work will focus on the model architecture for
incremental TTS to make its synthetic speech quality equivalent to that of
sentence-level TTS methods.

##  7 Acknowledgements

Part of this work was supported by JSPS KAKENHI Grant Number 17H06101 (system
development) and 18K18100 (system development), the MIC/SCOPE #182103104 (core
idea), and JST, Moonshot R&D Grant Number JPMJPS2011 (experiment).

## References

  * [1] J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y. Zhang, Y. Wang, R. Skerrv-Ryan, R. A. Saurous, Y. Agiomvrgiannakis, and Y. Wu,  “Natural TTS synthesis by conditioning WaveNet on mel spectrogram predictions,”  in Proc. ICASSP, Calgary, Canada, Apr. 2018, pp. 4779–4783.
  * [2] Y. Ren, C. Hu, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y. Liu,  “FastSpeech 2: Fast and high-quality end-to-end text to speech,”  in Proc. ICLR, 2021.
  * [3] R. J. Weiss, RJ. Skerry-Ryan, E. Battenberg, S. Mariooryad, and D. P. Kingma,  “Wave-tacotron: Spectrogram-free end-to-end text-to-speech synthesis,”  in Proc. ICASSP, Online, June 2021, pp. 5679–5683.
  * [4] J. Donahue, S. Dieleman, M. Binkowski, E. Elsen, and K. Simonyan,  “End-to-end adversarial text-to-speech,”  in Proc. ICLR, 2021.
  * [5] S. Bangalore, V. K. Rangarajan Sridhar, P. Kolan, L. Golipour, and A. Jimenez,  “Real-time incremental speech-to-speech translation of dialogs,”  in Proc. NAACL, Montreal, Canada, Jun 2012, pp. 437–445.
  * [6] K. Sudoh, T. Kano, S. Novitasari, T. Yanagita, S. Sakti, and S. Nakamura,  “Simultaneous speech-to-speech translation system with neural incremental ASR, MT, and TTS,”  arXiv, vol. abs/2011.04845, 2020.
  * [7] R. Zheng, M. Ma, B. Zheng, K. Liu, J. Yuan, K. Church, and L. Huang,  “Fluent and low-latency simultaneous speech-to-speech translation with self-adaptive training,”  in Proc. EMNLP Findings, Online, 2020, pp. 3928–3937.
  * [8] D. Bertero, F. B. Siddique, C.-S. Wu, Y. Wan, R. H. Y. Chan, and P. Fung,  “Real-time speech emotion and sentiment recognition for interactive dialogue systems,”  in Proc. EMNLP, Austin, Texas, Nov. 2016, pp. 1042–1047.
  * [9] T. Yanagita, S. Sakti, and S. Nakamura,  “Neural iTTS: Toward synthesizing speech in real-time with end-to-end neural text-to-speech framework,”  in Proc. SSW, Vienna, Austria, Sep. 2019, pp. 183–188.
  * [10] M. Ma, B. Zheng, K. Liu, R. Zheng, H. Liu, K. Peng, K. Church, and L. Huang,  “Incremental text-to-speech synthesis with prefix-to-prefix framework,”  in Proc. EMNLP, Online, Nov. 2020, pp. 3886–3896.
  * [11] T. Saeki, S. Takamichi, and H. Saruwatari,  “Incremental text-to-speech synthesis using pseudo lookahead with large pretrained language model,”  IEEE Signal Processing Letters, vol. 28, pp. 857–861, 2021.
  * [12] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever,  “Language models are unsupervised multitask learners,” https://openai.com/blog/better-language-models/, 2019.
  * [13] Y. K. Chia, S. Witteveen, and M. Andrews,  “Transformer to CNN: Label-scarce distillation for efficient text classification,”  Montreal, Canada, Dec. 2018.
  * [14] R. Tang, Y. Lu, L. Liu, L. Mou, O. Vechtomova, and J. Lin,  “Distilling task-specific knowledge from BERT into simple neural networks,”  arXiv, vol. abs/1903.12136, 2019.
  * [15] D. Chen, Y. Li, M. Qiu, Z. Wang, B. Li, B. Ding, H. Deng, J. Huang, W. Lin, and J. Zhou,  “Adabert: Task-adaptive bert compression with differentiable neural architecture search,”  in Proc. IJCAI, Yokohama, Japan, Jul. 2020, pp. 2463–2469.
  * [16] Y. Wang, R. J. Skerry-Ryan, D. Stanton, Y. Wu, Ron J. Weiss, N. Jaitly, Z. Yang, Y. Xiao, Z. Chen, S. Bengio, Q. Le, Y. Agiomyrgiannakis, R. Clark, and R. A. Saurous,  “Tacotron: Towards end-to-end speech synthesis,”  arXiv, vol. abs/1609.03499, 2017.
  * [17] N. Li, S. Liu, Y. Liu, S. Zhao, and M. Liu,  “Neural speech synthesis with transformer network,”  in Proc. AAAI, Honolulu, U.S.A., July 2019, pp. 6706–6713.
  * [18] Y. Ren, Y. Ruan, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y. Liu,  “FastSpeech: Fast, robust and controllable text to speech,”  in Proc. NeurIPS, Vancouver, Canada, Dec. 2019, pp. 3171–3180.
  * [19] B. Stephenson, L. Besacier, L. Girin, and T. Hueber,  “What the future brings: Investigating the impact of lookahead for incremental neural TTS,”  in Proc. Interspeech, Online, Oct. 2020, pp. 215–219.
  * [20] D. S. R. Mohan, R. Lenain, L. Foglianti, T. H. Teh, M. Staib, and A. Torresquintero,  “Incremental text to speech for neural sequence-to-sequence models using reinforcement learning,”  in Proc. Interspeech, Online, Oct. 2020, pp. 3186–3190.
  * [21] N. Ellinas, G. Vamvoukakis, K. Markopoulos, A. Chalamandaris, G. Maniati, P. Kakoulidis, S. Raptis, J. Sig Sung, H. Park, and P. Tsiakoulis,  “High Quality Streaming Speech Synthesis with Low, Sentence-Length-Independent Latency,”  in Proc. INTERSPEECH, Online, Oct. 2020, pp. 2022–2026.
  * [22] B. Stephenson, T. Hueber, L. Girin, and L. Besacier,  “Alternate Endings: Improving prosody for incremental neural tts with predicted future text input,”  arXiv, vol. abs/2102.09914, 2021.
  * [23] Q. He, Z. Xiu, T. Koehler, and J. Wu,  “Multi-rate attention architecture for fast streamable text-to-speech spectrum modeling,”  in Proc. ICASSP, Online, June 2021, pp. 5689–5693.
  * [24] J. Ba and R. Caruana,  “Do deep nets really need to be deep?,”  in Proc. NIPS, Montreal, Canada, Dec. 2014.
  * [25] G. Hinton, O. Vinyals, and J. Dean,  “Distilling the knowledge in a neural network,”  in Proc. NIPS Deep Learning and Representation Learning Workshop, Montreal, Canada, Dec. 2014.
  * [26] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova,  “BERT: Pre-training of deep bidirectional transformers for language understanding,”  in Proc. NAACL, Minneapolis, Minnesota, June 2019, pp. 4171–4186.
  * [27] Y. Wang, D. Stanton, Y. Zhang, R. Skerry-Ryan, E. Battenberg, J. Shor, Y. Xiao, F. Ren, Y. Jia, and R. A. Saurous,  “Style Tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis,”  arXiv, vol. abs/1803.09017, 2018.
  * [28] P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov,  “Enriching word vectors with subword information,”  Transactions of the Association for Computational Linguistics, vol. 5, pp. 135–146, 2017.
  * [29] R. Prenger, R. Valle, and B. Catanzaro,  “WaveGlow: A flow-based generative network for speech synthesis,”  arXiv, vol. abs/1811.00002, 2018.
  * [30] K. Ito and L. Johnson,  “The LJ Speech Dataset,” https://keithito.com/LJ-Speech-Dataset/, 2017.
  * [31] R. M. Ochshorn and M. Hawkins.,  “Gentle: A robust yet lenient forced aligner built on Kaldi,” https://lowerquality.com/gentle/, 2017.
  * [32] D. Kingma and B. Jimmy,  “Adam: A method for stochastic optimization,”  arXiv, vol. abs/1412.6980, 2014.
  * [33] V. Levenshtein,  “Binary codes capable of correcting deletions, insertions and reversals,”  Soviet Physics Doklady, vol. 10, no. 8, pp. 707–710, Feb. 1966\.
  * [34] S. Kim, T. Hori, and S. Watanabe,  “Joint CTC-attention based end-to-end speech recognition using multi-task learning,”  in Proc. ICASSP, New Orleans, U.S.A., Mar. 2017, pp. 4835–4839.
  * [35] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur,  “Librispeech: An ASR corpus based on public domain audio books,”  in Proc. ICASSP, South Brisbane, Australia, Apr. 2015, pp. 5206–5210.
  * [36] S. Watanabe, T. Hori, S. Karita, T. Hayashi, J. Nishitoba, Y. Unno, N. E. Y. Soplin, J. Heymann, M. Wiesner, N. Chen, A. Renduchintala, and T. Ochiai,  “ESPnet: End-to-end speech processing toolkit,”  arXiv, vol. abs/1804.00015, 2018.
  * [37] D. Seleskovitch,  Language and Cognition, pp. 333–341,  Springer US, Boston, MA, 1978.
  * [38] M. Brysbaert,  “How many words do we read per minute? a review and meta-analysis of reading rate,”  Journal of Memory and Language, vol. 109, pp. 104047, 2019.
  * [39] R. Streijl, S. Winkler, and D. Hands,  “Mean opinion score (MOS) revisited: methods and applications, limitations and alternatives,”  Multimedia Systems, vol. 22, pp. 213–227, Mar. 2016.
  * [40] M. Buhrmester, T. Kwang, and S. D. Gosling,  “Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data?,”  Perspectives on Psychological Science, vol. 6, no. 1, pp. 3–5, 2011\.