# Correcting Mispronunciations in Speech using Spectrogram Inpainting

###### Abstract

Learning a new language involves constantly comparing speech productions with
reference productions from the environment. Early in speech acquisition,
children make articulatory adjustments to match their caregivers’ speech.
Grownup learners of a language tweak their speech to match the tutor
reference. This paper proposes a method to synthetically generate correct
pronunciation feedback given incorrect production. Furthermore, our aim is to
generate the corrected production while maintaining the speaker’s original
voice.

The system prompts the user to pronounce a phrase. The speech is recorded, and
the samples associated with the inaccurate phoneme are masked with zeros. This
waveform serves as an input to a speech generator, implemented as a deep
learning inpainting system with a U-net architecture, and trained to output a
reconstructed speech. The training set is composed of unimpaired proper speech
examples, and the generator is trained to reconstruct the original proper
speech. We evaluated the performance of our system on phoneme replacement of
minimal pair words of English as well as on children with pronunciation
disorders. Results suggest that human listeners slightly prefer our generated
speech over a smoothed replacement of the inaccurate phoneme with a production
of a different speaker.

Index Terms: speech synthesis, speech therapy, language acquisition, audio
inpainting

##  1 Introduction

Learning a language involves constantly comparing one’s own speech production
to reference examples given by the speakers of the language. Children at the
early stages of language acquisition make articulatory adjustments to match
better the reference models, and grownups learning a new language adjust their
speech so as it becomes more similar to their teachers.

Children and adults who suffer from speech sound disorder (SSD) have
persistent difficulty pronouncing words or sounds correctly. They are often
referred to speech therapy, where the work is focused on learning the motor
skills of sound production by providing feedback in the form of the correct
production. The most critical part of the treatment is the feedback, which
helps develop a proper model of pronunciation [1].

In this paper, we would like to propose an algorithm to generate synthetic
feedback of proper pronunciation from the wrong pronunciation. Assume, for
example, that Ron has a problem pronouncing /r/ in American English. It can be
because he suffers from SSD or since his native language is Greek and he is
trying to learn English. The system will ask Ron to pronounce the word “
_right_.” Ron will say “ _white_ ,” mispronouncing /r/ with /w/. The system
will say to Ron: “You pronounced _white_ , but you should have pronounced
_right_ ” – where the word “ _right_ ” will be synthesized in Ron’s own voice.
Ron will be able to compare his production with the desired production and try
to fix it.

Our goal is to automatically synthesize a waveform with proper pronunciation
generated from the waveform of wrong pronunciation, which will serve as a
reference or feedback for the speaker. The generated speech should maintain
the original voice of the speaker, as it was found to be a critical factor in
providing effective feedback [1].

The most straightforward approach to this problem is to learn a mapping
between the wrongly pronounced phoneme to a correct production of the phoneme.
To do so, we need a collection of parallel aligned data of the same speaker.
That is a recording of the child’s voice pronouncing the same word wrongly and
correctly. Unfortunately, it is impractical to collect such data, and
therefore we are focused on designing an algorithm that can handle non-
parallel data.

Strömbergsson [2, 3, 1, 4] was the first to suggest a method for automatic re-
synthesis of recorded wrongly-pronounced children’s speech. In that line of
work, a unit selection approach was used to replace an initial voiceless
plosive in a recorded speech sample with another voiceless plosive, retrieved
from a recording of another child. The method was evaluated with adult
listeners, and preliminary results from an evaluation with children with
typical speech as speakers/listeners were reported [1]. While this technique
allows previously impracticable exploration of self-monitoring skills in
children producing misarticulated speech, e.g., velar fronting, it is an _ad-
hoc_ method, which is limited to the correction of utterances with incorrect
production of voiceless plosives. Our goal, however, is to be able to modify
any phoneme while maintaining the speaker’s voice.

Recently, Biadsy _et al._ [5] proposed an end-to-end speech-to-speech
conversion model called _Parrotron_ that maps an input spectrogram to a target
spectrogram, without utilizing any intermediate discrete representation. The
authors showed that the model could be trained to normalize speech from any
speaker, regardless of accent, prosody, and background noise, into the voice
of a single canonical target speaker with a fixed accent and consistent
articulation and prosody. This model is not suitable for our task. Firstly,
there is no parallel data for our task (Paratron is trained on 30,000 hours of
generic synthetically generated speech). Secondly, the _Parrotron_ model is
designed to produce a generic speech, while in our task, it is critical to
retain the child’s original voice. Similarly, algorithms for foreign accent
conversion [6] and pronunciation prediction [7] will not meet our lack of data
and the goal of maintaining the speaker voice.

Methods based on Automatic speech recognition (ASR) have been widely used to
improve and assist speech and language therapy in recent years [8, 9, 10].
Recent work provides feedback through games to encourage children’s engagement
in otherwise repetitive tasks while providing an indirect progress report for
evaluation [11, 12]. The feedback may be a real-time response like a simple
score or visual traffic light concept response or, in other cases, an
immediate real-time impact on the user’s progress in the game with remote
clinical interference like updating task [13]. This type of feedback does not
actively suggest to the users how to improve their pronunciation, and they are
just noted if their production is correct or not. More effective feedback was
proposed by [14]. The children are shown a form of the formants, and
interactively they need to reach a target goal [14].

Our work is related to inpainting works in vision and speech. The concept
matched our goal by removing a segment while retaining the surrounding
information for keeping the speaker’s identity. Our base model architecture is
inspired from [15]. There have been several studies on inpainting,
particularly on speech, that strengthen our concept of context information
with inpainting [16, 17, 18, 19, 20, 21, 22, 23].

Although inpainting provides a suitable _context encoder_ , it can not change
significant components, let alone correct speech segments without any
guidance.

This paper represents an inpainting method to correct speech while retaining
the speaker’s voice. Samples of our model can be found on the project website
https://github.com/MLSpeech/spoken_feedback.

##  2 Problem Setting

Recall the input speech is generated by prompting the user, who potentially
has a production error, to pronounce a word (or a phrase). Hence, the
canonical target word is known, and we are aware that it is most likely
different from the actual surface pronunciation. Throughout the paper, we will
assume that only a single phoneme is mispronounced111If the speaker
mispronounces several phonemes, the chosen target word sequence can be
designed to include only a single mispronounced phoneme at a time..

Formally, let $\bar{\bm{x}}=(\bm{x}_{1},\ldots,\bm{x}_{T})$ denote the input
speech utterance, where each $\bm{x}_{t}\in\mathbb{R}^{D}$ is a
$D$-dimensional vector for $1\leq t\leq T$. In this paper, this vector is
composed of the frequency bins of the Mel-spectrum, but other representations
could be used as well. The duration of the speech utterance, $T$, is not
fixed. We denote by
$\bar{\bm{x}}_{t_{1}}^{t_{2}}=(\bm{x}_{t_{1}},\ldots,\bm{x}_{t_{2}-1})$ the
portion of the speech utterance from time frame $t_{1}$ to time frame
$t_{2}-1$.

The desired _correct_ phoneme sequence corresponding to $\bar{\bm{x}}$ is
denoted as $\bar{p}=(p_{1},\ldots,p_{K})$, where $p_{k}\in\mathcal{P}$, and
$\mathcal{P}$ is the set of the phonemes in the language. We denote the
segmentation of the phonemes in the speech by $\bar{t}=(t_{1},\ldots,t_{K})$,
where $1\leq t_{k}<T$ is the start time (in number of frames) of the phoneme
$p_{k}$. The speech segment associated with the $k$-th phoneme $p_{k}$ is
denoted as $\bar{\bm{x}}_{t_{k}}^{t_{k+1}}$.

Phoneme alignment and segmentation can be obtained by using a forced-
alignment. In our experiments we found out that applying force-alignment with
the canonical phoneme sequence and the actual mispronounced speech, gave good-
enough segmentation for the correction process. Note that practically, we care
only for the phonemes in the vicinity of the mispronounced phoneme.

Denote by $\rho^{*}$ the phoneme symbol that is wrongly pronounced in
$\bar{\bm{x}}$ and needs to be corrected. Without loss of generality, let the
$k$-th phoneme in this sequence be the mispronounced phoneme, namely,
$p_{k}=\rho^{*}$. Our goal is to generate a new speech utterance
$\bar{\bm{y}}$ that is based on the input speech utterance $\bar{\bm{x}}$,
where the segment $\bar{\bm{x}}_{t_{k}}^{t_{k+1}}$ of the wrongly-pronounced
phoneme is replaced with a newly generated segment
$\bar{\bm{y}}_{t_{k}}^{t_{k+1}}$ representing the _desired pronunciation_ of
the phoneme $\rho^{*}$. The replaced phoneme might not necessarily be of the
same duration as the originally pronounced phoneme, and then $t_{k}$ and the
following time segments are changed accordingly.

##  3 Method

Our method first replaces the mispronounced phoneme with complete silence,
i.e., the samples that correspond to this phoneme are replaced with zeros.
Then a neural network called _generator_ gets this processed speech as input
and is aimed at reconstructing back the original speech waveform. During the
training the generator works on examples of correctly pronounced speech
segments, where a segment associated with a phoneme $\rho^{*}$ is replaced
with zeros. Its goal is to regenerate the missing speech. At inference time,
the generator is provided with misproducted speech, and it generates a correct
speech, since it was trained only on examples of correctly pronounced speech.

The generator works on input speech segments of a fixed duration. Let us
denote by $G$ the generator that gets as input a speech segment of duration
$\tau$. The output of the generator is a speech segment of the same duration
$\tau$. We set this duration to be longer by 30% than the maximum phoneme
$\rho^{*}$ duration so as to have enough information of the speaker’s voice in
the replacement context. The speech segment corresponding to the phoneme
$p_{k}$ with its neighborhood is denoted by $\bar{\bm{x}}_{k}^{\tau}$.

Define $\bar{\bm{m}}_{t_{1}}^{t_{2}}$ a binary masking vector of length $\tau$
whose elements are either $0$ or $1$. This vector is set to zeros for frames
in the region $[t_{1},t_{2}-1]$ and to ones otherwise. Specifically, we denote
$\bar{\bm{m}}_{k}=\bar{\bm{m}}_{t_{k}}^{t_{k+1}}$ to be the mask vector of
phoneme $k$. Similarly, we define
$\bar{\bm{p}}_{k}=\bar{\bm{p}}_{t_{k}}^{t_{k+1}}$ the phoneme embedding
sequence at the same region.

The input to the generator is the speech segment when the mask is applied to
it, $\bar{\bm{m}}_{k}\odot\bar{\bm{x}}_{k}^{\tau}$, where $\odot$ stands for
the element-by-element product, and a sequence of phoneme embedding of the
same length $\bar{\bm{p}}_{k}$. The generator’s output,
$\bar{\bm{y}}_{k}^{\tau}=G(\bar{\bm{x}}_{k}^{\tau},\bar{\bm{p}}_{k})$, is a
sequence of vectors with the same representation as $\bar{\bm{x}}_{k}^{\tau}$
(Mel-spectrum bins), and with the same duration. Importantly, the generator
uses the portions of the speech which was not set to zeros to recover the
missing phoneme and maintain the speech smoothness. We now turn to describe
how the generator $G$ is trained effectively.

###  3.1 Training objectives

The most straightforward way to train $G$ is simply by letting it recover the
acoustic signal spanned by the mispronounced phoneme $p_{k}$ and its
neighborhood, including the previous and the following phonemes. Our
motivation is to keep as much of the original signal and fill in the “missing”
phoneme, which is replaced with zeros. The basic model is based on encoder-
decoder-like architecture. We aim to train an encoder with a long-enough
receptive field that captures the contextual information of the phoneme’s
vicinity and a decoder that will be able to generate the missing phoneme while
preserving its surrounding accurately.

The model is trained on unimpaired speech. Specifically, we mask the speech
corresponding to the phoneme $p_{k}$ and train $G$ to recover it. In the basic
model we would like to fulfill two complementary objectives: (i) the recovered
phoneme $p_{k}$ should be similar to the desired phoneme $\rho^{*}$, which in
this case is the original phoneme prior to its masking; and (ii) the
neighborhood of the phoneme should be similar to its original neighborhood.

The first objective deals with the recovered phoneme. The output of the
generator $\bar{\bm{y}}_{k}^{\tau}$ is a speech signal that spans the
recovered phoneme as well as its environment. The span area of the recovered
phoneme is $(1-\bar{\bm{m}}_{k})\odot\bar{\bm{y}}_{k}^{\tau}$ and it should be
similar to the original signal
$(1-\bar{\bm{m}}_{k})\odot\bar{\bm{x}}_{k}^{\tau}$. Similarly, the second
objective deals with the phoneme neighborhood, where the recovered
neighborhood $\bar{\bm{m}}_{k}\odot\bar{\bm{y}}_{k}^{\tau}$ should be similar
to the original environment $\bar{\bm{m}}_{k}\odot\bar{\bm{x}}_{k}^{\tau}$. In
our basic setting we measure similarity using the $\ell_{1}$ loss function,
and the overall objective is:

$\lambda_{1}\Big{|}(1-\bar{\bm{m}}_{k})\odot\bar{\bm{y}}_{k}^{\tau}-(1-\bar{\bm{m}}_{k})\odot\bar{\bm{x}}_{k}^{\tau}\Big{|}\\\
~{}+~{}\lambda_{2}\Big{|}\bar{\bm{m}}_{k}\odot\bar{\bm{y}}_{k}^{\tau}~{},~{}\bar{\bm{m}}_{k}\odot\bar{\bm{x}}_{k}^{\tau}\Big{|}$
(1)

The generator was implemented as a Convolutional Neural Network (CNN) in a
U-net architecture [24]. Similar architecture was used in the task of
restoring missing speech as illustrated in [16, 21]. It seems, however, that
this objective and architecture is less suitable to our task. Empirically, our
system was able to recover the speech of a speaker that was not seen in the
training set. However, when the speaker had speech impairment, the recovered
speech was still impaired, and the mispronounced phoneme was recovered instead
of the desired phoneme $\rho^{*}$. The reason is that the objective in Eq. (1)
is motivated to recover speech in general and not the desired phoneme
$\rho^{*}$ in particular. In the next subsections we will elaborate on our
method of improving the basic model.

Table 1: Relative MOS gap - negative values indicate higher score for reference audio Audio type | Reference | MOS GAP | $\rho^{*}=S$ | $\rho^{*}=SH$ | $\rho^{*}=W$ | $\rho^{*}=R$
---|---|---|---|---|---|---
Vocoder only | Original | -0.045 $\pm$ 0.59 | -0.07 $\pm$ 0.50 | 0.01 $\pm$ 0.66 | -0.03 $\pm$ 0.61 | -0.09 $\pm$ 0.53
Generated | Vocoder only | -0.219 $\pm$ 0.64 | -0.22 $\pm$ 0.67 | -0.05 $\pm$ 0.60 | -0.24 $\pm$ 0.59 | -0.28 $\pm$ 0.68
Smooth concat. | Generated | -0.41 $\pm$ 0.80 | -0.11 $\pm$ 0.77 | -0.07 $\pm$ 0.73 | -0.75 $\pm$ 0.72 | -0.55 $\pm$ 0.79

###  3.2 Acoustic embedding

To further steer the generator to generate the correct phoneme rather than
reconstruct the mispronounced speech we used phoneme acoustic embedding. While
previously the phoneme embedding was created based on the phoneme symbol, now
we propose to use embedding which is based on the acoustic characteristic of
each phoneme. The acoustic phoneme embedding allows the model to measure
whether two acoustic inputs can be associated with the same phoneme or not.

We trained a Siamese structure with cosine similarity loss applied between the
embedding vectors. Siamese network uses the same weights while working in
tandem on two different input vectors to compute comparable output vectors. To
the overall objective of our generator, we add two loss terms. First loss is
aimed at minimizing the cosine distance between the acoustic embedding of the
generated phoneme $\rho^{*}$ and random samples of the same phoneme. For the
second loss term, we picked a contrastive phoneme $q$ and generated a speech
with this phoneme (and not $\rho^{*}$). The loss is aimed at minimizing the
cosine distance between the acoustic embedding of speech generated with
contrastive phoneme and random samples of that phoneme.

###  3.3 Inference

At inference time, we transform sample $\bar{\bm{x}}_{k}^{\tau}$ by feeding it
to the generator $G$ along with the target phoneme embeddings
$\bar{\bm{p}}_{k}$. The new sample $\bar{\bm{y}}_{k}^{\tau}$ is given by
$G(\bar{\bm{m}}_{k}\odot\bar{\bm{x}}_{k}^{\tau},\bar{\bm{p}}_{k})$.

While $G$ generates the desired spectrogram features, we are missing phase
information to invert back to audio. We examined a few possibilities to get
the corresponding phase. Although using the initial wave phase gave us the
best results on the surrounding area, it gave poor results on a different
generated phoneme. Our next approach was using the Griffin-Lim algorithm to
estimate the missing phase information, which still fell a little short. The
results with WaveGlow [25], a flow-based model that generates speech given Mel
spectrograms, were significantly better but some baseline noise was
undesirably added to the audio. We got the best results with HiFi-GAN and used
it to produce the final waveform [26]. An additional considerable advantage of
the HiFi-GAN compared to other methods is in training simplicity and inference
speed.

##  4 Empirical Evaluation

Our generator $G$ is composed of a U-net encoder-decoder. We trained our model
for 450 epochs using Adam optimizer, a batch size of 100, an initial learning
rate of $10^{-4}$ and early stopping. Each component includes 5
1D-convolutional layers with $p$-ReLU activation, in which 2 layers are down-
sampling/up-sampling layers respectively achieved with kernels of size 3,
stride 2. No batch norm was used since it showed no improvement or even
resulted in a slight decrease.

The Siamese network for the acoustic embedding consisted of one layer of
bidirectional GRU with hidden size of 300, followed by a linear layer with
ReLU.

We fine-tuned HiFi-GAN model with generated examples on LibriSpeech dataset.
The examples were generated by taking a sample $\bar{\bm{x}}_{k}^{\tau}$ and
feeding it to $G$ along with the original target phonemes $\bar{\bm{p}}_{k}$
and set $\bar{\bm{m}}_{k}$ the mask vector to all $1$. The new sample is given
by $G(\bar{\bm{m}}\odot\bar{\bm{x}}_{k}^{\tau},\bar{\bm{p}}_{k})$ as the
spectogram matching the original audio clip for fine-tuning the model.

All our models were trained on TIMIT dataset after they were resampled to 22.5
kHz (to match HiFi-GAN). We used the original train and test split, and
further split the training corpus to train and validation sets with a ratio of
0.8 and 0.2, respectively. We used the Mel-spectrogram of the original audio
as the input, with 80 bins, where each bin is normalized by the filter length.
The Mel-spectrogram parameters were FFT size 1024, hop size 256, and window
size 1024.

###  4.1 Results

This section will present our results and the testing methods we used. We
provide an evaluation on two datasets: (i) partial utterances from
LibriSpeech-clean-100 aligned with Montreal Forced Aligner [27]: mimicking
speech correction with minimal-pair words since mispronunciation usually
occurs between specific phonemes in every language, for example, _run, one_
,_ship, sip_ ; and (ii) audio recordings of patients diagnosed with SSD in
Hebrew. We evaluated our approach with qualitative experiments with human
listeners using a crowd-sourcing platform on both datasets.

####  4.1.1 Experiment setup

We tested on 106 utterances from LibriSpeech-clean-100 from $27$ females
speakers and $24$ males speakers. To assess our method, we created three
different complementing utterances for each Original utterance: (i) Generated
\- our model generated output on the corresponding minimal pair word. (ii)
Vocoder only \- applying only HiFi-GAN process on the original utterance.
(iii) Smooth concat. \- we randomly selected a speaker of the same gender and
with the target phoneme preferring utterances from the same target word. we
combined the audio sections using SoX toolkit (provides a cross-fade at the
join, and a wave similarity comparison to help determine the best place at
which to make the concatenation). This method was designed to automatically
simulate resemble the Strömbergsson method and compare the result to our
Generated results. Our evaluation assessed the perceived correctness and
quality of the generated audio by using Amazon mTurk crowd-source platform
with native American English raters. Each rater was presented between 60-100
audio samples for each experiment.

####  4.1.2 Minimal pair

Table 2 presents the accuracy result on phoneme identity texts using similar
testing to ABX testing with minimal-pair words. The test asked participants to
classify a random audio, from {Vocoder only, Generated, Smooth concat},
between the target word, the corresponding minimal pair word, a control option
- an example that was irrelevant and very different from any valid value and
“none of the above.” We assume that the cause for higher performance in the
Smooth concat. method compared to our Generated method is that in the former
contains the exact corrected phoneme while the generator produced an
approximate one.

Table 2: Minimal-pair word correction evaluation by mTurk. The rows with
$\rho^{*}$ indicate the accuracy of the correction with the target phoneme
$\rho^{*}$ .

Audio type | Vocoder only | Generated | Smooth concat.
---|---|---|---
accuracy |  96.0% |  76.4% |  83.7%
switched |  2.2% |  20.8% |  10.9%
none above |  1.7% |  2.8% |  5.4%
$\rho^{*}=S$ |  94.2% |  64.6% |  95.7%
$\rho^{*}=SH$ |  97.8% |  90.0% |  89.4%
$\rho^{*}=W$ |  97.9% |  69.7% |  74.6%
$\rho^{*}=R$ |  91.8% |  86.2% |  77.2%

####  4.1.3 Speaker verification

The duration of each utterance was $\sim 0.4-1.2$ seconds ($1-3$ words). While
cutting a small word from audio mid sentences could decrease the quality of
perception and recognition, adding the context tends to be problematic since
context can create biases in our perception, e.g., “chapter one” vs. “chapter
run”. Consequently, we only took unbiased context, which mainly resulted in
adding articles or conjunctions if possible and unnatural yet possible
combinations, e.g., “gave one” $\to$ “gave run,” or “when then ship” $\to$
“when the sip.”

In the experiment, first, raters listened to the random speech utterance, from
{Original, Vocoder only, Generated} and then to three speech utterances of a
specific speaker (we created three unique $5$ seconds utterances for each
speaker; those utterances were not used for any other tasks). Then, the raters
were asked to determine whether the random utterance is from the same speaker
as the other three utterances. We got accuracy rates of $74.1\%$ on Original,
$74.7\%$ on Vocoder only and $71.0\%$ on the Generated. We suspect that the
main reason for the low values for all audio types is that the tested
utterance is too short, and therefore did not contain enough information for
the speaker to be accurately recognized. Another possible factor is that
LibriSpeech is a book reading dataset in which the speakers produce different
voices for characters to match the book, confusing the raters.

####  4.1.4 Perceived quality evaluation

Mean Opinion Score (MOS) is a numerical measure of subjective human
evaluation, often used to measure synthesized speech quality. Raters were
asked to listen to a random speech utterance from any possible audio type and
choose a score from $1-5$ with increments of $0.5$. The score reflects audio
naturalness and quality. The mean and std MOS results are as follows: Original
$3.77\pm 0.96$, $3.56\pm 0.94$ on Vocoder only and $3.37\pm 0.96$ on the
Generated. participants were significantly more likely to prefer our generated
version than Smooth concat. which received a MOS score of $2.96\pm 1.06$. The
dataset we used (LibriSpeech-clean-100) is known for relatively very clean and
high-quality recordings, which suggests the raters scored relatively low
overall, as the results on the Original indicate.

In the last experiment we tested Conditional MOS: In contrast to standard MOS,
we used a pairwise comparison to verify the statistical significance of our
results. Based on a given audio reference, raters were asked to choose a score
for the second audio, given that the reference has a default score of $3.5$.
The score reflects how natural and of high perceived quality the audio sounds.
Table 1 shows the difference in mean MOS scores between the different audio
types.

####  4.1.5 SSD and ASD testing

Emphasizing the model’s capabilities and possible contribution, we tested our
model on untrained Hebrew language sample recordings with /s/ and /sh/ as
target phonemes. We focus on /s/ and /sh/ since those phonemes can be used on
a model that was train on English and tested on Hebrew.

We selected 51 utterances from 4 children and one adult who suffers from
impaired speech, and 18 utterances from 4 children with ASD. The experiment
was an A/B Test: raters listened to two speech utterances, one Generated and
the Vocoder only audio. The listener was tasked to select which recording
sounded correct to a target word and more natural. Every assignment was
sampled $12-16$ times. The results showed the participants classified the
Generated speech as the correct pronunciation and more natural audio by
$68.1\%$ of the time for all groups, $72.8\%$ for the ASD group, and $66.6\%$
for regular SSD. We computed the statistical significance of our result. Based
on a binomial test (N = 912), participants were significantly more likely to
select the Generated speech as correct than the Vocoder only speech (the worst
the p-value was $3.2e^{-12}$).

##  5 Conclusion

In this work, we present a first attempt to generated a correct production
from incorrect one, while maintaining the speaker’s voice. Results suggest
that our proposed model has a slight advantages in subjective listening tests.
Our future work will focus on improving the phoneme embedding by looking at
self-supervised learning algorithms to improve representation.

## References

  * [1] S. Strömbergsson, Å. Wengelin, and D. House, “Children’s perception of their synthetically corrected speech production,” _Clinical linguistics & phonetics_, vol. 28, no. 6, pp. 373–395, 2014.
  * [2] S. Strömbergsson, “Segmental re-synthesis of child speech using unit selection.” in _ICPhS_ , 2011, pp. 1910–1913.
  * [3] ——, “Synthetic correction of deviant speech–children’s perception of phonologically modified recordings of their own speech,” in _Thirteenth Annual Conference of the International Speech Communication Association_ , 2012\.
  * [4] S. Strömbergsson, G. Salvi, and D. House, “Acoustic and perceptual evaluation of category goodness of /t/ and /k/ in typical and misarticulated children’s speech,” _The Journal of the Acoustical Society of America_ , vol. 137, no. 6, pp. 3422–3435, 2015.
  * [5] F. Biadsy, R. J. Weiss, P. J. Moreno, D. Kanvesky, and Y. Jia, “Parrotron: An end-to-end speech-to-speech conversion model and its applications to hearing-impaired speech and speech separation,” in _The Annual Conference of the International Speech Communication Association (Interspeech)_ , 2019.
  * [6] G. Zhao, S. Ding, and R. Gutierrez-Osuna, “Foreign Accent Conversion by Synthesizing Speech from Phonetic Posteriorgrams,” in _Proc. Interspeech 2019_ , 2019, pp. 2843–2847.
  * [7] D. Soboleva, O. Skopek, M. Šajgalík, V. Cărbune, F. Weissenberger, J. Proskurnia, B. Prisacari, D. Valcarce, J. Lu, R. Prabhavalkar _et al._ , “Replacing human audio with synthetic audio for on-device unspoken punctuation prediction,” in _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 7653–7657.
  * [8] J. McKechnie, B. Ahmed, R. Gutierrez-Osuna, P. Monroe, P. McCabe, and K. Ballard, “Automated speech analysis tools for children’s speech production: A systematic literature review,” _International journal of speech-language pathology_ , vol. 20, no. 6, pp. 583–598, 2018.
  * [9] L. Furlong, S. Erickson, and M. E. Morris, “Computer-based speech therapy for childhood speech sound disorders,” _Journal of communication disorders_ , vol. 68, pp. 50–69, 2017.
  * [10] D.-V. Popovici and C. Buică-Belciu, “Professional challenges in computer-assisted speech therapy,” _Procedia-Social and Behavioral Sciences_ , vol. 33, pp. 518–522, 2012.
  * [11] M. Shahin, B. Ahmed, A. Parnandi, V. Karappa, J. McKechnie, K. J. Ballard, and R. Gutierrez-Osuna, “Tabby talks: An automated tool for the assessment of childhood apraxia of speech,” _Speech Communication_ , vol. 70, pp. 49–64, 2015.
  * [12] A. A. Navarro-Newball, D. Loaiza, C. Oviedo, A. Castillo, A. Portilla, D. Linares, and G. Álvarez, “Talking to teo: Video game supported speech therapy,” _Entertainment Computing_ , vol. 5, no. 4, pp. 401–412, 2014.
  * [13] B. Ahmed, P. Monroe, A. Hair, C. T. Tan, R. Gutierrez-Osuna, and K. J. Ballard, “Speech-driven mobile games for speech therapy: User experiences and feasibility,” _International journal of speech-language pathology_ , vol. 20, no. 6, pp. 644–658, 2018.
  * [14] O. Saz, S.-C. Yin, E. Lleida, R. Rose, C. Vaquero, and W. R. Rodríguez, “Tools and technologies for computer-aided speech and language therapy,” _Speech Communication_ , vol. 51, no. 10, pp. 948–967, 2009.
  * [15] G. Liu, F. A. Reda, K. J. Shih, T.-C. Wang, A. Tao, and B. Catanzaro, “Image inpainting for irregular holes using partial convolutions,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 85–100.
  * [16] A. Marafioti, N. Perraudin, N. Holighaus, and P. Majdak, “A context encoder for audio inpainting,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 27, no. 12, pp. 2362–2372, 2019.
  * [17] N. Perraudin, N. Holighaus, P. Majdak, and P. Balazs, “Inpainting of long audio segments with similarity graphs,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 26, no. 6, pp. 1083–1094, 2018\.
  * [18] A. M. Krémé, V. Emiya, and C. Chaux, “Phase reconstruction for time-frequency inpainting,” in _International Conference on Latent Variable Analysis and Signal Separation_. Springer, 2018, pp. 417–426.
  * [19] I. Toumi and V. Emiya, “Audio inpainting based on joint-sparse modeling,” 2019\.
  * [20] O. Mokrỳ, P. Záviška, P. Rajmic, and V. Veselỳ, “Introducing spain (sparse audio inpainter),” in _2019 27th European Signal Processing Conference (EUSIPCO)_. IEEE, 2019, pp. 1–5.
  * [21] Y.-L. Chang, K.-Y. Lee, P.-Y. Wu, H.-y. Lee, and W. Hsu, “Deep long audio inpainting,” _arXiv preprint arXiv:1911.06476_ , 2019.
  * [22] H. Zhou, Z. Liu, X. Xu, P. Luo, and X. Wang, “Vision-infused deep audio inpainting,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 283–292.
  * [23] M. Kegler, P. Beckmann, and M. Cernak, “Deep speech inpainting of time-frequency masks,” _arXiv preprint arXiv:1910.09058_ , 2019.
  * [24] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical image computing and computer-assisted intervention_. Springer, 2015, pp. 234–241.
  * [25] R. Prenger, R. Valle, and B. Catanzaro, “Waveglow: A flow-based generative network for speech synthesis,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2019, pp. 3617–3621.
  * [26] J. Kong, J. Kim, and J. Bae, “Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis,” _arXiv preprint arXiv:2010.05646_ , 2020.
  * [27] M. McAuliffe, M. Socolof, S. Mihuc, M. Wagner, and M. Sonderegger, “Montreal forced aligner: Trainable text-speech alignment using kaldi.” in _Interspeech_ , 2017, pp. 498–502.