# DNN-based Acoustic-to-Articulatory Inversion using Ultrasound Tongue Imaging
††thanks: The first and second authors were funded by Universidad Industrial
de Santander (2183 grant). The last author was partially funded by the
National Research, Development and Innovation Office of Hungary (FK 124584 and
PD 127915 grants). The Titan X GPU used for the deep learning experiments was
donated by the NVIDIA Corporation.

Dagoberto Porras1, Alexander Sepúlveda-Sepúlveda1 1Escuela de Ingenierías
Eléctrica,
Electrónica y Telecomunicaciones,
Universidad Industrial de Santander
Bucaramanga, Santander, Colombia
<EMAIL_ADDRESS><EMAIL_ADDRESS>Tamás Gábor Csapó2,3
2Department of Telecommunications and Media Informatics,
Budapest University of Technology and Economics
3MTA-ELTE Lendület Lingual Articulation Research Group
Budapest, Hungary
<EMAIL_ADDRESS>
###### Abstract

Speech sounds are produced as the coordinated movement of the speaking organs.
There are several available methods to model the relation of articulatory
movements and the resulting speech signal. The reverse problem is often called
as acoustic-to-articulatory inversion (AAI). In this paper we have implemented
several different Deep Neural Networks (DNNs) to estimate the articulatory
information from the acoustic signal. There are several previous works related
to performing this task, but most of them are using ElectroMagnetic
Articulography (EMA) for tracking the articulatory movement. Compared to EMA,
Ultrasound Tongue Imaging (UTI) is a technique of higher cost-benefit if we
take into account equipment cost, portability, safety and visualized
structures. Seeing that, our goal is to train a DNN to obtain UT images, when
using speech as input. We also test two approaches to represent the
articulatory information: 1) the EigenTongue space and 2) the raw ultrasound
image. As an objective quality measure for the reconstructed UT images, we use
MSE, Structural Similarity Index (SSIM) and Complex-Wavelet SSIM (CW-SSIM).
Our experimental results show that CW-SSIM is the most useful error measure in
the UTI context. We tested three different system configurations: a) simple
DNN composed of 2 hidden layers with 64x64 pixels of an UTI file as target; b)
the same simple DNN but with ultrasound images projected to the EigenTongue
space as the target; c) and a more complex DNN composed of 5 hidden layers
with UTI files projected to the EigenTongue space. In a subjective experiment
the subjects found that the neural networks with two hidden layers were more
suitable for this inversion task.

###### Index Terms:

articulatory, ultrasound, deep neural networks, inversion

##  I Introduction

Speech sounds (=acoustics) are produced as the coordinated movement of the
speaking organs (=articulation). There are several available methods to model
the relation of articulatory movements and the resulting speech signal
(acoustic-to-articulatory inversion [1, 2, 3] and articulatory-to-acoustic
mapping [4, 5, 6, 7]).

###  I-A Acoustic-to-Articulatory Inversion

The articulatory movements are directly linked with the acoustic speech signal
in the speech production process. The acoustic-to-articulatory inversion (AAI)
[1] methods estimate articulatory movements from the acoustic speech signal.
Recently, there has been a significant interest in AAI, because learning the
correlation between articulatory information and acoustics could improve the
performance of several tasks such as speech recognition [8, 9], speech
synthesis [10], Speaker Verification [11], and talking heads [12]. For
Acoustic-to-articulatory inversion there are two scopes : Subject/Speaker
Dependent (SD-AAI) and Independent (SI-AAI) [13]. In this paper, we deal with
SD-AAI.

Several methods have been proposed to tackle the SD-AAI problem including
codebooks [14], Gaussian Mixture Models (GMM’s) [2], Hidden Markov Models
(HMM’s) [15], Mixture Density Networks [16]. Furthermore, during the past few
years Deep Neural Networks (DNNs) improved accuracies in several applications
such as phone recognition [17]. Then, researchers started to use deep
architectures to accomplish an effective articulatory Inversion [18, 19], even
having such success at obtaining the state-of-the-art accuracy [20] in AAI
task.

Definitely, all these approaches need parallel acoustic-articulatory data for
training the AAI model. Hence, these studies are based on ElectroMagnetic
Articulography (EMA) data [2, 15, 16, 18, 19, 20]. However, Ultrasound Tongue
Imaging (UTI) is a technique of higher cost-benefit if we take into account
equipment cost, portability, safety and visualized structures.

In fact, phonetic research has employed 2D ultrasound for a number of years
for investigating tongue movements during speech [21, 22, 23]. Usually, when
the subject is speaking, the ultrasound transducer is placed below the chin,
resulting in mid-sagittal images of the tongue movement. The typical result of
2D ultrasound recordings is a series of gray-scale images in which the tongue
surface contour has a greater brightness than the surrounding tissue and air.
For a guide to tongue ultrasound imaging and processing, see [22]. An example
ultrasound image can be found in Fig. 1.

Figure 1: Vocal tract (left) and a sample ultrasound tongue image (right),
with the same orientation.

###  I-B Articulatory-to-Acoustic Mapping

In the relevant field of articulatory-to-acoustic mapping, already several
types of articulatory tracking equipment types have been used. Over the last
decade, there has been an increased interest in the analysis, recognition and
synthesis of silent speech, which is a form of spoken communication where an
acoustic signal is not produced, that is, the subject is just silently
articulating without producing any sound. Systems which can perform the
automatic articulatory-to-acoustic mapping are often referred to as “Silent
Speech Interfaces” (SSI) [4]. Such an SSI can be applied to help the
communication of the speaking impaired (e.g. patients after laryngectomy), and
in situations where the speech signal itself cannot be recorded (e.g.
extremely noisy environments or certain military applications). As the
articulatory recording equipment, typically ultrasound tongue imaging [24, 25,
26, 5, 27, 6, 28, 29, 30], electromagnetic articulography [31, 32, 33, 34],
permanent magnetic articulography (PMA) [35, 36] and surface electromyography
(sEMG) [37, 38, 39, 40, 41] are used. Of course, the multimodal combination of
these methods is also possible [42], and the above methods may also be
combined with a simple video recording of the lip movements [26].

###  I-C Deep neural networks in the inversion and mapping fields

As deep neural networks (DNNs) became dominant in more and more areas of
speech technology, such as speech recognition [43], speech synthesis [44] and
language modeling [45], it is natural that the recent studies have attempted
to solve the acoustic-to-articulatory inversion and articulatory-to-acoustic
conversion problems using deep learning.

In the inversion field, researchers started to deal with deep architectures to
accomplish an effective articulatory inversion. Uria and his colleagues
compared a deep neural network and a deep trajectory mixture density network,
and obtained better inversion accuracies with the latter than smoothing the
results of a DNN [18]. They claim to be the first demonstration of applying
deep architectures to the above complex, time-varying regression problem. Wu
and his colleagues used the DNN-based acoustic-to-articulatory inversion
framework for a real-time speech-driven talking avatar system [19]. Since the
mapping between the acoustic features and the articulatory movements is non-
linear, they tried four different kinds of models including GLM, GMM, ANN and
DNN to validate their ability in describing the mapping relationship; and the
DNN was found to be the best. Liu et al. tried bidirectional LSTMs and a deep
recurrent mixture density network for the AAI task [20], which take into
account the articulatory movements as a time series, as compared to the static
DNN approaches before. The advantage of the recurrent neural network in this
scenario is that it can learn proper context information on its own without
the requirement of externally specifying a context window.

In the mapping field, there was a study about sEMG speech synthesis in
combination with a deep neural network [39, 40]. In another paper a multimodal
Deep AutoEncoder was used to synthesize sung vowels based on ultrasound
recordings and a video of the lips [27]. Gonzalez and his colleagues compared
GMM, DNN and RNN [36] for PMA-based direct synthesis, Csapó et al. used DNNs
to predict the spectral parameters [6] and F0 [28] of a vocoder using UTI as
articulatory input. For the prediction of the V/U flag and F0 using
articulatory input, multiple DNN architectures were compared, including DNN,
RNN and LSTM neural networks [46, 47].

According to our literature survey, in the AAI field, we only registered a
single work in mapping between acoustic features of the speech signal and
ultrasound images of the tongue [3]. They employed Deep Auto-encoders to
obtain the data representations, then with a DNN framework obtained a better
performance than the GMM method. They were motivated by the well known success
in accuracy of deep architectures and the desire to go beyond the AAI results.
However, the study was quite limited as it only focused on Chinese vowels,
i.e. they did not test longer speech sequences.

###  I-D Current study

The current paper focuses on the mapping from speech acoustic features to
ultrasound images. We show that simple fully connected deep neural networks
are able to predict articulatory information from spectral features. Finally
we evaluate our methods with objective and subjective measures.

Figure 2: Framework where it is showed the two input articulatory features
used for the DNN’s.

##  II Methods

###  II-A Data set

The data acquisition setup in this work is the same as [6]. One Hungarian
female subject (31 years old) with normal speaking abilities was recorded
while reading sentences aloud from the PPSD database [48] in Hungarian
language (altogether 209 sentences, of which 200 were used for training and
validation; and nine for testing). The tongue movement was also recorded in
midsagittal orientation using a “Micro” ultrasound system (Articulate
Instruments Ltd.) with a 2-4 MHz / 64 element 20mm radius convex ultrasound
transducer at 82 fps. During the recordings, the transducer was fixed using an
ultrasound stabilization headset (Articulate Instruments Ltd.). Lip video
recording was also done [6], but was not used in the current study. The speech
signal was recorded with an Beyerdynamic TG H56 omnidirectional condenser
microphone that was mounted to the helmet approximately 5cm from the lips.
Both the microphone signal and the ultrasound synchronization signals were
digitized using an M-Audio – MTRACK PLUS external sound card at 44 100 Hz
sampling frequency and were resampled to 22 050 Hz. The ultrasound and the
audio signals were synchronized using the frame synchronization output of the
equipment with the Articulate Assistant Advanced software (Articulate
Instruments Ltd.). In the experiments below and similar to [6], the raw
scanline data of the ultrasound was used.

###  II-B Ultrasound Feature Extraction

The original raw ultrasound grayscale images of 64x842 pixels were resized to
64x64 using a bicubic interpolation. This reduction did not significantly
affect the visual content of the images; instead it allows a compact size
dataset, which decreases the DNNs training, synthesis time and the risk of
overfitting.

In order to obtain features from the ultrasound images we tried two
approaches, as can be seen in the Figure 2. First, all the 64x64 pixels from
the UTI are used as a feature. And second, we perform the ’Eigentongue’ (ET)
feature extraction method [49], which consists on finding a set of orthogonal
images (EigenTongues) that constitute a subspace for the representation of all
likely tongue configurations. In practice, the eigentongues are extracted by
applying PCA on the training data and the eigenvectors obtained are defined as
the eigentongue. For each image, 128 ETs are extracted. Figure 3 shows a
sample ultrasound image, one predicted by the DNN, and the first two principal
ETs (out of 128), respectively. The eigentongue components extracted are
supposed to encode the maximum amount of relevant information present in the
images such as tongue position, hyoid bone and muscles are also present.

Figure 3: Reconstructed frame on the top-right from the acoustic information.
Original frame is on top-left. ET1 and ET2 (the first two principal components
out of 128) are showed in the bottom.

###  II-C Acoustic Features

For each 12 ms long audio frame (dictated by 82 fps ultrasound rate), 25 Mel-
Frequency Cepstral Coefficients (MFCC) were extracted. Also, the first
derivative $\Delta$ (known as the speed component) was included. In total an
acoustic feature vector of dimension $50$ is calculated to represent the audio
information [50].

TABLE I: Evaluation of the three different DNN configurations and ultrasound images features Hidden Layers | UTI features | MSE | SSIM | CW-SSIM
---|---|---|---|---
Mean | Std.dev. | Mean | Std.dev. | Mean | Std.dev.
$5$ x $5000$ units |  $128$ ETs | $85.54$ | $11.32$ | $0.69$ | $0.09$ | $0.79$ | $0.06$
$2$ x $1000$ units |  $64x64$ pixels | $86.02$ | $11.04$ | $0.71$ | $0.08$ | $0.81$ | $0.06$
$2$ x $1000$ units |  $128$ ETs | $85.03$ | $11.43$ | $0.72$ | $0.09$ | $0.81$ | $0.06$

###  II-D DNN architecture

The main goal of this work is to obtain ultrasound images (being the target)
from the MFCC features (being the input of the neural network). For this
purpose, we trained two DNN architectures: one with 5 hidden layers, with each
hidden layer consisting of 5000 rectified neurons; and the other one with 2
hidden layers, each hidden layer consisting of 1000 rectified neurons. The
input dimension (MFCC) is 50 (see II-C) and the output is different for the
two approaches: 128 for the ETs and 64x64 pixels for the other hand, as Figure
2 shows. In all cases, the training criterion was the Mean Squared Error (MSE)
between the predicted and the target features. In addition, we did a
hyperparameter optimization to get better results; the details are in the
Experimental section IV. Finally, the DNN architectures are trained with 200
files containing 56 773 ultrasound images for training and 6 309 for
validating (10% from total number of images).

##  III Discussion

According to our literature survey, we think that our work cannot be compared
with any other research. The single study of [3] is the most similar work in
acoustic-to-articulatory inversion with ultrasound images, however they just
perform the AAI on vowel sounds. In our study we expected going beyond that
approach, performing an AAI over full sentences. Although the Complex-Wavelet
Structural Similarity Index (CW-SSIM) index seems to be the best quality
measure for comparing ultrasound images [51], we decided to include other two
Quality indexes: the SSIM and the CW-SSIM indexes [52].

Figure 4: Image Quality measures in one test sentence. In the legend the terms
before de stripe floor (”_”) means which image is the ground true, the term
after the stripe floor represent the image which we are comparing. for example
O_rec represent the original image (ground truth) compared with the DNN
predicted or reconstructed image. To be clear, PCA term is the original image
but recovered from the PCA representation or EigenTongue space (ground truth).

##  IV Experimental Results

###  IV-A Image Quality Objective Measurements

To measure the quality of the reconstructed ultrasound images, three metrics
were used. Each Quality measure is calculated over each frame, where the True
image $y$ is the resized original image and $\hat{y}$ is the estimated image
from the DNN architecture.

  * •

Mean Square Error (MSE)

$MSE(y,\hat{y})=\frac{1}{n_{samples}}\sum_{i=0}^{n_{samples}-1}(y_{i}-\hat{y_{i}})^{2}$
(1)

where $n_{samples}$ are the 64x64 pixels (4096).

  * •

The structural similarity index (SSIM) [53], which measures three kinds of
visual impact of changes in luminance $l$, contrast $c$ and structure $s$
between two images.

$SSIM(y,\hat{y})=[l(y,\hat{y})]^{\alpha}[c(y,\hat{y})]^{\beta}[s(y,\hat{y})]^{\gamma}$
(2)

In our experiment the SSIM index is calculated by 11x11 circular-symmetric
Gaussian weighting function, with standard deviation of 1.5 pixels. The
statistics for visual impact changes are calculated as follows in [51].

  * •

Complex wavelet structural similarity (CW-SSIM) [52] is an extension of the
SSIM method to the complex wavelet domain, which is a novel image similarity
measurement robust to small distortions.

$CWSSIM(y,\hat{y})=\frac{2\arrowvert\sum_{l=1}^{L}w_{y,l}w_{\hat{y},l}^{*}\arrowvert+K}{\sum_{l=1}^{L}|w_{y,l}|^{2}+\sum_{l=1}^{L}|w_{\hat{y},l}|^{2}+K}$
(3)

where $w$ represents the complex wavelet coefficients of the two images. The ∗
indicates the complex conjugate of $w$, and $K$ is a small positive
stabilizing constant [51]. In both SSIM and CW-SSIM, the value 1 means the
contents in the two images compared are the same, and 0 is the minimum value
(for the most diverse images).

To demonstrate the objective image quality measures, in Figure 4 we can see
the three quality measures applied to one sentence from our experiment. With
this figure we show that the CW-SSIM is better than the MSE and SSIM measures.
For this purpose, we draw three lines for each graph. The green line is
obtained by comparing the original image and the reconstructed (legend =
’O_rec’). As a similar result, in the yellow line we compare a smooth version
of the original image and the reconstructed (legend = ’PCA_rec’). And to be
coherent we expect as the best result the comparison with the original image
and the smooth version in the red line (legend = ’O_PCA’).

The red line shows what we expect to be the best result to be coherent,
because we compare the original image to the recovered PCA or ET version from
the original. We denote as ’smooth’ version the image which is represented in
the Eigen Tongue Space and then transformed to the original space (being a
64x64 pixel image). This process is done by PCA. The PCA-recovered image looks
the same as the original (the same tongue contour and visualized structures)
but without the speckle noise that the original image contains. So, the figure
4 suggests the CW-SSIM index could be the best measure in our experiments
because it is more coherent than the other indexes.

###  IV-B Reconstruction Results

We compared 3 systems for acoustic-to-articulatory inversion using the data of
a single female Hungarian speaker. The results are shown in Table I. Our
experimental results suggest that with a simple DNN of 2 layers and 1000
units, we can get slightly better results compared to a complex 5-DNN layer of
5000 units. Moreover, it seems that is not necessary to project the ultrasound
images to an EigenTongue space, because the performance is almost the same
when predicting the image pixels directly. In addition, a reconstructed frame
as the output of the DNN is shown in Figure 3 top right. Also a hyperparameter
optimization was performed. Several configurations were tested including the
number of neurons in the 5 hidden layers, optimizer (’rmsprop’, ’adam’, ’sgd’)
and batch size. However the results are very close to the showed in this work.
Finally we think that the results presented in the Table I are very close due
to our limited amount of data. So it will be necessary to get more testing
data.

###  IV-C Subjective test

Figure 5: Results of the subjective listening test concerning naturalness
(synchrony of tongue ultrasound video and speech). The errorbars show the 95%
confidence intervals.

In order to determine which proposed system is closer to natural articulatory
movement, we conducted an online MUSHRA-like (MUlti-Stimulus test with Hidden
Reference and Anchor) listening test [54]. The advantage of MUSHRA is that it
allows the evaluation of multiple samples in a single trial without breaking
the task into many pairwise comparisons. Our aim was to compare the natural
tongue ultrasound videos with the generated videos with the various approaches
and with a lower anchor. The lower anchor was a video with almost constant
tongue movement. In the tests, the listeners had to rate the naturalness
(i.e., whether the tongue movement is in synchrony with speech) of each
stimulus in a randomized order relative to the reference (which was the PCA-
compressed version of the natural sentence), from 0 (highly unnatural) to 100
(highly natural).

For the subjective test, 9 sentences were chosen which were not included in
the training of the DNNs. We chose three types of the DNN configurations
listed in Table I. Together with the natural, and lower anchor samples, 45
videos were included in the test (1 speaker $\cdot$ 5 types $\cdot$ 9
sentences).

Altogether six listeners participated in the main test (two females, four
males). Three of them were native speakers of Hungarian, and four of them had
background in speech research. The subjects were between 23–44 years (mean: 32
years). On average the whole test took 11 minutes to complete. The MUSHRA
scores of the listening test are presented in Fig. 5 for the natural, lower
anchor videos and the three DNN models. In general, ’Natural’ sentences should
yield 100% in MUSHRA type tests. The ’lower anchor’ type ranked the lowest, as
videos contained an almost constant tongue. The utterance types in which the
ultrasound images were predicted based on the spectral features were ranked
around 40–57%, indicating that they are better than the lower anchor but still
far away from the original tongue movement video.

The ratings of the listeners were compared by Mann-Whitney-Wilcoxon ranksum
tests as well, with a 95% confidence level, indicating that the 5DNN+ET system
was significantly different from the 2DNN and 2DNN+ET systems, while the
latter two were not found to be different from each other by the subjects.
However, Fig. 5 shows slightly better results for the DNN with 2 layers using
the EigenTongues method. According to the preference of the subjects of the
listening test, the ’2DNN+ET’ approach ranked the best, showing that a neural
network with two hidden layers was more suitable for this inversion task.

When checking the videos, we also observed that the resulting image sequence
is not always smooth, i.e. the consecutive tongue ultrasound images can
contain abrupt changes. This can happen because with the above DNN, we are not
taking into account the sequential nature of the articulatory and acoustic
data.

##  V Conclusions

In this work we have implemented three different kinds of AAI systems. For all
cases we used deep architectures. The results over the quality measures used
indicate that with a simple DNN consisting of two hidden layers can perform
the best accuracy in the articulatory inversion task. However there is no much
difference compared with the other system configurations, which include trying
with: EigenTongue and the UTI as representation of the articulatory images.

Moreover, we performed a hyperparameter optimization in order to obtain better
results, but again we did not find a significant improvement and we think it
could be due to our limited amount of data. Despite that, as a preliminary
result, we can conclude that the Eigen Tongue representation procedure is not
necessary.

In a similar work, Wei and his colleagues [3] predicted ultrasound tongue
images from speech input – using only several Chinese vowels as training data.
Compared to this, our results show that the ultrasound-based acoustic-to-
articulatory inversion is possible using continuous speech.

In conclusion, more testing data is necessary, and, training other deep
architectures can be useful, for example Convolutional Neural Networks (CNN)
[55, 56]. Also we think that one could get a significant improvement including
temporal information. For this purpose, we suggest using deep architectures
such as RNN, LSTM or BLSTM layers. In other hand, the CW-SSIM index seems to
be more coherent than the other quality measures. As a future work, we propose
to compare the results presented in this work with a GMM approach and include
temporal information to the features.

##  VI Acknowledgements

This work is the result of an internship done by Dagoberto Porras at Budapest
University of Technology and Economics (BME). It would not have been possible
with the support of Universidad Industrial de Santander (UIS), Colombia. We
thank the subjects taking part in the experiments.

## References

  * [1] K. Richmond, “Estimating articulatory parameters from the acoustic speech signal,” 2002.
  * [2] T. Toda, A. W. Black, and K. Tokuda, “Statistical mapping between articulatory movements and acoustic spectrum using a gaussian mixture model,” _Speech Communication_ , vol. 50, no. 3, pp. 215–227, 2008.
  * [3] J. Wei, Q. Fang, X. Zheng, W. Lu, Y. He, and J. Dang, “Mapping ultrasound-based articulatory images and vowel sounds with a deep neural network framework,” _Multimedia Tools and Applications_ , vol. 75, no. 9, pp. 5223–5245, 2016.
  * [4] B. Denby, T. Schultz, K. Honda, T. Hueber, J. M. Gilbert, and J. S. Brumberg, “Silent speech interfaces,” _Speech Communication_ , vol. 52, no. 4, pp. 270–287, 2010.
  * [5] T. Hueber, E.-l. Benaroya, B. Denby, and G. Chollet, “Statistical Mapping Between Articulatory and Acoustic Data for an Ultrasound-Based Silent Speech Interface,” in _Proc. Interspeech_ , Florence, Italy, 2011, pp. 593–596.
  * [6] T. G. Csapó, T. Grósz, G. Gosztolya, L. Tóth, and A. Markó, “DNN-Based Ultrasound-to-Speech Conversion for a Silent Speech Interface,” in _Proc. Interspeech_ , Stockholm, Sweden, 2017, pp. 3672–3676.
  * [7] L. Tóth, G. Gosztolya, T. Grósz, A. Markó, and T. G. Csapó, “Multi-Task Learning of Phonetic Labels and Speech Synthesis Parameters for Ultrasound-Based Silent Speech Interfaces,” in _Proc. Interspeech_ , Hyderabad, India, 2018, pp. 3172–3176.
  * [8] J. Frankel, K. Richmond, S. King, and P. Taylor, “An automatic speech recognition system using neural networks and linear dynamic models to recover and model articulatory traces,” 2000.
  * [9] S. King, J. Frankel, K. Livescu, E. McDermott, K. Richmond, and M. Wester, “Speech production knowledge in automatic speech recognition,” _The Journal of the Acoustical Society of America_ , vol. 121, no. 2, pp. 723–742, 2007\.
  * [10] Z.-H. Ling, K. Richmond, J. Yamagishi, and R.-H. Wang, “Integrating articulatory features into hmm-based parametric speech synthesis,” _IEEE Transactions on Audio, Speech, and Language Processing_ , vol. 17, no. 6, pp. 1171–1185, 2009.
  * [11] M. Li, J. Kim, A. Lammert, P. K. Ghosh, V. Ramanarayanan, and S. Narayanan, “Speaker verification based on the fusion of speech acoustics and inverted articulatory signals,” _Computer speech & language_, vol. 36, pp. 196–211, 2016.
  * [12] L. Wang, X. Qian, W. Han, and F. K. Soong, “Synthesizing photo-real talking head via trajectory-guided sample selection,” in _Eleventh Annual Conference of the International Speech Communication Association_ , 2010.
  * [13] A. Illa and P. K. Ghosh, “Low resource acoustic-to-articulatory inversion using bi-directional long short term memory,” _Proc. Interspeech 2018_ , pp. 3122–3126, 2018.
  * [14] S. Ouni and Y. Laprie, “Modeling the articulatory space using a hypercube codebook for acoustic-to-articulatory inversion,” _The Journal of the Acoustical Society of America_ , vol. 118, no. 1, pp. 444–460, 2005.
  * [15] L. Zhang and S. Renals, “Acoustic-articulatory modelling with the trajectory hmm,” 2008.
  * [16] K. Richmond, “A trajectory mixture density network for the acoustic-articulatory inversion mapping,” in _Ninth International Conference on Spoken Language Processing_ , 2006.
  * [17] A.-r. Mohamed, G. E. Dahl, G. Hinton _et al._ , “Acoustic modeling using deep belief networks,” _IEEE Trans. Audio, Speech & Language Processing_, vol. 20, no. 1, pp. 14–22, 2012.
  * [18] B. Uria, I. Murray, S. Renals, and K. Richmond, “Deep architectures for articulatory inversion,” in _Thirteenth Annual Conference of the International Speech Communication Association_ , 2012.
  * [19] Z. Wu, K. Zhao, X. Wu, X. Lan, and H. Meng, “Acoustic to articulatory mapping with deep neural network,” _Multimedia Tools and Applications_ , vol. 74, no. 22, pp. 9889–9907, 2015.
  * [20] P. Liu, Q. Yu, Z. Wu, S. Kang, H. Meng, and L. Cai, “A deep recurrent approach for acoustic-to-articulatory inversion,” in _Acoustics, Speech and Signal Processing (ICASSP), 2015 IEEE International Conference on_. IEEE, 2015, pp. 4450–4454.
  * [21] M. Stone, B. Sonies, T. Shawker, G. Weiss, and L. Nadel, “Analysis of real-time ultrasound images of tongue configuration using a grid-digitizing system,” _Journal of Phonetics_ , vol. 11, pp. 207–218, 1983.
  * [22] M. Stone, “A guide to analysing tongue motion from ultrasound images,” _Clinical Linguistics & Phonetics_, vol. 19, no. 6-7, pp. 455–501, jan 2005.
  * [23] T. G. Csapó and S. M. Lulich, “Error analysis of extracted tongue contours from 2D ultrasound images,” in _Proc. Interspeech_ , Dresden, Germany, 2015, pp. 2157–2161.
  * [24] B. Denby and M. Stone, “Speech synthesis from real time ultrasound images of the tongue,” in _Proc. ICASSP_. Montreal, Quebec, Canada: IEEE, 2004, pp. 685–688.
  * [25] B. Denby, J. Cai, T. Hueber, P. Roussel, G. Dreyfus, L. Crevier-Buchman, C. Pillot-Loiseau, G. Chollet, S. Manitsaris, and M. Stone, “Towards a Practical Silent Speech Interface Based on Vocal Tract Imaging,” in _9th International Seminar on Speech Production (ISSP 2011)_ , 2011, pp. 89–94.
  * [26] T. Hueber, E.-L. Benaroya, G. Chollet, G. Dreyfus, and M. Stone, “Development of a silent speech interface driven by ultrasound and optical images of the tongue and lips,” _Speech Communication_ , vol. 52, no. 4, pp. 288–300, 2010.
  * [27] A. Jaumard-Hakoun, K. Xu, C. Leboullenger, P. Roussel-Ragot, and B. Denby, “An Articulatory-Based Singing Voice Synthesis Using Tongue and Lips Imaging,” in _Proc. Interspeech_ , 2016, pp. 1467–1471.
  * [28] T. Grósz, G. Gosztolya, L. Tóth, T. G. Csapó, and A. Markó, “F0 Estimation for DNN-Based Ultrasound Silent Speech Interfaces,” in _Proc. ICASSP_ , 2018.
  * [29] K. Xu, P. Roussel, T. G. Csapó, and B. Denby, “Convolutional neural network-based automatic classification of midsagittal tongue gestural targets using B-mode ultrasound images,” _The Journal of the Acoustical Society of America_ , vol. 141, no. 6, pp. EL531–EL537, 2017.
  * [30] E. Tatulli and T. Hueber, “Feature extraction using multimodal convolutional neural networks for visual speech recognition,” in _Proc. ICASSP_ , 2017, pp. 2971–2975.
  * [31] J. Wang, A. Samal, J. R. Green, and F. Rudzicz, “Sentence Recognition from Articulatory Movements for Silent Speech Interfaces,” in _Proc. ICASSP_ , 2012, pp. 4985–4988.
  * [32] J. Wang, A. Samal, and J. Green, “Preliminary Test of a Real-Time, Interactive Silent Speech Interface Based on Electromagnetic Articulograph,” in _Proceedings of the 5th Workshop on Speech and Language Processing for Assistive Technologies_ , 2014, pp. 38–45.
  * [33] F. Bocquelet, T. Hueber, L. Girin, C. Savariaux, and B. Yvert, “Real-Time Control of an Articulatory-Based Speech Synthesizer for Brain Computer Interfaces,” _PLOS Computational Biology_ , vol. 12, no. 11, p. e1005119, nov 2016.
  * [34] M. Kim, B. Cao, T. Mau, and J. Wang, “Speaker-Independent Silent Speech Recognition From Flesh-Point Articulatory Movements Using an LSTM Neural Network,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 25, no. 12, pp. 2323–2336, dec 2017.
  * [35] M. J. Fagan, S. R. Ell, J. M. Gilbert, E. Sarrazin, and P. M. Chapman, “Development of a (silent) speech recognition system for patients following laryngectomy,” _Medical Engineering and Physics_ , vol. 30, no. 4, pp. 419–425, 2008.
  * [36] J. A. Gonzalez, L. A. Cheah, A. M. Gomez, P. D. Green, J. M. Gilbert, S. R. Ell, R. K. Moore, and E. Holdsworth, “Direct Speech Reconstruction From Articulatory Sensor Data by Machine Learning,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 25, no. 12, pp. 2362–2374, dec 2017.
  * [37] K. Nakamura, M. Janke, M. Wand, and T. Schultz, “Estimation of fundamental frequency from surface electromyographic data: EMG-to-F0,” in _Proc. ICASSP_ , Prague, Czech Republic, 2011, pp. 573–576.
  * [38] Y. Deng, J. T. Heaton, and G. S. Meltzner, “Towards a practical silent speech recognition system,” in _Proc. Interspeech_ , Singapore, Singapore, 2014, pp. 1164–1168.
  * [39] L. Diener, M. Janke, and T. Schultz, “Direct conversion from facial myoelectric signals to speech using Deep Neural Networks,” in _2015 International Joint Conference on Neural Networks (IJCNN)_. IEEE, jul 2015, pp. 1–7.
  * [40] M. Janke and L. Diener, “EMG-to-Speech: Direct Generation of Speech From Facial Electromyographic Signals,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 25, no. 12, pp. 2375–2385, dec 2017.
  * [41] G. S. Meltzner, J. T. Heaton, Y. Deng, G. De Luca, S. H. Roy, and J. C. Kline, “Silent speech recognition as an alternative communication device for persons with laryngectomy,” _IEEE/ACM Transactions on Audio Speech and Language Processing_ , vol. 25, no. 12, pp. 2386–2398, 2017.
  * [42] J. Freitas, A. J. Ferreira, M. A. T. Figueiredo, A. J. S. Teixeira, and M. S. Dias, “Enhancing multimodal silent speech interfaces with feature selection,” in _Proc. Interspeech_ , Singapore, Singapore, 2014, pp. 1169–1173.
  * [43] G. Hinton, L. Deng, D. Yu, G. Dahl, A.-r. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. Sainath, and B. Kingsbury, “Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups,” _IEEE Signal Processing Magazine_ , vol. 29, no. 6, pp. 82–97, nov 2012.
  * [44] Z.-H. Ling, S.-Y. Kang, H. Zen, A. Senior, M. Schuster, X.-J. Qian, H. Meng, and L. Deng, “Deep Learning for Acoustic Modeling in Parametric Speech Generation,” _IEEE Signal Processing Magazine_ , vol. 32, no. 3, pp. 35–52, 2015.
  * [45] E. Arisoy, T. N. Sainath, B. Kingsbury, and B. Ramabhadran, “Deep Neural Network Language Models,” in _Proceedings of the NAACL-HLT 2012 Workshop: Will We Ever Really Replace the N-gram Model? On the Future of Language Modeling for HLT_ , ser. WLM ’12, 2012, pp. 20–28.
  * [46] Z.-C. Liu, Z.-H. Ling, and L.-R. Dai, “Articulatory-to-acoustic conversion with cascaded prediction of spectral and excitation features using neural networks,” in _Proc. Interspeech_ , 2016, pp. 1502–1506.
  * [47] C. Zhao, L. Wang, J. Dang, and R. Yu, “Prediction of F0 based on articulatory features using DNN,” in _Proc. ISSP_ , Tienjin, China, 2017.
  * [48] G. Olaszy, “Precíziós, párhuzamos magyar beszédadatbázis fejlesztése és szolgáltatásai [development and services of a hungarian precisely labeled and segmented, parallel speech database],” _Beszédkutatás 2013 [Speech Res. 2013]_ , pp. 261–270, 2013.
  * [49] T. Hueber, G. Aversano, G. Cholle, B. Denby, G. Dreyfus, Y. Oussar, P. Roussel, and M. Stone, “Eigentongue feature extraction for an ultrasound-based silent speech interface,” in _2007 IEEE International Conference on Acoustics, Speech and Signal Processing-ICASSP’07_ , vol. 1. IEEE, 2007, pp. I–1245.
  * [50] B. McFee, C. Raffel, D. Liang, D. P. Ellis, M. McVicar, E. Battenberg, and O. Nieto, “librosa: Audio and music signal analysis in python,” in _Proceedings of the 14th python in science conference_ , 2015, pp. 18–25.
  * [51] K. Xu, T. Gábor Csapó, P. Roussel, and B. Denby, “A comparative study on the contour tracking algorithms in ultrasound tongue images with automatic re-initialization,” _The Journal of the Acoustical Society of America_ , vol. 139, no. 5, pp. EL154–EL160, 2016.
  * [52] M. P. Sampat, Z. Wang, S. Gupta, A. C. Bovik, and M. K. Markey, “Complex wavelet structural similarity: A new image similarity index,” _IEEE transactions on image processing_ , vol. 18, no. 11, pp. 2385–2401, 2009.
  * [53] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” _IEEE transactions on image processing_ , vol. 13, no. 4, pp. 600–612, 2004.
  * [54] “ITU-R Recommendation BS.1534: Method for the subjective assessment of intermediate audio quality,” 2001.
  * [55] M. T. McCann, K. H. Jin, and M. Unser, “A Review of Convolutional Neural Networks for Inverse Problems in Imaging,” oct 2017. [Online]. Available: http://arxiv.org/abs/1710.04011
  * [56] M. Feigin, D. Freedman, and B. W. Anthony, “A Deep Learning Framework for Single-Sided Sound Speed Inversion in Medical Ultrasound,” sep 2018. [Online]. Available: http://arxiv.org/abs/1810.00322