# The impact of differences in facial features between real speakers and 3D
face models on synthesized lip motions

Rabab Algadhy
University of Sheffield
United Kingdom
<EMAIL_ADDRESS>
Yoshihiko Gotoh
University of Sheffield
United Kingdom
<EMAIL_ADDRESS>
Steve Maddock
University of Sheffield
United Kingdom
<EMAIL_ADDRESS>

###### Abstract

Lip motion accuracy is important for speech intelligibility, especially for
users who are hard of hearing or second language learners. A high level of
realism in lip movements is also required for the game and film production
industries. 3D morphable models (3DMMs) have been widely used for facial
analysis and animation. However, factors that could influence their use in
facial animation, such as the differences in facial features between recorded
real faces and animated synthetic faces, have not been given adequate
attention. This paper investigates the mapping between real speakers and
similar and non-similar 3DMMs and the impact on the resulting 3D lip motion.
Mouth height and mouth width are used to determine face similarity. The
results show that mapping 2D videos of real speakers with low mouth heights to
3D heads that correspond to real speakers with high mouth heights, or vice
versa, generates less good 3D lip motion. It is thus important that such a
mismatch is considered when using a 2D recording of a real actor’s lip
movements to control a 3D synthetic character.
Keywords: Lip motions, Visual speech animation, 3D morphable model, Talking
heads.

##  1 Introduction

The external articulators of the face – lips, teeth and tongue – play an
important role in facial analysis and animation. They can provide a
significant proportion of the visual speech signal [1, 2] and improve the
intelligibility of speech for hearing impaired people [3, 4, 5] or non-native
listeners [6, 7] or in adverse listening conditions such as noisy environments
[8, 9, 10]. Accurate articulator movement is thus important for visual speech
animation and its many application areas. There are many approaches to produce
visual speech animation [11, 12, 13, 14, 15]. We use a data-driven approach
that maps tracked lip motions in 2D videos of a real speaker to corresponding
3D landmarks labelled on a 3D morphable model (3DMM) [16, 17] built using 3D
synthetic head poses. With such a visual speech animation approach, it is
important that any mismatches between the real and synthetic faces are
considered carefully as a mismatch can reduce the quality of the final
animation [18]. Our work focuses on lip motion and how this is affected by
similarities and differences in the facial features of the real faces and the
3DMMs. A greater understanding of the impact of any differences could help to
provide guidelines for choosing an appropriate real person when animating a
non-corresponding synthetic face, i.e. a face that is not the same as the real
actor, as might be the case for animating a historical character or a
recently-deceased person for a film or game, or even a humanoid character in a
film or computer game.

A classification approach for facial features is required. Since we are
focusing on lip motion, two mouth features are investigated in this paper:
mouth height and width. Based on an analysis of speakers in the Audio-Visual
Lombard Grid Speech Corpus [19], the relevant facial features of each speaker
are classified into three categories for each feature: low, middle and high.
This is used to investigate the mapping between real speakers and non-
corresponding 3DMMs belonging to the same or different classes.

We believe this is the first paper to provide an extensive study of the impact
of a mismatch between the facial features of recorded real faces and synthetic
3D faces on the resulting 3D lip motions. We show that when 2D videos of real
speakers who have low mouth height are mapped to 3D synthetic heads that
correspond to real speakers who have high mouth height, or vice versa, that
the resulting animation is adversely affected. Both quantitative and
qualitative evaluations are included in the results.

The remainder of this paper is structured as follows. Section 2 gives an
overview of related work. Section 3 presents metrical analysis of facial
features for the speakers in the audio-visual Lombard Grid speech corpus and
summarises the process of mapping 2D video data to a 3DMM. Sections 4 and 5
address quantitative and qualitative evaluation, respectively. Section 6
presents conclusions.

##  2 Related work

Generally, based on the input data to a synthetic speech generator,
audiovisual speech synthesis approaches can be classified into two main
categories: viseme driven approaches and data driven approaches. Viseme driven
approaches involve segmenting audio speech signals into phonemes which are
then classified into visual units called visemes. Viseme parameters are then
interpolated with co-articulation rules incorporated [12, 20, 21] or using
dominance functions [12, 22] or deep learning approaches [23, 24, 25, 26, 27,
28, 29]. The previous approaches are based on determining the weight of the
target phoneme against the neighboring segments and their influence on the
corresponding control parameters [12]. The rule-based models [21, 30, 31] take
into account only visemes that have an impact on the neighbouring ones
(backward and forward coarticulations). However, these models typically fail
to fully take co-articulation effects into account, thus leading to
unrealistic lip motions. For this reason, data-driven approaches are favored
more as they are based on animating faces according to captured data from real
speakers, which guarantee coarticulation effects are considered.

Data driven approaches involve capturing facial motion data occurring in
actual speech, which is then linked with a 3D face model either based on
phonetic information rules (sample-based approaches) [15, 32, 33, 34, 35, 36,
37, 38], or by using statistical models to control the facial motion that is
learned from the training data (learning-based approaches) [27, 39, 40, 41,
42].

These approaches require performance tracking for real faces and then the
reconstruction of corresponding 3D faces using the tracked data to produce
smooth mouth animations with co-articulation effects. Thus, the realism of the
resulting speech animation is highly dependent on how well the 3D face model
is reconstructed. There are two main traditional approaches used for
reconstructing 3D face models. The first is based on capturing performance of
real faces using RGB or RGB-D cameras and then reconstructing the 3D face
model using the captured data. The other approach involves scanning and then
blending 3D real faces [43]. Comprehensive research has been conducted on
reconstructing 3D face models from optical sensor measurements of a subject’s
performance. Face performance can be captured from the subject and represented
in a 3D domain either based on illumination data only [44, 45] or with the aid
of markers [46, 47, 48, 49]. These techniques involve capturing the face using
a camera and then reconstructing the face geometry either via triangulation or
colour and depth values based on the type of camera used. In the past,
calibrated dense camera arrays with complex indoor lighting setups, which are
expensive to set up and operate, were used [48, 50, 51]. Recently, low-cost
devices such as monocular RGB and RGB-D cameras have been used for offline and
online monocular face reconstruction and tracking [52, 53, 54]. However, the
quality of these techniques is affected by lighting conditions that may
produce undesired pixels with noisy depth values. This makes capturing faces
more complicated. In addition, these techniques are highly challenging since
they are based on forming the image by convolving multiple physical dimensions
in a single colour measurement. Therefore, current state-of-the-art approaches
employ face models and statistical analysis of the distribution of 3D facial
shapes.

A large body of research has been conducted on modelling the structure and
expression of faces based on low-dimensional subspaces. In some works, a
blendshape expression model based on a set of 3D face models, each
representing a particular expression, has been used. Facial animation can be
achieved by morphing between the neutral face and a specific expression, or by
morphing between various expressions. Several studies used delta variations to
linearly add each expression to the neutral face [39, 54, 55, 56, 57].
Blendshape models can be constructed using multi-monocular cameras in general
surroundings [39], monocular RGB cameras [54] or RGB-D sensor devices [55].
Although these methods can achieve globally pleasant results with regards to
the static realism of the face rendering, all of these approaches require
professional camera setups for gathering data to train the blendshape model,
and the lip shapes that can be acquired during speech are still not fully
included.

There is another body of research based on the most commonly used prior,
presented by Blanz and Vetter [16], which constructs a 3D face model by
learning from a low-dimensional face subspace created from high-resolution
laser scans of real faces with neutral expressions (3D morphable model
(3DMM)). Geometry and the illumination-corrected textures of the faces are
included in such models. These methods applied this reconstruction for 2D face
recognition, pose normalisation and illumination [58, 59], face reanimation
[60] and facial expression tracking [61] in 2D images, but they were rarely
extended to track lip motions during speech [62, 63] due to the expensive
devices that are required for gathering the data from real speakers, in
addition to the complexity of techniques for preprocessing the gathered data.
These methods lack person-specific facial characteristics. In addition, they
ignore anatomical and physical plausibility in the reconstructions. Some work
has been conducted to personalize the model, either by increasing the number
of 3D laser scans to model the skin reflectance, sex and age variations [58,
59, 64, 65], or by fitting person-specific shape correctives to the generic
face models [55, 56], where performance of the resulting animation is mainly
limited to noise levels and resolution of the input device. However, when
using 3DMMs for producing facial animation, the impact of facial feature
differences between real faces and 3D synthetic faces on the resulting 3D
animation does not appear to have been considered. There is a gap in the
literature that our work aims to address.

##  3 Methodology

Our approach tracks lip motions in 2D videos of real speakers and maps tracked
points to equivalent landmark points labelled on a 3D morphable model (3DMM),
built using 3D synthetic head poses, to produce animation. The 2D videos from
the Audio-Visual Lombard Grid Speech Corpus [19] are used as input. Section
3.1 describes this input data, how it is landmarked and how this is used to
classify the faces. Section 3.2 describes the approach used to construct a
3DMM for each real face used from the dataset. Section 3.3 describes the
mapping process from 2D video to 3DMM.

###  3.1 Input data and classification of facial features

To analyse the facial features of the Audio-Visual Lombard Grid Speech
Corpus’s speakers [19], a method presented by Roelfose et al. [66] was used.
They used morphometrical methods to classify the facial features of South
African males in photos to investigate common and rare features in this
community. The method is based on both measurements and morphology of the
face, which provides a reliable procedure for facial features classification.

The Audio-Visual Lombard Grid Speech Corpus consists of both front- and side-
view video of 54 speakers (30 female and 24 male) uttering sentences from the
GRID corpus [67] in both plain and Lombard conditions. Only front-view videos
of plain sentences are used from the corpus for mapping between 2D videos of a
real speaker and a 3DMM, while both front- and side-view videos are used for
creating the 3DMMs.

A pool of 27 speakers (12 male and 15 female) of the Audiovisual Lombard Grid
Speech corpus was selected to validate the performance of the 3D head models.
This pool contains videos of real speakers whose faces are not obscured by
glasses or facial hair, heads are not tilted downward, and bottom chin points
are visible. The purpose of this step was to facilitate the facial landmark
annotation process.

Because lip shapes are affected by facial movements such as smiling and
crying, the lips must be assessed when the subject has a neutral face shape
[68]. Thus, videos of the corpus were investigated for each speaker (27
speakers) to select the appropriate frame. The selected frames were processed
using Faceware Analyzer software to obtain x and y coordinates for each
landmark. Figure 1 (left) shows the landmarks (Lx) that were utilised to take
the facial measurements. Figure 1 (right) shows the 13 measurements (Dx) that
were taken from each frame using the Euclidean distance between the
predetermined facial landmarks. These distances were used to create indices,
irrespective of camera distance. Table 1 shows how the twelve indices were
computed using the calculated measurements.

Figure 1: Face landmarks (left) and measurements used for each video frame (right). Index | Calculation
---|---
I1-Facial | (100 * (D1/D2))
I2-Intercanthal | (100 * (D4/D3))
I3-Nasal | (100 * (D6/D5))
I4-Nasofacial | (100 * (D5/D1)
I5-Nose-face width | (100 * (D6/D2))
I6-Lip area | (100 * (D7/D8))
I7-Vertical mouth height | (100 * (D7/D1))
I8-Upper lip thickness | (100 * (D9/D1))
I9-Lower lip thickness | (100 * (D10/D1))
I10-Mouth width | (100 * (D8/D3))
I11-Chin size | (100 * (D11/D1))
I12-Nose-upper-lips | (100 * (D12/D13))
Table 1: Metrical features (indices) for the speakers in the audio-visual
Lombard grid speech corpus.

The ranges of each index were used to categorise the features into three
different morphological classes, low, middle and high, using the interquartile
range (IQR) and outliers (calculated using Q1-1.5*IQR and Q3+1.5*IQR, where Q1
and Q3 are the lower and upper quartile values, respectively). The
classification of the twelve indices for each speaker in the corpus was
calculated with 80% of the confidence intervals for each class of each index.
Figure 2 shows the classification of the twelve indices for the selected 27
real speakers from the audio-visual Lombard grid corpus, where the low, middle
and high classes are represented in yellow, orange and red filled circles
respectively. The number of speakers in each class of mouth height index and
mouth width index are shown in the relevant circles.

Figure 2: Classification of indices for each speaker of the audio-visual
Lombard grid speech corpus (80% confidence level for each class of each
index). The x axis shows the speaker’s ID (where M refers to male speaker and
F refers to female speaker) and the y axis shows the indices’ number. Number
of speakers in each class of mouth height and mouth width indices are shown in
the relevant circles at the top of the figure.

###  3.2 3D morphable model

A 3DMM is created for each real speaker used from the input data set. The real
speakers used were the selection classified under one of the three classes of
index 7 (mouth height) and index 10 (mouth width), as presented in Figure 2
that shows the three classes (low, middle and high, represented in filled
yellow, orange and red circles, respectively). The 3DMM is constructed using
synthetic faces. The commercial software FaceGen is used to create 161 head
poses, consisting of a neutral pose and 10 intensity variations of 16 visemes,
where each head mesh has vertex correspondence. The tongue and teeth meshes
are excluded. The initial neutral head pose can be generated using photo
images of a real person by placing facial landmarks either on a front-view
only photo or front- and side-view photographs. In this paper, the front- and
side-view photographs of each real speaker are used. Then the software is used
to deform the face into the required range of poses, where 161 poses are used
to train the 3DMM as explained in our previous work [69].

Principal Component Analysis (PCA) is applied to the vertices of the 161
Facegen head poses to generate a 3DMM. It is not applied to the texture
coordinates since all the Facegen heads have the same texture. The geometry of
the head is represented by a shape vector
$S=(X_{1},Y_{1},Z_{1},\ldots,X_{n},Y_{n},Z_{n})^{\top}$, containing the $X$,
$Y$, $Z$ coordinates of 5850 vertices. The 3DMM consists of a PCA model of the
shape, which is represented as:

$M:=\\{\overline{F},\sigma,V\\}$ (1)

where $\overline{F}\in R^{3N}$ is the mean vector of the example meshes (mean
head) with $N$ being the number of mesh vertices. $\sigma\in R^{n-1}$ denotes
the standard deviation, where $n$ is the number of Facegen heads used to build
the 3DMM, and $V=[v_{1},.....,v_{n-1}]\in R^{3N\times n-1}$ is a set of
principal components in the model.

A new head can be generated as follows:

$S=\overline{F}+\sum_{i=1}^{K}\alpha_{i}\sigma_{i}v_{i}$ (2)

where $K\leq n-1$ is the number of principal components and $\alpha_{i}\in
R^{K}$ is the shape coefficient [58].

###  3.3 Mapping process

The mapping between the 2D frames of a video and a 3DMM is achieved using the
camera matrix method presented by Huber et al [58]. In order to track the
facial features of a speaker in video, the random cascaded-regression copse
(R-CR-C) approach presented by Feng et al [70] is applied to generate a
learned landmark detection model using the Ibug-Helen test set [71].

Given 51 2D landmarks (of facial features used to animate the 3D mesh) and the
corresponding 3D landmarks, a pose of the face is estimated using the Gold
Standard Algorithm (more details in [58]). It computes the camera matrix that
is used to reconstruct the 3D shape. The most likely vector of PCA shape
coefficients $\alpha$ is found by minimising the following cost function:

$E=\sum\limits_{i=1}^{3N}\frac{(y_{m2D,i}-y_{i})^{2}}{2\sigma_{2D}^{2}}+\|\alpha\|_{2}^{2}$
(3)

where $N$ is the number of landmarks, $y$ is the 2D landmarks represented in
homogeneous coordinates, $\sigma_{2D}^{2}$ is an ad hoc variance of these
landmarks, and $y_{m2D,i}$ is the projected 3D landmarks to a 2D plane using
the camera matrix [58].

##  4 Quantitative evaluation

For each speaker from the selected pool (27 speakers), four plain sentences
from the front-view video files were chosen to be mapped to the corresponding
3D head and the non-corresponding 3D heads. The chosen sentences contain
different words (e.g different verbs (bin, lay, place and set) and letters (a,
b, etc.)), in order to contain the maximum number of English phonemes. For
example, referring to index 7 in Figure 2, a real speaker (ID: S17) was
classified into the high class. 2D videos of this speaker were mapped to the
corresponding 3D head and the non-corresponding 3D heads of other speakers in
the same class (high class) (i.e. speaker IDs: S19, S22, S35 and S46), middle
class (i.e. speaker IDs: S32, S42, S54 and S55) and low class (i.e. speaker
IDs: S23, S31, S47 and S48) (see Figure 3). Next, 2D videos of the resulting
3D lip motions of each head were compared with the original ground-truth 2D
videos. For comparison, Faceware Analyzer software was used to track the
facial features in the ground-truth 2D video and the front-view (2D) of the 3D
animation. This software can process video files as a batch, which speeds the
evaluation process. Two geometric articulatory measurements were calculated
from the extracted facial features. The first was a width measurement defined
by the horizontal distance between the right and left inner corners of the
lips. The second was a height measurement defined by the distance between the
top and the bottom middle of the inner mouth contour.

In order to correct the distance between the camera and the real speaker or
the talking 3D head, all the landmarks were normalised using the Euclidean
distance between the midpoint of the inner corners of the eyes and the nose
tip, since it is assumed these are not affected by the articulations [72]. All
visual articulatory features for the real speakers and their corresponding 3D
heads were normalised by their corresponding maximum and minimum mouth
measurements in the videos. Given the height and width values for each frame
of animation, for both the real video for a speaker and the resulting 3D
animation, the root mean square error (RMSE) over a sentence was used to
evaluate the effectiveness of each 3DMM.

[5pt]

Figure 3: An example of the mapping process between 2D video frames of a real
speaker (ID: S17) who classified under the high class of index 7, the
corresponding 3D head, and the non-corresponding 3D heads.

###  4.1 Mouth height (index 7)

Table 2 shows the RMSE results averaged over four sentences for the width and
height of the mouth aperture of real speakers in the low class of index 7 and
their corresponding, non-corresponding low, non-corresponding middle and non-
corresponding high 3D heads. From this table it is clear that the RMSE results
varied when 2D videos of real speakers were mapped to non-corresponding low 3D
heads or non-corresponding middle 3D heads. When the 2D videos were mapped to
the non-corresponding high 3D heads, the corresponding 3D heads gave the
lowest RMSE scores for height for all speakers and for width for two out of
four speakers who were classified in the low to middle class (ID: S31) and the
middle class (IDs: S48) of index 10 (mouth width). The corresponding 3D heads
of the real speakers (IDs: S23 and S47) failed to give the lowest score for
width because their corresponding real speakers were classified under the
middle to high class of index 10, which is very close to most of the non-
corresponding high 3D heads.

For the corresponding low 3D head of each real speaker and the non-
corresponding low 3D heads, a t-test suggested no significant difference in
RMSE results for width and height. Additionally, no significant difference was
found between the corresponding low 3D heads and the non-corresponding middle
3D heads for all speakers for height and three out of four speakers for width.
The significant difference in width given by the corresponding low 3D head of
a real speaker (ID: S31) was due to its low mouth width. A significant
difference was found between three out of four of the corresponding low 3D
heads and the non-corresponding high 3D heads for height. The corresponding
low 3D head of a real speaker (ID: S48) suggested no significant differences.
This may be due to the large distance between the nose tip and the upper lip
(index 12), which reduces the height of the mouth aperture. For width, three
out of four of the corresponding 3D heads showed a significant difference. The
corresponding low 3D head of a real speaker(ID: S23) suggested no significant
difference; this may be because it was classified in the middle to high class
of indices 10 and 12.

|  |  | Non-corresponding
---
3D head (low)
|  Corresponding 3D head (low) | S23 | S31 | S47 | S48 |  |  T-test (P value)
2D video | W | H | W | H | W | H | W | H | W | H |  |  | W | H
S23 | 0.270 | 0.131 |  |  | 0.232 | 0.114 | 0.253 | 0.137 | 0.188 | 0.145 |  |  | 0.1399 | 0.9195
S31 | 0.193 | 0.163 | 0.208 | 0.201 |  |  | 0.189 | 0.229 | 0.219 | 0.125 |  |  | 0.2946 | 0.5523
S47 | 0.218 | 0.099 | 0.212 | 0.081 | 0.227 | 0.149 |  |  | 0.140 | 0.082 |  |  | 0.4501 | 0.8448
S48 | 0.149 | 0.071 | 0.131 | 0.101 | 0.157 | 0.083 | 0.188 | 0.073 |  |  |  |  | 0.6168 | 0.2153
|  |  |  | Non-corresponding
---
3D head (middle)
|
|  |  | S32 | S42 | S54 | S55 |  |  |
| W | H | W | H | W | H | W | H | W | H |  |  | W | H
S23 | 0.270 | 0.131 | 0.241 | 0.135 | 0.256 | 0.168 | 0.265 | 0.185 | 0.277 | 0.152 |  |  | 0.2696 | 0.0734
S31 | 0.193 | 0.163 | 0.229 | 0.159 | 0.281 | 0.206 | 0.245 | 0.218 | 0.248 | 0.213 |  |  | 0.0132 | 0.0767
S47 | 0.218 | 0.099 | 0.250 | 0.091 | 0.234 | 0.131 | 0.170 | 0.103 | 0.173 | 0.098 |  |  | 0.6233 | 0.4975
S48 | 0.149 | 0.071 | 0.157 | 0.071 | 0.187 | 0.078 | 0.139 | 0.104 | 0.166 | 0.072 |  |  | 0.2762 | 0.2772
|  |  |  | Non-corresponding
---
3D head (high)
|
|  |  | S17 | S19 | S22 | S35 | S46 |  |
| W | H | W | H | W | H | W | H | W | H | W | H | W | H
S23 | 0.270 | 0.131 | 0.209 | 0.193 | 0.234 | 0.202 | 0.278 | 0.211 | 0.244 | 0.208 | 0.262 | 0.179 | 0.1059 | 0.0003
S31 | 0.193 | 0.163 | 0.321 | 0.185 | 0.278 | 0.223 | 0.246 | 0.286 | 0.243 | 0.216 | 0.251 | 0.213 | 0.0070 | 0.0209
S47 | 0.218 | 0.099 | 0.198 | 0.145 | 0.204 | 0.137 | 0.149 | 0.191 | 0.188 | 0.143 | 0.168 | 0.194 | 0.0226 | 0.0073
S48 | 0.149 | 0.071 | 0.198 | 0.083 | 0.184 | 0.084 | 0.164 | 0.145 | 0.157 | 0.101 | 0.156 | 0.096 | 0.0508 | 0.0532

Table 2: The RMS error averaged over 4 sentences for width (W) and height (H)
of the mouth of the real speakers classified under the low class of index 7
(mouth height), their corresponding 3D heads and the non-corresponding 3D
heads. Values in bold means the lowest RMS error. The last column shows the p
value of the t-test results between each corresponding 3D head and the non-
corresponding 3D heads for width and height.

Figure 4 provides an example of consecutive frames of the phoneme /ih/ during
the utterance of the word ”in” from the sentence ”bin white in O seven now”
for a real speaker (ID: S47) who was classified in the low class of index 7,
the corresponding 3D head, the non-corresponding low 3D head, the non-
corresponding middle 3D head and the non-corresponding high 3D head. This
figure illustrates how the non-corresponding high 3D head failed to detect the
uttered phoneme and that the mouth was completely closed due to lip thickness,
while the corresponding 3D head, the non-corresponding low 3D head and the
non-corresponding middle 3D head gave the closest mouth shapes to the real
speaker. These findings and the clear visual discrimination in the 3D lip
motions presented by each 3D head suggest that an appropriate animation can be
achieved by mapping between 2D videos of real speakers and the corresponding
low 3D head, the non-corresponding low 3D heads or the non-corresponding
middle 3D heads but not the non-corresponding high 3D heads.

Figure 5 shows the trajectories of the width and the height parameters of the
mouth aperture for the real speaker (ID: S31) classified under the low class
of index 7, the corresponding low 3D head, the non-corresponding middle 3D
head (ID: S32) and the non-corresponding high 3D head (ID: S19), whilst
uttering the sentence ”set white at D zero please”. Whilst all the
trajectories generated using the animation pipeline generally follow the real
speaker’s trajectory, the trajectories of the corresponding 3D head and the
non-corresponding middle 3D head are closer to the ground truth trajectories.
For the width, the trajectory of the non-corresponding high 3D head shows a
marked rise for the bilabial phoneme /p/, which confirms that the lips stretch
due to touch between the upper and lower lips caused by the lip thickness.
Also, for the height, the trajectory of this 3D head shows steep drops for
dental phonemes such as /s/, /t/ and /d/, which confirms that the lips are
semi-closed during uttering these phonemes.

Real speaker (ID: S47) |  |  |
---|---|---|---
The corresponding 3D head (ID: S47) |  |  |
The non-corresponding low 3D head (ID: S23) |  |  |
The non-corresponding middle 3D head (ID: S48) |  |  |
The non-corresponding high 3D head (ID: S22) |  |  |
Figure 4: Consecutive frames of the phoneme /ih/ during utterance of the word
”in” from sentence ”bin white in O seven now” for a real speaker (ID: S47) who
is classified under the low class of index 7, the corresponding 3D head, the
non-corresponding low, the non-corresponding middle and the non-corresponding
high 3D heads.

Figure 5: Width (upper) and height (lower) of mouth trajectories of 2D frames
of the real speaker (ID: S31) classified under the low class of index 7, the
corresponding 3D head, the non-corresponding middle 3D head (ID: S32) and the
non-corresponding high 3D head (ID: S19), whilst uttering the sentence ”set
white at D zero please”.

|  |  | Non-corresponding
---
3D head (middle)
|  Corresponding 3D head (middle) | S32 | S42 | S54 | S55 |  |  T-test (P value)
2D video | W | H | W | H | W | H | W | H | W | H |  |  | W | H
S32 | 0.111 | 0.056 |  |  | 0.122 | 0.089 | 0.106 | 0.112 | 0.174 | 0.059 |  |  | 0.3790 | 0.1837
S42 | 0.198 | 0.083 | 0.261 | 0.078 |  |  | 0.248 | 0.138 | 0.220 | 0.088 |  |  | 0.0653 | 0.4274
S54 | 0.281 | 0.132 | 0.255 | 0.152 | 0.263 | 0.130 |  |  | 0.256 | 0.149 |  |  | 0.0118 | 0.2324
S55 | 0.113 | 0.107 | 0.154 | 0.105 | 0.126 | 0.118 | 0.134 | 0.117 |  |  |  |  | 0.0953 | 0.2687
|  |  |  | Non-corresponding
---
3D head (low)
|
|  |  | S23 | S31 | S47 | S48 |  |  |
| W | H | W | H | W | H | W | H | W | H |  |  | W | H
S32 | 0.111 | 0.056 | 0.119 | 0.068 | 0.123 | 0.047 | 0.144 | 0.075 | 0.163 | 0.089 |  |  | 0.0820 | 0.2141
S42 | 0.198 | 0.083 | 0.185 | 0.075 | 0.224 | 0.125 | 0.255 | 0.092 | 0.208 | 0.130 |  |  | 0.2669 | 0.1870
S54 | 0.281 | 0.132 | 0.249 | 0.151 | 0.265 | 0.180 | 0.249 | 0.158 | 0.273 | 0.133 |  |  | 0.0351 | 0.0943
S55 | 0.113 | 0.107 | 0.119 | 0.096 | 0.119 | 0.111 | 0.112 | 0.108 | 0.122 | 0.104 |  |  | 0.0997 | 0.5385
|  |  |  | Non-corresponding
---
3D head (high)
|
|  |  | S17 | S19 | S22 | S35 | S46 |  |
| W | H | W | H | W | H | W | H | W | H | W | H | W | H
S32 | 0.111 | 0.056 | 0.185 | 0.132 | 0.145 | 0.102 | 0.135 | 0.193 | 0.176 | 0.145 | 0.161 | 0.123 | 0.0060 | 0.0055
S42 | 0.198 | 0.083 | 0.205 | 0.140 | 0.209 | 0.090 | 0.200 | 0.171 | 0.229 | 0.156 | 0.172 | 0.098 | 0.6150 | 0.0394
S54 | 0.281 | 0.132 | 0.226 | 0.148 | 0.209 | 0.147 | 0.241 | 0.129 | 0.267 | 0.136 | 0.131 | 0.194 | 0.0130 | 0.1931
S55 | 0.113 | 0.107 | 0.102 | 0.103 | 0.097 | 0.111 | 0.116 | 0.114 | 0.099 | 0.127 | 0.116 | 0.110 | 0.0832 | 0.2022

Table 3: The RMS error averaged over 4 sentences for width (W) and height (H)
of the mouth of the real speakers classified under the middle class of index 7
(mouth height), their corresponding 3D heads and the non-corresponding 3D
heads. Values in bold means the lowest RMS error. The last column shows p
value of the t-test results between each corresponding 3D head and the non-
corresponding 3D heads for width and height.

Table 3 shows the RMSE results averaged over four sentences for the width and
height of the mouth aperture of real speakers classified in the middle class
of index 7 (mouth height), their corresponding middle 3D heads, non-
corresponding low, non-corresponding middle and non-corresponding high 3D
heads. This shows variations in the RMSE results for width due to variations
in the mouth width of the real speakers. T-test results suggested no
significant difference in RMSE results for height for all speakers and for
three out of four speakers for width. The corresponding 3D head of a real
speaker (ID: S54) suggested a significant difference for the width; this may
be due to a large mouth width (index 10). For the non-corresponding high 3D
heads, the t-test results show a significant difference in the RMSE scores for
width for the corresponding 3D head of the real speaker (ID: S32); this is
probably due to the small mouth width.

For the height, t-test results showed a significant difference for two of the
corresponding middle 3D heads; this may be because their corresponding real
speakers (IDs: S32 and S42) were classified in the low class and the low to
middle class of index 10, respectively. This makes the mouth of the non-
corresponding high 3D heads shrink to fit the real speakers’ mouths; thus, the
lips are not closed or opened adequately. Figure 6 confirms these findings by
showing an example of consecutive frames of the phoneme /b/ during the
utterance of the word ”bin” from the phrase ”bin white at U three again” for a
real speaker (ID: S32) classified in the middle class of index 7, the
corresponding 3D head, the non-corresponding low 3D heads and the non-
corresponding high 3D heads. These findings may confirm that the resulting 3D
lip motions become sufficient and adequate when 2D videos of the real speakers
classified in the middle class of index 7 are mapped to the corresponding 3D
head, the non-corresponding middle 3D heads and the non-corresponding low 3D
heads and when they are mapped to the non-corresponding high 3D heads that
relate to real speakers who have a similar mouth width.

Real speaker (ID: S32) |  |  |
---|---|---|---
The corresponding 3D head (ID: S32) |  |  |
The non-corresponding low 3D head (ID: S23) |  |  |
The non-corresponding middle 3D head (ID: S54) |  |  |
The non-corresponding high 3D head (ID: S19) |  |  |
Figure 6: Consecutive frames of the phoneme /b/ during utterance of the word
”bin” from sentence ”bin white at U three again” for a real speaker (ID: S32)
who classified under the middle class of index 7, the corresponding 3D head,
the non-corresponding low 3D head, the non-corresponding middle 3D head and
the non-corresponding high 3D head.

Table 4 shows the RMSE results averaged over four sentences for the width and
height of the mouth aperture of real speakers classified in the high class of
index 7 (mouth height), their corresponding 3D heads, non-corresponding low,
non-corresponding middle and non-corresponding high 3D heads. It can be
observed that the corresponding 3D head of a real speaker (ID: S19) gave the
lowest RMSE scores for width for real speakers (IDs: S22 and S46); this may
due to similarities in mouth width. Additionally, the non-corresponding low 3D
heads (IDs: S23, S47 and S 48) gave the lowest scores because their
corresponding real speakers were classified in the middle class and the middle
to high class of index 10, and most of the real speakers (IDs: S17, S19, S22
and S46) were classified in the middle to high class or the high class. This
was confirmed by t-test results that showed no significant difference for
width between the corresponding 3D heads and the non-corresponding low 3D
heads.

For the height, the corresponding high 3D heads gave the lowest score for most
of the speakers from different classes. T-test results showed no significant
difference in the RMSE scores for height between four out of five of the
corresponding high 3D heads and the non-corresponding middle 3D heads.
However, there was a significant difference between three out of five of the
corresponding high 3D heads and the non-corresponding low 3D heads.

Based on these findings, it can be concluded that mapping between 2D videos of
real speakers classified in the high class of index 7 and the non-
corresponding low 3D heads cannot achieve any reasonable 3D lip animations.
While it is possible to achieve reasonable 3D lip motion using 2D videos of
real speakers who have a middle mouth height, to animate 3D heads
corresponding to real speakers who have a high mouth height, they must be
similar in other facial features, such as lower or upper lip thickness
(indices 8 and 9, respectively), mouth width (index 10) or the distance
between the nose and the upper lip (index 12). This is indicated by Figure 7
which shows how the lips of the non-corresponding 3D heads fail to give the
mouth shape of the phoneme /b/. This figure also shows how the non-
corresponding middle 3D head (ID: S54) gives a semi-opened mouth shape due to
its high mouth width, which is similar to that of the real speaker (ID: S22).
However, it fails to deliver the correct mouth shape due to its middle upper
and lower lip thickness.

Figure 8 shows an example of consecutive frames of the phoneme /p/ during
utterance of the word ”please” from the phrase ”lay white with A 5 please” for
a real speaker (ID: S46) classified in the high class of index 7, the
corresponding 3D head, the non-corresponding middle 3D heads and the non-
corresponding low 3D heads. This figure illustrates how the non-corresponding
middle 3D (ID: S54) heads gave a mouth shape more similar to that of the real
speaker due to similarities in mouth width (index 10) and middle lower lip
thickness. The non-corresponding low 3D heads (ID: S48) gave a more accurate
mouth shape (semi-closed mouth shape) because of the large distance between
the nose and the upper lip (index 12), the middle mouth width (index 10) and
the middle upper and lower lip thickness (indices 8 and 9), while the non-
corresponding low 3D heads (ID: S31) failed to give the correct mouth shape
due to the low to middle mouth width and upper and lower lip thickness. The
distortion in the texture around the mouth’s corners of the non-corresponding
low 3D heads (IDs: S48 and S31) was due to differences in mouth width between
the real speaker and the 3D heads.

Figure 9 shows the trajectories of the width and the height parameters of the
mouth aperture for the real speaker (ID: S22) classified under the high class
of index 7, the corresponding 3D head, the non-corresponding middle 3D head
(ID: S55) and the non-corresponding low 3D head (ID: S48), whilst uttering the
sentence ”set white with S 1 now”. The trajectories of the corresponding 3D
head are closer to the ground truth trajectories for both width and height.
For the width, what can be clearly seen in this figure is the steady decline
of the trajectories of the non-corresponding low 3D head. For the height, the
trajectories of the non-corresponding low 3D head show a marked increase for
the rounding lips phonemes such as /w/ and /aw/, alveolar phonemes such as
/th/, and dental phonemes such as /s/, /t/, and /n/, which confirms that the
lips are widely opened during uttering these phonemes that require semi-opened
mouth shape.

|  |  | Non-corresponding
---
3D head (high)
|  Corresponding 3D head (high) | S17 | S19 | S22 | S35 | S46 |  T-test (P value)
2D video | W | H | W | H | W | H | W | H | W | H |  |  | W | H
S17 | 0.092 | 0.095 |  |  | 0.101 | 0.130 | 0.112 | 0.113 | 0.107 | 0.111 | 0.111 | 0.110 | 0.0080 | 0.0210
S19 | 0.215 | 0.122 | 0.259 | 0.132 |  |  | 0.222 | 0.134 | 0.250 | 0.120 | 0.247 | 0.128 | 0.0337 | 0.1266
S22 | 0.172 | 0.137 | 0.136 | 0.150 | 0.125 | 0.184 |  |  | 0.127 | 0.157 | 0.149 | 0.156 | 0.0062 | 0.0469
S35 | 0.311 | 0.149 | 0.270 | 0.181 | 0.294 | 0.152 | 0.261 | 0.186 |  |  | 0.499 | 0.226 | 0.7465 | 0.0919
S46 | 0.231 | 0.118 | 0.198 | 0.142 | 0.166 | 0.136 | 0.227 | 0.070 | 0.244 | 0.108 |  |  | 0.2847 | 0.8234
|  |  |  | Non-corresponding
---
3D head (low)
|
|  |  | S23 | S31 | S47 | S48 |  |  |
| W | H | W | H | W | H | W | H | W | H |  |  | W | H
S17 | 0.092 | 0.095 | 0.087 | 0.110 | 0.097 | 0.116 | 0.112 | 0.128 | 0.111 | 0.110 |  |  | 0.2021 | 0.0158
S19 | 0.215 | 0.122 | 0.240 | 0.122 | 0.224 | 0.125 | 0.254 | 0.079 | 0.214 | 0.130 |  |  | 0.1337 | 0.5459
S22 | 0.172 | 0.137 | 0.126 | 0.149 | 0.111 | 0.160 | 0.139 | 0.142 | 0.179 | 0.153 |  |  | 0.1070 | 0.0338
S35 | 0.311 | 0.149 | 0.315 | 0.128 | 0.320 | 0.115 | 0.319 | 0.149 | 0.302 | 0.156 |  |  | 0.5214 | 0.2934
S46 | 0.231 | 0.118 | 0.267 | 0.148 | 0.237 | 0.166 | 0.267 | 0.137 | 0.215 | 0.138 |  |  | 0.3082 | 0.0224
|  |  |  | Non-corresponding
---
3D head (middle)
|
|  |  | S32 | S42 | S54 | S55 |  |  |
| W | H | W | H | W | H | W | H | W | H |  |  | W | H
S17 | 0.092 | 0.095 | 0.120 | 0.112 | 0.112 | 0.126 | 0.107 | 0.122 | 0.109 | 0.113 |  |  | 0.0060 | 0.0065
S19 | 0.215 | 0.122 | 0.262 | 0.125 | 0.224 | 0.127 | 0.238 | 0.140 | 0.281 | 0.131 |  |  | 0.0642 | 0.0783
S22 | 0.172 | 0.137 | 0.300 | 0.233 | 0.132 | 0.149 | 0.147 | 0.156 | 0.168 | 0.157 |  |  | 0.7269 | 0.1609
S35 | 0.311 | 0.149 | 0.325 | 0.130 | 0.325 | 0.153 | 0.293 | 0.156 | 0.313 | 0.141 |  |  | 0.7177 | 0.5501
S46 | 0.231 | 0.118 | 0.269 | 0.191 | 0.245 | 0.114 | 0.255 | 0.104 | 0.252 | 0.144 |  |  | 0.0171 | 0.3760

Table 4: The RMS error averaged over 4 sentences for width (W) and height (H) of the mouth of the real speakers classified under the high class of index 7 (mouth height), their corresponding 3D heads and the non-corresponding 3D heads. Values in bold means the lowest RMS error. The last column shows p value of the t-test results between each corresponding 3D head and the non-corresponding 3D heads for width and height. Real speaker (ID: S22) |  |  |
---|---|---|---
The corresponding 3D head (ID: S22) |  |  |
The non-corresponding low 3D head (ID: S47) |  |  |
The non-corresponding middle 3D head (ID: S55) |  |  |
The non-corresponding middle 3D head (ID: S54) |  |  |
Figure 7: Consecutive frames of the phoneme /b/ during utterance of the word ”bin” from sentence ”bin green by Q zero again” for a real speaker (ID: S22) who classified under the high class of index 7 (first row), the corresponding 3D head (second row), the non-corresponding low 3D head (third row) and the non-corresponding middle 3D head (last row). Real speaker (ID: S46) |  |
---|---|---
The corresponding 3D head (ID: S46) |  |
The non-corresponding middle 3D head (ID: S54) |  |
The non-corresponding low 3D head (ID: S48) |  |
The non-corresponding low 3D head (ID: S31) |  |
Figure 8: Consecutive frames of the phoneme /p/ during utterance of the word
”please” from sentence ”lay white with A 5 please” for a real speaker (ID:
S46) who classified under the high class of index 7 (first row), the
corresponding 3D head (second row), the non-corresponding middle 3D head
(third row) and the non-corresponding low 3D heads (last two rows).

Figure 9: Width (upper) and height (lower) of mouth trajectories of 2D frames
of the real speaker (ID: S22) classified under the high class of index 7, the
corresponding 3D head, the non-corresponding middle 3D head (ID: S55) and the
non-corresponding low 3D head (ID: S48), whilst uttering the sentence ”set
white with S 1 now”.

###  4.2 Mouth width (index 10)

Table 5 shows the RMSE results averaged over four sentences for width and
height of the mouth aperture of real speakers classified in the low class of
index 10 (mouth width), their corresponding 3D heads, non-corresponding low,
non-corresponding middle and non-corresponding high 3D heads. From this table,
it can be observed that the 3D head that corresponded to a real speaker (ID:
S32) gave the lowest RMSE score for three out of four speakers for width and
for all speakers for height, when it was fitted to 2D videos of real speakers
who were classified under the low class. This may be due to its middle mouth
height (i.e. index 7). This explains why it failed to give the lowest score
for width for a real speaker (ID: S35) with a high mouth height.

|  |  | Non-corresponding
---
3D head (low)
|  | Corresponding 3D
---
head (low)
S16 | S32 | S35 | S38 |  |  | T-test
---
(P value)
2D video | W | H | W | H | W | H | W | H | W | H |  |  | W | H
S16 | 0.173 | 0.097 |  |  | 0.162 | 0.097 | 0.176 | 0.108 | 0.191 | 0.149 |  |  | 0.7290 | 0.3157
S32 | 0.111 | 0.056 | 0.135 | 0.118 |  |  | 0.176 | 0.145 | 0.140 | 0.138 |  |  | 0.0930 | 0.0107
S35 | 0.311 | 0.149 | 0.258 | 0.143 | 0.325 | 0.130 |  |  | 0.306 | 0.167 |  |  | 0.5385 | 0.8495
S38 | 0.296 | 0.138 | 0.273 | 0.147 | 0.204 | 0.111 | 0.214 | 0.121 |  |  |  |  | 0.0928 | 0.3905
|  |  |  | Non-corresponding
---
3D head (middle)
|
|  |  | S7 | S15 | S24 | S48 | S55 |  |
| W | H | W | H | W | H | W | H | W | H | W | H | W | H
S16 | 0.173 | 0.097 | 0.127 | 0.120 | 0.137 | 0.100 | 0.154 | 0.093 | 0.212 | 0.096 | 0.181 | 0.103 | 0.5231 | 0.3162
S32 | 0.111 | 0.056 | 0.117 | 0.198 | 0.133 | 0.085 | 0.152 | 0.103 | 0.163 | 0.089 | 0.175 | 0.059 | 0.0235 | 0.1006
S35 | 0.311 | 0.149 | 0.277 | 0.200 | 0.274 | 0.140 | 0.334 | 0.145 | 0.302 | 0.156 | 0.313 | 0.141 | 0.3838 | 0.5471
S38 | 0.296 | 0.138 | 0.300 | 0.183 | 0.287 | 0.112 | 0.247 | 0.117 | 0.174 | 0.119 | 0.212 | 0.114 | 0.0901 | 0.5430
|  |  |  | Non-corresponding
---
3D head (high)
|
|  |  | S19 | S20 | S22 | S46 | S54 |  |
| W | H | W | H | W | H | W | H | W | H | W | H | W | H
S16 | 0.173 | 0.097 | 0.179 | 0.095 | 0.126 | 0.127 | 0.145 | 0.103 | 0.181 | 0.094 | 0.213 | 0.116 | 0.7957 | 0.1912
S32 | 0.111 | 0.056 | 0.145 | 0.102 | 0.134 | 0.072 | 0.135 | 0.193 | 0.161 | 0.123 | 0.106 | 0.112 | 0.0484 | 0.0325
S35 | 0.311 | 0.149 | 0.294 | 0.152 | 0.288 | 0.132 | 0.261 | 0.186 | 0.499 | 0.226 | 0.293 | 0.156 | 0.7312 | 0.2610
S38 | 0.296 | 0.138 | 0.269 | 0.111 | 0.254 | 0.111 | 0.254 | 0.105 | 0.229 | 0.145 | 0.199 | 0.123 | 0.0111 | 0.0561

Table 5: The RMS error averaged over 4 sentences for width (W) and height (H)
of the mouth of the real speakers classified under the low class of index 10
(mouth width), their corresponding 3D heads and the non-corresponding 3D
heads. Values in bold means the lowest RMS error. The last column shows p
value of the t-test results between each corresponding 3D head and the non-
corresponding 3D heads for width and height.

The RMSE results varied when 2D videos of the real speakers were mapped to the
non-corresponding middle 3D heads and the non-corresponding high 3D heads. For
mapping between 2D videos of real speakers and the non-corresponding middle 3D
heads, the 3D head of a real speaker (ID: S32) gave the lowest score for both
width and height because its middle mouth height was similar to or the same as
most of the non-corresponding middle 3D heads with middle to high (IDs: S7,
S15 and S24) or middle (ID: S55) mouth heights. What is striking in Table 5 is
that the corresponding 3D head of a real speaker (ID: S32) gave the lowest
RMSE score for both width and height. Additionally, some of the non-
corresponding high 3D heads gave the lowest scores for width for real speakers
with similar mouth heights. For example, the non-corresponding high 3D head
(ID: S20) gave the lowest score for a real speaker (ID: S16) with the same
mouth height (i.e. low to middle). Figure 10 gives an example of consecutive
frames for the real speaker (ID: S16), the corresponding 3D head and the non-
corresponding high 3D head (ID: S20) during utterance of the phoneme /ih/ of
the word “bin” from the phrase “bin white with M 2 soon”. Also, the non-
corresponding high 3D head (ID: S22) gave the lowest score for a real speaker
(ID: S35); this may because they both had high mouth heights. However, a
t-test suggested a significant difference in the RMSE results between the
corresponding 3D head of a real speaker (ID: S32) and the non-corresponding
high 3D heads for both width and height. There was also a significant
difference between the corresponding 3D head of a real speaker (ID: S38) and
the non- corresponding high 3D heads for width.

Real speaker (ID: S16) |  |  |
---|---|---|---
The corresponding 3D head (ID: S16) |  |  |
The non-corresponding middle 3D head (ID: S20) |  |  |

Figure 10: Consecutive frames of the phoneme /ih/ during utterance of the word
”bin” from sentence ”bin white with M 2 soon” for a real speaker (ID: S16) who
classified under the low class of index 10 (mouth width), the corresponding 3D
head (second row), the non-corresponding high 3D head (third row).

These finding confirm that 2D videos of real speakers who have middle or wide
mouth widths can be used to animate 3D heads that associate with real speakers
who have narrow mouth widths, as long as they have similar mouth heights, lip
thicknesses and distances between the nose and the upper lip. For example,
Figure 11 gives an example of consecutive frames of a real speaker (ID: S38)
classified in the low class of index 10, the corresponding 3D heads and the
non-corresponding 3D heads. This Figure reveals that the non-corresponding
high 3D head (ID: S54) produced a mouth shape more similar to the real speaker
than the non-corresponding middle 3D head (ID: S55) due to closer classes of
indices 7, 8, 9 and 12 of the corresponding real speakers, while the non-
corresponding high 3D head (ID: S22) failed to give a more accurate shape
because of its lip thickness.

Real speaker (ID: S38) |  |
---|---|---
The corresponding 3D head (ID: S38) |  |
The non-corresponding high 3D head (ID: S54) |  |
The non-corresponding middle 3D head (ID: S55) |  |
The non-corresponding high 3D head (ID: S22) |  |

Figure 11: Consecutive frames of the phoneme /uw/ during utterance of the word
“two” from sentence “bin white in I 2 soon” for a real speaker (ID: S38) who
classified under the low class of index 10 (mouth width), the corresponding 3D
head (second row), the non-corresponding middle 3D head (third row) and the
non-corresponding high 3D heads (last two rows).

|  |  | Non-corresponding
---
3D head (middle)
|  | Corresponding 3D
---
head (middle)
S7 | S15 | S24 | S48 | S55 |  | T-test
---
(P value)
2D video | W | H | W | H | W | H | W | H | W | H | W | H | W | H
S7 | 0.273 | 0.128 |  |  | 0.318 | 0.140 | 0.279 | 0.124 | 0.296 | 0.185 | 0.302 | 0.148 | 0.6312 | 0.1985
S15 | 0.131 | 0.087 | 0.158 | 0.191 |  |  | 0.186 | 0.132 | 0.164 | 0.151 | 0.179 | 0.134 | 0.0081 | 0.0177
S24 | 0.219 | 0.123 | 0.172 | 0.142 | 0.237 | 0.122 |  |  | 0.245 | 0.142 | 0.260 | 0.118 | 0.6584 | 0.3001
S48 | 0.149 | 0.071 | 0.158 | 0.188 | 0.133 | 0.081 | 0.144 | 0.095 |  |  | 0.166 | 0.072 | 0.8754 | 0.2506
S55 | 0.143 | 0.167 | 0.118 | 0.116 | 0.119 | 0.103 | 0.122 | 0.104 | 0.113 | 0.107 |  |  | 0.1241 | 0.3809
|  |  |  | Non-corresponding
---
3D head (low)
|
|  |  | S16 | S32 | S35 | S38 |  |  |
| W | H | W | H | W | H | W | H | W | H |  |  | W | H
S7 | 0.273 | 0.128 | 0.310 | 0.138 | 0.295 | 0.149 | 0.298 | 0.131 | 0.293 | 0.129 |  |  | 0.0064 | 0.1481
S15 | 0.131 | 0.087 | 0.179 | 0.139 | 0.183 | 0.130 | 0.175 | 0.139 | 0.174 | 0.136 |  |  | 0.0002 | 0.0002
S24 | 0.219 | 0.123 | 0.217 | 0.116 | 0.255 | 0.124 | 0.275 | 0.117 | 0.228 | 0.112 |  |  | 0.1558 | 0.1046
S48 | 0.149 | 0.071 | 0.122 | 0.090 | 0.157 | 0.071 | 0.157 | 0.101 | 0.113 | 0.127 |  |  | 0.4464 | 0.1105
S55 | 0.113 | 0.107 | 0.145 | 0.109 | 0.154 | 0.105 | 0.116 | 0.114 | 0.120 | 0.116 |  |  | 0.1122 | 0.2056
|  |  |  | Non-corresponding
---
3D head (high)
|
|  |  | S19 | S20 | S22 | S46 | S54 |  |
| W | H | W | H | W | H | W | H | W | H | W | H | W | H
S7 | 0.273 | 0.128 | 0.180 | 0.122 | 0.257 | 0.137 | 0.256 | 0.121 | 0.274 | 0.121 | 0.325 | 0.126 | 0.5645 | 0.4410
S15 | 0.131 | 0.087 | 0.146 | 0.136 | 0.173 | 0.134 | 0.164 | 0.163 | 0.166 | 0.136 | 0.170 | 0.149 | 0.0022 | 0.0005
S24 | 0.219 | 0.123 | 0.244 | 0.133 | 0.251 | 0.129 | 0.255 | 0.133 | 0.258 | 0.120 | 0.228 | 0.114 | 0.0062 | 0.5007
S48 | 0.149 | 0.071 | 0.184 | 0.078 | 0.156 | 0.068 | 0.164 | 0.145 | 0.156 | 0.096 | 0.139 | 0.104 | 0.2129 | 0.1107
S55 | 0.113 | 0.107 | 0.097 | 0.111 | 0.114 | 0.099 | 0.099 | 0.127 | 0.116 | 0.110 | 0.134 | 0.117 | 0.8886 | 0.2747

Table 6: The RMS error averaged over 4 sentences for width (W) and height (H)
of the mouth of the real speakers classified under the middle class of index
10 (mouth width), their corresponding 3D heads and the non-corresponding 3D
heads. Values in bold means the lowest RMS error. The last column shows p
value of the t-test results between each corresponding 3D head and the non-
corresponding 3D heads for width and height.

Table 6 shows the RMSE results averaged over four sentences for the width and
height of the mouth aperture of real speakers classified in the middle class
of index 10, their corresponding 3D heads, non-corresponding low, non-
corresponding middle and non-corresponding high 3D heads. From this Table, it
can be observed that three of the corresponding heads (IDs: S7, S15 and S55)
gave the lowest scores for width, when the 2D videos of real speakers were
mapped to the non-corresponding low 3D heads and the non-corresponding middle
3D heads. However, t-test results showed a significant difference in RMSE
results between the corresponding 3D head of a real speaker (ID: S15) versus
all the non-corresponding 3D heads for height and width. Also, there was a
significant difference in the RMSE results for width between the corresponding
3D head of a real speaker (ID: S7) and the non-corresponding low 3D heads and
between the corresponding 3D head of a real speaker (ID: S24) and the non-
corresponding high 3D heads. These findings may prove that 2D videos of real
speakers who have middle mouth width can be used to animate 3D heads that
correspond to real speakers that have narrow, middle or wide mouth widths, as
long as they have similar lip thicknesses or distances between the nose and
the upper lip. Figure 12 shows an example of consecutive frames of the phoneme
/th/ from the word “three” during phrase “bin white in N 3 now” by a real
speaker (ID: S24) classified in the middle class of index 10, the
corresponding 3D heads, the non-corresponding low 3D heads and the non-
corresponding high 3D heads. This figure shows how the 3D heads gave the
correct mouth shape regardless of the mouth width. The mouth aperture of the
non-corresponding middle 3D head (ID: S32) (third row) is slightly wide
compared to the real speaker due to its middle mouth height (index 7).

Real speaker (ID: S24) |  |  |
---|---|---|---
The corresponding 3D head (ID: S24) |  |  |
The non-corresponding low 3D head (ID: S32) |  |  |
The non-corresponding low 3D head (ID: S35) |  |  |
The non-corresponding high 3D head (ID: S19) |  |  |

Figure 12: Consecutive frames of the phoneme /th/ during utterance of the word
“three” from sentence “bin white in N 3 now” for a real speaker (ID: S24) who
classified under the middle class of index 10 (mouth width), the corresponding
3D head (second row), the non-corresponding low 3D head (third and forth rows)
and the non-corresponding high 3D heads (last row).

Table 7 shows the RMSE results averaged over four sentences for the width and
height of the mouth aperture of real speakers classified in the high class of
index 10 (mouth width), their corresponding 3D heads, non-corresponding low,
non-corresponding middle and non-corresponding high 3D heads. From this Table,
it can be noticed that the corresponding 3D head (ID: S19) gave the lowest
RMSE scores for width for most of the speakers when it was mapped to 2D videos
of real speakers who are classified under the high class. This may be due to
the middle distance between the nose tip and the upper lip and the high mouth
height. This also explains the significant difference suggested by the t-test
result for height. Also, it can be noticed that the non- corresponding low 3D
head (ID: S16) gave the lowest RMSE score for width for most of the speakers.
This is probably due to the large distance between the nose tip and the upper
lip. Another notable finding shown in this Table is that the non-corresponding
middle 3D head (ID: S 7) gave the lowest RMSE score for most of the speakers.
This may be due to the middle to high mouth height (index 7), upper lip
thickness (index 8), lower lip thickness (index 9) and distance between the
nose tip and the upper lip (index 12). The t-test results suggested no
significant difference for width and height between four out of five of the
corresponding 3D heads, the non-corresponding high 3D heads and the non-
corresponding middle 3D heads, while there is no significant difference
between the non-corresponding low 3D heads, the corresponding 3D heads for
height and three out of five of the corresponding 3D heads for width.

|  |  | Non-corresponding
---
3D head (high)
|  | Corresponding 3D
---
head (high)
S19 | S20 | S22 | S46 | S54 |  | T-test
---
(P value)
2D video | W | H | W | H | W | H | W | H | W | H | W | H | W | H
S19 | 0.215 | 0.122 |  |  | 0.218 | 0.129 | 0.222 | 0.134 | 0.247 | 0.128 | 0.238 | 0.140 | 0.0967 | 0.0297
S20 | 0.239 | 0.166 | 0.240 | 0.149 |  |  | 0.275 | 0.208 | 0.333 | 0.162 | 0.252 | 0.152 | 0.1933 | 0.9064
S22 | 0.172 | 0.137 | 0.125 | 0.184 | 0.117 | 0.134 |  |  | 0.149 | 0.156 | 0.147 | 0.156 | 0.0182 | 0.1391
S46 | 0.231 | 0.118 | 0.166 | 0.136 | 0.216 | 0.172 | 0.227 | 0.070 |  |  | 0.255 | 0.104 | 0.4785 | 0.9160
S54 | 0.281 | 0.132 | 0.209 | 0.147 | 0.242 | 0.166 | 0.240 | 0.129 | 0.250 | 0.131 |  |  | 0.0148 | 0.2814
|  |  |  | Non-corresponding
---
3D head (low)
|
|  |  | S16 | S32 | S35 | S38 |  |  |
| W | H | W | H | W | H | W | H | W | H |  |  | W | H
S19 | 0.215 | 0.122 | 0.268 | 0.130 | 0.262 | 0.125 | 0.250 | 0.120 | 0.242 | 0.151 |  |  | 0.0062 | 0.2575
S20 | 0.239 | 0.166 | 0.223 | 0.155 | 0.326 | 0.157 | 0.327 | 0.173 | 0.281 | 0.156 |  |  | 0.1332 | 0.2708
S22 | 0.172 | 0.137 | 0.124 | 0.138 | 0.300 | 0.233 | 0.127 | 0.157 | 0.147 | 0.142 |  |  | 0.9564 | 0.2634
S46 | 0.231 | 0.118 | 0.256 | 0.105 | 0.269 | 0.191 | 0.244 | 0.108 | 0.248 | 0.099 |  |  | 0.0244 | 0.7461
S54 | 0.281 | 0.132 | 0.255 | 0.135 | 0.255 | 0.152 | 0.267 | 0.136 | 0.293 | 0.133 |  |  | 0.2289 | 0.2081
|  |  |  | Non-corresponding
---
3D head (middle)
|
|  |  | S7 | S15 | S24 | S48 | S55 |  |
| W | H | W | H | W | H | W | H | W | H | W | H | W | H
S19 | 0.215 | 0.122 | 0.209 | 0.169 | 0.244 | 0.130 | 0.254 | 0.122 | 0.214 | 0.130 | 0.281 | 0.131 | 0.1284 | 0.1582
S20 | 0.239 | 0.166 | 0.208 | 0.350 | 0.274 | 0.158 | 0.308 | 0.171 | 0.294 | 0.148 | 0.309 | 0.163 | 0.1023 | 0.4491
S22 | 0.172 | 0.137 | 0.140 | 0.116 | 0.110 | 0.135 | 0.129 | 0.146 | 0.179 | 0.153 | 0.168 | 0.157 | 0.1011 | 0.5825
S46 | 0.231 | 0.118 | 0.203 | 0.116 | 0.255 | 0.149 | 0.261 | 0.122 | 0.215 | 0.138 | 0.252 | 0.144 | 0.6259 | 0.0679
S54 | 0.281 | 0.132 | 0.240 | 0.156 | 0.273 | 0.155 | 0.254 | 0.142 | 0.273 | 0.133 | 0.256 | 0.149 | 0.0255 | 0.0252

Table 7: The RMS error averaged over 4 sentences for width (W) and height (H)
of the mouth of the real speakers classified under the high class of index 10
(mouth width), their corresponding 3D heads and the non-corresponding 3D
heads. Values in bold means the lowest RMS error. The last column shows p
value of the t-test results between each corresponding 3D head and the non-
corresponding 3D heads for width and height.

These findings suggest that animating the wide mouths of 3D heads can be
achieved using 2D videos of real speakers who have narrow or middle mouth
widths as long as they have similar lip thicknesses, shapes and distances
between the nose and the upper lip. Figure 13 shows an example of consecutive
frames of a real speaker (ID: S54) classified in the high class of index 10,
the corresponding 3D head, the non-corresponding low 3D head and the non-
corresponding middle 3D head during utterance of the letter “B” from the
phrase “lay white by B 8 again”. This figure shows how all the 3D heads gave
the correct mouth shape for the phoneme /b/, including the non-corresponding
low 3D head.

Real speaker (ID: S54) |  |
---|---|---
The corresponding 3D head (ID: S54) |  |
The non-corresponding low 3D head (ID: S38) |  |
The non-corresponding middle 3D head (ID: S55) |  |

Figure 13: Consecutive frames of the phoneme /b/ during utterance of the
letter ”B” from sentence ”lay white by B 8 again” for a real speaker (ID: S54)
who classified under the high class of index 10 (mouth width), the
corresponding 3D head (second row), the non-corresponding low 3D head (third
row) and the non-corresponding middle 3D heads (last row).

##  5 Subjective evaluation

The subjective evaluation compares the naturalness of animations generated
using the mapping process described in Section 4. This test investigates the
impact of mouth height (index 7) and width (index 10) variation between real
speakers and 3DMMs on the resulting 3D lip motions. 2D video of a real speaker
from the Audio-Visual Lombard Grid corpus who is classified under one class of
index 7 or index 10 are used to animate the corresponding 3D head and the non-
corresponding 3D heads that relate to other speakers who are classified under
the other two classes. Videos of the animated 3D heads are synchronised with
the clean audio signal of the real speaker. The clean audio is used to enable
the participants to judge the extent to which the animated lip movement is as
smooth as a real speaker’s and how likely it was that the movement would
produce those sounds. The evaluation addresses two main points:

  * •

The impact of differences between the real speakers and the 3D heads in mouth
height and width on the resulting 3D lip animation.

  * •

The variation range in mouth height and width between real speakers and 3D
heads that provides sufficiently-3D lip motion.

###  5.1 Stimuli

2D front view videos of 24 real speakers were selected to be mapped to the
corresponding 3D heads and the non-corresponding 3D heads of the real speakers
who are classified under the other two classes. Twelve speakers were selected
based on classes of index 7, while the other twelve speakers were selected
based on classes of index 10 (Four speakers from each class of each index as
shown in Figure 14).

Figure 14: Structure of stimuli.

Based on this, 72 2D videos of 3D talking heads were used in this test. Three
separate animations were presented side by side for each set (24 sets in total
presented in a random order for each participant). Twelve sets showed the
resulting animation for mapping between non-corresponding faces based on the
classes of index 7, while the other 12 sets showed the resulting animation for
mapping between non-corresponding faces based on the classes of index 10. For
each set, three 2D videos of 3D animations were presented side by side. All
three heads were animated using 2D videos of one real speaker classified in
one of the three classes. One animation corresponded to that real speaker, and
the other two corresponded to different real speakers classified in the other
two classes. Figure 14 illustrates the structure of the stimuli. The
participants used a play button to repeat each sentence and watched each video
three times. After each set of videos, the participants were asked to choose
which 3D talking head had the most natural lip movements and which had the
least. The selection scores of a subject for the best and the worst choices
were used to evaluate the impact of the mouth height and the mouth width on
the resulting animation.

###  5.2 Participants

Two groups of participants with normal hearing and vision were recruited from
the Department of Computer Science, University of Sheffield, and tested
individually in an acoustically isolated booth with visual signals presented
on a computer screen combined with acoustic signals presented binaurally
through headphones. The first group consisted of 12 native English speakers
$E$ and the second group consisted of 15 non-native English speakers $N$ (from
different Arabic countries – Bahrain, Egypt, Iraq, Libya and Saudi Arabia –
with IELTS score $\epsilon$ [5.5,9]), where $E$ and $N$ denote native English
speakers and non-native English speakers, respectively. It was deemed
important to also test the results on non-native English speakers since such
animation has been used for pronunciation training systems [73, 74, 75, 76,
77, 78, 79] and has been utilised more widely for customer services and
entertainment such as films and games [80]. This study was ethically approved
via the University of Sheffield’s ethics review procedure.

###  5.3 Results

Figure 15 shows the most and least natural choices made by the two groups for
the corresponding 3D heads for index 7 (left) and index 10 (right). Generally,
the differences in mouth height (index 7) between real speakers and 3D head
models have a greater influence on the resulting animation, where the two
groups voted for the corresponding 3D heads as most natural with no
significant difference as confirmed by a t-test result (p=0.6514).

Figure 15: Results for the best, not chosen and worst rates for the
corresponding 3D heads for the $E$ and $N$ groups: Rates for index 7 (left);
Rates for index 10 (right).

For the mouth width (index 10), the $E$ group outperformed the $N$ group for
voting for the corresponding 3D heads as most natural. This may be because the
stimuli were presented in their language, suggesting that the differences in
mouth width have less impact on the 3D lip motions, which meant the non-native
group ($N$) were not able to select the corresponding 3D heads as the best
choice. However, no significant difference was found between the two groups
for voting for the corresponding 3D heads as most natural (p=0.1482).

[3pt]  [3pt]

Figure 16: Results for the best and worst choice rates for the corresponding
3D heads and the non-corresponding 3D heads for each class of index 7 for the
$E$ and $N$ groups: Rates of the best answer (top); Rates of the worst answer
(bottom).

Figure 16 shows the most and least natural choice rates for the corresponding
3D heads and the non-corresponding 3D heads for each class of index 7 for the
two groups. The $E$ group found the corresponding 3D heads of each class to be
most natural, while the $N$ voted only for the corresponding high 3D heads.
The two groups were able to distinctively choose the corresponding high 3D
heads as having the most natural lip motions. This is confirmed by a t-test
result that showed no significant difference between the two groups for
choosing the corresponding high 3D heads as having most natural lip motions
(p=0.0981). However, for the high class, t-test results showed no significant
difference between the corresponding 3D heads and the non-corresponding low 3D
heads for the $E$ group (p=0.3225) and the $N$ group (p=0.1643). Also, no
significant difference was found between the corresponding 3D heads and the
non-corresponding middle 3D heads for the $E$ group (p=0.2229) and the $N$
group (p=0.0611). The two groups found the animation generated by mapping 2D
videos of a real speaker classified in the low class to the non-corresponding
high 3D heads to be the least natural. A t-test result showed no significant
difference between the two groups for voting for the non-corresponding high 3D
heads as least natural (p=0.7882). This confirms the objective test results
provided in Table 2. However, no significant difference was found between the
non-corresponding high 3D heads and the corresponding low 3D heads for the two
groups (p=0.1362 for $E$ and p=0.1442 for $N$) or between the non-
corresponding high 3D heads and the corresponding middle 3D heads for the two
groups (p=0.1891 for $E$ and p=0.1781 for $N$).

Figure 17 shows the most and least natural choice rates for the corresponding
3D heads and the non-corresponding 3D heads for each class of index 10 for the
two groups. For the high class, t-test results showed a significant difference
between the corresponding 3D heads and the non-corresponding middle 3D heads
for the $E$ group (p=0.0280). The $E$ group found the corresponding 3D heads
of the middle and the high classes to be the most natural. For the low class,
the two groups found the non-corresponding high 3D heads to have the least
natural lip motions. This was confirmed by a t-test result that showed no
significant difference between the two groups for selecting the non-
corresponding high 3D head as having the least natural lip motions (p=0.7575).
A significant difference was suggested by the t-test results between the
corresponding 3D heads and the non-corresponding high 3D heads for the two
groups (p=0.0448 for the $E$ group and p=0.0046 for the $N$ group). Also, a
significant difference was found between the non-corresponding high 3D heads
and the non-corresponding middle 3D heads for the two groups (p=0.0012 for the
$E$ group and p=0.0009 for the $N$ group). This may prove that reasonable 3D
lip motions cannot be achieved when 2D videos of real speakers with narrow
mouth widths are mapped to 3D heads that relate to real speakers with wide
mouth widths.

[3pt]  [3pt]

Figure 17: Results for the best and worst choice rates for the corresponding
3D heads and the non-corresponding 3D heads for each class of index 10 for the
$E$ and $N$ groups: Rates of the best answer (top); Rates of the worst answer
(bottom).

###  5.4 Discussion

This study has investigated the impact of similarities and differences in the
facial features between real speakers and 3DMMs on the resulting 3D lip
motions and also defined the ranges of differences in facial features between
real speakers and 3D heads that allow adequate 3D lip motions to be achieved.

It was found that native English-speaking participants were able to
distinguish between the corresponding and non-corresponding 3D heads slightly
better than non-native English-speaking participants for the two tested
indices (indices 7, mouth height, and 10, mouth width). This may be because
the native participants had greater linguistic competence than the non-
natives.

The two groups were able to distinguish between the corresponding 3D heads and
the non-corresponding low 3D heads for the high class of index 7 (mouth
height), where the participants chose the lip motions of the corresponding
high 3D heads as the most natural. However, the results for the least natural
choice for this class were contrary for the $N$ group and convergent for the
$E$ group. This indicates that selecting one of three choices is more
difficult than choosing between two options. This could also apply to the most
natural choice answers for the low class, where the results were close for
each 3D head alternate, although the participants were able to select the non-
corresponding high 3D heads as the least natural. This indicates that the
difference between the corresponding 3D low heads and the non-corresponding
high 3D heads is distinguishable, which was also confirmed by the objective
test results.

For the low class of index 10 (mouth width), the two groups were able to
distinctly choose the non-corresponding high 3D heads as the least natural,
which confirms that the difference is significant and distinguishable between
these classes for this index. These findings are not comparable with the
objective test results due to an unbalanced number of male and female speakers
in each class of the tested indices. Consequently, presenting all possible
methods of mapping to the participants was restricted by this factor, as it is
not reasonable to display a 3D head of a real male speaker combined with a
female audio signal to the participants. This suggests that in order to
accurately investigate the effects of differences and similarities in the
facial features between real speakers and 3DMMs on the resulting 3D visual
speech animation, a large amount of data is essential. However, the most
natural choice answers for this class indicate confusion between the
corresponding 3D heads and the non-corresponding 3D heads. The performance of
the $N$ group for the high class of index 10 (mouth width) is mixed for the
most and least natural choice answers; this may be because the variations in
this index are not noticeable to non-native participants in comparison to
index 7 (mouth height), which has a greater effect on lip closure.

##  6 Conclusions

In this paper, an investigation of the effects of differences and similarities
in facial features between real speakers and 3DMMs on the mapping process was
presented. The facial features of real speakers were represented by 12
indices, and each index was classified into three classes: low, middle and
high. In this paper, two indices representing mouth height (index 7) and width
(index 10) were investigated separately by mapping between real speakers from
different classes to their corresponding 3D heads and 3D heads that
corresponded to different speakers in the same class or different classes.

The resulting 3D lip motions were evaluated quantitatively and qualitatively.
The results of the quantitative test suggest that, for index 7 (mouth height),
the mapping between real speakers with low mouth height and the 3D heads that
correspond to real speakers with high mouth height, or vice versa, leads to
unpleasant 3D lip motions. For index 10 (mouth width), the results varied
between the classes, which confirms that mouth width does not have significant
effect on the mapping process, whilst other facial features should be
considered, such as lip thickness. The qualitative evaluation results suggest
that native English-speaking participants are able to distinguish between the
corresponding and non-corresponding 3D heads slightly better than non-native
speakers. For the two tested indices, the two groups of participants chose the
non-corresponding high 3D heads as having the most unnatural lip motions when
they were mapped to real speakers classified in low classes. For index 7
(mouth height), the two groups selected the corresponding high 3D heads as
having the most natural lip motions. This is not the case with index 10 (mouth
width), where only the native-speaking participants were able to select the
corresponding high 3D heads as having the most natural lip motions. This may
confirm that mouth width does not have a considerable effect on the resulting
3D lip motions due to limited changes in this feature during speech in
comparison with index 7 (mouth height), which affects lip closure.

Based on these findings, it is thus important that any mismatch between a real
actor’s mouth and the 3D synthetic character’s mouth that is being animated
should be considered carefully. This has implications for training systems for
the hard of hearing and for animation production for entertainment
applications. Future work will consider the impact of other aspects of the
mouth and other facial proportions.

## Declaration of competing interest

The authors declare that they have no known competing financial interests or
personal relationships that could have appeared to influence the work reported
in this paper.

## Acknowledgements

This research has been supported by the Libyan Ministry of Higher Education
and Derna University.

## References

  * [1] M. McGrath, “An examination of cues for visual and audio-visual speech perception using natural and computer-generated faces,” Ph. D Thesis, Univ. of Nottingham, 1985.
  * [2] Q. Summerfield, “Lips, teeth, and the benefits of lipreading,” Handbook of research on face processing, pp. 223–233, 1989.
  * [3] C. L. Rogers, J. J. Lister, D. M. Febo, J. M. Besing, and H. B. Abrams, “Effects of bilingualism, noise, and reverberation on speech perception by listeners with normal hearing,” Applied Psycholinguistics, vol. 27, no. 3, p. 465, 2006.
  * [4] M. Stropahl and S. Debener, “Auditory cross-modal reorganization in cochlear implant users indicates audio-visual integration,” NeuroImage: Clinical, vol. 16, pp. 514–523, 2017.
  * [5] R. S. Tyler, H. Fryauf-Bertschy, D. M. Kelsay, B. J. Gantz, G. P. WOODWORTH, A. Parkinson, et al., “Speech perception by prelingually deaf children using cochlear implants,” Otolaryngology-Head and Neck Surgery, vol. 117, no. 3, pp. 180–187, 1997.
  * [6] V. Hazan, J. Kim, and Y. Chen, “Audiovisual perception in adverse conditions: Language, speaker and listener effects,” Speech Communication, vol. 52, no. 11-12, pp. 996–1009, 2010.
  * [7] V. Hazan, A. Sennema, M. Iba, and A. Faulkner, “Effect of audiovisual perceptual training on the perception and production of consonants by japanese learners of english,” Speech communication, vol. 47, no. 3, pp. 360–378, 2005.
  * [8] N. P. Erber, “Interaction of audition and vision in the recognition of oral speech stimuli,” Journal of speech and hearing research, vol. 12, no. 2, pp. 423–425, 1969.
  * [9] M. Middelweerd and R. Plomp, “The effect of speechreading on the speech-reception threshold of sentences in noise,” The Journal of the Acoustical Society of America, vol. 82, no. 6, pp. 2145–2147, 1987.
  * [10] W. H. Sumby and I. Pollack, “Visual contribution to speech intelligibility in noise,” The journal of the acoustical society of america, vol. 26, no. 2, pp. 212–215, 1954.
  * [11] S. Agarwal, D. Das, and B. Bhowmick, “Realistic lip animation from speech for unseen subjects using few-shot cross-modal learning,” in 2020 28th European Signal Processing Conference (EUSIPCO), pp. 690–694, IEEE, 2021.
  * [12] M. M. Cohen and D. W. Massaro, “Modeling coarticulation in synthetic visual speech,” in Models and techniques in computer animation, pp. 139–156, Springer, 1993.
  * [13] N. Liu, T. Zhou, Y. Ji, Z. Zhao, and L. Wan, “Synthesizing talking faces from text and audio: An autoencoder and sequence-to-sequence convolutional neural network,” Pattern Recognition, vol. 102, p. 107231, 2020.
  * [14] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” in Advances in neural information processing systems, pp. 3104–3112, 2014.
  * [15] S. L. Taylor, M. Mahler, B.-J. Theobald, and I. Matthews, “Dynamic units of visual speech,” in Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation, pp. 275–284, Eurographics Association, 2012.
  * [16] V. Blanz and T. Vetter, “A morphable model for the synthesis of 3d faces,” in Proceedings of the 26th annual conference on Computer graphics and interactive techniques, pp. 187–194, ACM Press/Addison-Wesley Publishing Co., 1999.
  * [17] B. Egger, W. A. Smith, A. Tewari, S. Wuhrer, M. Zollhoefer, T. Beeler, F. Bernard, T. Bolkart, A. Kortylewski, S. Romdhani, et al., “3d morphable face models–past, present and future,” arXiv preprint arXiv:1909.01815, 2019.
  * [18] S. Taylor, T. Kim, Y. Yue, M. Mahler, J. Krahe, A. G. Rodriguez, J. Hodgins, and I. Matthews, “A deep learning approach for generalized speech animation,” ACM Transactions on Graphics (TOG), vol. 36, no. 4, pp. 1–11, 2017.
  * [19] N. Alghamdi, S. Maddock, R. Marxer, J. Barker, and G. J. Brown, “A corpus of audio-visual lombard speech with frontal and profile views,” The Journal of the Acoustical Society of America, vol. 143, no. 6, pp. EL523–EL529, 2018.
  * [20] P. Edwards, C. Landreth, E. Fiume, and K. Singh, “Jali: an animator-centric viseme model for expressive lip synchronization,” ACM Transactions on Graphics (TOG), vol. 35, no. 4, p. 127, 2016.
  * [21] C. Pelachaud, N. I. Badler, and M. Steedman, “Generating facial expressions for speech,” Cognitive science, vol. 20, no. 1, pp. 1–46, 1996.
  * [22] D. Massaro, M. Cohen, M. Tabain, J. Beskow, and R. Clark, “12 animated speech: research progress and applications,” 2012.
  * [23] J. S. Chung, A. Jamaludin, and A. Zisserman, “You said that?,” arXiv preprint arXiv:1705.02966, 2017.
  * [24] Y. Chai, Y. Weng, L. Wang, and K. Zhou, “Speech-driven facial animation with spectral gathering and temporal attention,” Frontiers of Computer Science, vol. 16, no. 3, pp. 1–10, 2022.
  * [25] A. Choudhury, P. Roy, and S. Bandyopadhyay, “Review of various machine learning and deep learning techniques for audio visual automatic speech recognition,” in 2023 International Conference on Intelligent Systems, Advanced Computing and Communication (ISACC), pp. 1–10, IEEE, 2023.
  * [26] Y. Fan, Z. Lin, J. Saito, W. Wang, and T. Komura, “Faceformer: Speech-driven 3d facial animation with transformers,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18770–18780, 2022.
  * [27] T. Karras, T. Aila, S. Laine, A. Herva, and J. Lehtinen, “Audio-driven facial animation by joint end-to-end learning of pose and emotion,” ACM Transactions on Graphics (TOG), vol. 36, no. 4, p. 94, 2017.
  * [28] K. Vougioukas, S. Petridis, and M. Pantic, “Realistic speech-driven facial animation with gans,” International Journal of Computer Vision, vol. 128, no. 5, pp. 1398–1413, 2020.
  * [29] H. Zhu, M.-D. Luo, R. Wang, A.-H. Zheng, and R. He, “Deep audio-visual learning: A survey,” International Journal of Automation and Computing, vol. 18, no. 3, pp. 351–376, 2021.
  * [30] J. Beskow, “Rule-based visual speech synthesis,” in Fourth European Conference on Speech Communication and Technology, 1995.
  * [31] C. Pelachaud, Communication and coarticulation in facial animation.  University of Pennsylvania. Moore School of Electrical Engineering, 1991.  PhD theis, https://repository.upenn.edu/dissertations/AAI9211983.
  * [32] C. Bregler, M. Covell, and M. Slaney, “Video rewrite: driving visual speech with audio.,” in Siggraph, vol. 97, pp. 353–360, 1997.
  * [33] Y. Cao, P. Faloutsos, E. Kohler, and F. Pighin, “Real-time speech motion synthesis from recorded motions,” in Proceedings of the 2004 ACM SIGGRAPH/Eurographics symposium on Computer animation, pp. 345–353, Eurographics Association, 2004.
  * [34] E. Cosatto and H. P. Graf, “Sample-based synthesis of photo-realistic talking heads,” in Proceedings Computer Animation’98 (Cat. No. 98EX169), pp. 103–110, IEEE, 1998.
  * [35] H. P. Graf, E. Cosatto, and T. Ezzat, “Face analysis for the synthesis of photo-realistic talking heads,” in Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580), pp. 189–194, IEEE, 2000.
  * [36] M. Kawai, T. Iwao, A. Maejima, and S. Morishima, “Automatic generation of photorealistic 3d inner mouth animation only from frontal images,” Journal of information processing, vol. 23, no. 5, pp. 693–703, 2015.
  * [37] S. Kshirsagar and N. Magnenat-Thalmann, “Visyllable based speech animation,” in Computer Graphics Forum, vol. 22, pp. 631–639, Wiley Online Library, 2003.
  * [38] J. Xing, M. Xia, Y. Zhang, X. Cun, J. Wang, and T.-T. Wong, “Codetalker: Speech-driven 3d facial animation with discrete motion prior,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12780–12790, 2023.
  * [39] P. Garrido, M. Zollhöfer, C. Wu, D. Bradley, P. Pérez, T. Beeler, and C. Theobalt, “Corrective 3d reconstruction of lips from monocular video.,” ACM Trans. Graph., vol. 35, no. 6, pp. 219–1, 2016.
  * [40] H. X. Pham, Y. Wang, and V. Pavlovic, “End-to-end learning for 3d facial animation from speech,” in Proceedings of the 2018 on International Conference on Multimodal Interaction, pp. 361–365, ACM, 2018.
  * [41] J. Yu, C. Jiang, R. Li, C.-W. Luo, and Z.-F. Wang, “Real-time 3d facial animation: From appearance to internal articulators,” IEEE Transactions on Circuits and Systems for Video Technology, 2016.
  * [42] L. Yu, J. Yu, and Q. Ling, “Deep neural network based 3d articulatory movement prediction using both text and audio inputs,” in International Conference on Multimedia Modeling, p. 68, Springer, 2019.
  * [43] J. P. Lewis, K. Anjyo, T. Rhee, M. Zhang, F. H. Pighin, and Z. Deng, “Practice and theory of blendshape facial models.,” Eurographics (State of the Art Reports), vol. 1, no. 8, p. 2, 2014.
  * [44] A. Ghosh, G. Fyffe, B. Tunwattanapong, J. Busch, X. Yu, and P. Debevec, “Multiview face capture using polarized spherical gradient illumination,” in ACM Transactions on Graphics (TOG), vol. 30, p. 129, ACM, 2011.
  * [45] W.-C. Ma, T. Hawkins, P. Peers, C.-F. Chabert, M. Weiss, and P. Debevec, “Rapid acquisition of specular and diffuse normal maps from polarized spherical gradient illumination,” in Proceedings of the 18th Eurographics conference on Rendering Techniques, pp. 183–194, Eurographics Association, 2007.
  * [46] B. Bickel, M. Botsch, R. Angst, W. Matusik, M. Otaduy, H. Pfister, and M. Gross, “Multi-scale capture of facial geometry and motion,” in ACM transactions on graphics (TOG), vol. 26, p. 33, ACM, 2007.
  * [47] M. C. Doukas, E. Ververas, V. Sharmanska, and S. Zafeiriou, “Free-headgan: Neural talking head synthesis with explicit gaze control,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 8, pp. 9743–9756, 2023.
  * [48] X. Huang, S. Zhang, Y. Wang, D. Metaxas, and D. Samaras, “A hierarchical framework for high resolution facial expression tracking,” in Computer Vision and Pattern Recognition Workshop, 2004. CVPRW’04. Conference on, pp. 22–22, IEEE, 2004.
  * [49] E. Wood, T. Baltrušaitis, C. Hewitt, M. Johnson, J. Shen, N. Milosavljević, D. Wilde, S. Garbin, T. Sharp, I. Stojiljković, et al., “3d face reconstruction with dense landmarks,” in European Conference on Computer Vision, pp. 160–177, Springer, 2022.
  * [50] T. Beeler, F. Hahn, D. Bradley, B. Bickel, P. Beardsley, C. Gotsman, R. W. Sumner, and M. Gross, “High-quality passive facial performance capture using anchor frames,” in ACM Transactions on Graphics (TOG), vol. 30, p. 75, ACM, 2011.
  * [51] G. Fyffe, A. Jones, O. Alexander, R. Ichikari, and P. Debevec, “Driving high-resolution facial scans with video performance capture,” ACM Transactions on Graphics (TOG), vol. 34, no. 1, p. 8, 2014.
  * [52] P. Garrido, M. Zollhöfer, D. Casas, L. Valgaerts, K. Varanasi, P. Pérez, and C. Theobalt, “Reconstruction of personalized 3d face rigs from monocular video,” ACM Transactions on Graphics (TOG), vol. 35, no. 3, p. 28, 2016.
  * [53] J. Thies, M. Zollhöfer, M. Nießner, L. Valgaerts, M. Stamminger, and C. Theobalt, “Real-time expression transfer for facial reenactment.,” ACM Trans. Graph., vol. 34, no. 6, pp. 183–1, 2015.
  * [54] J. Thies, M. Zollhofer, M. Stamminger, C. Theobalt, and M. Nießner, “Face2face: Real-time face capture and reenactment of rgb videos,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2387–2395, 2016.
  * [55] S. Bouaziz, Y. Wang, and M. Pauly, “Online modeling for realtime facial animation,” ACM Transactions on Graphics (TOG), vol. 32, no. 4, p. 40, 2013.
  * [56] H. Li, J. Yu, Y. Ye, and C. Bregler, “Realtime facial animation with on-the-fly correctives.,” ACM Trans. Graph., vol. 32, no. 4, pp. 42–1, 2013.
  * [57] T. Weise, S. Bouaziz, H. Li, and M. Pauly, “Realtime performance-based facial animation,” in ACM transactions on graphics (TOG), vol. 30, p. 77, ACM, 2011.
  * [58] P. Huber, G. Hu, R. Tena, P. Mortazavian, W. P. Koppen, W. Christmas, M. Rätsch, and J. Kittler, “A multiresolution 3d morphable face model and fitting framework,” in Proceedings of the 11th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, 2016.
  * [59] P. Koppen, Z.-H. Feng, J. Kittler, M. Awais, W. Christmas, X.-J. Wu, and H.-F. Yin, “Gaussian mixture 3d morphable face model,” Pattern Recognition, vol. 74, pp. 617–628, 2018.
  * [60] V. Blanz, C. Basso, T. Poggio, and T. Vetter, “Reanimating faces in images and video,” in Computer graphics forum, vol. 22, pp. 641–650, Wiley Online Library, 2003.
  * [61] B. Amberg, R. Knothe, and T. Vetter, “Expression invariant 3d face recognition with a morphable model,” in Automatic Face & Gesture Recognition, 2008. FG’08. 8th IEEE International Conference on, pp. 1–6, IEEE, 2008.
  * [62] D. Cudeiro, T. Bolkart, C. Laidlaw, A. Ranjan, and M. J. Black, “Capture, learning, and synthesis of 3d speaking styles,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 10101–10111, 2019.
  * [63] U. Musti, S. Ouni, Z. Zhou, and M. Pietikäinen, “3d visual speech animation from image sequences,” in Proceedings of the 2014 Indian Conference on Computer Vision Graphics and Image Processing, p. 47, ACM, 2014.
  * [64] J. Booth, A. Roussos, A. Ponniah, D. Dunaway, and S. Zafeiriou, “Large scale 3d morphable models,” International Journal of Computer Vision, vol. 126, no. 2-4, pp. 233–254, 2018.
  * [65] J. Booth, A. Roussos, S. Zafeiriou, A. Ponniah, and D. Dunaway, “A 3d morphable model learnt from 10,000 faces,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5543–5552, 2016.
  * [66] M. M. Roelofse, M. Steyn, and P. J. Becker, “Photo identification: facial metrical and morphological features in south african males,” Forensic Science International, vol. 177, no. 2-3, pp. 168–175, 2008.
  * [67] M. Cooke, J. Barker, S. Cunningham, and X. Shao, “An audio-visual corpus for speech perception and automatic speech recognition,” The Journal of the Acoustical Society of America, vol. 120, no. 5, pp. 2421–2424, 2006.
  * [68] J. C. Carey, M. M. Cohen, C. J. Curry, K. Devriendt, L. B. Holmes, and A. Verloes, “Elements of morphology: standard terminology for the lips, mouth, and oral region,” American Journal of Medical Genetics Part A, vol. 149, no. 1, pp. 77–92, 2009.
  * [69] R. Algadhy, Y. Gotoh, and S. Maddock, “3d visual speech animation using 2d videos,” in Proceedings of ICASSP 2019, IEEE, 2019.
  * [70] Z.-H. Feng, P. Huber, J. Kittler, W. Christmas, and X.-J. Wu, “Random cascaded-regression copse for robust facial landmark detection,” IEEE Signal Processing Letters, vol. 22, no. 1, pp. 76–80, 2015.
  * [71] C. Sagonas, E. Antonakos, G. Tzimiropoulos, S. Zafeiriou, and M. Pantic, “300 faces in-the-wild challenge: Database and results,” Image and Vision Computing, vol. 47, pp. 3–18, 2016.
  * [72] O. Celiktutan, S. Ulukaya, and B. Sankur, “A comparative study of face landmarking techniques,” EURASIP Journal on Image and Video Processing, vol. 2013, no. 1, p. 13, 2013.
  * [73] A. Z. M. Ali, K. Segaran, and T. W. Hoe, “Effects of verbal components in 3d talking-head on pronunciation learning among non-native speakers,” Journal of Educational Technology & Society, vol. 18, no. 2, pp. 313–322, 2015.
  * [74] P. Dey, S. Maddock, and R. Nicolson, “A talking head for speech tutoring,” in Proceedings of the SSPNET 2nd International Symposium on Facial Analysis and Animation, pp. 14–14, 2010.
  * [75] P. Dey, S. C. Maddock, and R. Nicolson, “Evaluation of a viseme-driven talking head.,” in TPCG, pp. 139–142, 2010.
  * [76] O. Engwall, “Analysis of and feedback on phonetic features in pronunciation training with a virtual teacher,” Computer Assisted Language Learning, vol. 25, no. 1, pp. 37–64, 2012.
  * [77] O. Engwall and O. Bälter, “Pronunciation feedback from real and virtual language teachers,” Computer Assisted Language Learning, vol. 20, no. 3, pp. 235–262, 2007.
  * [78] D. W. Massaro, Perceiving talking faces: From speech perception to a behavioral principle, vol. 1.  Mit Press, 1998.
  * [79] L. Wang, H. Chen, S. Li, and H. M. Meng, “Phoneme-level articulatory animation in pronunciation training,” Speech Communication, vol. 54, no. 7, pp. 845–856, 2012.
  * [80] L. Xie, L. Wang, and S. Yang, “Visual speech animation,” Handbook of Human Motion, Springer (Ed.). Springer, pp. 1–30, 2016.