# Semi-Supervised Disentanglement of
Tactile Contact Geometry from Sliding-Induced Shear

Anupam K. Gupta1, Alex Church1 and Nathan F. Lepora1 *This work was supported
by an award from the Leverhulme Trust on “A biomimetic forebrain for robot
touch” (RL-2016-39)1The authors are with the Department of Engineering
Mathematics and the Bristol Robotics Laboratory, University of Bristol, UK
<EMAIL_ADDRESS>
###### Abstract

The sense of touch is fundamental to human dexterity. When mimicked in robotic
touch, particularly by use of soft optical tactile sensors, it suffers from
distortion due to motion-dependent shear. This complicates tactile tasks like
shape reconstruction and exploration that require information about contact
geometry. In this work, we pursue a semi-supervised approach to remove shear
while preserving contact-only information. We validate our approach by showing
a match between the model-generated unsheared images with their counterparts
from vertically tapping onto the object. The model-generated unsheared images
give faithful reconstruction of contact-geometry otherwise masked by shear,
along with robust estimation of object pose then used for sliding exploration
and full reconstruction of several planar shapes. We show that our semi-
supervised approach achieves comparable performance to its fully supervised
counterpart across all validation tasks with an order of magnitude less
supervision. The semi-supervised method is thus more computational and labeled
sample-efficient. We expect it will have broad applicability to wide range of
complex tactile exploration and manipulation tasks performed via a shear-
sensitive sense of touch.

Figure 1: Sensor, stimuli, model schematic and object shape reconstructions.
(a) TacTip sensor. (b) TacTip internal sensing surface. (c) 3D printed tactile
stimuli. Red circles highlight approximate location of data collection for
training. (d) & (e) Schematic of the model architecture for self-supervised
and supervised training phases respectively. $S_{\rm inp}$ and $U_{\rm inp}$
represent sheared data (distorted by motion-induced shear) and unsheared
(vertical tap) input samples. $S_{\rm rec}$ and $U_{\rm rec}$ represent
sheared and unsheared reconstructed samples. $U_{\rm GT}$ represents unsheared
ground truth samples used for fine-tuning sheared-unsheared mapping in the
supervised phase. P & S are latent representations of pose (contact-only) and
shear components of sensor response respectively. $L1_{\rm patch}$ loss is
computed between 100 random patches ($20\times 20$ pix.) of model-generated
unsheared images $U_{\rm rec}$ and its ground-truth counterparts $U_{\rm GT}$,
is used to enforce local image compliance. Detailed architectures in Fig. 2.
(f) Representative examples of full object shape reconstructions. For full set
of results, see Fig. 3.

##  I Introduction

The sense of touch is ubiquitous in everyday life. It allows us to interact,
explore and manipulate the environment around us using physical contact. The
physical contact allows us to measure the environment directly, which is not
possible with other sensory modalities like vision. However, this physical
contact also results in entangling of relevant information like contact
geometry with the manner in which the sensor-stimuli contact is established.
For example, when sliding or rubbing fingers across tactile stimuli, soft
tactile sensors are inevitably distorted by motion-dependent shear, making
their response history-dependent and generalization hard across tactile tasks.
This necessitates the removal of motion-induced global shear while preserving
the sensor distortions due to the contact-geometry or any other attribute of
interest. This issue plagues all shear-sensitive tactile sensors, in
particular camera-based soft tactile sensors that use the displacement of
markers to encode stimuli attributes [1, 2, 3, 4], complicating tactile-
dependent tasks like shape modelling and exploration.

Recently, this shear problem was tackled in [5, 6] for sliding exploration of
complex 2D or 3D objects, using a supervised deep learning model to learn
mappings between shear-distorted data and the target pose. However, this
approach to learn insensitivity to shear requires training separate models for
each tactile task, such as contact reconstruction or shape exploration. A
study by the authors of this paper instead used a supervised deep learning
model to remove shear distortion from the original sheared tactile image so
that quantities of interest (e.g. pose) can be predicted from unsheared
tactile data in a task-agnostic manner, which is more computational and data
efficient [7]. However, because these approaches are supervised, they suffer
from costly burden of requiring annotated data for learning. This limitation
can be overcome by the use of self- and semi-supervised approaches that can
learn rich representations from uncurated/unlabelled datasets, and are thus
more labeled sample-efficient. This mode of learning is closer to human
learning, as we likewise learn with only small amounts of labelled data and a
vast resource of self-supervised experiences.

In this work, we propose a semi-supervised learning approach to remove shear
from shear-distorted tactile images that uses only a small subset of annotated
data (10%) to achieve comparable performance to previous fully-supervised
approaches [7]. Self supervision with tactile data differs from computer
vision because it lacks attributes like colour, texture and the large-scale
structure of natural image data that have been exploited in computer vision to
generate supervisory signals without external supervision [8, 9, 10].
Moreover, the present work relies only on tactile data without recourse to
techniques that correlate data across multiple sensory modalities to generate
training signals without external supervision [11, 12, 13] (see Background).
In the presence of these limitations, we achieved self-supervision by first
disentangling sliding data into its contact-only and shear components,
followed by recombining the shear component with paired and novel canonical
(non-sheared or vertical tap) data to generate training signals without
external supervision.

Our experiments demonstrated that the proposed model could successfully
recover contact geometry from sheared images, allow continuous shape
exploration and also allow object reconstruction across multiple planar
objects. We also found that our proposed semi-supervised model achieved near
identical performance to its fully supervised counterpart [7] while using only
10% of labelled samples for training.

##  II Background

Soft optical tactile sensors like the TacTip [1] and other marker-based
sensors [2, 4] are vulnerable to sliding-induced global shear that distorts
the geometry of contact. Local shear caused by the stimulus imprint on the
sensor, represented in the lateral displacement of the markers, is
superimposed upon a global shear caused by contact motion such as sliding.
This distortion, by masking contact geometry, complicates challenging tactile
tasks like shape reconstruction and tactile exploration, which require spatial
information about the local geometry of an object free from motion-induced
distortion.

Recent works that use TacTip sensor (also used here) addressed this shear
problem by using a shear-insensitive pose-prediction deep network to slide
around complex planar shapes [5] and complex 3D objects [6, 14]. Shear-
insensitive training was achieved by mapping the shear-distorted samples
directly to target pose outputs. While effective, this approach necessitated
separate training for each new tactile task, for example surface- and contour-
following [14], greatly complicating the hyper-parameter tuning for accurate
network performance [6], and could not be applied to shape reconstruction. To
overcome these drawbacks, the authors of the present paper instead focused on
learning a mapping between shear-distorted tactile images and their unsheared
(vertical discrete tap) counterparts with a fully-supervised convolutional
neural network [7]. Once learned, this mapping can be reused to remove shear
for any number of downstream tasks, making this approach computational and
data efficient. However, the supervised nature of that approach necessitated
complex labelled-data collection. Here, we improve upon that work by proposing
a semi-supervised model that achieves comparable performance to the previous
supervised baseline, by training first in self-supervised phase followed by a
brief supervised phase (with a tenth of the labelled data from [7]).

Tactile data differs from natural images in being low-dimensional due to
absence of attributes like color and texture, alongside lacking the large-
scale global structure of natural image data. This is primarily due to the
localised nature of tactile sensing, unlike vision or audition which are
global sensing modalities over a scene. This increases the complexity of
generating a supervisory signal in the absence of external supervision; for
example, in computer vision, supervisory signals in prior studies have been
generated by: 1) learning to predict withheld data dimensions at the output,
such as in [8] two sub-networks are trained that are each tasked with
predicting one subset of the data channels (color or lightness channel) from
another (lightness or color channel); 2) image completion conditioned on the
immediate surroundings, as in [10] in which the input image has a missing part
that needs to be predicted at the output based on the surroundings; and 3)
predicting the correct arrangement of image subsections, as in [9] in which
the input image is first split into $N$ rectangular tiles whose locations are
shuffled to break spatial structure, and the network is then tasked to predict
the original image from a shuffled input image. These methods cannot be
readily extended to tactile data due to its low-dimensional structure and lack
of global structure as discussed previously.

Another approach to generate supervisory signals in absence of external
supervision is by correlating data from multiple sensory modalities like audio
and vision [11] or touch and vision [12, 13]. In this work, however we rely on
the information encoded by the tactile sensor alone.

To generate a supervisory signal, like in [7], the proposed model learns to
disentangle the sensor response components due to contact geometry and motion-
induced shear. Our approach, like [7], is inspired from similar works in
computer vision where disentanglement of object attributes and factors of
variation are considered important for learning robust representation that
improve generalizability on out-of-distribution (OOD) samples or novel
scenarios [16]. Several works in computer vision learn disentangled
representations via varitional autoencoders (VAE) [17, 18] and generative
adversarial networks (GAN) [19, 20, 21] for downstream applications such as
style transfer between images[22, 19]. Our approach is akin to style transfer
where the input image is decomposed into its content (geometry) and style
(color, intensity, texture) components. Once the content and style components
are disentangled, they can be used to transfer style to novel content. In our
work, content is akin to sensor response due to contact geometry and style to
response due to motion-induced shear. We are aware of no other prior work
exploring semi-supervised learning of disentangled representations for robust
representation learning in tactile sensing.

Figure 2: Network architecture. (a) Self-supervised training: The sheared
($S_{\rm inp}$) – unsheared ($U_{\rm rec}$), unsheared ($U_{\rm inp}$) –
sheared ($S_{\rm rec}$) and sheared ($S_{\rm inp}$) – sheared ($S_{\rm rec}$)
mappings $M$, $K$ and $N$ are learned via cycle consistency loss ($L_{\rm
cyc}$) [15] and reconstruction loss ($L_{\rm rec}$). $L_{\rm cycF}$ and
$L_{\rm cycB}$ are the components of ($L_{\rm cyc}$) corresponding to the
forward and backward cycle respectively. (b) Supervised training: The mappings
$M$ and $N$ previously learned in (a) are further fine-tuned via $L_{sup}$
loss (M), $L1_{\rm patch}$ (M) and reconstructions loss $L_{\rm rec}$ (N).
$L1_{\rm patch}$ loss, computed between 100 random patches ($20\times 20$
pix.) of model-generated unsheared images $U_{\rm rec}$ and its canonical
(non-sheared) counterparts $U_{\rm GT}$, is used to enforce local image
compliance.

##  III Methods

###  III-A Experimental Setup

The setup used in this work is similar to that of [7], where a soft biomimetic
optical tactile sensor – the TacTip [1] – is mounted on an industrial robot
arm (ABB IRB120) as an end effector. The tactile sensor encodes tactile
information in the shear displacement of marker pins on the inner side of the
sensing surface caused by the soft sensor skin deformation. The morphology of
this TacTip consisted of a 3D-printed soft rubber-like hemispherical dome (40
mm dia., TangoBlack+) with its inner surface covered by 331 hard white tip
pins arranged in a concentric circular grid fashion (Figs 1a,b). The soft dome
was filled with an optically-clear silicon gel to mimic the compliance of a
human fingertip. An RGB camera (ELP 1080p module) captures the inner surface
of the sensor dome. For more details, we refer to Ref. [1].

For test stimuli, we used seven distinct planar shapes with various
morphologies, including two acrylic spiral shapes that differ in frictional
properties to the other five 3D-printed ABS plastic shapes (Fig. 3, second
row). For model training we used three shapes: the circular disk, clover and
teardrop shown in Fig. 1 (c) (red circles show approximate location of data
collection). All shapes were securely fastened to the workspace to prevent
accidental motion during experiments.

###  III-B Data Collection

This work proposes a semi-supervised deep learning model, as an alternative to
the fully-supervised model proposed in [7], to remove motion-induced shear
distortion from tactile images. This motion-induced shear is governed by the
manner of contact between the sensor and object, distorting the sensor
response induced by the stimulus geometry during contact.

Our semi-supervised model is trained in two phases: self-supervised and
supervised. To ease data collection for these model training phases, we reused
the paired data (tap or canonical and sheared data with same relative pose)
previously collected in [7] to train the fully-supervised model. We generate
unpaired data, used for self-supervised training phase, by randomizing the
pose pairings (every epoch) between the canonical (non-sheared) tactile images
$U_{\rm inp}$ (collected by vertically tapping on the stimulus) and sheared
tactile images $S_{\rm inp}$ with random global shear (collected by sliding
across the stimulus), for all but a small subset (10%) of paired data held
back for supervised training phase.

To collect non-sheared canonical tap data, the sensor is brought in the
contact with the stimuli vertically to minimise global shear. For sliding
data, the sensor is brought to the target pose while in contact with the
object from a random offset location along either the $x$\- or $y$-axis or
both lateral directions simultaneously. To obtain paired data, the tap and
sliding data are collected for same relative target poses between the sensor
and stimulus (Fig. 1c). In total, a set of 200 random poses were collected,
with the sensor location relative to the initial contact location randomly
sampled from a uniform distribution spanning a range $[-5,5]$ mm in the two
lateral directions ($x$\- and $y$-axes), $[-45,45]$ deg in yaw
($\theta{\bm{n}}_{z}$) and $[-6,-1]$ mm in depth ($z$-axis). Finally, the
tactile images captured are cropped and subsampled to a $256\times 256$-pixel
region and binarized to minimise any effect of lighting changes inside the
sensor.

####  III-B1 Canonical Data

The canonical tapping dataset had 30,000 samples in total: 50 instances of
each of the 200 random poses for the three stimuli shapes used for training
(Fig. 1 (c)). The instances were generated by varying the indentation depth
($z$-axis) randomly between $[-1,1]$ mm from a set indentation depth. This
dataset was used as the target (reference) to which the sliding data should be
restored.

####  III-B2 Sheared Data

The sheared data had 90,000 samples in total: 150 instances for each of the
200 random poses for each of the three stimuli shapes (Fig. 1 (c)). To
generate instances, the sensor was first brought in contact with the stimulus
at a lateral offset location sampled randomly between $[-2.5,2.5]$ mm
laterally (along the $x$\- or $y$-axes or both), then sliding to the target
pose.

####  III-B3 Training and Test Data

The canonical and sheared data were first partitioned randomly into training,
validation and test sets in the ratio 60:20:20. Thus, for each stimulus shape
(Fig. 1), 120 poses were aligned to training set and 40 poses each to
validation and test sets.

The paired training set (canonical-sheared) was further randomly split in the
ratio 90:10. The larger subset (90%) was shuffled to generate random paired
data for the first self-supervised phase of model training. The shuffling was
repeated before every training epoch to increase variability in training data.
The other subset (10%) kept the pairing between sheared data and a canonical
instance of the same pose, for the supervised phase of training.

###  III-C Formulation

Our goal is to learn mapping functions between two domains, tap (T) and
sheared (S), given training samples comprising canonical (tap) data
$\\{x_{t}\\}_{i=1}^{N}$ where $x_{t}\in X_{t}$ and shear-transformed training
samples $\\{x_{s}\\}_{i=1}^{N}$ where $x_{s}\in X_{s}$. Our model includes
three mappings, $M:X_{s}\rightarrow X_{t}$, $N:X_{s}\rightarrow X_{s}$ and
$K:X_{t}\rightarrow X_{s}$. Our objective has four loss terms: reconstruction
loss to match the input $S_{\rm inp}$ and reconstructed sheared samples
$S_{\rm rec}$ to learn the mapping $N$, an L2 loss to match the reconstructed
$U_{\rm rec}$ and canonical samples $U_{\rm GT}$ in the supervised training
phase to fine tune the learned mapping $M$, an L1 patch loss to enforce image
similarity locally between the reconstructed $U_{\rm rec}$ and canonical
samples $U_{\rm GT}$ in the supervised training phase to fine tune the learned
mapping $M$ and cycle consistency loss to learn mappings $M$ and $K$ and
prevent them from contradicting each other.

The encoder $E_{S}$ (Fig. 2) disentangles the latent representation
($\mathbf{z\in Z}$) of input sheared samples $S_{\rm inp}$ into a pose code
($\mathbf{z_{t}\in Z_{t}}$) and shear code ($\mathbf{z_{s}\in Z_{s}}$). This
disentanglement of sensor response not only aids in learning of robust mapping
$M$ as shown in [7], but also allows recombination of unpaired shear and pose
codes to synthesize novel sheared samples necessary for generating training
signals via cycle consistency loss [15] without external supervision.

##### cycle Consistency Loss ($L_{\rm cyc}$)

We apply the cycle consistency loss $\mathbb{L}_{\rm cyc}$ to train the model
(Fig. 2) in the self-supervised phase of training. This loss aids learning of
mappings $M$ and $K$, and prevents them contradicting each other. Enforcing
cycle-consistency reduces the space of possible mapping functions that can be
learned. For each image $x_{s}$ from the domain $S$, the forward cycle should
be able to bring the input image $x_{s}$ back to original image i.e.
$x_{s}\rightarrow M(x_{s})\rightarrow K(M(x_{s}))\approx x_{s}$. Similarly,
the backward cycle should satisfy $x_{t}\rightarrow K(x_{t})\rightarrow
M(K(x_{t}))\approx x_{t}$ for each $x_{t}\in T$. This behaviour is enforced
via the cycle consistency loss:

$\begin{split}\mathbb{L}_{\rm cyc}(M,K)&=\mathbb{E}_{x_{s}\sim
X_{s}}[\|x_{s}-K(M(x_{s}))\|_{2}]\\\ &+\mathbb{E}_{x_{t}\sim
X_{t}}[\|x_{t}-M(K(x_{t})\|_{2}].\end{split}$ (1)

##### Reconstruction Loss ($L_{\rm rec}$)

We apply the reconstruction loss $\mathbb{L}_{\rm rec}$ to learn the mapping
$N:X_{s}\rightarrow X_{s}$ between input sheared samples $S_{\rm inp}$ and
reconstructed output sheared samples $S_{\rm rec}$ through a decoder $D_{S}$,
in both self-supervised and supervised phases of training. This loss aids the
disentanglement of sheared-input samples into pose and shear codes:

$\mathbb{L}_{\rm rec}(N)=\mathbb{E}_{x_{s}\sim
X_{s}}[\|x_{s}-N(x_{s})\|_{2}].$ (2)

#####  $L_{sup}$ Loss

We apply an L2 loss ($\mathbb{L}_{\rm sup}$) to fine tune the mapping
$M:X_{s}\rightarrow X_{t}$ between input sheared samples $S_{\rm inp}$ and
model-generated unsheared images $U_{\rm rec}(M(x_{s}))$.

$\mathbb{L}_{\rm sup}(M)=\mathbb{E}_{x_{s}\sim X_{s},x_{t}\sim
X_{t}}[\|x_{t}-M(x_{s})\|_{2}].$ (3)

#####  $L1_{\rm patch}$ Loss

We apply $L1_{\rm patch}$ loss $\mathbb{L}_{\rm patch}$ to fine tune the
mapping $M:X_{s}\rightarrow X_{t}$. Unlike $\mathbb{L}_{\rm sup}$, this loss
helps enforce image similarity locally, because the change in tactile sensor
response with contact conditions is predominantly local in nature. The loss
was computed between 100 random crops of size $20\times 20$ of model-generated
unsheared images $U_{\rm rec}(M(x_{s}))$ and its canonical (tap) counterparts
$U_{GT}(x_{t})$ .

$\begin{split}\mathbb{L}_{\rm patch}(M)=\mathbb{E}_{x_{s}\sim X_{s},x_{t}\sim
X_{t}}[\|&{\rm crops}(x_{t})-{\rm crops}(M(x_{s}))\|_{1}].\end{split}$ (4)

##### Full Objective

Our full objectives for Encoders $E_{S}$, $E_{T}$ and Decoders $D_{S}$,
$D_{T}$ are:

$\displaystyle\mathbb{L}_{E_{S}}=\mathbb{L}_{\rm cyc}(M,K)+\mathbb{L}_{\rm
rec}(N)+\mathbb{L}_{\rm sup}(M)+\lambda\cdot\mathbb{L}_{\rm patch}(M)$ (5)
$\displaystyle\mathbb{L}_{E_{T}}=\mathbb{L}_{\rm cyc}(M,K)$ (6)
$\displaystyle\mathbb{L}_{D_{T}}=\mathbb{L}_{\rm cyc}(M,K)+\mathbb{L}_{\rm
sup}(M)+\lambda\cdot\mathbb{L}_{\rm patch}(M)$ (7)
$\displaystyle\mathbb{L}_{D_{S}}=\mathbb{L}_{\rm rec}(N)$ (8)

where $\lambda=0.1$ is the relative loss scale factor.

The fully-supervised counterpart [7] of the above model is akin in
architecture and training methodology to supervised phase of the proposed
model except in the amount of annotated data (10$\times$) used for training.

###  III-D Implementation

####  III-D1 Network Architecture

##### Shear Encoder

The shear encoder $E_{S}$ compressed the sheared input $S_{\rm inp}$
($256\times 256$-pixel tactile image) using four convolutional (Conv) layers,
each followed by batch normalization (BN) and rectified linear unit (ReLU)
activation layers respectively (architecture in Fig. 2). The convolution
layers had 32, 64, 64 and 128 filters respectively. The output of the last
convolutional layer was passed to two additional Conv layers whose outputs
represent the two latent codes: Pose ($8\times 8\times 64$) and Shear
($8\times 8\times 64$).

##### Pose (Tap) Decoder

The pose decoder $D_{T}$ upsampled the pose code to reconstruct unsheared
output $U_{\rm rec}$ ($256\times 256$-pixel tactile image). It had five
transposed convolutional layers (Tconv), each followed by a BN and ReLU
activation layers respectively except the output layer which used sigmoid
activation function. The five transposed convolution layers had 64, 64, 32, 32
and 1 filter respectively.

##### Shear Decoder

The shear decoder $D_{S}$ upsampled the pose and shear code to reconstruct
sheared input $S_{\rm rec}$ ($256\times 256$-pixel tactile image). It had the
same overall architecture as the pose decoder $D_{T}$ except the number of
filters used in Tconv layers. The five transposed convolution layers had 128,
64, 64, 32 and 1 filter respectively.

##### Pose (Tap) Encoder

The pose encoder $E_{T}$ compressed the unsheared output of $D_{T}$ to
$8\times 8\times 64$ using five Conv layers each followed by a BN and ReLU
activation layer. The five convolution layers had 32, 32, 64, 64 and 64
filters.

All encoders and decoders used a kernel of $4\times 4$ with stride of 2. For
detailed architecture, see Fig. 2.

####  III-D2 Training Details

The input and output tactile data (binary $256\times 256$-pixel images) were
scaled to the range $[0,1]$. The network weights were initialized from a zero-
centered normal distribution with standard deviation 0.02.

For training, a batch size of 32 was used. All network layers used L1/L2
regularization along with random image shifts of 1-2% to prevent overfitting.
We used ADAM optimizer [24] with $\beta_{1}$ = 0.5, $\beta_{2}$ = 0.999 and
learning rate of $5\times 10^{-5}$ for self-supervised (SS) and $2.5\times
10^{-5}$ for supervised (S) training phases respectively. The model was
trained for 50 epochs in the SS phase and for another 50 epochs in the S
phase. We used learning rate scheduling for S training phase with training
rate reduced to one-tenth after 25 epochs. The training and optimization of
the networks was implemented in the Tensorflow 2.0 library on a NVIDIA GTX
1660 (6 GB memory) hosted on a Ubuntu machine.

##  IV Experimental Results

###  IV-A Disentanglement of Latent Representations

We did an ablation study to verify the successful disentanglement of sensor
response components due to the geometry of contact (pose code) and that
induced by motion-induced shear (shear code). To do this, we reconstructed
unsheared samples ($U_{\rm rec}$) from the shear code instead of the pose
code. This resulted in a $\sim$5-fold increase in mean-squared error (MSE)
between test set model-generated unsheared images $U_{\rm rec}$ and their
canonical counterparts $U_{\rm GT}$ (0.11 from 0.024) as well as $>70$%
reduction in image similarity (SSIM) (18% from 92.5%), indicating the two
images are structurally very dissimilar [27]. This shows that the appropriate
information to reconstruct unsheared samples ($U_{\rm rec}$) resides in the
pose code. Similarly, we tested the reconstruction of sheared samples ($S_{\rm
rec}$) from only the pose code, which resulted in a 60-fold increase in MSE
error between the input $S_{\rm inp}$ and the reconstructed $S_{\rm rec}$
sheared samples, and $>70$% reduction in SSIM between $S_{\rm inp}$ and
$S_{\rm rec}$ (18% from 93%), showing that the both shear and pose code are
jointly required for reconstruction of $S_{\rm rec}$. These two experiments
jointly demonstrate the successful disentanglement of the sensor response by
our model (Fig. 2).

###  IV-B Motion-induced Shear Removal from Tactile Images

First, we used multi-scale SSIM [27] to test the effectiveness of our approach
in removing motion-induced global shear. To do so, we computed the average
image similarity between the model-generated unsheared images $U_{\rm rec}$
and their test set sheared $S_{\rm inp}$ and canonical (non-sheared) $U_{\rm
GT}$ counterparts. On comparison, we found a closer match between $U_{\rm
rec}$ and $U_{\rm GT}$ (92.5% with 10% supervision), dropping to 74% in
absence of any supervision) than $U_{\rm rec}$ and $S_{\rm inp}$ (32%). The
corresponding values for fully supervised baseline [7] were 93% and 32%
respectively. Thus, our model not only successfully removes global shear from
tactile images but is also economical with labelled data (using just 10% of
the previous supervised method [7]). Degraded performance was seen both in the
supervised baseline [7] and semi-supervised approach proposed in this work on
reducing the labelled data used for training. For example, a 20% reduction in
labelled training data lead to a drop of approximately 2% (from 93%) and 0.5%
(from 92.5%) in image similarity between the model-generated unsheared images
and their test set canonical (non-sheared counterparts) for supervised
baseline [7] and proposed semi-supervised approaches respectively.

We also found that omitting $L1_{\rm patch}$ loss leads to a 1.5% drop in
image similarity between test set canonical images $U_{\rm GT}$ and their
model-generated unsheared counterparts $U_{\rm rec}$ (91.5% from 93%)at 10%
supervision.

TABLE I: Mean Absolute Error (MAE) of Pose Prediction

Pose Prediction Error
---
|  | Tap Data
---
| Sheared Data
---
| Unsheared Data
---
(Supervised)
| Unsheared Data
---
(Semi-Supervised)
| horizontal, $\tau_{x}$
---
(mm)
0.43 | 2.72 | 0.64 | 0.71
| yaw, $\theta n_{Z}$
---
(degrees)
2.13 | 22.20 | 4.38 | 4.81

###  IV-C Local Contact Geometry Reconstruction

Motion-induced shear masks the contact geometry requiring removal of global
shear for its faithful reconstruction (Fig. 3, top row). To reconstruct the
approximate indentation field, we used a Voronoi-based method [23] that
transforms the inner sensor surface with tactile markers into hexagonal cells,
thus tessellating the grid of the markers [23, Fig. 4]. The change in cell
areas from an undeformed reference represent the magnitude of local skin
deformation that correlates with the indentation due to contact geometry.

A 3D surface was fitted to the $(x,y)$ centroid coordinates of the Voronoi
cells with cell areas as the corresponding height values (Fig. 3, top row).
The fitted surface from the test sheared data $S_{\rm inp}$ is visibly
distorted by shear (Fig. 3, top row). Once global shear is removed in the
model-generated unsheared images $U_{\rm rec}$, the fitted surface closely
resembles the true representation from the original canonical (non-sheared)
data $U_{\rm GT}$ and the unsheared images obtained via supervised baseline
[7] across multiple stimuli (Fig. 3, top row). This again demonstrates the
effectiveness of our approach in removing global shear vis-á-vis the
supervised baseline [7].

Figure 3: Contact geometry reconstructions, contour following and full object
shape reconstructions. Top row: Local contact geometry reconstruction: The
approximate contact geometry can be recovered from sheared data using the
model-generated unsheared images but not from the original sheared data.
Middle row: Trajectories, overlaid on the objects, under robust sliding using
pose estimation from model-generated unsheared images. Red and green
trajectories correspond to semi-supervised and supervised [7] approaches
respectively. Red dot is the starting point of trajectory. For both, contact
reconstruction and contour following, supervised [7] & semi-supervised
approaches achieve comparable performance. Bottom row: Full object
reconstructions from combining the unsheared local contact geometry
reconstructions along the sliding trajectories.

###  IV-D 2D Continuous Shape Exploration

We further confirmed the model performance by computing the mean-absolute
error (MAE) between the predicted pose (lateral position $\tau_{x}$ and in-
plane rotation $\theta n_{z}$), from test set model-generated unsheared images
$U_{\rm rec}$, sheared images $S_{\rm inp}$ and canonical (non-sheared) images
$U_{\rm GT}$, and target pose using a pose prediction network PoseNet trained
on canonical (non-sheared) images previously used in [7]. In comparison, the
prediction error with $S_{\rm inp}$ was over 4-fold higher than with $U_{\rm
rec}$ (Table I), showing the successful removal of global shear. Our results
also show that the model-generated unsheared images from the present model and
from the fully supervised baseline [7] have similar prediction errors (Table
I).

To test the real-time performance and model generalizability to novel
situations, we used the PoseNet [7] for sliding exploration of various planar
shapes (Fig. 3, second row). This testing included four novel shapes absent
from training, including two acrylic shapes (spirals 1 and 2, Fig. 3, second
row) with distinct friction to the 3D-printed stimuli. In addition, for all
test shapes, we tested the performance by varying the location of initial
contact, depth of indentation, exploration speed, and also the relative
location between the object and the sensor tip. The robot successfully traced
contours around all test shapes (Fig. 3, second row, red) by a combination of
model-generated unsheared images, PoseNet for prediction of pose parameters
and a simple servo control policy [7], demonstrating successful removal of
global shear. Thus, our model generalizes well to novel experimental
conditions. Moreover, comparison of the contours using the current approach
(red) and supervised baseline (green) [7] show near-identical performance
(Fig. 3, second row).

###  IV-E 2D Object Reconstruction

As a final test, we demonstrate our model’s utility in the removal of global
shear from sheared images by reconstruction of full object shape. This
required: 1) faithful contact-geometry reconstruction and 2) successful
contour tracing, with both tasks adversely impacted by motion-induced global
shear. To obtain full object reconstructions, we first fused together the
contact information extracted from unsheared images of sliding contacts
recorded during shape exploration and later interpolated them on a rectangular
grid (Fig. 3, third row) to obtain smooth object reconstructions from discreet
sliding contacts. These results demonstrate not only the successful removal of
global shear but also the effectiveness of learning the sheared-unsheared
mapping once, for later reuse on multiple downstream tasks. The faithful
object reconstructions with successful shape exploration across multiple
shapes and experimental conditions discussed in section IV.D also show that
our model generalizes to novel situations (Fig. 3, second and third row).

##  V Discussion

This work proposed a semi-supervised approach that preserved the sensor
deformations due to stimulus contact geometry while removing the shear-
distortion caused by motion-dependent shear. We showed that the proposed
approach achieves comparable performance to its fully supervised counterpart
[7] with an order-of-magnitude less annotated data, simplifying data
collection considerably.

We validated our approach, similar to [7] by: 1) demonstrating a good match
(92.5%) between the model-generated unsheared images (from sliding contacts)
and their non-sheared counterpart taken from vertical tapping, using the
structural similarity index measure; 2) reconstructing an approximate stimulus
contact geometry from model-generated unsheared images that was previously
masked by sliding-induced shear (Fig. 3, top row); 3) predicting local object
pose from a pose prediction network [7] trained on non-sheared (tapping) data
(Table I). This pose prediction was then used for sliding exploration of
multiple planar objects (Fig. 3, second row), and so result 4) combined 2) and
3) to reconstruct full object shapes for several planar objects (Fig. 3, third
row).

One limitation of our study is that it considered only planar objects. We
expect our approach will extend to 3D objects, like recent work that has
successfully implemented tactile servoing to slide over complex 3D surfaces (3
pose components) and edges (5 pose components) [5, 6]. An extension would be
to sliding shape exploration and full 3D object reconstruction, which are open
problem under active investigation [25, 26]. This would open interesting
avenues for future research, such as: 1) the fusion of visual and tactile
sensory modalities for robust object representations and 2) tactile object
recognition by exploiting prior visual experience of the object. An additional
limitation is the requirement of annotated data, although much less than the
fully supervised counterpart [7]. An unsolved problem for future work is to
completely eliminate external supervision. Finally, our training was limited
to translational shear even though servo control introduces rotational shear
from the angular motion while sliding over objects. That the methods worked
well both for sliding exploration and contact geometry reconstruction in the
absence of trying to compensate for rotational shear shows that some shear
removal carries over from translation to rotational shear. Our expectation is
that bringing rotational shear into the training will become important when
extending the methods to 3D objects.

Overall, we expect that our methods should apply to various other exploration
and manipulation tasks using soft optical tactile sensors adversely affected
by motion-induced shear.

## References

  * [1] N. F. Lepora, ”Soft Biomimetic Optical Tactile Sensing with the TacTip: A Review,” in IEEE Sensors Journal, vol. 21, no. 19, pp. 21131-21143, 2021.
  * [2] X. Lin, L. Willemet, A. Bailleul and M. Wiertlewski, ”Curvature sensing with a spherical tactile sensor using the color-interference of a marker array,” 2020 IEEE International Conference on Robotics and Automation (ICRA), 2020, pp. 603-609.
  * [3] K. Kamiyama, K. Vlack, T. Mizota, H. Kajimoto, K. Kawakami and S. Tachi, ”Vision-based sensor for real-time measuring of surface traction fields,” in IEEE Computer Graphics and Applications, vol. 25, no. 1, pp. 68-75, Jan. 2005.
  * [4] C. Sferrazza and R. D’Andrea. ”Design, Motivation and Evaluation of a Full-Resolution Optical Tactile Sensor” Sensors, vol. 19, no. 4, pp. 928, 2019.
  * [5] N. F. Lepora, A. Church, C. de Kerckhove, R. Hadsell and J. Lloyd, ”From Pixels to Percepts: Highly Robust Edge Perception and Contour Following Using Deep Learning and an Optical Biomimetic Tactile Sensor,” in IEEE Robotics and Automation Letters, vol. 4, no. 2, pp. 2101-2107, April 2019.
  * [6] N. F. Lepora and J. Lloyd, ”Optimal Deep Learning for Robot Touch: Training Accurate Pose Models of 3D Surfaces and Edges,” in IEEE Robotics & Automation Magazine, vol. 27, no. 2, pp. 66-77, June 2020.
  * [7] A. K. Gupta, L. Aitchison and N. F. Lepora, ”Tactile Image-to-Image Disentanglement of Contact Geometry from Motion-Induced Shear,” 5th Conference on Robot Learning (CoRL), 2021.
  * [8] R. Zhang, P. Isola and A. A. Efros, ”Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction,” 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 645-654, 2017.
  * [9] M. Noroozi and P. Favaro, ”Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles,” In: Leibe B., Matas J., Sebe N., Welling M. (eds) Computer Vision – ECCV 2016. ECCV 2016. Lecture Notes in Computer Science, vol. 9910, Springer, Cham.
  * [10] D. Pathak, P. Krähenbühl, J. Donahue, T. Darrell and A. A. Efros, ”Context Encoders: Feature Learning by Inpainting,” 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2536-2544, 2016.
  * [11] A. Owens, A. A. Efros, ”Audio-Visual Scene Analysis with Self-Supervised Multisensory Features,” In: Ferrari V., Hebert M., Sminchisescu C., Weiss Y. (eds) Computer Vision – ECCV 2018. ECCV 2018. Lecture Notes in Computer Science, vol 11210. Springer, Cham.
  * [12] M. Zambelli, Y. Aytar, F. Visin, Y. Zhou and R. Hadsell, ”Learning rich touch representations through cross-modal
self-supervision,” 4th Conference on Robot Learning (CoRL), 2020.

  * [13] M. A. Lee, Y. Zhu, K. Srinivasan, P. Shah, S. Savarese, L. Fei-Fei, A. Garg and J. Bohg, ”Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks,” 2019 International Conference on Robotics and Automation (ICRA), pp. 8943-8950, 2019.
  * [14] N. Lepora and J. Lloyd, ”Pose-Based Tactile Servoing: Controlled Soft Touch Using Deep Learning,” in IEEE Robotics & Automation Magazine, vol. 28, no. 4, pp. 43-55, 2021.
  * [15] J. Zhu, T. Park, P. Isola and A. Efros, ”Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks,” IEEE International Conference on Computer Vision (ICCV), 2017, pp. 2242-2251.
  * [16] Y.Bengio. Deep Learning of Representations: Looking Forward. In: Dediu, AH., Martín-Vide, C., Mitkov, R., Truthe, B. (eds) Statistical Language and Speech Processing. SLSP 2013. Lecture Notes in Computer Science, vol. 7978. Springer, Berlin, Heidelberg.
  * [17] I. Higgins, L. Matthey, A. Pal, C. P. Burgess, X. Glorot, M. M. Botvinick, S. Mohamed and A. Lerchner, ”beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework”, International Conference on Learning Representations (ICLR), 2016.
  * [18] H. Kim and A. Mnih, ”Disentangling by Factorising”, Proceedings of the 35th International Conference on Machine Learning (PMLR) vol. 80, pp. 2649-2658, 2018.
  * [19] N. Siddharth, B. Paige, J.-W. van de Meent, A. Desmaison, N. D. Goodman, P. Kohli, F. Wood and P. H. S. Torr, ”Learning Disentangled Representations with Semi-Supervised Deep Generative Models”, 31st Conference on Neural Information Processing Systems (NIPS), 2017.
  * [20] X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever and P. Abbeel, ”InfoGAN: interpretable representation learning by information maximizing generative adversarial nets”, 30th Conference on Neural Information Processing Systems (NIPS), 2016.
  * [21] I. Jeon, W. Lee, M. Pyeon and G Kim, ” IB-GAN: Disentangled Representation Learning with Information Bottleneck Generative Adversarial Networks”, Proceedings of the AAAI Conference on Artificial Intelligence (AAAI-21), vol. 35, no. 9, pp. 7926-7934, 2021.
  * [22] T. Park, J.-Y. Zhu, O. Wang, J. Lu, E. Shechtman, A. A. Efros, R. Zhang, ”Swapping Autoencoder for Deep Image Manipulation”, 34th Conference on Neural Information Processing Systems (NeurIPS), 2020.
  * [23] L. Cramphorn, J. Lloyd and N. F. Lepora, ”Voronoi Features for Tactile Sensing: Direct Inference of Pressure, Shear, and Contact Locations,” 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 2752-2757, 2018.
  * [24] D.P. Kingma and J.L. Ba,” Adam: A method for stochastic optimization,” CoRR, vol. abs/1412.6980, 2015.
  * [25] S. Suresh, M. Bauza, K. T. Yu, J, G. Mangelson, A. Rodriguez and M. Kaess, ”Tactile SLAM: Real-time inference of shape and pose from planar pushing,” 2018 IEEE International Conference on Robotics and Automation (ICRA), 2018.
  * [26] M. Bauza, O. Canal and A. Rodriguez, ”Tactile Mapping and Localization from High-Resolution Tactile Imprints,” 2019 International Conference on Robotics and Automation (ICRA), pp. 3811-3817, 2019.
  * [27] Z. Wang, E. Simoncelli and A. Bovik, ”Multiscale structural similarity for image quality assessment,” The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, vol. 2, pp. 1398-1402, 2003.


# A Novel Speech-Driven Lip-Sync Model with CNN and LSTM

Xiaohong Li, Xiang Wang, Kai Wang and Shiguo Lian  AI Innovation and
Application Center,
China Unicom, Beijing, China

###### Abstract

Generating synchronized and natural lip movement with speech is one of the
most important tasks in creating realistic virtual characters. In this paper,
we present a combined deep neural network of one-dimensional convolutions and
LSTM to generate vertex displacement of a 3D template face model from
variable-length speech input. The motion of the lower part of the face, which
is represented by the vertex movement of 3D lip shapes, is consistent with the
input speech. In order to enhance the robustness of the network to different
sound signals, we adapt a trained speech recognition model to extract speech
feature, and a velocity loss term is adopted to reduce the jitter of generated
facial animation. We recorded a series of videos of a Chinese adult speaking
Mandarin and created a new speech-animation dataset to compensate the lack of
such public data. Qualitative and quantitative evaluations indicate that our
model is able to generate smooth and natural lip movements synchronized with
speech.

###### Index Terms:

speech-driven lip-sync; facial animation synthesis; convolutional neural
network; LSTM

##  I Introduction

Creating lifelike and emotional digital human has wide applications in many
fields, such as film and games, facial repair and therapy, child education and
so on. Nowadays, it is often seen a digital human serving as a newscaster or a
narrator. Facial movement, especially the lip movement when speaking, is one
of the most important component for digital human to express themselves. The
lip shape must match the pronunciation, otherwise it will make audience feel
uncomfortable or even fake, leading to the uncanny valley effect. In recent
years, a lot of research on speech-driven lip-sync has been carried out. These
works can be divided into two categories: 2D and 3D, depending on whether the
output is 2D video or a 3D animation. In this work, we focus on the 3D facial
animation generation, which is widely used in video games and films.

Although various works on 3D facial animation have been proposed in recent
years [13, 9, 4], few works have addressed the issue of animating a 3D model
speaking Chinese due to the lack of open dataset and adapted algorithm. To
compensate the lack of training data, we used a web camera to record several
hours of the video of a male Chinese character speaking Mandarin, with rate of
60 frames per second (fps). The recorded data includes images and the
synchronized voice. Afterwards, we created a 3D mesh model of a human head,
which was used to transfer the facial motion captured from the human performer
to a digital face with the help of a facial tracking software and a
retargeting software from Faceware Technologies [7].

To effectively make use of the collected data, we designed a deep neural
network which takes speech feature as input and outputs the 3D vertex
displacement. The speech feature is extracted with a pre-trained speech
recognition model, and the vertex displacement is used to drive 3D mesh models
to generate accurate facial movements synchronized with the input speech
sound. The proposed network combines a one-dimensional convolution and LSTM
(Long Short-Term Memory) and is able to generate realistic, smooth and natural
facial animations. Note that although we only collected the voice and motion
of a male character as training data, the facial animation generated by our
model can be applied to 3D virtual characters of different styles and genders.
In addition, our model is robust to the speech of different people with
different voices.

The remaining part of the paper is organized as follows: Section II reviews
the related work of speech-driven facial animation methods. The details of the
data collection and the proposed network is introduced in Section III. Section
IV shows the experimental results and the conclusion is given in Section V.

##  II Related Work

Speech-driven facial animation has received intensive attentions in the past
decades. Existing methods can be classified into viseme-driven and data-driven
approaches. The viseme-driven approaches adopt a two-phase strategies: speech
recognition algorithms are first used to segment speech into phonemes, which
are then mapped to visual units a.k.a visemes. Contrarily, the data-driven
approaches learnt from a large amount of data and learnt a model which
directly transfers from speech or text to facial animations.

Mattheyses et al. declared that an accurate mapping from phonemes to visemes
should be many-to-many due to the coarticulation effect [10]. They introduced
a many-to-many phoneme-to-viseme mapping scheme using tree-based and k-means
clustering approaches, and achieved better visual result. Edwards et al.
depicted the many-valued phoneme-to-viseme mapping using two visually distinct
anatomical actions of jaw and lip, and proposed the JALI viseme model [6].
They first computed a sequence of phonemes from the speech transcript and
audio, and then extracted the jaw and lip motions for individual phonemes as
viseme action units and blended the corresponding visemes into coarticulated
action units to produce animation curves. At last, a viseme compatible rig was
driven by the computed viseme values. Zhou et al. applied the JALI model and
further introduced a three-stage LSTM network to predict compact animator-
centric viseme curves[16].

As it is difficult to accurately identify phonemes and the coarticulation
effect is difficult to simulate with simple functions, more and more
researchers preferred the data-driven methods. Shimba et al. trained a
regression model from speech to talking head with LSTM network and applied
lower level audio features instead of phonemes as input [13]. The output of
their network was Active Appearance Model (AAM) parameters [3]. Suwajanakorn
et al. also used a recurrent neural network to learn the mapping from raw
audio features to mouth shapes based on many hours of high quality videos of
the target person [14]. They synthesized mouth textures and composed them with
proper images to generate a 2D video that matched the input speech. Unlike the
above 2D methods, Cudeiro et al. proposed an audio-driven 3D facial animation
method [4]. They trained a neural network on their 4D scans dataset captured
from 12 speakers, and used a subject label to control the generated speaking
style. In addition, they integrated the speech feature extraction approach
DeepSpeech [8] into their model to improve the robustness to different audio
sources. However, their trained model is only applicable to targets of the
FLAME model [9], which is a statistical head model. Meanwhile, their model
performs better on English than other languages because their training data
only includes English speech.

##  III Method

When dealing with temporal and sequential tasks, such as speech recognition,
machine translation and text processing with relevance to the context, the
Recurrent Neural Networks (RNNs) are often used considering its advantage over
the traditional feed-forward neural networks which cannot exhibit temporal
dynamic behavior. The RNNs are a class of neural networks that allow previous
output be used as input to the recurrent layer. However, it is difficult to
solve problems that require learning long-term temporal dependencies using
standard RNNs due to the vanishing gradient phenomenon. Therefore, a memory
cell that is able to maintain information for long period is introduced, which
is known as the Long Short-Term Memory (LSTM) unit. In this work, we use LSTM
as the backbone network to map the speech feature extracted with DeepSpeech to
vertex offsets. Specifically, we use unidirectional LSTM to model this mapping
and make further improvements on this basis.

A common LSTM unit contains a cell, an input gate, an output gate and a forget
gate. The cell remembers information from previous intervals and the three
gates control the memorizing process. Let $x_{t}$ denote the input of a LSTM
unit at time $t$. The activation of this unit is updated as follows:

$\displaystyle f_{t}=\sigma(W_{f}\cdot[h_{t-1},x_{t}]+b_{f})$ (1)
$\displaystyle i_{t}=\sigma(W_{i}\cdot[h_{t-1},x_{t}]+b_{i})$ (2)
$\displaystyle o_{t}=\sigma(W_{o}\cdot[h_{t-1},x_{t}]+b_{o})$ (3)
$\displaystyle\tilde{C}_{t}=tanh(W_{C}\cdot[h_{t-1},x_{t}]+b_{C})$ (4)
$\displaystyle C_{t}=f_{t}*C_{t-1}+i_{t}*\tilde{C}_{t}$ (5) $\displaystyle
h_{t}=o_{t}*tanh(C_{t}),$ (6)

where $f$, $i$, $o$, $C$, $h$ represent the forget gate, the input gate, the
output gate, the cell state and the cell output respectively. $\sigma$ denotes
the sigmoid activation function. $W$ and $b$ are the weights and biases
respectively.

Next, we will introduce our dataset generation method, the process of speech
feature extraction, our network framework and the loss function in details.

###  III-A Dataset Generation

To the best of our knowledge, there has not been any public Chinese dataset
that can be used for the training of speech-driven 3D facial animation. So we
produced our own dataset. We applied the THCHS-30 corpus [15], which is an
open Chinese speech database published by Center for Speech and Language
Technology (CSLT) at Tsinghua University, as our speech script. It contains
1000 sentences and covers most of the phonemes in Chinese. A male character
was invited to speak these sentences with normal speed in front of a camera.
This camera contains a microphone and is able to record synchronized video and
sound. It was used to capture the image data at a rate of 60 frames per second
and record the surrounding sound signals at the same time. We collected
totally four hours of video data.

Next, we utilized two tools from the Faceware software kits [7] to convert the
collected videos into 3D animations. One is the face tracking tool Analyzer,
which tracks facial landmarks from video in a markerless way (see Figure 1).
The other is Retargeter, which is a high quality facial animation solving
software that retargets facial motion from a tracked video onto a 3D
character. It needs a prepared 3D model with proper rigs and generates facial
animations by controlling the rig movement. We used these two tools to
transfer the facial animations from the captured video to a 3D character, and
obtained the 3D face data by recording the vertex position of the head of the
animated 3D character in every frame. Figure 2 shows the retargeting result
from video frames to a virtual 3D character. The prepared virtual 3D character
is regarded as a template, which is a mesh model in ”zero pose” (see Figure
3). All the animated 3D faces have the same topology as the template.

Figure 1: Facial landmarks tracked with Faceware Analyzer. Figure 2: Facial
motions transferred from video to 3D character with Faceware Retargeter.
Figure 3: 3D virtual character used to generate 3D facial animation data.

We split the generated 3D facial animation data and accompanied audio data
into a training set (900 sentences), a validation set (50 sentences) and a
test set (50 sentences). The ground truth of the vertex displacement is
calculated by subtracting the template from the generated 3D faces.

###  III-B Speech Feature Extraction

In order to improve the robustness of our model to input speech signals, we
use a pre-trained speech recognition model to extract speech feature. Like
VOCA [4], we adapt the DeepSpeech model [8], which is trained on hundreds of
hours of voice data and generalized to different speakers, voice speed,
environment noise, etc. We used the pre-trained model provided at the
DeepSpeech GitHub releases page, which used Mel Frequency Cepstral
Coefficients (MFCCs) [5] to extract the input audio features, and replaced the
fourth recurrent layer with a LSTM unit. The output of the model is a sequence
of character probabilities, which is used as the input of our network.

Given an audio clip with a length of $T$ seconds, we first resample the audio
to fixed 16kHZ, and calculate its MFCCs. After normalization, we feed them to
the pre-trained DeepSpeech model to extract audio features and resample them
to 60 fps, which is consistent with the frame rate of videos in our dataset.
Finally, the output is a two-dimensional array with size of $60T\times D$,
where $D$ is the number of letters in alphabet plus a blank label.

###  III-C Network Architecture

The architecture of our network is inspired by WaveNet [11] proposed by Aaron
van den Oord et al., which is a deep generation model of raw audio waveforms.
It is able to generate speech that mimics many different speakers, and make it
sound more naturally than lots of existing text-to-speech systems. It realized
a time sequence generation with only a series of convolutional layers stacked,
which is faster to train than the recurrent network. However, its receptive
field is limited. To overcome this limitation, we combine the convolutional
layers with LSTM blocks to construct our network structure, as shown in Figure
4.

Figure 4: Network architecture

The network is implemented using Tensorflow framework [1]. The first part of
the network is composed of two one-dimensional convolutional layers, four
unidirectional LSTM blocks and two fully connected layers, which are used
together to transform speech features extracted by DeepSpeech to a low-
dimensional embedding. The latter part of the network is a decoder consisting
of a fully connected layer with linear activation. The decoder maps the
embedding into a high-dimensional space of 3D vertex displacement, whose
dimension is $5713\times 3$ in our case, as the 3D facial model we used in
training set has $5713$ vertices, and the position of each vertex is
represented by the Cartesian coordinate in 3D space. Table I shows the
specific parameters of our network.

TABLE I: The proposed network architecture Type | Kernel | Stride | Output | Activation | Num
---|---|---|---|---|---
DeepSpeech | - | - | 29 | - | -
Convolution | 5$\times$1 | 1$\times$1 | 32 | ReLU | 2
LSTM | - | - | 128 | - | 2
LSTM | - | - | 64 | - | 2
Fully connected | - | - | 128 | tanh | -
Fully connected | - | - | 50 | linear | -
Fully connected | - | - | 5713$\times$3 | linear | -

During inference, we take variable-length audio clips as input and output
vertex displacement at 60 fps. Then the computed displacement is added to the
position of each vertex of a 3D face template to drive the face deformation.
The topology of the face template here needs to be the same as that of the one
we used for training data generation.

###  III-D Loss Function

Suppose the vertex displacement sequence output by the network during training
is $\\{\tilde{y}_{t}\\}$, $t=1..T$, and the ground truth of the vertex
displacement is $\\{y_{t}\\}$, $t=1..T$, where $T$ is the number of video
frames. We train the model by minimizing the following loss function:

$Loss=\omega_{1}*L_{p}+\omega_{2}*L_{v},$ (7)

where $L_{p}$ is the reconstruction loss, $L_{v}$ is the velocity loss, and
$\omega_{1}$ and $\omega_{2}$ are the weight coefficients respectively.
$L_{p}$ is defined in Eq. 8, in which $F$ represents the Frobenius Norm. It
calculates Euclidean distance of vertices between the predicted output and the
real facial animation, and thus used to constrain the gap between the
predicted vertex coordinates and the ground truth.

$L_{p}={\parallel y_{t}-\tilde{y}_{t}\parallel}_{F}^{2}.$ (8)

The velocity loss $L_{v}$ is defined as Eq. 9. It uses backward finite
differences of the mesh vertices of adjacent frames to estimate the
deformation speed of the face vertices, and calculates the difference between
the predicted value and the ground truth. It has a smoothing effect. When only
reconstruction loss is used, the mouth movement is obvious, while lip jitter
cannot be avoided. Through qualitative and quantitative analysis, it is found
that the velocity loss can reduce the lip shaking and improve the model
accuracy significantly.

$L_{v}={\parallel(y_{t}-y_{t-1})-(\tilde{y}_{t}-\tilde{y}_{t-1})\parallel}_{F}^{2}.$
(9)

The weight coefficients for $L_{p}$ and $L_{v}$ have to be set carefully,
otherwise the mouth shape will be inaccurate. We set the weights of the
reconstruction loss and velocity loss to $1.0$ and $0.5$ respectively. More
details can be found in the follow-up comparative experiments in Section IV-B.

##  IV Experiments

We compared our network with other methods and conduct ablation experiments
for velocity loss term. Both qualitative and quantitative evaluations and
analysis will be introduced in this section.

###  IV-A Qualitative Evaluation

Although we only used the voice of a male for training, our method generalizes
well for female and even synthesized voices. The test results for different
speech sources can be found in the supplementary
video111https://www.dropbox.com/s/71oayo97aywd3l9/A%20Novel%20Speech-
Driven%20Lip-Sync%20Model%20with%20CNN%20and%20LSTM.mp4?dl=0. We compared our
network with LSTM and VOCA, whose results are also shown in the supplementary
video. We used the pre-trained VOCA model which is claimed to work for any
language, and use the same test audio spoken in Mandarin as input.

Figure 5 shows some samples. The first column on the left shows the video
frames of the character, and the second column is the ground truth of the 3D
face which is reconstructed using Faceware. Facial animations generated by
LSTM, VOCA and our model are shown in the third, fourth and the last columns
respectively. The test speech of the first three rows comes from the test data
set, which is the same male voice as the training data. The test speech of the
last two rows is female voice and there is no ground truth of 3D face. It can
be obviously seen that our model shows the best performance and generates more
accurate mouth shapes than the other two methods.

Figure 5: Samples generated by different models. From left to right, the
columns show the face appearance of the character, the ground truth of 3D face
generated by Faceware, and results of LSTM, VOCA and our model respectively.

###  IV-B Quantitative Evaluation

Two metrics are used to measure the accuracy of generated facial animation and
lip movements for different models: the positional error and the velocity
error of the 2D facial landmarks. Two network structures with and without
velocity loss are compared, and the results are shown in Table II.

TABLE II: Error metrics of four models on the test data | LSTM | LSTM+v_loss | Conv+LSTM | Conv+LSTM+v_loss
---|---|---|---|---
| position error of the 2D facial landmarks (unit:pixel)
| 2.449 | 2.401 | 2.275 | 2.296
| position error of the 2D mouth landmarks (unit:pixel)
| 4.920 | 4.812 | 4.514 | 4.592
| velocity error of the 2D facial landmarks
| 4.022 | 3.593 | 3.719 | 3.397
| velocity error of the 2D mouth landmarks
| 5.704 | 4.537 | 4.881 | 4.204

It can be seen from the qualitative results that the combination of
convolution layers with LSTM achieves higher action accuracy than the
application of LSTM alone, both on the result of the landmark position error
and that of the motion error. The comparison results of the velocity loss
function is a bit complicated. For the LSTM network, the introduction of the
velocity loss function has shown positive effects both on positional error and
motion error. For the network with the combination of convolution layers and
LSTM blocks, the velocity loss function reduces the motion error of the
vertices, but it does not decrease the positional error of the landmarks.
Actually, the positional error of the landmarks can only represent the
difference between the facial shape output of the network and the ground
truth, rather than the reality and naturalness of the facial animation. For
example, the amplitude of the mouth movement is different when different
people speak. The mapping between the speech and the face shape is not unique.
Therefore, it is not appropriate to judge the contribution of the velocity
loss function to the fidelity of the facial animations from the positional
error of landmarks. On the other hand, as the velocity error of the landmarks
is able to reflect the error of facial motion, we believe that the
introduction of the velocity loss function is beneficial to the network we
proposed.

We also conducted experiment to prove if the velocity loss we added is able to
reduce lip jitter and make the animation transition between words much
smoother. Similar to [12], we render 3D facial animations into images and
detect 2D facial landmarks with a facial behavior analysis toolkit OpenFace
[2] (see Figure 6). We extract the positions of a feature point located in the
middle of the upper lip, and draw the movement curve of this feature point in
the $y$ direction. Figure 7 and Figure 8 show the movement curves generated by
models with and without the velocity loss term respectively. The base model
adopted in Figure 7 uses our network architecture which combines convolutional
layers with LSTM blocks, and that adopted in Figure 8 uses the LSTM network.
It can be seen from the two figures that the introduced velocity loss term
significantly reduces the lip jitter and smooths the lip movement for both
networks. It is also noted that the velocity loss reduced the motion range of
the mouth. Therefore, it is necessary to comprehensively consider the trade-
offs to select an appropriate weight for the velocity loss term.

Figure 6: Rendered facial animations and the detected 2D facial landmarks.
Figure 7: Movement curves of the selected lip feature point generated by our
models trained with and without the velocity loss term. Figure 8: Movement
curves of the selected lip feature point generated by LSTM models trained with
and without the velocity loss term.

##  V Conclusion

In this work, we created a new speech-animation dataset which contains the
video and animation data of an adult speaking Mandarin, and proposed a novel
network for the generation of facial animation from speech. We conducted both
quantitative and qualitative evaluations to show that our network is able to
generate more accurate and smooth facial animation, and is more robust to
various audio sources. It can be further improved by incorporating more
movement data of the upper face part, such as eyebrows and eyes, to make the
facial animation of digital human more natural and realistic.

## References

  * [1] Abadi, M., Agarwal, A., Barham, P., et al, _Tensorflow: Large-scale machine learning on heterogeneous distributed systems_. arXiv preprint arXiv:1603.04467, 2016.
  * [2] Baltrusaitis, T., Zadeh, A., Lim, Y. C., and Morency, L. P., _Openface 2.0: Facial behavior analysis toolkit_. In 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp. 59-66. IEEE, 2018.
  * [3] Cootes, T.F., Edwards, G.J. and Taylor, C.J., _Active appearance models_. IEEE Transactions on pattern analysis and machine intelligence, 23(6), pp.681-685, 2001.
  * [4] Cudeiro, D., Bolkart, T., Laidlaw, C., Ranjan, A. and Black, M.J., _Capture, learning, and synthesis of 3D speaking styles_. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, (pp. 10101-10111), 2019.
  * [5] Davis, S. and Mermelstein, P., _Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences_. IEEE transactions on acoustics, speech, and signal processing, 28(4), pp.357-366, 1980.
  * [6] Edwards, P., Landreth, C., Fiume, E. and Singh, K., _JALI: an animator-centric viseme model for expressive lip synchronization_. ACM Transactions on graphics (TOG), 35(4), pp.1-11, 2016.
  * [7] Faceware, _http://support.facewaretech.com/home_.
  * [8] Hannun, A., Case, C., Casper, J., et al, _Deep speech: Scaling up end-to-end speech recognition_. arXiv preprint arXiv:1412.5567, 2014.
  * [9] Li, T., Bolkart, T., Black, M.J., Li, H. and Romero, J., _Learning a model of facial shape and expression from 4D scans_. ACM Trans. Graph., 36(6), pp.194-1, 2017.
  * [10] Mattheyses, W., Latacz, L. and Verhelst, W., _Comprehensive many-to-many phoneme-to-viseme mapping and its application for concatenative visual speech synthesis_. Speech Communication, 55(7-8), pp.857-876, 2013.
  * [11] Oord, A.V.D., Dieleman, S., Zen, H., et al., _Wavenet: A generative model for raw audio_. arXiv preprint arXiv:1609.03499, 2016.
  * [12] Richard, A., Lea, C., Ma, S., Gall, J., de la Torre, F. and Sheikh, Y., _Audio-and gaze-driven facial animation of codec avatars_. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 41-50, 2021.
  * [13] Shimba, T., Sakurai, R., Yamazoe, H. and Lee, J.H., _Talking heads synthesis from audio with deep neural networks_. In 2015 IEEE/SICE International Symposium on System Integration (SII) (pp. 100-105). IEEE. 2015.
  * [14] Suwajanakorn, S., Seitz, S.M. and Kemelmacher-Shlizerman, I., _Synthesizing obama: learning lip sync from audio_. ACM Transactions on Graphics (ToG), 36(4), pp.1-13, 2017.
  * [15] Wang, D. and Zhang, X., _Thchs-30: A free chinese speech corpus_. arXiv preprint arXiv:1512.01882, 2015.
  * [16] Zhou, Y., Xu, Z., Landreth, C., Kalogerakis, E., Maji, S. and Singh, K., _Visemenet: Audio-driven animator-centric speech animation_. ACM Transactions on Graphics (TOG), 37(4), pp.1-10, 2018.