# Mesh-Tension Driven Expression-Based Wrinkles for Synthetic Faces

Chirag Raman
Delft University of Technology
chiragraman.com Charlie Hewitt
Microsoft
chewitt.me Erroll Wood
Microsoft
errollw.com Tadas Baltrušaitis
Microsoft
<EMAIL_ADDRESS>
###### Abstract

Recent advances in synthesizing realistic faces have shown that synthetic
training data can replace real data for various face-related computer vision
tasks. A question arises: how important is realism? Is the pursuit of
photorealism excessive? In this work, we show otherwise. We boost the realism
of our synthetic faces by introducing dynamic skin wrinkles in response to
facial expressions, and observe significant performance improvements in
downstream computer vision tasks. Previous approaches for producing such
wrinkles either required prohibitive artist effort to scale across identities
and expressions, or were not capable of reconstructing high-frequency skin
details with sufficient fidelity. Our key contribution is an approach that
produces realistic wrinkles across a large and diverse population of digital
humans. Concretely, we formalize the concept of mesh-tension and use it to
aggregate possible wrinkles from high-quality expression scans into albedo and
displacement texture maps. At synthesis, we use these maps to produce wrinkles
even for expressions not represented in the source scans. Additionally, to
provide a more nuanced indicator of model performance under deformations
resulting from compressed expressions, we introduce the 300W-winks evaluation
subset and the Pexels dataset of closed eyes and winks.

##  1 Introduction

Figure 1:  Final renders for a diverse set of synthetic identities and
expressions. For each identity we illustrate renders using the base method of
Wood et al. [1] (left), and our added technique for generating expression-
based wrinkling effects (right). For the same expression parameters, our
method produces varied wrinkling effects across distinct identities (middle
and bottom row).

Synthetic data has been commonly employed for a variety of computer vision
tasks including object recognition [2, 3, 4, 5], scene understanding [6, 7, 8,
9], eye tracking [10, 11], hand tracking [12, 13], and full body analysis [14,
15, 16]. However, the complexity of modeling the human head has largely
precluded the generation of full-face synthetics for face-related machine
learning. While realistic digital humans have been created for movies and
video games, they usually entail significant artist effort per character [17,
18]. Consequently in literature, the synthesis of facial training data has
been accompanied by simplifications, or a focus on parts of the face such as
the eye region [19, 20] or the hockey mask [21, 22, 23, 24]. This has resulted
in a domain gap—a difference in distributions between real and synthetic
facial data that makes generalization challenging. Efforts towards bridging
this domain gap have mainly utilized domain adaptation to refine synthesized
images [25] or domain-adversarial training where models are encouraged to
ignore domain differences [26]. As such, generating realistic face data has
been considered so challenging that it is assumed that synthetic data cannot
fully replace real data for in-the-wild tasks [25].

To directly address the challenge, Wood et al. [1] attempted to minimize the
domain gap at the source, by generating synthetic faces with unprecedented
realism. Their method procedurally combines a parametric 3D face model with a
comprehensive library of high-quality artist-created assets including
textures, hair, and clothing. In doing so, the method overcomes a key
bottleneck in techniques employed by the Visual Effects (VFX) industry for
synthesizing realistic humans—that of scale. The procedural sampling can
randomly create and render novel 3D faces without manual intervention. Machine
learning systems trained on the synthesized data for landmark localization and
face parsing achieved performance comparable with the state-of-the-art without
using a single real image.

However, one limitation of the method proposed by Wood et al. [1] is the lack
of dynamic, expression dependent wrinkles. The method generates textures using
only the neutral-expression scans, which remain static for all deformations of
the underlying face mesh resulting from expression changes. In this work we
propose a simple yet effective method for incorporating expression-based
wrinkles. Our central idea is to capture complex wrinkling effects for an
identity from high-resolution scans of their posed expressions. We store all
these possible wrinkles into albedo and displacement textures we refer to as
wrinkle maps. At synthesis, for any arbitrary expression beyond those
represented in the source scans, we blend between the neutral and wrinkle
textures using a notion of the tension in the face mesh to obtain dynamic
wrinkling effects. Figure 1 contrasts the results of our method against the
current state-of-the-art (SOTA) approach for face synthetics. We also include
animated sequences in the Supplementary Material.

The term wrinkle maps was first used by early VFX approaches to refer to
artist-defined bump or normal maps for simulating animated wrinkles [27, 28,
29, 30]. However, these approaches suffer from three drawbacks. First, the
bump and normal maps only simulate underlying geometry changes; the silhouette
and shadows which are of relevance for face related tasks such as landmark
localization remain unaffected. Second, the methods do not affect the albedo
or diffuse textures. Finally, the most crucial drawback is scale. The methods
entail manual definition of wrinkle maps and masks for every blendshape for
every character. In contrast, our automatic mesh-tension driven method
naturally scales with the number of identities and expressions, while
incorporating real wrinkles for both albedo and displacement textures from
scans. Furthermore, we also handle identities without expression scans,
transferring plausible wrinkles from the most similar neutral textures.

To advance the development of synthetics for face-related tasks, we make the
following concrete contributions:

  * •

A system for dynamic, expression-based wrinkles that scales easily with
increasing identities and expressions.

  * •

A demonstration of empirical qualitative and quantitative improvement over the
SOTA synthetics system on face-keypoint localization and surface-normal
estimation.

  * •

Novel evaluation data and metrics for keypoint localization in the eye region
where wrinkles are especially relevant for learning tasks.

##  2 Background: Synthesizing Faces

We build upon the work of Wood et al. [1] for synthesizing face images for
downstream machine learning tasks. Their method involved sampling from a
generative 3D blendshape-based face model learned from 3D scans of $511$
individuals with neutral expression. The sampled face is then dressed up with
samples from a large collection of hair, clothing, and accessory assets. For
each synthesized face, the authors employ three textures that remain fixed
across all expressions: one albedo map for skin color; one coarse displacement
map to encode scan geometry not captured by the sparsity of the vertex-level
identity model; and one meso-displacement map to approximate skin-pore level
detail built by high-pass filtering the albedo texture. In contrast, we
automatically compute an additional sets of albedo and displacement wrinkle
textures from expression scans to support dynamic wrinkling effects.

Figure 2:  Method Overview. The state-of-the-art method for face synthetics
[1] generates albedo and displacement textures using only the neutral-
expression scan for an identity (middle row, also see Figure 1). In contrast,
we automatically compute expanded and compressed texture maps to aggregate
wrinkling effects in the face and neck regions across available posed-
expression scans for the identity. At synthesis, for a given set of arbitrary
expression parameters we compute the local tension at every vertex in the
corresponding face mesh: we depict expansion in green and compression in red.
This mesh tension serves as weights to dynamically blend between the neutral,
expanded, and compressed texture maps to synthesize the wrinkling effect at
that vertex. Note that our method can thereby generate wrinkles for
expressions even beyond those represented in the source scans.

##  3 Related Work

#### Wrinkle Maps

Oat [27] proposed using a pair of bump maps to render animated wrinkles on
virtual characters. These bump maps—called wrinkle maps—store surface normals
for an expanded (or stretched) and compressed (or scrunched-up) expression,
typically obtained from artist sculpted high-resolution meshes. A base normal
map stores fine surface details such as pores. In order to achieve
independently controlled wrinkles, the face is divided into multiple regions.
Each region is specified by an artist-defined mask stored in a texture map. An
animated scalar wrinkle weight in the range $[-1,1]$ then interpolates between
the two wrinkle maps for each masked region: at either end of the range one of
the wrinkle maps is at its full influence, with a weight of $0$ corresponding
to no influence on the base normal map. A similar method was later
independently proposed by Duque Reis et al. [31] using a single wrinkle map.
Jimenez et al. [29] expanded on the scheme proposed by Oat [27], allowing for
the use of any number of wrinkle maps, with a weight in the range of $[0,1]$
defining the influence of each map. Subsequent improvements to make the
technique amenable in real-time or performance driven settings involved the
dynamic generation of either the region masks [30] or the wrinkle weights
[28]. Both approaches relied on using a skinned mesh attached to bones.
Dutreve et al. [30] proposed generating dynamic region masks by using the bone
influence weights from a set of artist defined reference poses. Oat [28]
proposed generating dynamic wrinkle weights by comparing each mesh triangle’s
area before and after skinning, a technique derived from Microsoft’s DirectX
10 Sparse Morph Targets demo [32]. While the term wrinkle maps in literature
has been alternatively used to refer to bump or normal maps, in this work we
use the term to collectively refer to the textures used for synthesizing
wrinkles: the albedo and displacement maps corresponding to the expanded and
compressed textures.

#### Simulation Based Approaches

While the use of wrinkle maps is the most common methodology when artistic
control is of importance, several alternate techniques have been proposed for
simulating wrinkles on 3D surfaces. These methods can broadly be grouped into
physical and geometric simulation of wrinkles. An early physical simulation
based approach employed a biomechanical perspective, considering the skin as
an elastic membrane and modeling the deformations using linear plastic model
[33]. Boissieux et al. [34] extended the elastic membrane perspective by
modeling the skin as a volumetric substance comprising layers of different
materials and using a finite element method for computing deformations. Finite
element modeling was also employed in subsequent works to simulate forearm
skin wrinkling [35], and skin aging [36]. Wang et al. [37] and Venkataraman et
al. [38] proposed energy based approaches. Here, wrinkle deformations are
produced by minimizing an energy function indicating flexure properties of a
governing curve on a surface. To produce wrinkles on dynamic meshes such as
simulated cloth, Müller and Chentanez [39] proposed attaching a higher
resolution wrinkle mesh to the coarse base mesh and determining the deviations
of the wrinkle mesh vertices using a static solver [40]. Geometric simulation
based approaches typically involve expressing the wrinkles using some
geometric primitives. Bando et al. [41] represented wrinkles using a cubic
Bezier curve, generating their furrows from a sequence of starting points
along a user specified direction field. Other proposed techniques involved the
use of length preserving constraints on planar curves along with artist placed
features at locations on an animated mesh where wrinkling is desired [42, 43].
Ilie et al. [44] employed a Hermite spline interpolation along with a modified
Rayleigh distribution function to simulate wrinkling activity in facial
animations. Subsequent methods extracted wrinkle curves automatically from
images [45, 46]. Finally, Gui et al. [47] used both a muscle model and a
geometric wrinkle shape function to simulate 3D facial wrinkles.

#### Machine Learning Approaches

More recently, several methods for expression and texture synthesis, and
facial performance capture have addressed the synthesis of wrinkles. As part
of their performance capture system, Cao et al. [48] trained regressors for
mapping local image appearance to wrinkle displacements to augment a coarse
face mesh tracked in real-time. Zeng et al. [24] and Richardson et al. [22]
proposed convolutional networks based refinement architectures to reconstruct
detailed facial geometry from a single image. Nagano et al. [49] proposed a
conditional generative adversarial network architecture for the synthesis of
image-based dynamic 3D avatars. Given a single neutral-face input image, their
system can generate novel photo-real expressions from alternate viewpoints,
including variable details such as wrinkles. More directly, Deng et al. [50]
proposed a variational autoencoder architecture to synthesize plausible fine-
scale wrinkles on a variety of coarse-scale 3D faces.

##  4 Synthethizing Expression-Based Wrinkles

Figure 3:  Mesh Tension. We illustrate our computation of mesh tension for
various deformations of a simple cylinder. Expansion is depicted in green and
compression in red. Black shading corresponds to zero tension.

Figure 2 illustrates an overview of our approach. The underlying idea is that
wrinkles can by synthesized additively over the neutral-expression textures.
We formalize the concept of mesh tension and use it to automatically aggregate
wrinkling effects in a data-driven manner across all expression scans of an
identity. We store these possible wrinkles corresponding to the expansion and
compression deformations of the face in separate albedo and displacement
textures, which we collectively refer to as wrinkle maps in this work. Note
that displacement maps modify the underlying geometry unlike bump or normal
maps that simply simulate the geometry changes. At synthesis, we sample a face
mesh from a generative face model [1] and randomly select a set of neutral and
wrinkle textures corresponding to an identity from the available scans. We
then compute the tension in the face mesh to drive the blending between the
neutral and wrinkle maps to obtain dynamic wrinkling effects. In contrast with
previous learning-based wrinkling methods [23, 22, 24, 50], we do not build a
generative model for the textures since such models struggle to reconstruct
high frequency details such as wrinkles compared to directly extracting them
from scans.

###  4.1 Mesh Tension

We formalize mesh tension to capture the amount of compression or expansion at
each vertex of a 3D polygon mesh resulting from a deformation. More
concretely, we express mesh tension as a function of the mean change in the
length of the edges connected to a vertex as a result of the deformation.
Consider an undeformed mesh $\macc@depth\char
1\relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a
111{}=(\macc@depth\char
1\relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a
111{V},\macc@depth\char
1\relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a 111{E})$
with a sequence of vertices $\macc@depth\char
1\relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a 111{V}$
and sequence of edges $\macc@depth\char
1\relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a 111{E}$,
that undergoes a deformation to result in the mesh $\mathbf{X}=(V,E)$. We only
consider deformations such that $\macc@depth\char
1\relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a 111{}$
and $\mathbf{X}$ possess the same topology. For vertex $v_{i}\in V$, let
$(e_{1},\ldots,e_{K})$ denote the sequence of $K$ edges connected to $v_{i}$,
with $(\macc@depth\char
1\relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a
111{e}_{1},\ldots,\macc@depth\char
1\relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a
111{e}_{K})$ denoting the corresponding edges in $\macc@depth\char
1\relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a 111{X}$
connected to $\macc@depth\char
1\relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a
111{v}_{i}$. We then define the mesh tension at $v_{i}$ as

$t_{v_{i}}\coloneqq
1-\frac{1}{K}\sum_{k\in[K]}\frac{\|e_{k}\|}{\|\macc@depth\char
1\relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a
111{e}_{k}\|},$ (1)

where $[K]=\\{1,\ldots,K\\}$, and $\|.\|$ denotes edge length. Note that we
subtract from $1$ so that positive values of $t_{v_{i}}$ indicate compression,
negative values indicate expansion, and a value of $0$ indicates no change.

|  |
---|---|---
Figure 4:  Data—High-resolution 3D Scans. For each identity, we illustrate:
the raw neutral scan (top-left), the manually-cleaned neutral scan to remove
sensor noise and hair (top-right), and two raw expression scans (bottom).

In practice, for finer manual control we introduce the parameters of strength
$s$ to scale the tension, and bias $b$ to artificially favor expansion or
compression, computing the weighted tension at $v_{i}$ as
$t^{\prime}_{v_{i}}=s\cdot t_{v_{i}}+b$. Further, we allow for artificial
propagation of expansion and compression effects through the mesh. For each
effect we introduce a parameter denoting the number of iterations for a
morphological dilation (positive values) or erosion (negative values)
operation. The propagation of each effect is first performed independently
over the mesh, and the resulting tension values are added for vertices that
end up with both expansion and compression. Figure 3 illustrates these effects
for a simple cylindrical mesh. See Appendix A for additional illustrations of
the effect of the tension parameters. Code as a Blender [51] add-on is
available at https://github.com/chiragraman/mesh-tension

###  4.2 Data and Preprocessing

Raw |  Coarse Mask |  Fine Mask |  Cleaned |  Base
---|---|---|---|---
|  |  |  |
|  |  |  |
Figure 5:  Cleaning Raw Textures. We illustrate the cleanup of albedo (top)
and displacement (bottom) textures on the surprise expression. We
automatically remove the hair and sensor noise artifacts in the raw textures
around the head, neck, and cheeks while preserving the desired wrinkles in the
nose, forehead, and mouth regions (compared to the base mesh, with neutral
albedo and without displacement respectively, for the same expression).

We start with a set of high-quality commercially available 3D scans of $208$
individuals. All $208$ identities contain scans with neutral expressions,
while $52$ contain additional scans for posed expressions. The neutral scans
were manually cleaned for removing noise and hair artifacts, and registered to
the topology of the 3D face model proposed by Wood et al. [1], resulting in a
mesh of $7,667$ vertices and $7,414$ polygons. Figure 4 illustrates the scans.

#### Automatic Cleaning of Expression Scans

The manual cleaning of scans is a labor-intensive process. To automate the
process of masking the noise and hair artifacts from the expression scans, we
utilize the difference between the raw and manually-cleaned neutral scans.
Concretely, we employ a two-stage masking procedure illustrated in Figure 5.
First, we apply an identity-agnostic coarse mask to filter most artifacts
outside of the hockey-mask and neck regions where expression-based wrinkling
occurs. Next, to capture the manual changes made by the artists in the
cleaning of each neutral scan, we employ a Gaussian Mixture Model-based
background subtraction technique [52]. Treating the clean neutral textures as
background and the raw original ones as foreground, we obtain an identity-
specific mask of the noise and hair artifacts for every identity. We apply
this fine mask to clean the textures from the corresponding expression scans
for each identity.

###  4.3 Data-Driven Wrinkle Maps

Figure 6:  Generating Wrinkle Maps from Scans. We illustrate the computation
of albedo wrinkle maps with three raw expression scans (top). We compute the
tension maps corresponding to the scans (middle), depicting expansion in green
and compression in red. Finally, the expression albedo textures (bottom) are
linearly combined using the normalized tension as weights to obtain the
expanded and compressed albedo wrinkle maps. A similar procedure is applied to
obtain the displacement wrinkle maps.

#### Tension-Weighted Wrinkle Maps

Figure 6 illustrates our method for generating wrinkle maps from the face
scans. Our underlying idea is to use the tension at each vertex as weights in
a linear combination of the cleaned textures across expressions, with zero
tension corresponding to the neutral textures. (Figure 6 depicts raw textures
for easier visual correspondence with the scans.) We begin by fitting the
generative face model from Wood et al. [1] to the raw scans and compute the
tension maps from the resulting meshes. The individual expansion and
compression maps are then normalized using the $\mathrm{softmax}$ function.
Finally, we linearly combine expression textures using the normalized tension
as weights to obtain the expanded and compressed wrinkle maps. The same
procedure is applied to obtain both the albedo and displacement wrinkle maps.

#### Identities With Missing Expression Scans

How do we compute wrinkle maps for the identities without posed expression
scans? We employ a simple wrinkle-grafting procedure. For a target identity
without wrinkle maps, we find the source identity with wrinkle maps that has
the most similar neutral albedo map, measured by mean squared error in pixel
color. For the source identity, we compute the wrinkling effects as the
difference between the neutral and wrinkle maps (for both albedo and
displacement). We then add this difference to the neutral textures for the
target identity to obtain the target wrinkle maps. We illustrate the grafting
procedure for the compressed albedo maps in Figure 7, and final example
renders with grafted wrinkles in Figure 8.

Figure 7:  Grafting Wrinkles. For an identity with missing expression scans
(target), we find the identity from among those with expression scans that has
the most similar neutral albedo map (source). We then graft the wrinkles from
the source’s wrinkle map onto the target’s neutral texture to obtain the
target wrinkle maps (here illustrating the compressed albedo).  Figure 8:
Final Renders for Some Identities with Grafted Wrinkles. We computed the
wrinkle maps for these identities by grafting wrinkles from identities with
expression scans (see Figure 7). We illustrate two expressions for each
identity, without (left) and with wrinkles (right).

##  5 Experiments and Results

Table 1:  Landmark Localization on 300W. We normalize mean error using interocular distance. Lower is better.  Method | Common | Challenging | Private
---|---|---|---
| NME | NME | FR10%
Trained on Real Data |  |  |
LAB [53] | $2.98$ | $5.19$ | $0.83$
AWING [54] | $\mathbf{2.72}$ | $\mathbf{4.52}$ | $\underline{0.33}$
ODN [55] | $3.56$ | $6.67$ | -
3FabRec [56] | $3.36$ | $5.74$ | $\mathbf{0.17}$
LUVLi [57] | $\underline{2.76}$ | $5.16$ | -
Trained on Synthetic Data |  |  |
No wrinkles [1] | $3.11$ | $4.84$ | $\underline{0.33}$
Ours (wrinkles) | $3.10$ | $\underline{4.83}$ | $\mathbf{0.17}$

We evaluate our proposed mesh-tension driven wrinkles both quantitatively and
qualitatively on two face analysis tasks: landmark detection (Section 5.1) and
normal estimation (Section 5.2). We compare adding mesh-tension to the
existing SOTA method for full-face synthetics, and compare the performance of
models trained on the resulting data against SOTA approaches in the field for
these tasks.

###  5.1 Landmark Localization

Experimental Details. We use direct regression based facial landmark detection
[58] with an off-the-shelf ResNet 101 [59]. We use a $256\times 256$ px RGB
image as input to predict $703$ dense facial landmarks. We additionally employ
label translation [1] to deal with systematic inconsistencies between our
$703$ predicted dense landmarks and the $68$ sparse landmarks labeled as
ground truth in our evaluation datasets (this is done only for Table 1).

As a training dataset we rendered $100k$ synthetic images, consisting of $20k$
identities with $5$ frames for each identity (different view-points,
expressions, and environments). We also generated ground-truth annotations of
$703$ dense 2D landmarks from the face-meshes to accompany each image. We
train our models for $300$ epochs using PyTorch Lightning, starting with a
learning rate of $1\mathrm{e}{-3}$ and halved every 100 epochs.

Evaluation Datasets and Metrics. We use the 300W dataset [60] (with common,
challenging and private subsets), and employ the standard normalized mean
error (NME) and failure rate (FR10%) error metrics [60].

While the 300W dataset provides evaluation of overall landmark detection
performance, it is not sensitive enough to detect improvements in specific
parts of the face or during particular expressions. We identify a small subset
of $30$ images from 300W that contain winks and compressed face expressions
(300W-winks) to provide a more nuanced indication of performance under such
deformations. We report errors for eyelid-landmarks by taking a point-to-line
distance from every predicted eyelid landmark to the corresponding polyline
defining an eyelid in ground truth. This metric allows us to better understand
eye region error and to use different landmark definitions in training and
evaluating models (e.g. from our 703 landmark model or from 98 landmark models
[54]). See Appendix E for the list of images in 300W-winks.

Table 2:  Landmark Localization - Eyes. We report eye-opening errors for Pexels, and eyelid point-to-polyline errors for 300W and the winks subset. In all cases normalized by bounding-box diagonal. Lower is better.  Method | Pexels | 300W | 300W-winks
---|---|---|---
Trained on Real Data |  |  |
AWING [54] | 1.06 | 0.62 | $\mathbf{0.69}$
3FabRec [56] | 3.60 | 0.81 | 1.32
Trained on Synthetic Data |  |  |
No wrinkles [1] | $0.97$ | $0.51$ | $0.86$
Ours (wrinkles) | $\mathbf{0.86}$ | $\mathbf{0.48}$ | $0.74$

We also introduce a Pexels dataset which contains 318 images of fully closed
eyes (because of blinking, scrunching or compressing the face) and 105 images
with only a single eye closed (winking). This allows us to asses model
performance under such conditions which are rare in other datasets. To collect
the data we used a stock photography website 111https://www.pexels.com/ using
search terms wink/blink/compress/scrunched and similar image searches. We
select only semi-frontal images with no or limited occlusion of the eyes to
best evaluate performance in that region. The URLs of the images selected can
be found in Appendix F. Knowing which images contain fully closed eyes or just
a single eye closed allows us to measure eyelid accuracy without explicit
landmark annotations. We define the eye opening error as the mean eye aperture
of both eyes in the eye-closed case and eye aperture of closed eye in the wink
case. See Appendix B for illustrations of the above two metrics.

Baselines. We compare against recent SOTA methods trained on images of real
faces. For subsequent nuanced analysis on 300W-winks and Pexels we consider
the methods of Wang et al. [54] and Browatzki and Wallraven [56] since they
collectively yield the best performance on 300W.

Results. From Table 1 we see that our proposed mesh-tension driven wrinkles
provide a marginal improvement for landmark localization. However, when we
look at specific eye region results on 300W, 300W-winks and Pexels in Table 2,
we see that improvement is much larger for the eye region and our synthetic-
only trained approaches outperform real-data based models. Also see Figure 9
and Appendix C.

Ablation. We further analyze the importance of the albedo and displacement
wrinkling components for landmark detection. From Figure 10 and Table 3 we see
that displacement plays a more important role than albedo in improving
performance, but best results are achieved through a combination of both.

Trained on Real Data | Trained on Synthetic Data
---|---
AWING [54] |  3FabRec [56] |  No Wrinkles [1] |  Ours (Wrinkles)

Figure 9:  Qualitative results for landmark localization on Pexels. Training on synthetic faces with our expression-based wrinkles is crucial for localizing keypoints in compressed regions of the face.  Table 3:  Landmark Localization Ablation. We report eye-opening errors for Pexels, and eyelid point-to-polyline errors for 300W and the winks subset. Lower is better.  Dataset | Base | Disp. Only | Albedo Only | Full
---|---|---|---|---
300W | $0.51$ | $0.51$ | $0.50$ | $\mathbf{0.48}$
300W-winks | $0.86$ | $0.76$ | $0.80$ | $\mathbf{0.74}$
Pexels | $0.97$ | $\mathbf{0.86}$ | $0.89$ | $\mathbf{0.86}$
Base |  Disp. Only |  Albedo Only |  Albedo & Disp.
---|---|---|---
|  |  |
Figure 10:  Expression-Based Wrinkle Components. We add wrinkles through two
components: displacement and albedo. Here we show each in isolation.
Displacement is critical for achieving realistic lighting of wrinkles.
Especially note the forehead (zoomed) and neck regions.

###  5.2 Surface-Normals Prediction

Surface normals can be used to infer 3D information about a surface from 2D
images, and have been used in several human-centered vision tasks such as
clothing [61] and face-shape [62] reconstruction and relighting [63].

We train a U-Net [64] with a ResNet 18 [59] encoder to predict camera-space
surface normals of the face. As input we use $256\times 256$ px RGB images
from a dataset of $50k$ synthetics images. The network is trained for $200$
epochs using cosine similarity loss with a learning rate of $1\mathrm{e}{-3}$.
Camera-space surface normal images rendered as part of our synthetic data
pipeline are used as ground-truth.

Input Image |  No Wrinkles [1] |  Ours (Wrinkles) |  [1] Top Inset / Ours Bottom
---|---|---|---

Figure 11:  Qualitative Surface-Normals Predictions on Pexels. The model
trained on synthetic faces with wrinkles recovers significantly more high-
frequency details.

Results on real images are shown in Figure 11; the network trained on images
synthesized with our method recovers more high-frequency detail on the face.
As shown in Figure 12, we achieve comparable results to other recent methods
for face surface-normals prediction [62, 63]. Further comparisons are provided
in Appendix D.

Input Image |  CMDFN [62] |  SfSnet [63] |  Ours
---|---|---|---

Figure 12:  Qualitative Comparison against SOTA. Our synthetic data-only U-Net
yields predictions comparable to SOTA while being less noisy and more robust
to lighting.

##  6 Conclusion

We have presented a method for introducing dynamic expression-based wrinkles
to synthetic faces that yields improved performance on the downstream tasks of
landmark localization and surface-normals estimation, especially for regions
of the face most deformed by expressions.

Our use of tension in the face mesh is key in the automatic scaling of our
method with identities and expressions, which has been a bottleneck for past
wrinkling approaches that rely on prohibitive artist effort. In addition, our
data-driven approach also enables the capturing of real wrinkles from scans
which doesn’t require artistic judgment.

By boosting the realism of synthesized faces with dynamic wrinkles, we have
made an explicit case for synthetic data: our method yields improved
performance for models on downstream tasks. In addition, synthesizing data
with diverse faces across races and genders involves significantly less effort
than collecting representative datasets in the wild. Consequently, downstream
real-life systems developed using such synthetic data are less likely to
suffer from unfair biases along these sensitive variables.

## Acknowledgments

Chirag would like to thank: Tom Cashman, Stephan Garbin, and Panagiotis
Giannakopoulos for the insightful discussions; Sebastian Dziadzio for help
with fitting the face model; Sarah Roberts for being an infallible remover of
obstacles; and Steve Miller (BA: @shteeve) for an initial implementation of
the Blender mesh tension add on.

## References

  * Wood et al. [2021] Erroll Wood, Tadas Baltrušaitis, Charlie Hewitt, Sebastian Dziadzio, Matthew Johnson, Virginia Estellers, Thomas J. Cashman, and Jamie Shotton.  Fake It Till You Make It: Face analysis in the wild using synthetic data alone.  _arXiv:2109.15102 [cs]_ , October 2021.
  * Yao et al. [2020] Yue Yao, Liang Zheng, Xiaodong Yang, Milind Naphade, and Tom Gedeon.  Simulating Content Consistent Vehicle Datasets with Attribute Descent.  _arXiv:1912.08855 [cs]_ , July 2020.
  * Hodan et al. [2019] Tomas Hodan, Vibhav Vineet, Ran Gal, Emanuel Shalev, Jon Hanzelka, Treb Connell, Pedro Urbina, Sudipta N. Sinha, and Brian Guenter.  Photorealistic Image Synthesis for Object Instance Detection.  In _arXiv:1902.03334 [Cs]_ , pages 66–70, February 2019.
  * Qiu et al. [2017] Weichao Qiu, Fangwei Zhong, Yi Zhang, Siyuan Qiao, Zihao Xiao, Tae Soo Kim, and Yizhou Wang.  UnrealCV: Virtual Worlds for Computer Vision.  In _Proceedings of the 25th ACM International Conference on Multimedia_ , MM ’17, pages 1221–1224, New York, NY, USA, October 2017\. Association for Computing Machinery.  ISBN 978-1-4503-4906-2.  doi: 10.1145/3123266.3129396.
  * Rozantsev et al. [2015] Artem Rozantsev, Vincent Lepetit, and Pascal Fua.  On Rendering Synthetic Images for Training an Object Detector.  _Computer Vision and Image Understanding_ , 137:24–37, August 2015.  ISSN 10773142.  doi: 10.1016/j.cviu.2014.12.006.
  * Kar et al. [2019] Amlan Kar, Aayush Prakash, Ming-Yu Liu, Eric Cameracci, Justin Yuan, Matt Rusiniak, David Acuna, Antonio Torralba, and Sanja Fidler.  Meta-Sim: Learning to Generate Synthetic Datasets.  _arXiv:1904.11621 [cs]_ , April 2019.
  * Gaidon et al. [2016] Adrien Gaidon, Qiao Wang, Yohann Cabon, and Eleonora Vig.  Virtual Worlds as Proxy for Multi-Object Tracking Analysis.  _arXiv:1605.06457 [cs, stat]_ , May 2016.
  * Richter et al. [2016] Stephan R. Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun.  Playing for Data: Ground Truth from Computer Games.  _arXiv:1608.02192 [cs]_ , August 2016.
  * Ros et al. [2016] German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M. Lopez.  The SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 3234–3243, Las Vegas, NV, USA, June 2016. IEEE.  ISBN 978-1-4673-8851-1.  doi: 10.1109/CVPR.2016.352.
  * Wood et al. [2015] Erroll Wood, Tadas Baltrusaitis, Xucong Zhang, Yusuke Sugano, Peter Robinson, and Andreas Bulling.  Rendering of Eyes for Eye-Shape Registration and Gaze Estimation.  _arXiv:1505.05916 [cs]_ , May 2015.
  * Świrski and Dodgson [2014] Lech Świrski and Neil Dodgson.  Rendering synthetic ground truth images for eye tracker evaluation.  In _Proceedings of the Symposium on Eye Tracking Research and Applications_ , pages 219–222, Safety Harbor Florida, March 2014. ACM.  ISBN 978-1-4503-2751-0.  doi: 10.1145/2578153.2578188.
  * Mueller et al. [2017] Franziska Mueller, Florian Bernard, Oleksandr Sotnychenko, Dushyant Mehta, Srinath Sridhar, Dan Casas, and Christian Theobalt.  GANerated Hands for Real-time 3D Hand Tracking from Monocular RGB.  _arXiv:1712.01057 [cs]_ , December 2017.
  * Simon et al. [2017] Tomas Simon, Hanbyul Joo, Iain Matthews, and Yaser Sheikh.  Hand Keypoint Detection in Single Images using Multiview Bootstrapping.  _arXiv:1704.07809 [cs]_ , April 2017.
  * Varol et al. [2017] Gül Varol, Javier Romero, Xavier Martin, Naureen Mahmood, Michael J. Black, Ivan Laptev, and Cordelia Schmid.  Learning from Synthetic Humans.  _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 4627–4635, July 2017.  doi: 10.1109/CVPR.2017.492.
  * Shotton et al. [2011] Jamie Shotton, Andrew Fitzgibbon, Mat Cook, Toby Sharp, Mark Finocchio, Richard Moore, Alex Kipman, and Andrew Blake.  Real-time human pose recognition in parts from single depth images.  In _CVPR 2011_ , pages 1297–1304, June 2011.  doi: 10.1109/CVPR.2011.5995316.
  * Huazhong Ning et al. [2008] Huazhong Ning, Wei Xu, Yihong Gong, and Thomas Huang.  Discriminative learning of visual words for 3D human pose estimation.  In _2008 IEEE Conference on Computer Vision and Pattern Recognition_ , pages 1–8, Anchorage, AK, USA, June 2008. IEEE.  ISBN 978-1-4244-2242-5.  doi: 10.1109/CVPR.2008.4587534.
  * Hendler et al. [2018] Darren Hendler, Lucio Moser, Rishabh Battulwar, David Corral, Phil Cramer, Ron Miller, Rickey Cloudsdale, and Doug Roble.  Avengers: Capturing thanos’s complex face.  In _ACM SIGGRAPH 2018 Talks_ , pages 1–2, Vancouver British Columbia Canada, August 2018. ACM.  ISBN 978-1-4503-5820-0.  doi: 10.1145/3214745.3214766.
  * Karis, Brian et al. [2016] Karis, Brian, Antoniades, Tameem, Caulkin, Steve, and Mastilovic, Vladimir.  Digital Humans: Crossing the Uncanny Valley in UE4.  In _Game Developers Conference_ , 2016.
  * Wood et al. [2016] Erroll Wood, Tadas Baltrušaitis, Louis-Philippe Morency, Peter Robinson, and Andreas Bulling.  Learning an appearance-based gaze estimator from one million synthesised images.  In _Proceedings of the Ninth Biennial ACM Symposium on Eye Tracking Research & Applications_, pages 131–138, Charleston South Carolina, March 2016. ACM.  ISBN 978-1-4503-4125-7.  doi: 10.1145/2857491.2857492.
  * Sugano et al. [2014] Yusuke Sugano, Yasuyuki Matsushita, and Yoichi Sato.  Learning-by-Synthesis for Appearance-Based 3D Gaze Estimation.  In _2014 IEEE Conference on Computer Vision and Pattern Recognition_ , pages 1821–1828, Columbus, OH, USA, June 2014. IEEE.  ISBN 978-1-4799-5118-5.  doi: 10.1109/CVPR.2014.235.
  * Sela et al. [2017] Matan Sela, Elad Richardson, and Ron Kimmel.  Unrestricted Facial Geometry Reconstruction Using Image-to-Image Translation.  _arXiv:1703.10131 [cs]_ , March 2017.
  * Richardson et al. [2017] Elad Richardson, Matan Sela, Roy Or-El, and Ron Kimmel.  Learning Detailed Face Reconstruction from a Single Image.  _arXiv:1611.05053 [cs]_ , April 2017.
  * Richardson et al. [2016] Elad Richardson, Matan Sela, and Ron Kimmel.  3D Face Reconstruction by Learning from Synthetic Data.  _arXiv:1609.04387 [cs]_ , September 2016.
  * Zeng et al. [2019] Xiaoxing Zeng, Xiaojiang Peng, and Yu Qiao.  DF2Net: A Dense-Fine-Finer Network for Detailed 3D Face Reconstruction.  In _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 2315–2324, Seoul, Korea (South), October 2019. IEEE.  ISBN 978-1-72814-803-8.  doi: 10.1109/ICCV.2019.00240.
  * Shrivastava et al. [2017] Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Josh Susskind, Wenda Wang, and Russ Webb.  Learning from Simulated and Unsupervised Images through Adversarial Training.  _arXiv:1612.07828 [cs]_ , July 2017.
  * Ganin et al. [2016] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky.  Domain-Adversarial Training of Neural Networks.  _arXiv:1505.07818 [cs, stat]_ , May 2016.
  * Oat [2007a] Christopher Oat.  Animated wrinkle maps.  In _ACM SIGGRAPH 2007 Courses_ , SIGGRAPH ’07, pages 33–37, New York, NY, USA, 2007a. Association for Computing Machinery.  ISBN 978-1-4503-1823-5.  doi: 10.1145/1281500.1281667.
  * Oat [2007b] Christopher Oat.  Real-Time Wrinkles, 2007b.
  * Jimenez et al. [2011] Jorge Jimenez, Jose I. Echevarria, Christopher Oat, and Diego Gutierrez.  Practical and Realistic Facial Wrinkles Animation.  In Wolfgang Engel, editor, _GPU pro 2_ , chapter Practical and Realistic Facial Wrinkles Animation. AK Peters Ltd., 2011.
  * Dutreve et al. [2009] Ludovic Dutreve, Alexandre Meyer, and Saida Bouakaz.  Real-Time Dynamic Wrinkles of Face for Animated Skinned Mesh.  In _ISVC’ 09: 5th International Symposium on Visual Computing_ , Advances in Visual Computing, pages 25–34, Las Vegas, USA, United States, November 2009. Springer.  doi: 10.1007/978-3-642-10520-3˙3.
  * Duque Reis et al. [2008] Clausius Duque Reis, G Reis, José De Martino, and Harlen Batagelo.  _Real-Time Simulation of Wrinkles_.  February 2008.
  * Team [2007] Microsoft DirectX 10 SDK Team.  Sparse Morph Targets Sample, 2007.
  * Wu et al. [1997] Yin Wu, Prem Kalra, and Nadia Magnenat Thalmann.  Physically-based Wrinkle Simulation & Skin Rendering.  In W. Hansmann, W. T. Hewitt, W. Purgathofer, Daniel Thalmann, and Michiel van de Panne, editors, _Computer Animation and Simulation ’97_ , pages 69–79. Springer Vienna, Vienna, 1997.  ISBN 978-3-211-83048-2 978-3-7091-6874-5.  doi: 10.1007/978-3-7091-6874-5˙5.
  * Boissieux et al. [2000] Laurence Boissieux, Gergo Kiss, Nadia Magnenat Thalmann, and Prem Kalra.  Simulation of Skin Aging and Wrinkles with Cosmetics Insight.  In W. Hansmann, W. Purgathofer, F. Sillion, Nadia Magnenat-Thalmann, Daniel Thalmann, and Bruno Arnaldi, editors, _Computer Animation and Simulation 2000_ , pages 15–27. Springer Vienna, Vienna, 2000.  ISBN 978-3-211-83549-4 978-3-7091-6344-3.  doi: 10.1007/978-3-7091-6344-3˙2.
  * Flynn and McCormack [2008] Cormac Flynn and B. A. O. McCormack.  Finite element modelling of forearm skin wrinkling.  _Skin Research and Technology_ , 14, 2008.
  * Flynn and McCormack [2010] Cormac Flynn and Brendan A. O. McCormack.  Simulating the wrinkling and aging of skin with a multi-layer finite element model.  _Journal of Biomechanics_ , 43(3):442–448, February 2010.  ISSN 0021-9290.  doi: 10.1016/j.jbiomech.2009.10.007.
  * Wang et al. [2006] Yu Wang, Charlie CL Wang, and Matthew MF Yuen.  Fast energy-based surface wrinkle modeling.  _Computers & Graphics_, 30(1):111–125, 2006\.
  * Venkataraman et al. [2005] Kartik Venkataraman, Suresh Lodha, and Raghu Raghavan.  A kinematic-variational model for animating skin with wrinkles.  _Computers & Graphics_, 29(5):756–770, October 2005.  ISSN 0097-8493.  doi: 10.1016/j.cag.2005.08.024.
  * Müller and Chentanez [2010] Matthias Müller and Nuttapong Chentanez.  Wrinkle meshes.  In _Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation_ , pages 85–92, 2010.
  * Müller et al. [2007] Matthias Müller, Bruno Heidelberger, Marcus Hennix, and John Ratcliff.  Position based dynamics.  _J. Vis. Comun. Image Represent._ , 18(2):109–118, April 2007.  ISSN 1047-3203.  doi: 10.1016/j.jvcir.2007.01.005.
  * Bando et al. [2002] Y. Bando, T. Kuratate, and T. Nishita.  A simple method for modeling wrinkles on human skin.  In _10th Pacific Conference on Computer Graphics and Applications, 2002. Proceedings._ , pages 166–175, Beijing, China, 2002\. IEEE Comput. Soc.  ISBN 978-0-7695-1784-1.  doi: 10.1109/PCCGA.2002.1167852.
  * Larboulette and Cani [2004] Caroline Larboulette and Marie-paule Cani.  Real-Time Dynamic Wrinkles, 2004.
  * Li et al. [2007] Ming Li, BaoCai Yin, DeHui Kong, and XiaoNan Luo.  Modeling Expressive Wrinkles of Face For Animation.  In _Fourth International Conference on Image and Graphics (ICIG 2007)_ , pages 874–879, August 2007.  doi: 10.1109/ICIG.2007.22.
  * Ilie et al. [2012] Mihai Daniel Ilie, Cristian Negrescu, and Dumitru Stanomir.  A robust mathematical model for simulating wrinkle activity in 3D facial animations.  In _2012 10th International Symposium on Electronics and Telecommunications_ , pages 271–274, November 2012.  doi: 10.1109/ISETC.2012.6408082.
  * Li et al. [2011] Li Li, Fei Liu, Congbo Li, and Guoan Chen.  Realistic wrinkle generation for 3D face modeling based on automatically extracted curves and improved shape control functions.  _Computers & Graphics_, 35(1):175–184, February 2011.  ISSN 00978493.  doi: 10.1016/j.cag.2010.08.003.
  * Vanderfeesten and Bikker [2018] Ron Vanderfeesten and Jacco Bikker.  Example-Based Skin Wrinkle Displacement Maps.  In _2018 31st SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI)_ , pages 212–219, Parana, October 2018. IEEE.  ISBN 978-1-5386-9264-6.  doi: 10.1109/SIBGRAPI.2018.00034.
  * Gui et al. [2016] Jiamin Gui, Yue Zhang, and Shaobin Li.  Realistic 3D Facial Wrinkles Simulation Based on Tessellation.  In _2016 9th International Symposium on Computational Intelligence and Design (ISCID)_ , volume 1, pages 250–254, December 2016.  doi: 10.1109/ISCID.2016.1064.
  * Cao et al. [2015] Chen Cao, Derek Bradley, Kun Zhou, and Thabo Beeler.  Real-time high-fidelity facial performance capture.  _ACM Transactions on Graphics_ , 34(4):1–9, July 2015.  ISSN 0730-0301, 1557-7368.  doi: 10.1145/2766943.
  * Nagano et al. [2019] Koki Nagano, Jaewoo Seo, Jun Xing, Lingyu Wei, Zimo Li, Shunsuke Saito, Aviral Agarwal, Jens Fursund, and Hao Li.  paGAN: Real-time avatars using dynamic textures.  _ACM Transactions on Graphics_ , 37(6):1–12, January 2019.  ISSN 0730-0301, 1557-7368.  doi: 10.1145/3272127.3275075.
  * Deng et al. [2021] Qixin Deng, Luming Ma, Aobo Jin, Huikun Bi, Binh Huy Le, and Zhigang Deng.  Plausible 3D Face Wrinkle Generation Using Variational Autoencoders.  _IEEE Transactions on Visualization and Computer Graphics_ , pages 1–1, 2021.  ISSN 1077-2626, 1941-0506, 2160-9306.  doi: 10.1109/TVCG.2021.3051251.
  * [51] Blender.  https://www.blender.org/.  Accessed: 2021-09-30.
  * Zivkovic and van der Heijden [2006] Zoran Zivkovic and Ferdinand van der Heijden.  Efficient adaptive density estimation per image pixel for the task of background subtraction.  _Pattern Recognition Letters_ , 27(7):773–780, 2006.  ISSN 0167-8655.  doi: 10.1016/j.patrec.2005.11.005.
  * Wu et al. [2018] Wayne Wu, Chen Qian, Shuo Yang, Quan Wang, Yici Cai, and Qiang Zhou.  Look at boundary: A boundary-aware face alignment algorithm.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2129–2138, 2018.
  * Wang et al. [2019] Xinyao Wang, Liefeng Bo, and Li Fuxin.  Adaptive wing loss for robust face alignment via heatmap regression.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 6971–6981, 2019.
  * Zhu et al. [2019] Meilu Zhu, Daming Shi, Mingjie Zheng, and Muhammad Sadiq.  Robust facial landmark detection via occlusion-adaptive deep networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3486–3496, 2019.
  * Browatzki and Wallraven [2020] Bjoern Browatzki and Christian Wallraven.  3fabrec: Fast few-shot face alignment by reconstruction.  In _CVPR_ , 2020.
  * Kumar et al. [2020] Abhinav Kumar, Tim K Marks, Wenxuan Mou, Ye Wang, Michael Jones, Anoop Cherian, Toshiaki Koike-Akino, Xiaoming Liu, and Chen Feng.  Luvli face alignment: Estimating landmarks’ location, uncertainty, and visibility likelihood.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8236–8246, 2020.
  * Wood et al. [2022] Erroll Wood, Tadas Baltrusaitis, Charlie Hewitt, Matthew Johnson, Jingjing Shen, Nikola Milosavljevic, Daniel Wilde, Stephan Garbin, Toby Sharp, Ivan Stojiljkovic, Tom Cashman, and Julien Valentin.  3d face reconstruction with dense landmarks, 2022.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _CVPR_ , 2016.
  * Sagonas et al. [2016] C Sagonas, E. Antonakos, G Tzimiropoulos, S Zafeiriou, and M Pantic.  300 faces In-the-wild challenge: Database and results.  _Image and Vision Computing (IMAVIS)_ , 2016.
  * Alldieck et al. [2019] Thiemo Alldieck, Gerard Pons-Moll, Christian Theobalt, and Marcus Magnor.  Tex2shape: Detailed full human body geometry from a single image.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 2293–2303, 2019.
  * Abrevaya et al. [2020] Victoria Fernández Abrevaya, Adnane Boukhayma, Philip HS Torr, and Edmond Boyer.  Cross-modal deep face normals with deactivable skip connections.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4979–4989, 2020.
  * Sengupta et al. [2018] Soumyadip Sengupta, Angjoo Kanazawa, Carlos D Castillo, and David W Jacobs.  Sfsnet: Learning shape, reflectance and illuminance of faces in the wild’.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 6296–6305, 2018.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _International Conference on Medical image computing and computer-assisted intervention_ , pages 234–241. Springer, 2015.

Mesh-Tension Driven Expression-Based Wrinkles for Synthetic Faces
Appendices
Chirag Raman1 Charlie Hewitt2 Erroll Wood2 Tadas Baltrušaitis2

1Delft University of Technology, 2Microsoft

##  Appendix A Illustrating Tension Parameters

|  |  |  |  |  |  |
---|---|---|---|---|---|---|---
s: 3 b: 0 |  s: 5 b: 0 |  s: 10 b: 0 |  s: 10 b: -0.6 |  s: 10 b: 0.6 |  s: 10 b: 0 |  s: 10 b: 0 |  s: 10 b: 0
e: 0 c: 0 |  e: 0 c: 0 |  e: 0 c: 0 |  e: 0 c: 0 |  e: 0 c: 0 |  e: 3 c: -3 |  e: -3 c: 3 |  e: -3 c: -3
Figure 13:  Tension Parameters - Cylinder. Illustrating the effect of varying
tension parameters on a simple cylinder mesh. Legend: s \- strength, b \-
bias, e \- iterations for dilating/eroding expansion, c \- iterations for
dilating/eroding compression.

Strength
---
Low |  Default |  High
|  |
|  |

Iterations
---
Low |  Default |  High
|  |
|  |

Figure 14:  Tension Parameters - Face. Illustrating the effect of varying
tension parameters on a face mesh.

##  Appendix B Eye-Region Landmark Metrics

To deal with different landmark annotation conventions (e.g. $68$, $98$, $703$
landmarks), we use a point to polyline distance. For each eyelid point in the
prediction, we measure its distance to the relevant polyline, e.g. for a
predicted point on upper-left eyelid we measure the distance from it to upper-
left eyelid polyline (illustrated in Figure 16). This allows us to compare
models with different annotation schemes and to have a better understanding of
eye region error.

In cases where we do not have landmark annotations, but we know that both eyes
are closed (or a single eye is closed), we can use the eye opening/aperture
error instead. This is illustrated in Figure 16. The limitation of this
approach is that it measure the relative openness of eye only and will have a
low error even if the location of eyelid is wrong (but the aperture is
correct). However, in combination with other metrics it provides a good signal
to how well the models deal in detecting winks and blinks.

Figure 15:  Point-to-Line Distance Metric. Green: ground truth annotation
polyline. Blue: predicted eyelid landmarks. Red: residual distance. Top-left
corner: associated error.

Figure 16:  Eye Opening / Aperture Error. Green: prediction. Red: distance
between the top and bottom lid that measures eye opening. Top-left corner:
eye-opening of the left and right eye and the subsequent error.

##  Appendix C Landmark Predictions on 300W

We present examples of predictions on 300W dataset from models trained on real
and synthetic data in Figure 17.

Input Image |  AWING [54] |  3FabRec [56] |  No Wrinkles [1] |  Ours (Wrinkles)
---|---|---|---|---

Figure 17:  Qualitative Comparison against SOTA. Comparing prediction on 300W
against SOTA models for facial landmark detection, our synthetic-only model
often results in better accuracy for eye region, with improved performance for
wink detection with mesh-based tension data.

##  Appendix D Surface-Normals Predictions

In Figure 18 we show further comparisons to the recent face surface-normals
prediction techniques of Abrevaya et al. [62] and Sengupta et al. [63].

Input Image |  CMDFN [62] |  SfSnet [63] |  No Wrinkles [1] |  Ours (wrinkles)
---|---|---|---|---

Figure 18:  Surface-Normals Comparison. Qualitative results for surface-
normals compared with two recent approaches. Note that we use our own face-
alignment resulting in slight offset of the predicted ROI.

Figure 19 shows failure cases from Abrevaya et al. [62] and our results on the
same images. It is clear that our technique results in a significantly more
robust model which can deal better with extreme lighting conditions,
occlusions and darker skin tones. Note that when training our model we take
the surface of glasses lenses into account, though it is also possible to
ignore these and predict for the face underneath depending on how the
rendering pipeline is configured. In all figures relating to surface-normals
prediction we use our own face alignment to select the region of interest
(ROI) to input to the normals prediction U-Net model, which causes some
misalignment with the ROI used in other techniques.

Figure 19:  Surface-Normals Robustness. Comparison of surface-normals
prediction to failure cases from Abrevaya et al. [62], showing input (top),
Abrevaya et al. (middle) and Ours (bottom). Our approach is significantly more
robust in cases of extreme lighting, occlusion and darker skin.

##  Appendix E The 300W-winks Subset

The subset of 300W images that make up 300W-winks is:
indoor_048, indoor_052, indoor_053, indoor_054, indoor_055, indoor_089,
indoor_094, indoor_099, indoor_180, indoor_226, indoor_242, indoor_253,
indoor_264, indoor_267, indoor_278, indoor_280, indoor_282, indoor_286,
outdoor_073, outdoor_076, outdoor_077, outdoor_089, outdoor_097, outdoor_145,
outdoor_165, outdoor_209, outdoor_243, outdoor_249, outdoor_251, outdoor_292.

##  Appendix F The Pexels Winks and Blinks Dataset

All images can be accessed by appending to https://www.pexels.com/photo/

###  F.1 Blinks subset

abhishek-sinari-2026945, adrienne-andersen-2552127, adrienne-andersen-2552131,
alekke-blazhin-7448048, alekke-blazhin-8450287, alekke-blazhin-8450288,
alekke-blazhin-8450290, alekke-blazhin-8450296, alena-darmel-6940463, alena-
shekhovtcova-7036537, alesia-kozik-7295537, alex-green-6626087, alexander-
krivitskiy-4383786, alexander-stemplewski-2906663, alexandr-podvalny-1540408,
alexandra-patrusheva-6806789, ali-karimiboroujeni-11381826, alina-
blumberg-6925493, alyona-pastukhova-11495069, amar-preciado-10820282, amr-
osman-10665375, anastasia-ilinamakarova-10832112, anastasia-shuraeva-7539962,
anastasia-trofimczyk-10311002, anastasiia-chaikovska-11834502, anastasiia-
shevchenko-10568846, andre-porto-7753232, andrea-gulotta-11140482, andrea-
piacquadio-3757942, andrea-piacquadio-3760262, andrea-piacquadio-3760611,
andrea-piacquadio-3764535, andrea-piacquadio-3768163, andrea-
piacquadio-3768724, andrea-piacquadio-3771813, andrea-piacquadio-3786522,
andrea-piacquadio-3799096, andrea-piacquadio-3799787, andrea-
piacquadio-3799830, andrea-piacquadio-3807762, andrea-piacquadio-3811603,
andrea-piacquadio-3811663, andrea-piacquadio-3812746, andrea-
piacquadio-3831645, andrea-piacquadio-941693, andres-ayrton-6578880, anete-
lusina-4793357, anete-lusina-5723189, angela-roma-7479819, angelica-
reyn-11893387, anh-tuan-9889769, anna-shvets-3746281, anna-shvets-3852192,
anna-shvets-4557467, anna-shvets-4611655, anna-shvets-4971107, anna-
shvets-5034475, anna-shvets-5069470, anna-shvets-5069493, anna-shvets-5069609,
anna-tarazevich-5155727, anna-zaykina-8452431, antoni-shkraba-5890702, antoni-
shkraba-7484863, arianna-jade-2896823, arina-krasnikova-6663361, arina-
krasnikova-6663367, arina-krasnikova-6914826, arina-krasnikova-6914833, arina-
krasnikova-6998572, arina-krasnikova-7752573, arina-krasnikova-7752693, armin-
rimoldi-5269495, arsham-haghani-3423024, artyom-malyukov-11896104, azraq-al-
rezoan-11763863, azraq-al-rezoan-11763868, barathan-amuthan-2723624, ben-
mack-6775289, blue-bird-7210441, breno-santos-10060165, brett-sayles-4095246,
caique-araujo-10218049, camilla-gari-10306657, charles-wundengba-3609781,
cliff-booth-4057336, cottonbro-10049355, cottonbro-10140838,
cottonbro-10678800, cottonbro-4727484, cottonbro-5020308, cottonbro-5386370,
cottonbro-5561559, cottonbro-5561563, cottonbro-5850831, cottonbro-5976145,
cottonbro-6700116, cottonbro-6700119, cottonbro-6700142, cottonbro-6700144,
cottonbro-6753360, cottonbro-6753370, cottonbro-6753371, cottonbro-7407129,
cottonbro-8102360, cottonbro-8142260, cottonbro-9063608, cottonbro-9063624,
cottonbro-9063626, cottonbro-9316296, cottonbro-9467199, cottonbro-9577189,
cottonbro-9955927, craig-adderley-2306203, craig-adderley-2306210, craig-
adderley-2306213, cup-of-couple-6634443, cup-of-couple-6962575, cup-of-
couple-6963527, daria-nekipelova-9665517, daria-rem-1977055, darina-
belonogova-7886748, darina-belonogova-8386475, davner-toledo-4574403, dziana-
hasanbekava-6851631, efigie-lima-marcos-11831324, ehsan-7538807, ekaterina-
bolovtsova-7113346, ekaterina-bolovtsova-7113362, elina-fairytale-3865731,
elina-fairytale-3865763, elina-fairytale-3865765, eman-genatilan-5348809,
eman-genatilan-8589781, emmy-pua-10196907, engin-akyurt-5059305, eric-
deine-11781294, estelle-umaes-11734787, evelina-zhu-6286063, faruk-
tokluoglu-8777603, fireberrytech-6683091, flint-huynh-11804619, gary-
barnes-6248993, gary-barnes-6249024, greta-hoffman-7675722, guilherme-
almeida-1858175, hebert-santos-5485599, ichad-windhiagiri-7616249, imad-
clicks-11742222, imad-clicks-11742223, ivan-mudruk-10400317, ivan-
samkov-6968814, ivan-samkov-8952728, jamie-saw-10029674, jeandaniel-
francoeur-7678688, jennifer-enujiugha-1904674, jill-burrow-6758033, joao-
vitor-heinrichs-1787039, joshua-abner-3605015, joshua-mcknight-3290242, julia-
avamotive-1070967, julia-tatyanenko-11855943, juliana-marinina-9957288,
kampus-production-6298293, kampus-production-6298321, kampus-
production-7928134, kampus-production-8871412, karolina-grabowska-4378486,
karolina-grabowska-4498195, kat-smith-568021, ketut-subiyanto-4473864, ketut-
subiyanto-4545165, ketut-subiyanto-4584390, ketut-subiyanto-4584601, kindel-
media-7298396, kindel-media-7298459, kindel-media-7938549, kirill-
palii-3545783, klaus-nielsen-6303717, korede-adenola-11785507, kseniya-
buraya-10008858, kwesiblaq-10986569, leah-kelley-3722162, leo-acevedo-3261142,
lucas-souza-1964442, lucas-souza-3608010, maksim-goncharenok-4892914, marcelo-
chagas-2535859, maria-eduarda-loura-magalhaes-4340053, maria-luiza-
melo-11819746, maria-orlova-4946635, maria-orlova-4947740, marija-7737766,
marlon-schmeiski-11193234, mart-production-7880131, matheus-bertelli-11749497,
matheus-ferrero-11470717, matheus-henrin-11360455, maycon-marmo-4346013,
michelle-leman-6774345, mike-cabugao-8503888, mikhail-nilov-6707031, mikhail-
nilov-6943956, mikhail-nilov-6945088, mikhail-nilov-6968191, mikhail-
nilov-6968331, mikhail-nilov-7776528, mikhail-nilov-8343016, mikhail-
nilov-8350479, ming-zimik-5861623, miriam-alonso-7623727, monica-
turlui-8218377, monstera-5063295, monstera-5273734, monstera-5302897,
monstera-5384518, monstera-6781240, monstera-6973715, monstera-6974031,
monstera-6977869, monstera-7352909, mosei-films-9209576, nadin-sh-11872307,
nguyen-phuong-linh-6211165, nicola-barts-7925781, nikita-nikitin-11008044,
nikita-semezhin-9787604, oleg-magni-1669154, olia-danilevich-8964938, olya-
prutskova-7179057, orione-conceicao-2983464, ozan-culha-11850759, ozan-
culha-11858978, ozan-culha-11866492, pavel-danilyuk-7267691, pavel-
danilyuk-7267700, pavel-danilyuk-7406040, pexels-user-9281097, pnw-
production-8980983, pnw-production-8981313, polina-chistyakova-9052464,
polina-kovaleva-5885655, polina-kovaleva-7090394, polina-tankilevitch-6630835,
polina-tankilevitch-8210939, rachel-claire-4992586, rafael-freire-5714746,
rafael-portraits-9281360, raquel-silva-11870922, renthel-cueto-11131698,
renthel-cueto-11131703, rfstudio-3843292, rheyan-glenn-dela-cruz-
manggob-10210334, rodnae-productions-7402945, rodnae-productions-8173525,
rodnae-productions-8173543, roman-odintsov-11760366, roman-odintsov-11760376,
roman-odintsov-11760378, roman-odintsov-8018975, ron-lach-10139616, ron-
lach-10321431, ron-lach-8159655, run-ffwpu-11757051, ruslan-rozanov-11585357,
samer-daboul-4506967, samson-katt-5256085, santiago-jose-calvo-11757764,
sasha-lazarev-3578326, shiny-diamond-3762659, shotpot-6338298, shvets-
production-6974955, shvets-production-6975262, shvets-production-6975383,
shvets-production-6975413, shvets-production-6984635, shvets-
production-8005151, si-luan-pham-8778439, sound-on-3756943, svetlana-10311383,
taina-bernard-3482526, tanya-gorelova-3855199, thiago-alencar-10154765,
thiago-matos-10359136, thirdman-6958390, thirdman-7237074, thirdman-7268229,
thirdman-7268234, thirdman-7268483, thirdman-8053704, thomas-nguka-10163670,
thomas-nguka-7562643, tieu-bao-truong-8298108, tiffany-freeman-11038435, tima-
miroshnichenko-5118496, tima-miroshnichenko-6670752, tubarones-
photography-2737046, tubarones-photography-2943689, tubarones-
photography-3065450, valdemar-9546870, vanessa-loring-5082946, vika-
kirillova-10119334, vika-kirillova-11067905, vinicius-altava-2657594, vitoria-
santos-1913161, vitoria-santos-2838831, vlada-karpovich-8528898, vlada-
karpovich-8939842, vladimir-konoplev-11323367, vladimir-konoplev-11323376,
vladimir-vasilev-7640302, wesley-carvalho-4126255, yan-krukov-6617027, yan-
krukov-7155545, yana-sperry-11810044, yaroslav-shuraev-6281021, zayceva-
tatiana-11210581, zayceva-tatiana-11698072

###  F.2 Winks subset

airam-datoon-9637814, alena-darmel-7322312, alena-darmel-8153597, alexander-
krivitskiy-6471731, alexander-krivitskiy-6828450, amina-filkins-5560027,
amina-filkins-5560029, amina-filkins-5561443, amina-filkins-5561455, andrea-
piacquadio-3764391, andrea-piacquadio-3777558, andrea-piacquadio-3777563,
andrea-piacquadio-3778216, andrea-piacquadio-3778673, andrea-
piacquadio-3779420, andrea-piacquadio-3783107, andrea-piacquadio-3979192,
arina-krasnikova-6663385, beatriz-braga-10461875, bia-sousa-2191056, brianna-
amick-2069008, bruno-ticianelli-1889787, chandrashekar-hosakere-matt-707449,
cleyder-duque-3690938, cottonbro-5416365, cottonbro-6144420, darija-
shelkovich-5010665, dazzle-jam-2020992, debendra-das-5429301, deepak-
digwal-3577006, ekaterina-glushenko-8993965, evgeny-zuchman-9986411,
furkanfdemir-7020329, ganeshbabu-arun-580012, gustavo-fring-4254148, gustavo-
fring-6050330, gustavo-fring-7447022, ike-louie-natividad-3208616, jekaterina-
glushenko-8993965, jevgenija-shuhman-9986411, john-valette-10785650, julia-
larson-6113247, julia-larson-6113626, julia-larson-6113631, julia-
larson-6113639, julia-larson-6113641, kaushal-moradiya-3400573, kenneth-
gorzal-surillo-8177834, ketut-subiyanto-4584180, ketut-subiyanto-4584197,
ketut-subiyanto-4584387, ketut-subiyanto-4584604, koolshooters-6976854,
koolshooters-7142955, koolshooters-7143075, koolshooters-7143155,
koolshooters-7143217, ksenia-chernaya-6616201, laura-tancredi-7065531,
matheus-bertelli-7510832, melissa-jansen-van-rensburg-2255462, mikhail-
nilov-8922330, mikhail-nilov-8922333, mikhail-nilov-8923584, monstera-7114452,
monstera-7114634, monstera-7139819, nataliya-vaitkevich-4813824, nataliya-
vaitkevich-4813859, nichole-sebastian-3264351, nikita-saif-11494236, pavel-
danilyuk-8422489, pavel-danilyuk-8422507, pavel-danilyuk-8422515,
pixabay-40565, polina-tankilevitch-4723521, polina-tankilevitch-4723538,
polina-tankilevitch-4725082, polina-tankilevitch-4725084, polina-
tankilevitch-4725108, polina-tankilevitch-4725153, polina-
tankilevitch-6630839, polina-tankilevitch-6988592, polina-zimmerman-3958873,
rahmi-aksoz-9957220, rich-ortiz-5661730, rodnae-productions-10503462, rodnae-
productions-6709127, ron-lach-8159657, ron-lach-8989996, sarah-chai-7262397,
shvets-production-6975619, shvets-production-7525145, sora-shimazaki-5938614,
th-team-7516292, the-weddingfog-9084064, thirdman-6109560, victoria-
strelkaph-11034423, yan-krukov-4964933, yan-krukov-4964936, yan-
krukov-4964937, yan-krukov-7793112, yuliya-shabliy-388517, zura-
modebadze-4922053, zura-modebadze-4922064