# Analyzing the Impact of Shape & Context on the Face Recognition Performance
of Deep Networks††thanks: *Work done while at Notre Dame

Sandipan Banerjee*1, Walter Scheirer2, Kevin Bowyer2, and Patrick Flynn2
1 Samsung Research America, 2 University of Notre Dame

###### Abstract

In this article, we analyze how changing the underlying 3D shape of the base
identity in face images can distort their overall appearance, especially from
the perspective of deep face recognition. As done in popular training data
augmentation schemes, we graphically render real and synthetic face images
with randomly chosen or best-fitting 3D face models to generate novel views of
the base identity. We compare deep features generated from these images to
assess the perturbation these renderings introduce into the original identity.
We perform this analysis at various degrees of facial yaw with the base
identities varying in gender and ethnicity. Additionally, we investigate if
adding some form of context and background pixels in these rendered images,
when used as training data, further improves the downstream performance of a
face recognition model. Our experiments demonstrate the significance of facial
shape in accurate face matching and underpin the importance of contextual data
for network training.

##  I Introduction

Deep learning [32] models have made significant contributions to face
recognition research culminating in state-of-the-art performance [39, 57, 16]
on extremely challenging datasets like MegaFace [30], IJB-B [56] and IJB-C
[37]. Such high accuracy numbers are typically generated by models trained on
millions of images that belong to $\sim$500K identity classes [12, 17].
However, many of these training datasets are either private [47, 50] or have
been decommissioned [1] due to privacy issues [24, 13, 30] and license misuse
[41]. A potential solution to bypass this issue but still inject diversity to
model training is to artificially augment [54] an existing training set. One
such popular augmentation technique is to introduce graphically generated
renderings of face images using 3D face models (_i.e._ texture wrapping) into
the downstream recognition model training [25, 35, 5]. Such graphical
renderings, when used as supplemental training data, can present variations in
facial pose and expression of existing [34] or synthetic identities [3, 5] to
a recognition model and help it learn robust representations, boosting its
test performance [35, 5].

While this augmentation approach has been shown to improve network models, it
does not essentially preserve the underlying shape of the base identity as the
3D models are either randomly chosen [25, 34] or selectively picked based on a
fit-estimation function [5]. The generated renderings typically look realistic
but a discussion on how much of an appearance change this produces for any
base identity is missing from the literature. Should the base identity and the
corresponding 3D face model come from the same gender and ethnic group or
should the latter be picked based on some rigid filtering in the attribute
space? Using two popular techniques [34, 5], we try to answer this question
and understand how changing the underlying shape can perturb the appearance of
both real and synthetic subjects, quantified using the authentication
performance of an established network model [26, 13].

Additionally, due to the missing 2D $\rightarrow$ 3D correspondences outside
of the internal facial mask (_i.e._ convex hull of the outer facial landmark
points [31]), these texture wrapping methods typically do not produce
realistic contextual pixels (_e.g._ forehead, hair, neck, clothes) or render
any at all [36, 25, 5]. In such a case, the training set is augmented with
novel views of the salient facial region but masked out (_i.e._ zero
intensity) context and background pixels. We question the optimality of this
approach and artificially add contextual pixels [6] to the rendered masked
face images of both real and synthetic identities. When compared to the masked
face images, we find the set with synthesized context and background to
generate a higher performance boost when used to supplement existing face
image training data [58].

While we do not present a novel framework or newly collected dataset in this
paper, the originality of this work lies in our experimental analysis of face
image data, and associated augmentation operations, for training CNN models.
Our contributions can be summarized as follows:

  1. 1.

We evaluate if using randomly chosen [25, 35] or well-fitted [5] 3D face
models, when rendering 2D face images for training data augmentation, results
in different realism [27] & face matching [13] scores (III).

  2. 2.

We analyze how changing the underlying shape of the base identities in these
images can affect the face matching performance of deep network models during
authentication experiments. We perform this analysis at various levels of
facial yaw with subjects varying in gender and ethnicity (IV).

  3. 3.

We investigate if adding some form of contextual pixels in these artificially
rendered images, when used to train a model, further improves its verification
performance during testing (V).

Figure 1: Sample results generated by rendering a 2D synthetic texture with
well-fitting ($dist(\text{I},\text{B})$ $\leq$ $m$) and ill-fitting
($dist(\text{I},\text{B})$ $>$ $m$) 3D models for different gender and
ethnicity groups. As can be seen, there is no noticeable difference in
rendering quality between the two sets.

##  II Related Work

Face Recognition: While face recognition research started with mostly
handcrafted features [43], it has made significant progress with the advent of
deep learning [53]. Models like FaceNet [47], VGG16 [39], LightCNN [57] and
ResNet [26] (coupled with ArcFace loss [16]) have achieved state-of-the-art
verification performance on multiple benchmark datasets [28, 30, 37]. Such
networks are typically trained with millions of web-scraped images, either
hosted privately by commercial entities [47, 50, 35] or shared publicly for
academic use [58, 13, 24]. However, many of these public datasets [39, 24, 30]
have been decommissioned due to privacy related issues [1] and are unavailable
at present [41].

Training Data Augmentation: To improve the diversity and coverage in training
sets, researchers typically augment existing datasets before training [54].
Augmentation techniques include simple image transformations (_e.g._ flip,
rotation, scaling) to GAN based attribute translations (_e.g._ modifying
expression, accessories, hairstyle) [14, 4]. While GANs can generate
photorealistic face images [29], they do not always retain subject-specific
traits [35], especially without fine-tuning during test time [18]. The
synthesized face images might also leak identity information from the subject
set used to train the model [52], which might pollute the downstream
recognition task. Another approach makes use of graphics based renderings of
existing faces [34] or synthesized textures [3] to augment the depth and width
of the training set. Either generic [25, 34] or best-fitting [5] 3D face
models are used to create novel views of a given facial texture by varying
their facial shape, pose and expression.

Facial Shape: Extensive research on analyzing how facial shape affects the
machine, and especially human, perception of overall facial appearance has
been carried out over the last two decades. [19] analyzes how the eyes, nose,
mouth and jaw features affect human judgement (_i.e._ emotion, attractiveness)
of face images. Relative contributions of facial shape and surface information
towards the human categorization of facial expressions have been investigated
in [48]. Further studies on evaluating the race-specific face shape
contributions and how they modulate neural responses have been shared in [2].
As one can imagine, changes in facial shape occur frequently as people age or
gain (or lose) weight. The effect of such changes in body weight and facial
adiposity on the face recognition performance of different algorithms has been
presented in [55, 51]. Since it affects both human and machine performance,
many models have been developed that focus on learning the facial shape of
individual subjects for face detection and recognition tasks [11, 15, 20].
Thus, artificially augmenting the training dataset by varying the intrinsic 3D
shape of a base identity can make face recognition models more robust towards
such changes in unseen data [35, 5].

Context & Background: Along with facial shape, the presence of contextual
pixels in face images also play an important role in establishing their
overall appearance. For example, external facial regions (context) are shown
to be key in mapping accurate face matching by human examiners when looking at
composite images [21, 22]. Altering these external features (_e.g._ hair,
ears, hooded top) in the composite images have also been shown to negatively
affect face recognition performance of witnesses [10, 23]. When internal
facial features are not useful, in case of occlusion or bad lighting, human
raters are shown to rely heavily on external regions for correct face matching
[46]. As demonstrated in [42], the presence of contextual pixels also
positively impacts face recognition performance of pre-trained deep networks
[39]. Preserving more contextual information when pre-processing the training
and testing face images (_i.e._ loose cropping) also produces better
verification performance in deep learning models, especially at lower false
acceptance rates [7]. Leveraging the contextual information in a scene can
benefit models responsible for downstream tasks like emotion recognition as
well [33].

TABLE I: Quantitative Analysis: Comparing the realism and face matching scores of graphical renderings of 2D face images synthesized using well-fitting 3D face models (_i.e._ picked via a rigorous fit-based thresholding) and ill-fitting (_i.e._ picked based on only gender and ethnicity conformity) 3D face models. Metric |  | Female Asian
---
| Female Caucasian
---
| Male Asian
---
| Male Caucasian
---
FID [27] (well-fitting) | 53.19 | 42.33 | 42.94 | 46.53
FID [27] (ill-fitting) | 52.29 | 42.48 | 42.00 | 50.31
Similarity Score [26] (well-fitting) | 0.627 | 0.603 | 0.601 | 0.571
Similarity Score [26] (ill-fitting) | 0.592 | 0.602 | 0.618 | 0.597

While these works explore the importance of facial shape on the overall
appearance of a human subject, they do not shed light on how much reshaping
existing face images can perturb the underlying base identity. We perform an
extensive set of experiments to specifically answer this question.
Additionally, we estimate how the presence of contextual pixels in face
images, when used for training a face recognition model, can improve its
downstream verification performance.

##  III Do We Really Need Best-fitting 3D Models for Rendering Facial Texture?

To render novel views of a given face image (_i.e._ 2D facial texture) the
additional depth information in a 3D face model is typically utilized. While
3D data corresponding to 2D images is present in some datasets [44, 40], such
information is absent in popular face datasets used to train deep networks
[58, 24]. In the absence of such depth information, researchers take advantage
of proxy 3D face models available in public datasets [40, 5]. This proxy 3D
model can be either generic [25], randomly chosen [34] or selectively picked
based on some fitting function between the 3D model and the 2D image [5]. In
this section, we experimentally judge if a rigorous model-fitting function is
really required to generate realistic renderings or can a simple gender and
ethnicity based model selection approach achieve the desired result.

To approximate the compatibility of a 3D face model to a 2D face image, we use
the fitting-function proposed in [5]. For a given facial texture $\text{{\bf
I}}_{s}$, of a synthetically generated identity in [5], a distance score
$dist$ is calculated as follows:

$dist(\text{{\bf I}}_{s},\text{B})=w_{1}*\left\|\text{{\bf v}}^{\text{{\bf
I}}_{s}}-\text{{\bf v}}^{\text{B}}\right\|_{1}+w_{2}*\left\|\text{{\bf
f}}^{\text{{\bf I}}_{s}}-\text{{\bf f}}^{\text{B}}\right\|_{1},$ (1)

where B is a 2D scan of the same gender and race as the base identity in
$\text{{\bf I}}_{s}$ and serves as a 2D substitute for a 3D model of the same
subject from the Notre Dame Synthetic Face Dataset [5]. $\text{{\bf
v}}^{\text{I}}$ represents the 68 landmark points of the image I, extracted
using Dlib [31]. $\text{{\bf f}}^{\text{I}}$ serves as the textural
representation of I and is extracted from the pre-trained VGGFace network
[39]. Given this shape and textural information of each image,
$dist(\text{{\bf I}}_{s},\text{B})$ captures the dissimilarity in appearance
between facial texture and 3D models in these attribute spaces. In [5], the
$k$ 3D models corresponding to the scans producing the lowest $dist$ values
are deemed best-fitting to $\text{{\bf I}}_{s}$ and used for rendering.

For this study, we accumulate the $dist$ scores for all the 2D facial textures
from the synthetic dataset released in [5] and then compute the mean $m$ from
this set and set it as a threshold for filtering out ill-fitting 3D models.
Mathematically this can be written as:

$m=\frac{1}{kn}\sum_{i=1}^{n}\sum_{j=1}^{k}dist(\text{I}^{i},\text{B}_{j}),$
(2)

where $n$ is the total number of 2D textures, $\text{I}_{i}$ is the $i$-th
texture, and $\text{B}_{j}$ is the $j$-th best-fitting model for
$\text{I}_{i}$.

The mean $m$ can be treated as an approximate threshold for 2D-3D fitness that
is computed dynamically once we have all the fitness scores available. Even if
$m$ is numerically different across datasets, the overall fitting behavior
should not change drastically, as the associated features for each gender-
ethnicity bucket and the 2D-3D fitting functions remain static. Any 3D model
corresponding to a 2D scan (B) that generates a $dist$ above $m$ can be
labeled as ill-fitting while still keeping the ones that generate $dist$
within $m$ as well-fitting. When enforcing a rigid goodness of fit, we can
simply filter out the ill-fitting models before rendering.

Figure 2: Sample results generated by (a) [5] (synthetic identity), and (b)
[34] (real identity). Each row of a sub-figure shows the same facial texture
(identity) rendered using a 3D face model at different facial poses. Notice
how different the same facial texture looks when rendered with different 3D
models (_i.e._ variable shape). An analysis of how this variability in facial
shape affects recognition performance can be found in IV.

To estimate the effect of this filtering approach, we use the following
metrics: (a) FID [27] to approximate the realness of the novel views produced
by 3D rendering, and (b) a similarity score between the 2D synthetic texture
and the final rendered views using the pre-trained ResNet50 [26, 13] model to
capture the structural closeness between the two. Specifically, we compute
these metrics for both the well-fitting ($dist(\text{I},\text{B})$ $\leq$ $m$)
and ill-fitting ($dist(\text{I},\text{B})$ $>$ $m$) 3D models for a particular
2D synthetic texture I. To gain a deeper perspective, we also estimate this
separately for each gender-ethnicity group annotated in the synthetic dataset
from [5]. To focus solely on the facial region, the facial mask of the 2D
synthetic texture is cropped out using the convex hull of its landmark points,
as shown in Fig. 1. The results are shared in Table I.

As can be seen, the thresholding does not dramatically improve the realism or
the structural similarity of the 3D rendered views. This can be attributed to
the fact that both the feature and landmark spaces used to compute $dist$ in 1
are already normalized based on gender and ethnic attributes [5] and hence any
best-fitting 3D model is already withing an implicit threshold for a given
facial texture. Therefore, employing a simple gender-ethnicity based model
selection approach, instead of an explicit fit-estimation function [5], can
effectively estimate the wellness of the 2D-3D fit.

We also investigated the effect of gender and race in 2D-3D fitting by
relaxing the gender-ethnicity constraint while finding compatible 3D models
for a given base identity. In most cases the best fitting 3D models did belong
to individuals in the same gender-ethnicity bucket as the base identity.
Consequently, we added this constraint to reduce computation time.

##  IV Impact of Facial Shape on Subject Identity

In this section, we seek to answer the following questions:

  1. 1.

How much of an influence does the facial shape have on a subject’s identity?
More specifically, if different 3D models are used to render the same facial
texture, how much does the appearance change?

  2. 2.

Is the appearance change, if any, consistent at different facial poses?

  3. 3.

Is the appearance change, if any, noticeably different across varying
rendering methods like [34, 5]?

  4. 4.

Is the appearance change, if any, similar across different genders and ethnic
groups?

As discussed before, a popular technique for augmenting training datasets is
to render 2D face images with different 3D models at different facial yaw
values [25, 34, 5]. It is argued that although the facial shape changes, the
facial texture remains intact, and therefore the synthetic images are assigned
the same identity label (an example can be seen in Fig. 2). These synthetic
views infuse more variation in facial shape and pose and augments the depth of
the training dataset. As image pairs of a test subject during authentication
experiments can vary in age [38, 30], and consequently the facial shape [55],
these synthetic views in training can make the network more robust to such
fluctuations, as shown in [36, 5]. However, the amount of variation in
appearance this augmentation introduces is not analyzed, nor is how different
the face looks overall even with the facial texture intact. To answer these
questions, we randomly sample three different sets of 1,000 synthetic
identities from the Notre Dame Synthetic Face Dataset [5], with each facial
texture (identity) rendered with three different 3D models, at yaw values [0°,
-30°, -60°, -90°]. Similarly, we sample three sets of 1,000 real identities
from a larger gallery of images [44, 3], with each facial texture (identity)
rendered with three randomly chosen 3D models (of the 10 available in [40]),
at yaw values [0°, -22°, -55°, -75°]. The yaw values are picked for both sets
as consistently as possible to each other.

Figure 3: Verification rate with varying facial shape at different facial yaw
with synthetic images generated using [5] Figure 4: Verification rate with
varying facial shape at different facial yaw with synthetic images generated
using Masi et al’s method [34]

For a particular facial yaw value in each of the two datasets, we perform a
verification experiment using face images (_i.e._ the 2D facial textures)
rendered with: (1) the same 3D model resulting in a consistent shape, and (2)
different 3D models generating variable shapes. Once generated, these rendered
views are passed through the ResNet50 model [26], pre-trained on the VGGFace2
dataset [13], and the 256-dimensional descriptor from the _feat_extract_ layer
is used as their representative features. We then use the Pearson correlation
coefficient to compute match score between different feature pairs. Since each
facial texture, real (Masi et al.’s method [34]) or synthetic (SREFI [5]), is
rendered with three different 3D models, we perform the experiment separately
for each 3D model. The resulting verification scores are shared in Figures 3
and 4 respectively.

As can be seen, the verification performance drops rapidly as the facial pose
moves away from 0°yaw. This effect is much more prominent in the SREFI
generated synthetic images, compared to the ones generated by Masi et al.’s
method. The reason behind this is the normalization along the jaw-line that is
present in Masi et al.’s method, which makes the variation in shape less
noticeable. SREFI, on the other hand, keeps the 3D model intact and therefore
the variation in facial shape becomes more noticeable at higher yaws (as
apparent from Fig. 2). It is interesting to note that for SREFI, the three 3D
models chosen for rendering are the most compatible with the facial texture
(using Eq. 1), and therefore are similar to each other to a certain degree.
Even in that case, the variation in appearance between two different
renderings is quite large as evident from Fig. 2:a and the plots in Fig. 3.
Such a variation in facial shape, when used for training a network, is
definitely beneficial to its feature learning as exhibited in [36, 34, 3, 5]
and Table I in the supplementary text accompanying this manuscript.

One thing to note in regards to facial shape is that SREFI [5] estimates
compatible 3D face models for synthetic textures and hence the 3D renderings
exhibit plausible appearance variations of the actual synthetic identity.
This, in turn, boosts the recognition efficacy of a network when used for
training [5]. However, using generic 3D face models, as done in [25, 36, 34],
might generate implausible results (_e.g._ female facial texture rendered with
a male 3D face model), and make the network confuse one identity with another
due to similar facial shape. Such trained networks can become more vulnerable
to presentation attacks, _i.e._ , distractors, with different people wearing
the same facial texture mask to generate false matches [8, 45].

###  IV-A Gender-Ethnicity Analysis

We further analyze the role of gender and ethnicity while estimating the
impact of consistent and variable 3D facial shape on face recognition
performance. For this experiment, we select synthetic identities from each of
the gender-ethnic groups present in the Notre Dame Synthetic Face Dataset [5]
\- Female Asian (FA), Female Caucasian (FC), Male Asian (MA) and Male
Caucasian (MC). Since there are 593 synthetic identities altogether in the FA
group, we randomly select 593 identities from the FC, MA, and MC groups, with
each facial texture (identity) rendered with three different 3D models, at yaw
values [0°, -30°, -60°, -90°]. It is to be noted however that each of these
three 3D models are from individuals of the same gender and ethnicity (as
discussed in III), consistent with that of the generated synthetic texture.

Figure 5: Verification rate with varying facial shape at different facial yaw
values with Female-Asian synthetic images generated using [5]. Figure 6:
Verification rate with varying facial shape at different facial yaw values
with Female-Caucasian synthetic images generated using [5]. Figure 7:
Verification rate with varying facial shape at different facial yaw values
with Male-Asian synthetic images generated using [5]. Figure 8: Verification
rate with varying facial shape at different facial yaw values with Male-
Caucasian synthetic images generated using [5].

For a particular facial yaw value and a certain gender-ethnic group, we
perform a verification experiment with facial textures rendered with the same
(consistent shape) and different 3D models (variable shape), same as before.
The verification scores, resulting from this experiment, can be seen in
Figures 5, 6, 7, and 8. They show a similar trend of diminishing performance
when the same synthetic facial texture, rendered with different 3D models, are
matched with each other. The drop in performance is even more prominent in the
Asian identities (Figures 5 and 7) than the Caucasian identities (Figures 6
and 8), possibly due to the presence of significantly more Caucasian 3D models
in the Notre Dame Synthetic Face Dataset dataset [5]. With a larger set of 3D
models, the chances are higher that the three 3D models, compatible to the
synthetic texture, will be more closer in facial shape and generate a higher
match score.

TABLE II: Distribution of the training datasets created for Experiment 5. The contextual pixels for the datasets are either preserved from a frontal view [34] or synthesized artificially [6]. | Training
---
Data
| CASIA [58]
---
Images
| Masi [34] Images
---
(wo/ context)
| Masi [34] Images
---
(w/ context)
| SREFI [5] Images
---
(wo/ context)
| SREFI [5] Images
---
(w/ context)
Masi_woContext |  | 111,276
---
(1,452 real
identities)
| 111,276
---
(1,452 real
identities)
0 | 0 | 0
SREFI_woContext |  | 111,276
---
(1,452 real
identities)
0 | 0 |  | 111,276
---
(1,452 synth.
identities)
0
Masi_Context |  | 111,276
---
(1,452 real
identities)
0 |  | 111,276
---
(1,452 real
identities)
0 | 0
SREFI_Context |  | 111,276
---
(1,452 real
identities)
0 | 0 | 0 |  | 111,276
---
(1,452 synth.
identities)

##  V Impact of Context & Background on Face Recognition Performance

In this section, we attempt to answer these questions:

  1. 1.

Does the presence of context (forehead, hair, neck, clothes) and background in
the training data (real or synthetic) for deep networks generate a higher test
performance, when compared with masked face images (_i.e._ no context)?

  2. 2.

In a similar vein, does the presence of context and background in the
supplemental training data, generated by SREFI [5] or Masi et al.’s method
[34], generate a higher test score?

For this experiment, we prepare two hybrid datasets using real face images
from the CASIA-WebFace dataset [58] and synthetic images generated using (1)
Masi et al.’s method [34] and (2) SREFI [5]. [34] warps a given face image
using a static set of ten 3D face models from the Basel dataset [40] at pre-
determined yaw values [0°, -22°, -40°, -55°, -75°]. Therefore, for a given
face image of a real identity, it can generate 50 (10$\times$5) new views
varying in facial shape and pose. However, it cannot create new identities and
to generate the hybrid dataset we again use the external gallery from [44]. It
contains a total of 15,807 face images of 1,452 real subjects and using [34]
we render 790,350 synthetic views of the same 1,452 subjects varying facial
shape and pose, and coin it as the “Masi Dataset”. These images are rendered
without any context (_e.g._ masked faces in Fig. 2.b).

Since Masi et al.’s method [34] cannot generate new identities, we re-sample
the real training data from CASIA-WebFace [58] to keep the number of subjects
the same with the Masi Dataset. Specifically, we randomly select 111,276 face
images of 1,452 subjects from the head, _i.e._ half of the dataset containing
the subjects with the most number of images of CASIA-WebFace’s distribution
[36]. For the supplemental data generated by [34], a total of 111,276 images
is selected from the 1,452 real subjects from the Masi Dataset. Similarly for
SREFI [5], we sample three sets of 111,276 images of 1,452 synthetic subjects
from the Notre Dame Synthetic Face Dataset [5]. When combined with the CASIA-
WebFace samples, these synthetic samples form a pair hybrid datasets of masked
face images, as listed in the first two rows of Table II.

Figure 9: (a) Synthetic face images (synthetic identities) generated by SREFI
[5] with context and background hallucinated automatically using [6], (b)
synthetic face images (real identities) generated by Masi et al.’s method [34]
preserving the background. All the images are 128$\times$128 in size.

Since Masi et al.’s method [34] has the functionality to preserve the context
and background while rendering a frontal face image, we prepare another hybrid
dataset (“Masi_Context” in Table II) where the context of the same synthetic
images are preserved. The SREFI method [5] however does not render context and
background pixels at non-frontal poses due to missing 2D $\rightarrow$ 3D
correspondences for these external facial regions. To hallucinate the context
pixels for these images, we use the trained snapshot of the multi-scale GAN
model from [6]. This model is composed of a cascaded network of GAN blocks,
each tasked with hallucination of missing pixels at a particular resolution
while guiding the synthesis process of the next GAN block. It can generate the
missing pixels automatically, by taking cues from the features of the provided
face mask, without requiring any human supervision. We generate context and
background pixels for the 111,276 face images of the 1,452 SREFI-based
synthetic identities, and prepare a hybrid dataset with sampled images from
CASIA-WebFace [58] (“SREFI_Context” in Table II). Sample synthetic face images
with such artificially rendered context and background can be found in Fig. 9.

Once these datasets are prepared, we fine-tune the ResNet-50 network [26],
pre-trained on the VGGFace2 dataset [13], separately in four different
training sessions, using the four datasets from Table II. We use a polynomial
decay policy [9] for training each network with the same batch size = 16, base
learning rate = 0.001, gamma = 0.96, momentum = 0.009 and a step size of 32K
training iterations. We stop training when the validation loss plateaus across
an epoch.

For testing each trained network, we use the IJB-B verification protocol [56]
as our performance metric. Each still image or video frame from a _template_
is first aligned about its eye center and then fed to each of the four trained
networks for feature extraction (_i.e._ 256-D descriptor from _feat_extract_
layer). We generate an average feature vector for a template using _video_ and
_media pooling_ operations, described in [34], and match a pair of templates
using a Pearson correlation coefficient metric ($\rho$) between their feature
vectors as:

$\rho=\frac{Cov(F_{i},F_{j})}{\sigma_{F_{i}}\sigma_{F_{j}}},$ (3)

where _Cov_ denotes covariance, $F_{i}$ and $F_{j}$ are the pooled feature
vectors of the $i$-th and $j$-th templates respectively. In an ideal situation
pooled features of two templates of the same identity should match perfectly,
_i.e._ , $\rho$ should be close to 1. The resulting ROC curves are presented
in Fig 10.

Figure 10: Verification performance of the ResNet-50 model [26, 13] on the
IJB-B [56] dataset, when fine-tuned with training data with and without the
presence of context and background pixels. As the curves suggest, presence of
facial context benefits network training.

As the curves clearly suggest, training the network with supplemental face
images containing contextual pixels makes it more robust to visual changes
during testing. This can be attributed to the additional information and
variation the presence of context and background provides to the network
during training. Such variations in the training data actually makes the model
robust to intra-class variance [36] and less prone to overfitting, improving
its test performance over only training with face masks. Moreover, these
results are along the same lines of previous research that underpin the
importance of context in face recognition by humans [46, 42] and deep networks
[7].

##  VI Training Data Augmentation Results

Similar to the dataset preparation in Section V, we prepare augmented version
of a randomly sampled subset of the CASIA-WebFace [58] using synthetic face
images from both Masi et al.’s method [34] and SREFI [5]. To gauge the
statistical significance of the experimental results, we sample three
different sets (not necessarily subject disjoint) of 111,276 images of 1,452
subjects from CASIA-WebFace [58], the “Masi Dataset” (described in Section V
of the main paper) and the Notre Dame Synthetic Face Dataset. The synthetic
images from the Masi dataset are posed at [0°, -22°, -40°, -55°, -75°] yaw
while the Notre Dame Synthetic Face Dataset consists of multiples of
30°increments in facial yaw for a synthetic identity. We do not explicitly
check for consistency in the pose distribution in each of these datasets. As
the synthetic images are graphically rendered, they are essentially masked
outside the salient facial region (_i.e._ zero intensity), and do not have any
contextual information embedded in them.

TABLE III: Effectiveness of Masi et al.’s method [34] and SREFI [5] as data augmentation modules, tested on IJB-B [56], using sampled CASIA-WebFace (CW) [58] and the ResNet50 model [26]. We highlight the best two verification scores. Training Data |  | CW [58] Images
---
(real identities)
| Synthetic Masi images
---
(real identities)
| Synthetic SREFI images
---
(synthetic identities)
| IJB-B [56] Performance
---
(TAR@FAR = 0.01)
Dataset 1 | 111,276 (1,452) | 0 | 0 | 0.925 $\pm$ 0.04
Dataset 2 | 111,276 (1,452) | 111,276 (1,452) | 0 | 0.939 $\pm$ 0.14
Dataset 3 | 0 | 111,276 (1,452) | 0 | 0.857 $\pm$ 0.18
Dataset 4 | 111,276 (1,452) | 0 | 111,276 (1,452) | 0.937 $\pm$ 0.09
Dataset 5 | 0 | 0 | 111,276 (1,452) | 0.835 $\pm$ 0.22

We fine-tune the ResNet-50 network [26], pre-trained on VGGFace2 [13],
separately with these datasets. The hyper-parameters, scheduling and
termination policy is set the same as described in Section V of the main text.
For testing each trained network we use the IJB-B verification protocol [56]
and match templates post _video_ and _media pooling_ operations using a
Pearson correlation. As shown in Table III, both the augmentation methods
(Datasets 2 and 4) trigger a boost in ResNet50’s verification score when
trained on just CASIA-WebFace (Dataset 1). Furthermore, when the number of
training images and identities is kept the same, the test performance remains
approximately the same. Although Masi et al. [34] generated images visually
look different than those generated by SREFI [5], as can be seen in Fig. 2 of
the main text (forehead, jawline, prominence of 3D shape, etc.), they produce
similar results when similar amount of supplemental data (_i.e._ same number
of images and identities) is chosen from each set. This suggests the visual
appearance of the supplemental data in face recognition experiments is not as
important as the size of the overall training dataset; this is consistent with
recent findings in other domains [49]. However, synthetic face images alone
fail to capture the full range of visual diversity found in real images
(Dataset 1) and consequently garner lower verification scores when used by
themselves (Datasets 3 and 5) to train the ResNet50 model.

##  VII Conclusion

In this paper, we perform an extensive set of experiments to quantify the
impact the underlying facial shape has on the overall appearance of a
particular subject. For this work, we chose two data augmentation methods from
[34] and [5] that use different 3D models to graphically render novel views of
2D face images (real or synthetic) with varying facial pose and shape.
Utilizing these rendered images as variations of the original image (and
subject) we analyze how much of a perturbation this process injects into the
base identity. This analysis is performed using FID [27] as the realism metric
and the ResNet50 [26, 13] model as an estimator of facial similarity. Since
the intrinsic shape of a profile face is more visible to the network, we find
the subject identity is disturbed further as we move away from a frontal pose
when variable 3D models are used to render the same face image. We also
observe that simply choosing 3D models based on congruence in gender and
ethnicity to the base identity generates synthetic views that score similarly
in realism and identity retention when compared to a strict fit-estimation
function. Additionally, we quantify the effect context and background pixels
have on model performance when used to train a deep network. We find the
presence of such pixels in the training data, even when artificially
hallucinated [6], to inject vital information to the model representation and
improve its downstream authentication performance.

Reproducibility: Most of the datasets, 3D face models and network models used
in this study are publicly available at present, as listed below:

  1. 1.

The Notre Dame Synthetic Face Dataset [5]: https://cvrl.nd.edu/projects/data/.

  2. 2.

Gallery for “Masi Dataset”: shared in [44].

  3. 3.

Masi Renderer [34]: https://github.com/iacopomasi/face_specific_augm.

  4. 4.

ResNet50 [26] trained on VGGFace2 [13]: https://github.com/ox-vgg/vgg_face2.

The trained snapshot of the context generation GAN [6], used to hallucinate
context and background in Section V of the main paper, is however not publicly
available and was generously shared with us by its authors for this work.

## References

  * [1] Exposing.ai. available here: https://exposing.ai/.
  * [2] B. Balas and CA. Nelso.  The role of face shape and pigmentation in other-race face perception: an electrophysiological study.  Neuropsychologia, 48(2):498–506, 2010.
  * [3] S. Banerjee, J. Bernhard, W. Scheirer, K. Bowyer, and P. Flynn.  Srefi: Synthesis of realistic example face images.  In IJCB, 2017.
  * [4] S. Banerjee, A. Joshi, P. Mahajan, S. Bhattacharya, S. Kyal, and T. Mishra.  Legan: Disentangled manipulation of directional lighting and facial expressions whilst leveraging human perceptual judgements.  In CVPR Workshops, 2021.
  * [5] S. Banerjee, W. Scheirer, K. Bowyer, and P. Flynn.  Fast face image synthesis with minimal training.  In WACV, 2019.
  * [6] S. Banerjee, W. Scheirer, K. Bowyer, and P. Flynn.  On hallucinating context and background pixels from a face mask using multi-scale gans.  In WACV, 2020.
  * [7] A. Bansal, C. Castillo, R. Ranjan, and R. Chellappa.  The do’s and don’ts for cnn-based face verification.  ICCV Workshops, 2017.
  * [8] S. Bhattacharjee, A. Mohammadi, and S. Marcel.  Spoofing deep face recognition with custom silicone masks.  In BTAS, 2018.
  * [9] L. Bottou.  Large-scale machine learning with stochastic gradient descent.  In COMPSTAT. 2010.
  * [10] C. Brown, E. Portch, FC. Skelton, C. Fodarella, H. Kuivaniemi-Smith, K. Herold, PJB. Hancock, and CD. Frowd.  The impact of external facial features on the construction of facial composites.  Ergonomics, 62(4):575–592, 2019.
  * [11] R. Brunelli and T. Poggio.  Face recognition: features versus templates.  IEEE Trans. on Pattern Analysis and Machine Intelligence, 15(10):1042–1052, 1993.
  * [12] J. Cao, Y. Li, and Z. Zhang.  Celeb-500k: A large training dataset for face recognition.  In ICIP, 2018.
  * [13] Q. Cao, L. Shen, W. Xie, O. M. Parkhi, and A. Zisserman.  Vggface2: A dataset for recognizing faces across pose and age.  In arXiv:1710.08092.
  * [14] Y. Choi, Y. Uh, J. Yoo, and JW. Ha.  Stargan v2: Diverse image synthesis for multiple domains.  In CVPR, 2020.
  * [15] TF. Cootes, CJ. Taylor, DH. Cooper, and J. Graham.  Active shape models – their training and application.  CVIU, 61:38–59, 1995.
  * [16] J. Deng, J. Guo, N. Xue, and S. Zafeiriou.  Arcface: Additive angular margin loss for deep face recognition.  In CVPR. 2019.
  * [17] J. Deng, J. Guo, D. Zhang, Y. Deng, X. Lu, and S. Shi.  Lightweight face recognition challenge.  In ICCV Workshops, 2019.
  * [18] Y. Deng, J. Yang, D. Chen, F. Wen, and X. Tong.  Disentangled and controllable face image generation via 3d imitative-contrastive learning.  In CVPR, 2020.
  * [19] JA. Diego-Mas, F. Fuentes-Hurtado, V. Naranjo, and M. Alcañiz.  The influence of each facial feature on how we perceive and interpret human faces.  i-Perception, 2020.
  * [20] IL. Dryden and KV. Mardia.  Statistical shape analysis.  John Wiley, 1998.
  * [21] C. Frowd, V. Bruce, A. McIntyre, and P. Hancock.  The relative importance of external and internal features of facial composites.  British Journal of Psychology, 98(1):61–77, 2007.
  * [22] C. Frowd, W. Erickson, J. Lampinen, F. Skelton, A. McIntyre, and P. Hancock.  A decade of evolving composite techniques: Regression-and meta-analysis.  Journal of Forensic Practice, 2015.
  * [23] C. Frowd and G. Hepton.  The benefit of hair for the construction of facial composite images.  The British Journal of Forensic Practice, 2009.
  * [24] Y. Guo, L. Zhang, Y. Hu, X. He, and J. Gao.  Ms-celeb-1m: A dataset and benchmark for large-scale face recognition.  In ECCV, 2016.
  * [25] T. Hassner, S. Harel, E. Paz, and R. Enbar.  Effective face frontalization in unconstrained images.  In CVPR, 2015.
  * [26] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  CVPR, 2016.
  * [27] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  In NeurIPS, 2017.
  * [28] G. B. Huang, M. Ramesh, T. Berg, and E. Learned-Miller.  Labeled faces in the wild: A database for studying face recognition in unconstrained environments.  In Technical Report 07–49, 2007.
  * [29] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila.  Analyzing and improving the image quality of StyleGAN.  In CVPR, 2020.
  * [30] I. Kemelmacher-Shlizerman, S. Seitz, D. Miller, and E. Brossard.  The megaface benchmark: 1 million faces for recognition at scale.  In CVPR, 2016.
  * [31] D. E. King.  Dlib-ml: A machine learning toolkit.  In JMLR, volume 10, pages 1755–1758, 2009.
  * [32] Y. LeCun, Y. Bengio, and G. Hinton.  Deep learning.  Nature, 521(7553):436–444, 2015.
  * [33] J. Lee, S. Kim, S. Kim, J. Park, and K. Sohn.  Context-aware emotion recognition networks.  ICCV, 2019.
  * [34] I. Masi, T. Hassner, A. T. Tran, and G. Medioni.  Rapid synthesis of massive face sets for improved face recognition.  FG, 2017.
  * [35] I. Masi, AT. Tran, T. Hassner, G. Sahin, and G. Medioni.  Face-specific data augmentation for unconstrained face recognition.  IJCV, 127:642–667, 2019.
  * [36] I. Masi, A. T. Tran, J. T. Leksut, T. Hassner, and G. Medioni.  Do we really need to collect millions of faces for effective face recognition?  In ECCV, 2016.
  * [37] B. Maze and et al.  Iarpa janus benchmark–c: Face dataset and protocol.  In ICB, 2018.
  * [38] G. Panis, A. Lanitis, N. Tsapatsoulis, and T. Cootes.  Overview of research on facial ageing using the fg-net ageing database.  IET Biometrics, 5(2):37–46, 2016.
  * [39] O. M. Parkhi, A. Vedaldi, and A. Zisserman.  Deep face recognition.  In BMVC, 2015.
  * [40] P. Paysan, R. Knothe, B. Amberg, S. Romdhani, and T. Vetter.  A 3d face model for pose and illumination invariant face recognition.  In AVSS, 2009.
  * [41] K. Peng, A. Mathur, and A. Narayanan.  Mitigating dataset harms requires stewardship: Lessons from 1000 papers.  arXiv:2108.02922.
  * [42] P.J. Phillips.  A cross benchmark assessment of a deep convolutional neural network for face recognition.  In FG, 2017.
  * [43] P.J. Phillips, P. Flynn, T. Scruggs, K. Bowyer, J. Chang, K. Hoffman, J. Marques, J. Min, and W. Worek.  Overview of the face recognition grand challenge.  In CVPR, 2005.
  * [44] P. J. Phillips, P. Flynn, and K. Bowyer.  Lessons from collecting a million biometric samples.  IVC, 2016.
  * [45] R. Ramachandra, S. Venkatesh, KB. Raja, S. Bhattacharjee, P. Wasnik, S. Marcel, and C. Busch.  Custom silicone face masks: Vulnerability of commercial face recognition systems & presentation attack detection.  In IWBF, 2019.
  * [46] A. Rice, P.J. Phillips, V. Natu, X. An, and A.J. O’Toole.  Unaware person recognition from the body when face identification fails.  Psychological Science, 24:2235–2243, 2013.
  * [47] F. Schroff, D. Kalenichenko, and J. Philbin.  Facenet: A unified embedding for face recognition and clustering.  In CVPR, 2015.
  * [48] M. Sormaz, AW. Young, and TJ. Andrews.  Contributions of feature shapes and surface cues to the recognition of facial expressions.  Vision Research, 127:1–10, 2016.
  * [49] C. Summers and M.J. Dinneen.  Improved mixed-example data augmentation.  In WACV, 2019.
  * [50] Y. Taigman, M. Yang, M. Ranzato, and L. Wolf.  Deepface: Closing the gap to human-level performance in face verification.  In CVPR, 2014.
  * [51] P. Terhörst, JN. Kolf, M. Huber, F. Kirchbuchner, N. Damer, A. Morales, J. Fierrez, and A. Kuijper.  A comprehensive study on face recognition biases beyond demographics.  In arXiv:2103.01592, 2021.
  * [52] P. Tinsley, A. Czajka, and P. Flynn.  This face does not exist… but it might be yours! identity leakage in generative models.  In WACV, 2021.
  * [53] M. Wang and W. Deng.  Deep face recognition: A survey.  Neurocomputing, 429:215–244, 2021.
  * [54] X. Wang, K. Wang, and S. Lian.  A survey on face data augmentation for the training of deep neural networks.  Neural Computing and Applications, 32:15503–15531, 2020.
  * [55] L. Wen, G. Guo, and X. Li.  A study on the influence of body weight changes on face recognition.  In IJCB, 2014.
  * [56] C. Whitelam, E. Taborsky, A. Blanton, B. Maze, J. Adams, T. Miller, N. Kalka, A. K. Jain, J. A. Duncan, K. Allen, J. Cheney, and P. Grother.  Iarpa janus benchmark-b face dataset.  In CVPR Workshops, 2017.
  * [57] X. Wu, R. He, Z. Sun, and T. Tan.  A light cnn for deep face representation with noisy labels.  arXiv:1511.02683, 2017.
  * [58] D. Yi, Z. Lei, S. Liao, and S. Z. Li.  Learning face representation from scratch.  In arXiv:1411.7923.