# Can we truly transfer an actor’s genuine happiness to avatars? An
investigation into virtual, real, posed and spontaneous faces

Vitor Miguel Xavier Peres, Greice Pinho Dal Molin and Soraia Raupp Musse
Email<EMAIL_ADDRESS><EMAIL_ADDRESS>School
of TechnologyPontifical Catholic University of Rio Grande do SulPorto
AlegreBrazil

(2023)

###### Abstract.

“A look is worth a thousand words” is a popular phrase. And why is a simple
look enough to portray our feelings about something or someone? Behind this
question are the theoretical foundations of the field of psychology regarding
social cognition and the studies of psychologist Paul Ekman. Facial
expressions, as a form of non-verbal communication, are the primary way to
transmit emotions between human beings. The set of movements and expressions
of facial muscles that convey some emotional state of the individual to their
observers are targets of studies in many areas. Our research aims to evaluate
Ekman’s action units (AU) in datasets of real human faces, posed and
spontaneous, and virtual human faces resulting from transferring real faces
into Computer Graphics (CG) faces. In addition, we also conducted a case study
with specific movie characters, such as She-Hulk and Genius. We intend to find
differences and similarities in facial expressions between real and CG
datasets, posed and spontaneous faces, and also to consider the actors’
genders in the videos. This investigation can help several areas of knowledge,
whether using real or virtual human beings, in education, health,
entertainment, games, security, and even legal matters. Our results indicate
that AU intensities are greater for posed than spontaneous datasets,
regardless of gender. Furthermore, there is a smoothing of intensity up to 80%
for AU6 and 45% for AU12 when a real face is transformed into CG.

Action Units; Facial Expression; 3D facial, CG Faces, Virtual, Real, Posed,
Spontaneous

††journalyear: 2023††copyright: acmlicensed††conference: 22nd Brazilian
Symposium on Games and Digital Entertainment; November 6-9, 2023; Rio Grande
(RS), Brazil††booktitle: 22nd Brazilian Symposium on Games and Digital
Entertainment (SBGames 2023), June 11-September 11, 2023, Rio Grande (RS),
Brazil††price: 15.00††doi: 10.1145/3631085.3631231††isbn:
979-8-4007-1627-0/23/06††ccs: Computing methodologies Animation††ccs:
Computing methodologies Image-based rendering††ccs: Computing methodologies
Perception††ccs: Computing methodologies Reconstruction††ccs: Computing
methodologies 3D imaging

##  1\. Introduction

Expressing emotions is a crucial component of face-to-face communication, as
highlighted in a seminal study by psychologist Ekman et al.
(ekman1969repertoire, ). As a result, our perception as humans is critical
regarding the facial expressions of virtual humans and even robots. Studies on
human facial expressions are essential in creating more realistic features for
game and movie characters, enhancing user experiences. To this end,
performance-driven animations (Lance1990, ; Barros2019, ) have been employed
in the game and movie industries to transfer facial motion from real actors to
virtual characters. While evaluating such characters typically occurs through
movie and game reviews via audience opinions, this paper aims to investigate
how the transformation between the real and virtual domains takes place. For
instance, is a real human face movement truly represented when transferred to
a virtual face? So, the main investigation we intend to pursue in this work
concerns the difference/similarities between real and correspondent CG faces.

Numerous researchers have explored facial expression analysis. Barrett et al.
(barrett2019emotional, ) conducted a literature review of the six categories
of emotions, showing how people move their faces during various emotional
states and assessing which emotions can be inferred from facial movements.
This study is called emotional perception. Park et al.’s (park:2020, ) work
with real faces has shown that spontaneous smiles have higher intensity in the
Upper face region (eyes) than the Lower face (nose and mouth), whereas posed
smiles have more intense Lower-face activation. Through the extraction of the
Facial Action Coding System (FACS) (ekman1978, ), a comprehensive anatomical
taxonomy designed to categorize all facial movements (consisting of 46
distinct Action Unit) and the union of these movements (Action Unit) given a
target emotion, it becomes possible to distinguish between posed and
spontaneous images by quantifying the relative intensity of Upper and Lower
facial expressions (park:2020, ).

According to Mäkäräinen et al. (makarainen2014exaggerating, ), simplifying
facial expressions in the transfer from real to Computer Graphics (CG) domains
can provide artists with greater freedom to enhance the final facial
expression. The authors also believe that reducing realism can decrease
emotional intensity in facial expressions, and exaggeration can compensate for
this reduction.

In this work, we aim to compare real and correspondent CG faces when
expressing happiness, according to ANOVA and Pearson Correlation metrics.
Firstly, we want to analyze the discrete activation of AUs in the Upper and
Lower face regions, as proposed by Park et al.’s (park:2020, ); secondly, we
want to study if the transferring from Real to CG faces changes if the facial
expression is Spontaneous or Posed; and finally, we want to investigate if the
actors’ gender change the way the facial emotions are transferred from real to
CG faces. This article seeks to answer the following research questions:

Figure 1. Overview of our method: 1) obtaining real faces (Posed and
Spontaneous dataset) and CG Characters (detailed in Section 3.1 and 3.2), 2)
Split all faces by gender, 3) 3D face reconstruction (Deep3D
(deng2019accurate, ), Deca (DECA:Siggraph2021, ) and Emoca (EMOCA:CVPR:2021,
)) (detailed in Section 3.3), 4) Data extraction using OpenFace
(amos2016openface, )(detailed in Section 3.4), 5) Organize the data between
the Upper and Lower regions, and 6) Data analysis (detailed in Section 4).

  * •

RQ1: Are the real faces and their respective correspondences in CG similar
considering the activated AU (Lower or Upper parts of the face), the actors’
gender (Women or Men), and the type of dataset (Spontaneous or Posed),
considering only the happiness emotion?

  * •

RQ2: Is there a smooth effect when transforming from Real to CG faces?

  * •

RQ3: How do the attributes defined in RQ1 (detailed in Section 4.2) work for
existing animated characters from the movie industry?

To facilitate a comparison with Park’s method (park:2020, ), we have limited
our study to the emotion of happiness. This research explores how the Facial
Action Coding System (FACS) behaves when analyzing faces by region (Upper and
Lower) and across male and female genders. We also tested computer-generated
(CG) faces to compare real and virtual images comprehensively. Additionally,
we will also compare our results with the work of Mäkäräinen et al.
(makarainen2014exaggerating, ), concerning the smoothing expected when
transferring real to CG faces. Our work’s main contribution is to provide a
methodology for analyzing an actor’s facial movements and the corresponding
animated face to explore the similarities and differences. A possible
application could be to guide the needed exaggeration of final expressions
when transferring from real to CG faces. This analysis would enable post-
processing by artists to more accurately express the facial expressions of
virtual characters concerning the real actors’ actions. The findings of this
research will help guide the creation of virtual characters and their
expressions to reflect human emotions with more accuracy.

##  2\. Related Work

Studies on evaluating facial expressions and Action Units in real and CG faces
are the focus of this research. A literature review is done by Barret et al.
(barrett2019emotional, ) about the six categories of emotions. Their studies
show how people move their faces during different moments of emotions,
performing evaluations on which emotions are inferred through facial
movements. That study was called Emotional Perception. Park et al.
(park2020differences, ) explored differences in facial expressions of happy
emotion. They performed an analysis to measure changes in the dynamics of the
68 landmark regions. They obtained results indicating that the spontaneous
smile is more intense on the upper face than the Lower one, contrary to what
happens in faces with a Posed smile. These findings suggest an application of
emotion recognition based on landmarks.

The study by Fan et al. (fan2021demographic, ) aims to investigate the
demographic effects on the expression of facial emotions through the facial
action units of happiness indicated by the intensity levels of two distinct
facial action units, the Cheek Raiser (AU6) and the Lip Corner Puller (AU12).
They statistically analyze the intensity of happiness through the evaluation
of each AU and the interaction of three demographic factors, such as gender,
age, and race. Their studies indicated a higher intensity of AU12 in women
than men. Their research also shows that AI-powered social Rendering can be
used to measure the facial expression of happiness by assessing the happiness
intensity of AU, to validate the theory of human emotions into three closely
related fields: psychology, anthropology, and social studies.

For Sohre et al. (sohre2017data, ), the study aims to generate a non-
parametric face classifier to predict the level of happiness to be perceived.
They comment that humans use and expect faces to produce a variety of cues to
communicate intent through intonation and emotion. Happiness, for example, is
among one of the most basic and important emotions the human face conveys.
Since smiles vary widely within and between individuals, creating attractive
virtual characters must also exhibit these types of diversity. In this
research, it is observed that facial performance capture, although effective,
can involve parameter calibration and requires state-of-the-art equipment, as
well as actors who can produce all the desired facial expressions.

Regarding the transferring from real to CG faces, an interesting study in the
context of the present work is proposed by Mäkäräinen et al.
(makarainen2014exaggerating, ). The authors believe that simplifying facial
expressions, when transferring from the real to the CG domain, gives the
artist more freedom to enhance expressions. Just as they believe that facial
expressions lose emotional intensity when the level of realism is reduced,
exaggeration can compensate for this reduction.

In the present work, we want to answer the questions related to the main
research question: Are we truly transferring the actor’s genuine happiness to
avatars? For this, we define some metrics that should be used to answer this
question and compare our results with the work of Park et al.
(park2020differences, ), regarding the Upper and Lower parts of the face; with
the study by Fan et al. (fan2021demographic, ) which indicated a higher
intensity of AU12 in women than men, during happiness emotion; and finally,
with the work of by Mäkäräinen et al. (makarainen2014exaggerating, ) to
evaluate the smoothness of emotion when transferring from real to CG faces.

Figure 2. Visual representation of datasets used in this paper, given the
Posed and Spontaneous datasets. And the CG characters’ faces footage from
Aladdin (2019) and She-Hulk (2022).

##  3\. Data Acquisition and Generation

In literature, there are two types of datasets of real facial expressions:
Posed and Spontaneous (motley1988facial, ). In addition to the analysis
performed in the real faces datasets, as it is going to be discussed in the
next sections, we also transferred real to CG faces using the Deep3D
(deng2019accurate, ), Emoca (EMOCA:CVPR:2021, ) and Deca (DECA:Siggraph2021, )
software, generating two more datasets (CG Posed and CG Spontaneous). We
extract the Action Units for all studied datasets using OpenFace
(amos2016openface, ), a toolkit capable of facial landmark detection, head
pose estimation, facial action units recognition, and eye-gaze estimation.

The overview of our method is illustrated in Figure 1. Firstly, we obtained
the real faces from existing datasets; secondly, we split such datasets using
gender notations. The next phase is responsible for transferring the real
faces into CG faces through the Deep3D(deng2019accurate, ), Emoca
(EMOCA:CVPR:2021, ) and Deca (DECA:Siggraph2021, ). Then, all datasets (Real
and CG) extract AUs using OpenFace (amos2016openface, ). Finally, the datasets
are again separated regarding the activation of AUs (Lower and Upper parts of
the face). The following sections detail the used data.

###  3.1. Real Faces Datasets

In this paper, we use four Posed datasets (CFD (Ma:2015, ) and CFD-INDIA
(Lakshmi2021TheIF, ), FEI (THOMAZ2010902, ) and London (debruine2017face, ))
and two Spontaneous (Reacts (peres2021towards, ) and DISFA (disfa:2013, ))
datasets, where we choose only the footages that display the emotion
Happiness. The Chicago Face Database (CFD) (Ma:2015, ) and CFD-INDIA
(Lakshmi2021TheIF, ), consist of 587 and 142, respectively, high-resolution
standardized photographs of Black and White males and females between the ages
of 18 and 40 years. The CFD (Ma:2015, ) was expanded to include multiracial
faces and Indian faces photographed CFD-INDIA (Lakshmi2021TheIF, ) and
presents the equal distribution of samples for men and women.

FEI (THOMAZ2010902, ) is a dataset of frontal and pre-aligned images of a face
database maintained by the Department of Electrical Engineering of FEI. It
contains 200 faces (100 men and 100 women), where each subject has two frontal
images (one with a neutral expression and the other with a smiling facial
expression) and is between 19 and 40 years old with distinct appearance,
hairstyle, and adornments. London (debruine2017face, ) makes available 102
adult faces 1350x1350 pixels in full color. Self-reported age, gender, and
ethnicity are included, as are attractiveness ratings (on a scale of 1-7, from
”much less attractive than average” to ”much more attractive than average”).

Reacts (peres2021towards, ) presents a method of creating a spontaneous facial
expression dataset from YouTube reaction videos. From 18,400 images (frames)
extracted from the YouTube reaction videos, the authors selected 732 images,
with the six basic emotions and 80% as the threshold minimum in the VGG16
accuracy. DISFA (disfa:2013, ) contains approximately 130,000 annotated frames
from 27 subjects. For every frame in this dataset, the intensity of the 12
action units was manually annotated on a six-point ordinal scale (0 and 5
increasing intensity levels). Also, we only use faces that represent Happiness
emotion.

To ensure that all analyzed faces in the datasets include the corresponding
action units (AU) for happiness emotions, we employ the OpenFace
(amos2016openface, ) Confidence variable as a threshold. This variable
provides an estimate of the reliability of the current landmark detection. We
only consider faces with a confidence level of 90% or higher in the emotion
classification, ensuring their reliability for analysis. Finally, we utilized
a dataset consisting of 554 images of Posed faces, with 281 images of men and
273 images of women. Additionally, we included 464 images of Spontaneous
faces, comprising 247 images of men and 217 images of women.

###  3.2. Character’s Faces Footages

To complement our analysis, we collected images from two popular Hollywood
movie characters: Genius from Aladdin (2019) and She-Hulk from the Marvel TV
series (2022). The collection of these characters’ faces in CG was based on
detecting their faces in the YouTube video trailers111Genius -
https://www.youtube.com/watch?v=zWgBZ7erqDo222She-Hulk -
https://www.youtube.com/watch?v=a-oeACwhVI0. Figure 2 illustrates the footage
collected in this paper. As the main goal is to analyze only faces that
express the emotion of happiness, we use the pyFeat (pyfeat:2021, ) tool,
which classifies all the faces collected from the videos among the six basic
emotions since we do not have the emotions’ annotation of the faces extracted
from each film.

The collections contained images of the human form (real faces) of the
characters played by actress Tatiana Maslany and Will Smith, respectively, and
the CGI faces of the characters reproduced by the animators. This allowed us
to compare faces animated by the studio and faces produced by 3D
reconstruction models.

###  3.3. Transforming Real Faces to CG Faces

In addition to the real faces, we are interested in studying correspondent CG
faces. Therefore, we propose reconstructing CG faces based on real ones using
the three following facial reconstruction models: Deep3D (deng2019accurate, ),
Deca (DECA:Siggraph2021, ) and Emoca(EMOCA:CVPR:2021, ).

Deep3D (deng2019accurate, ) proposed a novel multi-image face reconstruction
aggregation method using Convolutional Neural Rendering without any explicit
label. Figure 2 illustrates the process proposed in this paper. Each dataset
containing real faces is transformed to CG, even for the media characters She-
Hulk and Genius. Deca (DECA:Siggraph2021, ) (Detailed Expression Capture and
Animation) is a model that learned from in-the-wild images with no paired 3D
supervision. Deca reconstructs a 3D head model with detailed facial geometry
from a single input image. Emoca (EMOCA:CVPR:2021, ) (Emotion Capture and
Animation) is a neural network that learns an animatable face model from in-
the-wild images without 3D and introduces a novel perceptual emotion
consistency loss that encourages the similarity of emotional content between
the input and rendered reconstruction supervision.

###  3.4. Data Processing using OpenFace

Once the datasets are organized as illustrated in Figure 1, we use OpenFace
(amos2016openface, ) to find the AU intensities in each face (real or CG).
OpenFace is an open-source facial recognition software that uses deep learning
to determine facial landmarks, head poses, and Action Units (AU). As presented
in literature (ekman1978, ; Keltner:2019, ), the six basic emotions are
described as a set of activated AU (Action Units). However, as mentioned
before, in our paper, we are analyzing only Happiness, which is compounded by
AU06 (cheek raiser) and AU12 (lip corner puller) (Ghayoumi, ). For all the
analyzed faces, Real or CG, Posed or Spontaneous, we split them into new
groups of features. We also consider the face gender and the upper (AU6) and
lower face parts (AU12) activated in the expressions.

The Convolutional Expert Constrained Local Model (CE-CLM) (8265507, ) and
Multi-task Convolutional Neural Network (MTCNN) (7553523, ) model
configurations were set for detecting facial landmarks and detecting faces in
OpenFace. The following section details the data analysis proposed in this
work.

##  4\. Data Analysis

This section includes a quantitative comparison of 3D reconstruction
techniques (ANOVA) concerning the type of dataset (Posed and Spontaneous),
image domain (Real and CG), and actor’s gender (Men and Women). We also use
the Cosine Similarity and Structural Similarity Index (SSIM) metrics to
evaluate the behavior of faces generated by 3D reconstruction models when
compared with their respective real faces. In addition, we present a case
study of two media characters from Blockbuster’s productions: Genius (Aladdin
- 2019) and She-Hulk (She-Hulk: Attorney at Law).

Figure 3. Mean and Median sum of all Intensities of Actions Units, generated
in each model and the real one. Separated by genre and for the posed dataset.

Figure 4. Mean and median of the intensity of the Actions units that compound
the expression of happiness (AU6 and AU12), generated in each model and the
real one, from dataset posed men.

Figure 5. Mean and median of the intensity of the Actions units that compound
the expression of happiness (AU6 and AU12), generated in each model and the
real one, from dataset posed women.

###  4.1. Quantitative Comparison of 3D Models

To evaluate the reconstructed 3D faces, we employ two metrics on 2D images to
compute the similarity between the input images (real faces) and the rendered
images (3D faces). We use the Cosine Similarity (Equation 1) of features
vectors (17 Facial action units) (cai2021multi, ), which are extracted from
OpenFace (amos2016openface, ), where $\left\|A\right\|$ represents the length
of the vector A, $\theta$ denotes the angle between $A$ and $B$, and ‘.’
denotes the dot product operator.

(1) $\cos(\theta)=\dfrac{A\cdot B}{\left\|A\right\|x\left\|B\right\|.}$

The Structural Similarity Index (SSIM) (lin2020towards, ) was used, where
$\mu_{x}$, $\mu_{y}$, $\sigma_{x}$, $\sigma_{y}$ and $\sigma_{x}y$ are the
local means, standard deviations and cross-covariance for images $x,y$. SSIM
enables us to evaluate results at the pixel level. Equation 2 defines the
SSIM:

(2)
$SSIM(x,y)=\dfrac{(2\mu_{x}\mu_{y}+c_{1})(2\sigma_{x}y+c_{2})}{(\mu_{x}^{2}+\mu_{y}^{2}+c_{1})(\sigma_{x}^{2}+\sigma_{y}^{2}+c_{2})}$

The SSIM’s goal is to assess perceptual image quality traditionally attempted
to quantify the visibility of errors between a distorted image and a reference
image using a variety of known properties of the human visual system
(nilsson2020understanding, ; 1284395, ). The Human visual perception can
identify structural information from a scene and hence identify the
differences between the information extracted from a reference and a sample
scene (1284395, ). The quantitative comparison in shown in Tables 1 and 2.

Dataset | Deep3D (deng2019accurate, ) | Emoca (EMOCA:CVPR:2021, ) | Deca (DECA:Siggraph2021, )
---|---|---|---
Posed Men |  0.6432 $\uparrow$ | 0.602 | 0.4587
Posed Women |  0.7430 $\uparrow$ | 0.7412 | 0.5839
Spont Men |  0.6574 $\uparrow$ | 0.6340 | 0.5305
Spont Women | 0.6471 |  0.7108 $\uparrow$ | 0.5542
She-Hulk | 0.5843 |  0.6288 $\uparrow$ | 0.4779
Genius |  0.6516 $\uparrow$ | 0.4913 | 0.4022
Table 1. Quantitative comparison of mean cosine similarity features (Facial Action Units) for each model 3D reconstructed through the Real faces. ”Spont” states for Spontaneous. The arrow up $\uparrow$, represents the highest achieved values of Cosine Similarity. Dataset | Deep3D (deng2019accurate, ) | Emoca (EMOCA:CVPR:2021, ) | Deca (DECA:Siggraph2021, )
---|---|---|---
Posed Men |  58.8% $\uparrow$ | 46.4% | 21.0%
Posed Women | 43.9% | 18.9% |  56.3% $\uparrow$
Spont Men | 30.0% | 24.0% |  34.3% $\uparrow$
Spont Women |  42.3% $\uparrow$ | 24.3% | 31.1%
She-Hulk |  24.9% $\uparrow$ | 20.5% | 19.5%
Genius |  39.8% $\uparrow$ | 17.4% | 18.9%
Table 2. Quantative comparison of mean Structural Similarity Index (SSIM), for
each model 3D reconstructed through the Real faces. ”Spont” states for
Spontaneous. The arrow up $\uparrow$, represents the the highest achieved
levels of SSIM.

The results indicate that the mean of Cosine Similarity between real and CG
faces was higher when reconstructing Posed (both genders), Spontaneous (Men),
and Genius, using Deep3D. Emoca obtained higher values for Spontaneous (Women)
and She-Hulk. Notably, the Deca model obtained the lowest results in all
datasets. Considering the SSIM value, we observe that Deep3D faces are closer
when processing the Posed (Men), Spontaneous (Women), and She-Hulk and Genius
in pixel levels.

The process of 3D reconstruction shows the effect of decreasing intensities
AUs intensities when compared to the real faces, as shown in Figures 5, 5 and
5. Figure 5 displays the mean (and median) sum of all Action Units (AUs)
generated by OpenFace for each set of faces generated for each 3D model and
the real faces from the posed dataset. In Figures 5 and 5, we present the mean
difference (and median) in AU intensities for the emotion of happiness (AU6
and AU12) in the posed dataset, encompassing both genres. Based on the SSIM
and Cosine Similarity metrics and the AU intensities, which exhibited greater
proximity compared to the real faces, we proceeded to the following analysis
using the data generated by the Deep3D model.

###  4.2. Comparative Analysis

The statistical analysis ANOVA was computed with the intensity values from AU6
(Upper) and AU12 (Lower), as shown in Figure 6. In such a figure, we present
the median, standard deviation, and variance of activated AU6 and AU12 in the
faces included in our datasets. In all statistical analyses, we used 5% of
significance level (Factorial ANOVA and Tukey HSD tests). The independent
variables include Gender (Men x Women), Image domain (Real x CG), and Dataset
(Posed and Spontaneous). The dependent variables are AU6 and AU12 intensity,
as shown in Table 3.

(a) Upper Face (AU6) Real

(b) Upper Face (AU6) CG

(c) Lower Face (AU12) Real

(d) Lower Face (AU12) CG

Figure 6. Comparisons between Posed and Spontaneous happy faces median, variance, and standard-deviation intensities from Upper (AU6) and Lower (AU12) Faces. This is presented in addition to the actors’ gender and Real/CG image features.  Factor | Sum of squares | df | F |  $f$-value
---|---|---|---|---
(a) Three-way ANOVA on AU6 intensity
Gender | 0.31371 | 1.0 | 0.92145 | 0.33721
Domain | 1039.04742 | 1.0 | 3051.94215 |  $<$ 0.00000
Dataset | 59.78617 | 1.0 | 175.60695 |  $<$ 1.6753e-38
Gender x Domain | 2.66359 | 1.0 | 7.82362 |  $<$ 0.0052
Gender x Dataset | 0.11457 | 1.0 | 0.33653 | 0.5619
Domain x Dataset | 0.14343 | 1.0 | 0.42128 | 0.5164
Gender x Domain x Dataset | 3.32974 | 1.0 | 9.78029 |  $<$ 0.0017
(b) Three-way ANOVA on AU12 intensity
Gender | 7459.18140 | 1.0 | 14965.25303 |  $<$ 4.7764e-18
Domain | 38.07696 | 1.0 | 76.39329 |  $<$ 1.7164e-189
Dataset | 535.61834 | 1.0 | 1074.60371 |  $<$ 3.0268e-60
Gender x Domain | 142.87125 | 1.0 | 286.64063 | 0.6078
Gender x Dataset | 0.13135 | 1.0 | 0.26353 |  $<$ 0.0360
Domain x Dataset | 2.19434 | 1.0 | 4.40247 |  $<$ 0.0005
Gender x Domain x Dataset | 0.16744 | 1.0 | 0.33592 | 0.56226
Table 3. Three-way (Gender x Domain x Dataset) ANOVA of the main effect and
the interaction effect on (a) AU6 intensity and (b) AU12 intensity, where
gender is men/women, the Domain is CG/real and Dataset is posed/spontaneous.
The significant variables and their corresponding p-values are highlighted in
bold. p-values $<$ 0.05 are considered statistically significant.

Regarding the AU6, the intensity values are presented in Figure 6 (a) and (b),
respectively, in real and CG faces. Specifically, we observed that the median
intensities for both Posed and Spontaneous datasets are similar between
genders (Men/Women) in CG and Real images, with higher values for Posed
datasets than Spontaneous ones. These findings suggest no significant
differences in AU6 intensity between men and women. This is supported by the
results of the three-way ANOVA (Table 3), which indicate that the Gender
factor (F=0.9214, $f$-value=0.3372) has a low significant effect. On the other
hand, we observed significant results in the domain (CG/Real) and dataset
(Posed/Spontaneous) features as Domain factor (F=3051.94215, $f$-value=0.0)
and Dataset factor (F=175.60695, $f$-value=1.6753e-38). In addition, the
interactions containing Gender x Domain (F=7.82362, $f$-value= 0.0052) and
Gender x Domain x dataset (F=9.78029, $f$-value=0.0017) were also evaluated.
For cases of significant interactions, we performed the post-hoc Tukey test to
assess the intensity of the AU6, which shows that, out of 6 gender and domain
interactions, the $p$-value was considered below $0.05$, with only two groups
having $p$-adj with similar means, not influencing the $p$-value of the Tukey
test.

The median intensity of AU12 expressions (Figure 6 (c) and (d)) is higher in
Posed datasets, for both genders, compared to the Spontaneous dataset, as
observed also in AU6. In this case, as shown in Table 3, the factors Gender
(F=14965.25303, $f$-value=4.7764e-18), Domain (F=76.39329, $F$-value=1.7164e
-189) and Dataset (F=1074.60371, $f$-value=3.0268e-60) are significant in the
AU12 analysis, while the interactions indicate significant values only in
Gender x Dataset (F=0.26353, $f$-value= 0.0360) and Domain x Dataset
(F=4.40247, $f$-value=0.0005). Tukey’s post-hoc test was performed for cases
of significant interactions to assess the intensity of the AU12, which shows
that, out of 6 gender and domain interactions, the $p$-value was considered
below 0.05. There was no interaction with $p$-adj greater than $0.05$,
indicating that this interaction influences this $p$-value. For the Domain x
Dataset interaction, we have a similar situation.

In addition to the ANOVA analysis, Figure 7 displays the Pearson correlation
analysis between the combination of AU intensities (AU6-Upper, AU12-Lower,
AU6/AU12, and All AUs). On the left (Figure 7 (a)), we note that the Posed
datasets with Real faces exhibited the strongest correlation in the AU06
(Upper) when we analyzed both genders and domains (Real and CG). Upon
analyzing the face’s Lower region (AU12) separately, a weak correlation is
explicitly observed in the women factor when comparing Real with CG face. This
weak correlation is not observed in the other combinations. So, we can observe
that the Upper part of the face better correlates with Real and CG actors than
the Lower part for posed datasets.

In Figure 7 (b), we noted that the AUs activated in the Spontaneous datasets
present a strong correlation between the combination of both Genders and in
Real and CG faces (CG men x CG women and Real men x Real women). It is
important to note that there is an inversion of correlations observed when
comparing AU6 and AU12 between Posed and Spontaneous faces. Specifically, AU12
presents higher correlations in all compared groups in Spontaneous datasets,
while AU6 correlates better with the Posed datasets. Indeed, our results
indicate that Spontaneous real faces have greater intensity at the Upper of
the face than in the Lower part, which is confirmed by Park et al. (park:2020,
) work. On the other hand, Park found that the posed smiles were more intense
on the Lower part of the face. However, this finding was not confirmed in our
study. Indeed, in the present work, we observed that CG faces presented such
characteristics, i.e., the lower part of the face got higher intensity values
than the upper part.

(a)

(b)

Figure 7. Pearson’s correlations regarding the AU intensity values for AU6,
AU12, the both of them and all AU.

###  4.3. Smoothing Intensities in CG faces

Analyzing the CG faces, transformed from Deep3D (deng2019accurate, ), in both
AU6 and AU12, comparing Figure 6 (a) with (b), and (c) with (d), it is
noticeable the smoothing of the intensity medians, as well as the standard
deviation and the variance. The intensity values for AU6 and AU12 in Posed
faces decreased by approximately 70% and 32%, respectively, for both genders,
while for Spontaneous faces, the smoothing rates for AU6 and AU12 were
observed to reach as high as 80% and 45%, respectively, for both genders.

This observation was previously discussed by Mäkäräinen et al.
(makarainen2014exaggerating, ), who hypothesized that when a facial expression
is directly copied from one domain to another, it may appear less intense on a
more abstract face. Our results are in accordance with the authors once the AU
intensity levels have been decreased when transferred to CG faces.

The factor that may contribute to the smoothing observed in the data is the
result (rendering) of the 3D reconstruction models, which still cannot achieve
the desired realism. Here, we hypothesized that the used software may smooth
the expression intensities. To verify that, we proceed with a case study
evaluating two media characters using our methodology. Our goal is to evaluate
the smoothness that happens when real faces are transferred to CG.

###  4.4. Case Studies: Genius and She-Hulk

(a) All AU Genius x She-Hulk

(b) Upper and Lower Face (AU6+AU12) Genius x She-Hulk

(c) Upper Face (AU6) Genius x She-Hulk

(d) Lower Face (AU12) Genius x She-Hulk

Figure 8. The graphs show the standard deviation (STD), median and variance
which are statistical measures used to evaluate the intensity values of AU
referring to She-Hulk and Genius characters in Real face domains, CG Movie and
CG generated in Deep3D (deng2019accurate, ).

In this section, we evaluate what happened to the facial expressions when we
transformed real faces to CG with the Deep3D tool (deng2019accurate, ) and
compared it with the CG presented in the studied films.

As shown in the last analysis, there is also a reduction of AU intensity
values when compared Real x CG, which is easily observed in Fig 8(a), Fig
8(b), Fig 8(c) and Fig 8(d). Mäkäräinen et. al. (makarainen2014exaggerating,
)’s studies indicate that smoothing out facial expressions not only
generalizes the emotion (being less specific) but also allows the post-
processing artist greater freedom to enhance the CG’s facial expression of
emotion. In this case, the representation of the virtual face is less faithful
to real movement but allows extra artistic work. In our tests, it is easy to
see that intensity values reduce in the median, standard deviation, and
variance.

The standard deviation (STD) and the variance presented in Figure 8(a), Fig
8(b), Fig 8(c) and Fig 8(d) shows that for the Genius and She-Hulk faces
values are always higher in Real faces, then in CG Movies and lastly in the
faces generated using Deep3D tool (deng2019accurate, ). So, in addition to the
high values of the median, there is more variation also in the Real faces of
the actors, confirming that independent of the software we used to transform
the domain of images from Real faces to CG, there is a smoothness in the AU
intensities.

Still, in the study by Mäkäräinen et al. (makarainen2014exaggerating, ), the
exaggeration approach is recognized as one of the most well-known animation
principles. According to studies by Johnston et al. (thomas1995illusion, ),
its use is relevant to the perception of emotional expressions. These
fundamental principles of animation have been applied in 3D computer
animation, as indicated by Lasseter et al. (lasseter1987principles, ) and in
other applications such as robotics.

Our studies indicate that while faces in the CG domain that corresponds to
Real faces tend to be smoothed, media characters in movies, that artists have
postprocessing worked on their animation, can amplify the final AU
intensities.

Factor | Sum of squares | F |  $f$-value
---|---|---|---
(a) two-way ANOVA on AU6 intensity
Gender | 41.70979 | 215.86032 |  $<$ 1.3627e-44
Domain | 93.55025 | 242.07490 |  $<$ 3.1802e-87
Gender x Domain | 0.50547 | 1.30797 | 0.2708
(b) two-way ANOVA on AU12 intensity
Gender | 27.58650 | 68.00717 |  $<$ 4.8163e-16
Domain | 39.48873 | 48.67447 |  $<$ 6.0499e-21
Gender x Domain | 18.20199 | 22.43608 |  $<$ 2.8709e-10
Table 4. Two-way (Gender x Domain) ANOVA of the main effect and the
interaction effect on (a) AU6 intensity and (b) AU12 intensity. The
significant variables and their corresponding $p$-values are highlighted.
$p$-value ¡ 0.05 is considered as statistically significant.

Table 4 provides an overview of the two variables and their effects on the
intensity of AU6 and AU12. Independent variables include Gender (Men and
Women) and Domain (Real, CG Movie, CG Deep3D). A two-way analysis of variance
(ANOVA) (Gender × Domain) is performed on the intensities AU6 and AU12
separately (Table 4). The two-way ANOVA (Table 4) shows that Gender
(F=215.86032, $f$-value=1.3627e-44) and Domain (F=242.07490,
$f$-value=3.1802e-87) have a significant effect on the intensity of AU6. This
no longer occurs for the Gender x Domain interaction (F=1.30797,
$f$-value=0.2708), as it is not significant.

(a)

(b)

(c)

Figure 9. Pearson’s correlations regarding the AU intensity values for AU6,
AU12 and the both of them.

Regarding the AU12 intensity values, the two-way ANOVA (Table 4) shows that
Gender (F=68.00717, $f$-value=4.8163e-16), Domain (F =48.67447,
$f$-value=6.0499e-21) and the Gender x Domain interaction (F=22.43608,
$f$-value=2.8709e-10) have a significant effect. For the domain, we can
confirm this in Figure 8(b). In the case of AU6, the ANOVA shows that the
Gender x Domain interaction is not significant (F=1.30797, $f$-value=0.2708)
and we can also confirm this through Figure 9(a) because the correlations are
all high. It is possible to verify in Figure 9(b) that there is only a small
correlation when the domain is Real and CG Movie/CG Deep3D for the Women,
otherwise the correlation is very high.

Figure 9(c), considering the intensities of the Upper and Lower AU, shows that
there is a strong correlation between CG Deep3D tool (deng2019accurate, ) and
CG Movie for She-Hulk and Genius. The opposite happens when the Pearson
comparison is made between Real faces and corresponding CG Deep3D/CG movie
faces. Considering only the intensity of the Upper part (AU6) 9(a), we can see
that all comparisons show high correlations, while the intensity of the Lower
face (AU12) 9(b) shows a low correlation in the comparisons between Real x CG
Movie and for Real x CG Deep3D in the actress.

##  5\. Final Considerations

This paper presented a study about Real and CG faces expressing Happiness
emotion in Women/Men and Posed/Spontaneous faces. Additionally, we analyzed
two media characters that are animated based on real actors’ performances.
This work first allowed us to generate datasets of real people having the
following features: i) Posed/Spontaneous, ii) in CG or Real domains, generated
with the Deep3D tool. In addition, 3 more datasets of faces of two known
actors were generated, the first dataset being composed of real faces, the
second using faces generated by Deep3D and the last one using animated CG
images of the actors in their respective films, She-Hulk and Genius.

We performed a three-way and two-way ANOVA analysis to assess the significance
of the Gender, Domain, and Dataset factors. The Pearson correlation between
the factors was also performed and graphs were created to analyze the standard
deviation, median, and variance of the AU’ intensities values.

Our main findings are:

  * •

Upper part of the face expresses higher values of intensities than the Lower
part, in Real faces (also according to Park et al (park:2020, ));

  * •

AU intensities of Posed faces are always higher than Spontaneous ones;

  * •

The intensities of AUs are higher in the Lower part of the face (detailed in
Section 4.2) in the CG faces than in the Upper part, as indicated in the study
by Park et al (park:2020, ) for the Posed dataset;

  * •

The Gender factor did not reveal significant differences (Table 3) in the
intensity of AU6 between Men and Women (Figure 6) in both the Posed and
Spontaneous datasets and in real/CG faces;

  * •

Through the median, we can see the smoothing of the intensity of the AU6 and
AU12 as discussed in Section 4.4, with respect to the media characters She-
Hulk and Genius. The smoothing is higher in the domain of CG Deep3D and then
CG Movie. This result is in accordance with the studies by Mäkäräinen et al.
(makarainen2014exaggerating, ), which discusses the smoothing of facial
expressions as a way to allow the artist to have greater freedom in post-
processing the animations to enhance the facial expression of emotions of the
CG characters; and finally

  * •

We observed that the Upper part of the face better correlates Real and CG
actors than the Lower part for posed datasets. On the other hand, the lower
part of the face presents higher values of Pearson correlation when comparing
Real and CG faces, in general.

In the present work, we aimed to answer the questions related to the main
research question: Are we really transferring the actor’s genuine emotions to
the avatars?

  * •

RQ1: Are the real faces and their respective correspondences in CG similar
considering the activated AUs (Lower or Upper parts of the face), the actors’
gender (women or men), and the type of dataset (Spontaneous or Posed),
considering only the happiness emotion?

ANSWER: Real and CG faces are differently impacted by the Upper and Lower
parts of the face. The actors’ gender is insignificant in the difference, and
AU intensities are always higher in Posed datasets (in real and CG faces).

  * •

RQ2: Is there a smooth effect when transforming from Real to CG faces?

ANSWER: Yes, there is a smoothing effect when we transfer from Real to CG
faces. Furthermore, some patterns were observed in the transformed datasets,
i.e., facial expressions of real humans are always more intense than CG faces.

  * •

RQ3: How the metrics defined in RQ1 work for existing animated characters from
the movie industry?

ANSWER: Indeed, the same patterns observed in CG faces transformed using
Deep3D were observed with CG actors manipulated by artists, e.g., in the cases
of She-Hulk and Genius.

We verified that the intensity of AU6 and AU12 in the dataset (spontaneous and
posed) suggests that the difference in intensity of these AUs concerning the
actors’ gender is very subtle, as we can see in Figure 6 (c) and (d ),
contradicting the work by Fan et al. (fan2021demographic, ) who indicated that
the female group had a higher AU12 intensity than the male group. However,
when observing the use case of She-Hulk and Genius in Figure 8(d), we can see
that there is a greater intensity of AU12 in She-Hulk compared to Genius, as
proposed by Fan et al. (fan2021demographic, ). We can also observe that the
She-Hulk in CG Movie indicates an AU12 intensity much closer to the Real
domain than the CG Deep3D.

Finally, as per our assessments, we believe that we are not transferring the
actor’s genuine emotions to the avatars, using the reconstruction software.
The transformation between Real and CG domains still needs tweaking, so the
presence of the artist is still critical to exaggerating the facial expression
in CG. However, we believe that even the artist still has limitations to
transfer the actor’s genuine emotion to the avatars.

In the future, we intend to explore other emotions and mixtures of emotions,
adding them to our real and CG datasets. In this way, we can suggest an
analysis method for the transformation from real to CG that can make it more
reliable with human facial expressions. Indeed, we intend to provide a
methodology that allows us to analyze real and virtual faces, providing
information about which part of the face should be exaggerated to better
reflect real faces’ motion. Another aspect we want to investigate is the
recognition of action units and their intensity values using models trained
with annotated CG faces.

##  6\. Acknowledgments

We would like to thank CNPq and CAPES for partially supporting this work.

## References

  * [1] Paul Ekman and Wallace V Friesen.  The repertoire of nonverbal behavior: Categories, origins, usage, and coding.  semiotica, 1(1):49–98, 1969.
  * [2] Lance Williams.  Performance-driven facial animation.  SIGGRAPH Comput. Graph., 24(4):235–242, sep 1990.
  * [3] Jilliam María Díaz Barros, Vladislav Golyanik, Kiran Varanasi, and Didier Stricker.  Face it!: A pipeline for real-time performance-driven facial animation.  In 2019 IEEE International Conference on Image Processing (ICIP), pages 2209–2213, 2019.
  * [4] Lisa Feldman Barrett, Ralph Adolphs, Stacy Marsella, Aleix M Martinez, and Seth D Pollak.  Emotional expressions reconsidered: Challenges to inferring emotion from human facial movements.  Psychological science in the public interest, 20(1):1–68, 2019.
  * [5] Seho Park, Kunyoung Lee, Jae-A Lim, Hyunwoong Ko, Taehoon Kim, Jung-In Lee, Hakrim Kim, Seong-Jae Han, Jeong-Shim Kim, Soowon Park, Jun-Young Lee, and Eui Chul Lee.  Differences in facial expressions between spontaneous and posed smiles: Automated method by action units and three-dimensional facial landmarks.  Sensors, 20:1199, 02 2020.
  * [6] P. Ekman and W.V. Friesen.  Facial Action Coding System: Manual.  Number v. 1-2. Consulting Psychologists Press, 1978.
  * [7] Meeri Mäkäräinen, Jari Kätsyri, and Tapio Takala.  Exaggerating facial expressions: A way to intensify emotion or a way to the uncanny valley?  Cognitive Computation, 6:708–721, 2014.
  * [8] Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong.  Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set.  In IEEE Computer Vision and Pattern Recognition Workshops, 2019.
  * [9] Yao Feng, Haiwen Feng, Michael J. Black, and Timo Bolkart.  Learning an animatable detailed 3D face model from in-the-wild images.  volume 40, 2021.
  * [10] Radek Danecek, Michael J. Black, and Timo Bolkart.  EMOCA: Emotion driven monocular face capture and animation.  In Conference on Computer Vision and Pattern Recognition (CVPR), pages 20311–20322, 2022.
  * [11] Brandon Amos, Bartosz Ludwiczuk, and Mahadev Satyanarayanan.  Openface: A general-purpose face recognition library with mobile applications.  Technical report, CMU-CS-16-118, CMU School of Computer Science, 2016.
  * [12] Seho Park, Kunyoung Lee, Jae-A Lim, Hyunwoong Ko, Taehoon Kim, Jung-In Lee, Hakrim Kim, Seong-Jae Han, Jeong-Shim Kim, Soowon Park, et al.  Differences in facial expressions between spontaneous and posed smiles: Automated method by action units and three-dimensional facial landmarks.  Sensors, 20(4):1199, 2020.
  * [13] Yingruo Fan, Jacqueline CK Lam, and Victor OK Li.  Demographic effects on facial emotion expression: an interdisciplinary investigation of the facial action units of happiness.  Scientific reports, 11(1):1–11, 2021.
  * [14] Nick Sohre, Moses Adeagbo, and Stephen Guy.  Data-driven variation for virtual facial expressions.  2017\.
  * [15] Michael T Motley and Carl T Camden.  Facial expression of emotion: A comparison of posed expressions versus spontaneous expressions in an interpersonal communication setting.  Western Journal of Communication (includes Communication Reports), 52(1):1–22, 1988.
  * [16] Debbie Ma, Joshua Correll, and Bernd Wittenbrink.  The chicago face database: A free stimulus set of faces and norming data.  Behavior research methods, 47, 01 2015.
  * [17] Anjana Lakshmi, Bernd Wittenbrink, Joshua Correll, and Debbie S. Ma.  The india face set: International and cultural boundaries impact face impressions and perceptions of category membership.  Frontiers in Psychology, 12, 2021.
  * [18] Carlos Eduardo Thomaz and Gilson Antonio Giraldi.  A new ranking method for principal components analysis and its application to face image analysis.  Image and Vision Computing, 28(6):902–913, 2010.
  * [19] Lisa DeBruine and Benedict Jones.  Face research lab london set.  Psychol. Methodol. Des. Anal, 2017.
  * [20] Vitor Miguel Xavier Peres and Soraia Raupp Musse.  Towards the creation of spontaneous datasets based on youtube reaction videos.  In George Bebis, Vassilis Athitsos, Tong Yan, Manfred Lau, Frederick Li, Conglei Shi, Xiaoru Yuan, Christos Mousas, and Gerd Bruder, editors, Advances in Visual Computing, pages 203–215, Cham, 2021. Springer International Publishing.
  * [21] S. M. Mavadati, M. H. Mahoor, K. Bartlett, P. Trinh, and J. F. Cohn.  Disfa: A spontaneous facial action intensity database.  IEEE Transactions on Affective Computing, 4(2):151–160, 2013.
  * [22] Jin Hyun Cheong, Tiankang Xie, Sophie Byrne, and Luke J. Chang.  Py-feat: Python facial expression analysis toolbox.  CoRR, abs/2104.03509, 2021.
  * [23] Dacher Keltner, Disa Sauter, Jessica Tracy, and Alan Cowen.  Emotional expression: Advances in basic emotion theory.  Journal of Nonverbal Behavior, 43, 06 2019.
  * [24] Mehdi Ghayoumi and Arvind Bansal.  Unifying geometric features and facial action units for improved performance of facial expression analysis.  06 2016.
  * [25] Amir Zadeh, Yao Chong Lim, Tadas Baltrušaitis, and Louis-Philippe Morency.  Convolutional experts constrained local model for 3d facial landmark detection.  In 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), pages 2519–2528, 2017.
  * [26] Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao.  Joint face detection and alignment using multitask cascaded convolutional networks.  IEEE Signal Processing Letters, 23(10):1499–1503, 2016.
  * [27] Xingjuan Cai, Yihao Cao, Yeqing Ren, Zhihua Cui, and Wensheng Zhang.  Multi-objective evolutionary 3d face reconstruction based on improved encoder–decoder network.  Information Sciences, 581:233–248, 2021.
  * [28] Jiangke Lin, Yi Yuan, Tianjia Shao, and Kun Zhou.  Towards high-fidelity 3d face reconstruction from in-the-wild images using graph convolutional networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5891–5900, 2020.
  * [29] Jim Nilsson and Tomas Akenine-Möller.  Understanding ssim, 2020.
  * [30] Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli.  Image quality assessment: from error visibility to structural similarity.  IEEE Transactions on Image Processing, 13(4):600–612, 2004.
  * [31] Frank Thomas, Ollie Johnston, and Frank Thomas.  The illusion of life: Disney animation.  Hyperion New York, 1995.
  * [32] John Lasseter.  Principles of traditional animation applied to 3d computer animation.  In Proceedings of the 14th annual conference on Computer graphics and interactive techniques, pages 35–44, 1987.