# Graph-based Facial Affect Analysis: A Review

Yang Liu, Xingming Zhang, Yante Li, Jinzhao Zhou, Xin Li, and Guoying Zhao Y.
Liu, X. Zhang and J. Zhou are with the School of Computer Science and
Engineering, South China University of Technology, Guangzhou 510006, China.
E-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>X. Li
is with the Department of Electrical and Computer Engineering, Rutgers
University, Piscataway, NJ-08854, United States. E-mail:
<EMAIL_ADDRESS>Y. Liu, Y. Li and G. Zhao are with the Center for
Machine Vision and Signal Analysis, University of Oulu, Oulu, FI-90014,
Finland. E-mail<EMAIL_ADDRESS>
###### Abstract

As one of the most important affective signals, facial affect analysis (FAA)
is essential for developing human-computer interaction systems. Early methods
focus on extracting appearance and geometry features associated with human
affects while ignoring the latent semantic information among individual facial
changes, leading to limited performance and generalization. Recent work
attempts to establish a graph-based representation to model these semantic
relationships and develop frameworks to leverage them for various FAA tasks.
This paper provides a comprehensive review of graph-based FAA, including the
evolution of algorithms and their applications. First, the FAA background
knowledge is introduced, especially on the role of the graph. We then discuss
approaches widely used for graph-based affective representation in literature
and show a trend towards graph construction. For the relational reasoning in
graph-based FAA, existing studies are categorized according to their non-deep
or deep learning methods, emphasizing the latest graph neural networks.
Performance comparisons of the state-of-the-art graph-based FAA methods are
also summarized. Finally, we discuss the challenges and potential directions.
As far as we know, this is the first survey of graph-based FAA methods. Our
findings can serve as a reference for future research in this field.

###### Index Terms:

Facial Expression Recognition, Micro-expression Recognition, Action Unit
Detection, Graph Representation, Graph Relational Reasoning, Graph Neural
Network.

##  1 Introduction

Facial affects relate to $55\%$ of messages when people perceive others’
feelings and attitudes [1, 2] because it conveys critical information that
reflects emotional states and reactions in human communications [3, 4, 5].
Many facial affect analysis (FAA) methods have been explored during the past
decade, benefiting from interdisciplinary studies of affective computing,
computer vision, and psychology [6, 7, 8]. Some have been extended to many
applications, including medical diagnosis [9], social media [10], and video
generation [11]. Meanwhile, competitions such as FERA [12], EmotiW [13], Aff-
Wild [14], ABAW [15], EmotioNet [16], AVEC [17], and MuSe [18] are regularly
held to evaluate the latest progress and propose frontier research trends.

Historically, FAA methods have undergone a series of evolutions. Initial
studies usually rely on hand-crafted design or classic machine learning to
obtain useful affective features without structural information [19, 7].
Psychological findings indicate that the human cognition of facial information
is realized through a dual system composed of analytic processing and holistic
processing [4]. The former acquires multi-dimensional cluster features by
analyzing local areas, while the latter aims to generate a holistic
representation to perceive the overall structure [4, 20]. Such an analytic-
holistic working system is similar to a topology-like structure, so it is
reasonable for machine vision researchers to model it into a graph.
Accordingly, many state-of-the-art studies have been dedicated to generating a
facial graph with local-to-global affective features [21, 22, 23, 24].

If the above evidence reveals the feasibility of using graph-based methods for
FAA, research on how facial muscles participate in affective expression
further demonstrates its possibility as a necessary condition [25, 26, 27].
There are latent relationships among different facial areas and contexts,
which are vital clues [28, 29]. A few non-graph-based deep models have partly
captured these relationships and improved performance [30, 31, 32]. The
underlying assumption is that explicit mappings that reflect this relationship
can be directly learned [33]. However, these mappings are not solid enough in
the real world because they differ from subject to subject and even from one
condition to another [34, 35]. Recently, graph-based methods have shown that
they represent facial anatomy and simultaneously fit latent relationships in
facial affects [36, 37, 38]. Some pilot studies have also suggested that the
graph-based method can even move beyond to deal with challenging tasks such as
analyzing occluded faces [39, 40] and ambiguous facial affects [41, 42].

By searching on Google Scholar using keywords of ’graph’ and Index Terms in
this survey, we have counted the number of relevant published papers from 2010
to the present. As presented in Fig. 1, the graph-based FAA has gained
increasing attention, especially in the past five years (publications in 2021
increased by 600 year-on-year).

Figure 1: The growth trend of papers related to graph-based FAA.

Based on theoretical support, outstanding performance and quantity of existing
work, and potential for future development, it is necessary to review the
state of graph-based FAA methods. Although many reviews have discussed FFA’s
historical evolutions [7, 43, 33] and recent advances [44, 45, 46], including
some on specific problems like occluded expression [47], multi-modal affect
[48] and micro-expression [49], this is the FIRST systematic and in-depth
survey for the graph-based FAA field as far as we know. We emphasize
representative research proposed after 2010. The goal is to present a novel
perspective on FAA and its latest trends.

This review is organized as follows: Section 2 provides a brief background on
FAA and discusses the unique role of graph-based methods in FAA research.
Section 3 presents a taxonomy of mainstream graph-based methods for affective
representation. Section 4 reviews classical and advanced approaches for graph
relational reasoning and discusses their pros and cons in FAA tasks. Section 5
summarizes public databases, main FAA applications, and current challenges
based on a detailed comparison of related literature. Finally, Section 6
concludes with a general discussion and identifies potential directions.

##  2 Facial Affect Analysis

###  2.1 Affective Desription Model

As early as the 1970s, Ekman and Friesen [50] proposed the definition of six
basic affects, i.e., _happiness_ , _sadness_ , _fear_ , _anger_ , _disgust_ ,
and _surprise_ , based on an assumption of the universality of human affective
display [25]. In addition, compound affects [51] defined by different
combinations of basic affects (e.g., _sadly surprise_ and _happily surprise_)
are proposed to depict more complex affective situations [52]. Another kind of
famous description, called Facial Action Coding System (FACS), is designed for
a broader range of affects, which consists of a set of atomic Action Units
(AUs) [20, 29]. Fig. 2 shows an example of six basic affects plus _neutral_
and activated AUs in each facial affect. Besides categorical models, a
continuous affective model named VAD Emotional State Model [53] is also
suggested [54, 55, 56]. The VAD model has three dimensions, i.e., valence (how
positive or negative an affect is), arousal (the activation intensity of an
affect), and dominance (how submissive or in-control a person is in an
affective display). Recent studies consider that the continuous model is more
appropriate for describing dynamic changes of human affects in the real-world
[57, 44]. Please refer to [7, 43] for a more detailed discussion about this
topic.

Figure 2: An example of basic facial affects and related AU marks. AU0 denotes
no activated AU. All annotations are made by FACS certificated experts. Images
are from BU-4DFE [58] database.

###  2.2 General Pipeline

A standard FAA method can be broken down into fundamental components: face
preprocessing, affective representation, and task analysis. As a new branch of
FAA, the graph-based method also follows this generic pipeline (see Fig. 3).
Face detection and registration are two necessary pre-steps that first locate
faces and normalize facial variations, sometimes also providing facial
landmarks [59, 60]. Fig. 4 presents an illustration of the preprocessing
steps. Early methods like _Viola and Jones_ [61], _Mixtures of Trees_ [62],
and _Active Appearance Model (AAM)_ [63] have been widely used for this
purpose. Recently, cascaded deep approaches with real-time performance are
popular, such as _Multi-Task Cascaded Convolutional Network_ [64], _Hyperface_
[65], and _Supervision by Registration and Triangulation_ [66]. Please refer
to [67, 68, 69] for more specific information.

Figure 3: The pipeline of graph-based facial affect analysis methods. Figure
4: An illustration of face detection and face registration. The _happy_ image
is from CK+ database [70].

Compared to other existing methods, the graph-based FAA pays more attention to
representing facial affects with graphs and obtaining affective features from
such representation by graph reasoning.

In mathematical terms, a graph can be denoted as $G=(V,E)$. The node set $V$
contains all the representations of the entities in the graph, and the edge
set $E$ contains all the structure information between two entities. Thus,
when $E$ is empty, $G$ becomes an unstructured collection of entities (e.g.,
independent local facial areas [30]). Meanwhile, we could also define some
initial graph structure ahead of the relational model, which is a general
practice in many affective graph representations [71, 39, 24].

Given this unstructured collection, performing relational reasoning requires
the model to infer the structure of these entities before predicting the
property or category of an object. Naturally, generic approaches need to be
adjusted depending on affective graph representations or propose new graph-
based approaches to infer the latent relationship and extract the final
affective feature.

The two components can perform separately or arranged as an end-to-end
framework. They are expected to exhibit better performance and generalization
capability by manually or automatically providing richer information through
prior knowledge. Hence, the advantages and limitations of different graph
generation methods and their relational reasoning approaches are two main
topics of this survey.

##  3 Graph-based Affective Representations

Affective representation is a crucial procedure for most graph-based FAA
methods. Depending on the domain that an affective graph models, we categorize
the strategy as Spatial graphs, Spatio-temporal graphs, AU-level graphs, and
Sample-level graphs. Fig. 5 illustrates a detailed summary of the literature
using different graph representations. Note that many graph-based
representations contain pre-extracted geometric or/and appearance features.
Whether hand-crafted or learned, these feature descriptors are not essentially
different from those used in non-graph-based affective representations.
Interested readers can refer to [7, 43, 45] for a systematic understanding of
this topic.

Figure 5: Taxonomy for Graph-based Facial Representation.

_Kaltwang et al. 72, Zhao et al. 73Liu et al. 35_ _Mohseni et al. 74Zhong et
al. 21_
_Liu et al. 75, Kumar and Bhanu 71_
_Dapogny et al. 39Lei et al. 76_ _Zhong et al. 21, Lei et al. 76_ _Mohseni et
al. 74, Kumar and Bhanu 71_
_Zhao et al. 73_ _Dapogny et al. 39Liu et al. 35_
_Pei and Zha 77, Kotsia and Pitas 78_
_Zhang et al. 23Liu et al. 79_
_Jin et al. 80Fan et al. 37_
_Liu et al. 81_ _Yao et al. 82_ _Zhang et al. 23, Yao et al. 82_
_Liu et al. 81Fan et al. 37_
_Liu et al. 79_ _Zafeiriou and Pitas 83Zhang et al. 84_
_Xie et al. 85_ _Zhang et al. 84_
_Xie et al. 85Zafeiriou and Pitas 83_
_Zhou et al. 86, 40Chen et al. 87_
_Chen et al. 88_ _Rivera and Chae 89_ _Liu et al. 90Shirian et al. 91_ _Zhou
et al. 86, 40, Chen et al. 87Chen et al. 88_ _Rivera and Chae 89Liu et al. 90,
Shirian et al. 91_ _Tong et al. 92, Zhu et al. 93, Lei et al. 94Cui et al. 41_
_Zhu et al. 93, Cui et al. 41, Tong et al. 92_
_Lei et al. 94_ _Li et al. 22, Corneanu et al. 95Niu et al. 96, Xie et al. 38_
_Chen et al. 42, Song et al. 97, 24, 98Walecki et al. 99_ _Walecki et al.
99Song et al. 97Song et al. 24, 98_
_Li et al. 22, Xie et al. 38Niu et al. 96Chen et al. 42_ _Chien et al. 100Chen
et al. 101He and Jin 102_

###  3.1 Spatial Graph Representations

Non-graph-based spatial methods usually treat a facial affect as a whole
representation or pay attention to variations among main face components or
crucial facial parts [103, 30, 104]. For spatial affective graphs, facial
changes are considered while their co-occurring relationships and affective
semantics are represented as essential cues [82, 21, 76]. These approaches can
be divided into landmark-level graphs and region-level graphs. Fig. 6
illustrates frameworks of different spatial graph representations.

Figure 6: Spatial graphs. (a) Landmark-level graph with FACS-based edges [74];
(b) Landmark-level graph with automatic triangle edges [39]; (c) ROI graph
based on correlation without landmarks [82]; (d) NPI graph with individual
nodes [83]. Zoom in for better view.

####  3.1.1 Landmark-level graphs

Facial landmarks are one of the most critical geometry that reflects the shape
of face components and the structure of facial anatomy [105]. Thus, it is
natural to use facial landmarks as base nodes to generate a graph
representation.

Limited by the detection performance, only a few landmarks that locate basic
face components were applied in early graph representations [106]. Recently,
graphs using more facial landmarks (e.g., 68 landmarks [107]) are proposed to
depict fine-grained facial shapes. For example, in [35] and [73], the authors
associated 68 landmarks with the AUs in FACS and made graph-based
representations. The difference is that the former additionally employed local
appearance features extracted by _Histograms of Oriented Gradients (HOG)_
[108] as node attributes; the latter proposed three landmark knowledge
encoding strategies for enhanced geometric representations. Alternatively,
[72] formulated a _Latent Tree (LT)_ where 66 landmarks were set as parts of
leaf nodes accompanied by several other leaf nodes of AU targets and hidden
variables, which reflected the joint distribution of targets and features.

Furthermore, some current methods select landmarks with significant
contributions to avoid redundant information [76, 109]. Landmarks locating
external contour and nose are frequently discarded [74, 39] (see Figs. 6a, b)
because they are considered irrelevant to facial affects. [21] chose to remove
the landmarks of the facial outline and applied a small window around each
remaining landmark as one graph node, while the local features were extracted
by _Gabor_ filter [110]. Since these local areas were segmented to introduce
facial appearance into the graph representation rather than as independent
nodes, similar to [35], these methods are still classified in landmark-level
graphs. On the other hand, adding extra reasonable landmarks was designed to
generate comprehensive graph representations [71, 111], which could keep an
appropriate dimension and represent sufficient affective information.

A fully connected graph is the most intuitive way to form edges [21, 76].
However, the number of edges is $n(n-1)/2$ for a complete graph with $n$
nodes, which means the complexity of the spatial relationship will increase as
the number of nodes increases. This positive correlation is not helpful
because landmarks in a facial component mostly move in concert rather than
arbitrarily when conveying facial affects [112]. Studies of point-light
displays in emotion perception also show that more complex representations
seem to be redundant [113]. To this end, work like [74, 71, 111, 73] manually
reduced edges based on muscle anatomy and FACS. Another type of approach is
exploiting triangulation algorithms [39], such as _Delaunay_ triangulation
[35], to generate graph edges consistent with true facial muscle distribution
and uniform for different subjects. Similarly, the landmark-level graph with
triangulation is also utilized in generating a sparse or dense facial mesh for
3D FAA [77, 78]. The Euclidean distance is the simplest and most dominant
metric for edge attributes of the above facial graphs, even with multiple
normalization methods like inner-eyes distance [35, 73]. The _Hop_ distance
has also been explored as edge attributes to model spatial relationships [86,
40]. Besides, several learning-based edge generation methods (e.g., _LT_ [72],
_Conditional Random Field (CRF)_ [99]) have been proposed to extract semantic
information from facial graphs automatically. This part is discussed in detail
in Sec. 4.1 and 4.4.

####  3.1.2 Region-level graphs

Like geometric information, appearance information, especially in local facial
regions, can also contribute to FAA [114, 115]. Using graph structures is an
excellent choice to encode spatial relationships while representing texture
changes in facial components [82, 83]. There are two categories of region-
level affective graphs: region of interest (ROI) graphs and non-prior
information (NPI) graphs.

ROI graphs partition a set of facial areas as graph nodes related to affective
display. Coordinates of facial landmarks are commonly applied to locate and
segment ROIs. Unlike a few landmark-level graphs that only use texture near
all landmarks as supplementary information, ROI graphs explicitly select
meaningful areas as graph nodes, and edges do not entirely depend on
established landmark relationships. [23] employed a _High-Resolution Network
(HRN)_ [116] to regress ROI maps spotted by representative landmarks. Each
spatial location in the extracted feature map was considered one graph node,
while edges were induced among node pairs according to mappings between ROIs
and AUs. Another example in [37] utilized feature maps of landmark-based ROIs
outputted by the _ResNet50_ [117] as nodes to construct a _K-Nearest-Neighbor
(KNN)_ graph. For each node, its pair-wise semantic similarities were
calculated, and the nodes with the closest _Euclidean_ distance were connected
as initial edges. Similarly, [81] also employed landmark-based ROIs, but the
_KNN_ graph was generated in _optical-flow_ space to encode the local manifold
structure for a sparse representation [75]. Due to chained reactions among
multiple AUs and the symmetrical structure of the human face, [80] proposed a
parts-based graph that had manually linked edges by taking FACS and landmarks
as references. The nodes were ROIs with _Local Binary Pattern (LBP)_ [19] or
deep features as attributes. In addition, the method of obtaining ROIs without
relying on facial landmarks has also been studied [82] (see Fig. 6c).

Different from ROI graph representations, nodes in NPI graphs are evenly
distributed in raw images or generated in a fully automatic manner without
external knowledge. [83] created a reference bunch graph by evenly overlaying
a rectangular graph on object images (see Fig. 6d). _Gabor_ filters were
utilized for each graph node to compute a set of feature vectors for different
facial instances. Recently, several methods have tried to introduce regions
beyond facial parts or single face images as context nodes. [84] exploited the
_Region Proposal Network (RPN)_ [118] with _VGG16_ [119] to extract regions-
level nodes, including the target face and its contexts, while edges were
affective relationships calculated based on feature vectors. [85] built two
NPI graphs for cross-domain FAA. First, holistic and local features were
extracted as nodes for source and target domains. Then, global-to-global
connection, global-to-local connection, and local-to-local connection were
computed according to statistical feature distribution acquired by _K-means_
algorithm.

###  3.2 Spatio-Temporal Graph Representations

Spatio-temporal representations deal with a sequence of frames within a
temporal window and describe the dynamic evolution of facial variations [120].
In particular, introducing temporal information allows nodes to interact with
each other at different times and generates a more complex affective graph.
Fig. 7 presents frameworks of various spatio-temporal graph representations.

Figure 7: Spatio-temporal graphs. (a) Landmark-level graph with adaptive edges
[40]; (b) Region-level graph with edges based on transitional masks [89]; (c)
Frame-level graph [90]. Zoom in for better view.

Extend spatial graphs to the spatio-temporal domain is currently the main
route. [89] exploited weighted compass masks to obtain 2D directional number
responses and 3D space-time directional edge responses corresponding to each
of the symmetry planes of a cube. The two masks of given local neighborhoods
were nodes in a spatio-temporal _Directional Number Transitional Graph (DNG)_
, which could represent salient facial changes and statistic frequency of
affective behaviors over time (see Fig. 7a).

Several representations have been proposed to define temporal connections
between landmarks, which can be seen as landmark-level spatio-temporal graphs.
[88] developed a context-aware facial multi-graph where intra-face edges were
initialized based on morphological and muscular relationships, and inter-frame
edges were created by linking the same node between consecutive frames.
Similar landmark-based edge initialization in the temporal domain was also
utilized in [87, 86]. In [40], authors introduced a connectivity inference
block that could automatically generate dynamic edges for a spatio-temporal
situational graph of part-occluded affective faces (see Fig. 7b).

Unlike landmark-level graphs, [90] first extracted a holistic feature of each
frame and set them as individual nodes to establish a fully connected graph
(see Fig. 7c), which could be seen as a frame-level spatio-temporal graph.
Similar work includes [91] that took _Discrete Cosine Transform (DCT)_
features as node attributes. Edge connections of these methods would be
established by learning the long-term dependency of nodes in time series
(discussed in Sec. 4).

###  3.3 AU-level Graph Representations

Apart from using knowledge of AUs and FACS in the above two types of affective
graphs, many graph-based representations have been proposed to model affective
information from the perspective of AUs themselves. We divide these approaches
into two categories: AU-label graph and AU-map graph. Fig. 8 shows frameworks
of different AU-level graph representations.

Figure 8: AU-level and Sample-level graph representations. (a) AU-label graph
with edges generated from training data [92]; (b) AU-map graph with FACS based
edges [22]; (c) Auxiliary graphs of AUs and landmarks [42]; (d) Sample-level
multi-modal graph of visual and physiological signals [100]. Zoom in for
better view.

####  3.3.1 AU-label graphs

Unlike spatial and spatio-temporal graphs, AU-label graphs were built from the
label distribution of training data [93, 36]. [92] computed the co-occurrence
and co-absence dependency between every AU pair from the existing database
(see Fig. 8a). Since the dependency is not always symmetric, these AU label
relationships were used as edges to construct a _Directed Acyclic Graph
(DAG)_. In [94], an AU-label graph was built with a data-driven asymmetrical
adjacency matrix that denoted the conditional probability of co-occurring AU
pairs. AU labels were transformed into high-dimensional node vectors as node
attributes [121]. On the other hand, [41] established a _DAG_ where object-
level labels (affect categories) and property-level labels (AUs) were regarded
as parent nodes and child nodes, respectively. The conditional probability
distribution of each node to its parents was measured to obtain graph edges
for correcting existing labels and generating unknown labels. A similar idea
was achieved in [42] to boost affective feature learning in large-scale FAA
databases (see Fig. 8b).

####  3.3.2 AU-map graphs

AU-map graphs are intuitively close to region-level spatial graphs, especially
ROI graphs, because they both employ local feature maps as graph nodes. [22]
is an example in between. Twelve AUs features were learned through landmark-
based ROI features cropped from a multi-scale global appearance feature [119].
These AU features and the AU relationships gathered from training data and
manually pre-defined edge connections [122] were combined to construct a
knowledge graph (see Fig. 8c). However, the significant difference is that AUs
define a set of facial muscle actions, which means there might be multiple AUs
in the same ROI. Like in Fig. 2, AU12 and AU15 co-occur at lip corners but
refer to ’ _puller_ ’ and ’ _depressor_ ’, respectively. Therefore, for many
AU graphs, their definition of nodes is independent of those in ROI graphs,
even though they are similar in feature map extraction. For instance, graph
nodes in [38] were AU features directly obtained by _ResNet_ without defining
ROIs. The homologous protocol was also conducted in [98].

Some special AU-map graphs have been proposed to introduce structure learning
for more complex FAA tasks. For AU intensity estimation, [99] trained a
_Convolutional Neural Network (CNN)_ to learn deep AU features from multiple
databases jointly. The _copula functions_ [123] were applied to model pair-
wise AU dependencies in a _CRF_ graph. In addition, _Bayesian networks (BNs)_
are also used to capture the AU inherent dependencies for this task [98, 24].
To account for indistinguishable affective faces, [95] designed a _VGG_ -like
patch prediction module plus a fusion module to predict the probability of
each AU. A prior knowledge taken from the given databases and a mutual gating
strategy were used simultaneously to generate initial edge connections. To
model uncertainty samples in real-world databases, [97] established an
uncertain graph, in which a weighted probabilistic mask that followed
_Gaussian_ distribution was imposed on each AU feature map. By doing this, the
importance of edges and the underlying uncertain information could be encoded
in the graph representation. Another attempt in [96] boosted semi-supervised
AU recognition for labeled and unlabeled face images. The parameters of two AU
classifiers were used as graph nodes to share the latent relationships among
AUs.

###  3.4 Sample-level Graph Representations

Recently, several graph representations beyond a single sample have been
proposed, which indicates that this is still an open research field. In [102],
a correlation graph with word-embedded affective labels as nodes was built for
distribution learning. Its edges could be generated either by psychologically
normalized _Gaussian_ function or conditional probabilities. To combine
signals from multiple corpora, [100] proposed a dual-branch framework, in
which the visual semantic features were extracted in source and target sets.
These features were then retrieved with correlation coefficients to generate
positive edge connections for a learnable visual semantic graph (see Fig. 8d).
Besides, [101] constructed a _KNN_ graph with edges of binary weights to
preserve the intrinsic geometrical structure of source and target data, which
can seek more latent common information to reduce the distribution difference
and make representations more discriminative.

###  3.5 Discussion

As a significant part of the graph-based FAA method, different affective graph
representations have their merits, shortcomings, and requirements (see Table
I).

TABLE I: An Overview of Affective Graph Representations Category |  Branch |  Strength |  Limitation |  Demand
---|---|---|---|---
Spatial | Landmark-level |  Effective facial geometry embedding; Flexible structural relationships | Sensitive to the landmark detection accuracy | High-quality face registration
Region-level |  Versatile local texture extraction; Underlying correlation beyond locations |  Landmark sensitive for related ROI graphs; Redundant or missing regions for NPI graphs | Suitable for various situations
Spatio-Temporal | Extra dynamic evolution information | Relatively fixed edges in the time domain | Video/Sequence input
AU-level | AU-label |  Meaningful semantic dependencies; Explicit prior knowledge introduction | Cannot be an end-to-end framework | Reliable & sufficient AU annotations
AU-map | Unstable co-occurring distributions
Sample-level |  Modular to existing architectures; Cross-corpus information | Lack of in-face modelling | Large-scale/Multiple databases

_Spatial graph representations:_ Conceptually, landmark-level graphs model the
facial shape variations of fiducial points and easily generate the internal
structural relationships of different affective displays. However, most
methods are sensitive to facial landmarks’ detection errors, thereby failing
in uncontrolled conditions. On the other hand, the selection of landmarks and
the connection of edges have not yet formed a standard rule. Their effects on
the graph representation are rarely reported, even though some FACS-based
strategies have been designed. Region-level graphs explicitly regard an
affective face as multiple local crucial facial areas compared with landmark-
level graphs. The spatial relationships among selected regions are measured
through feature similarity instead of manual initialization based on facial
geometry. The circumstance resulting from inaccurate or unreasonable landmarks
will also impact related ROI graphs. Since most NPI graphs utilize a region
searching strategy, the problem is how to avoid the loss of target face and
how to exclude invalid regions.

_Spatio-temporal graph representations:_ With extra dynamic affective
information, spatio-temporal graphs can help aggregate evolution features in
continuous time. For landmark-level methods, the current initialization
strategy of edges is to link the facial landmark with the same index frame by
frame. Unfortunately, no research has been reported to learn the interaction
of landmarks with different indexes in the temporal dimension. Besides, in
addition to _Euclidean_ distance and _Hop_ distance, other edge attributes
measurement methods should also be explored to model the semantic context both
spatially and temporally. For the frame-level methods, embedding domain
knowledge related to affective behaviours like the muscular activity by graph
structure is not explicitly considered in recent work. Therefore, building a
hybrid spatio-temporal graph is a practical way to simultaneously encode the
two levels of affective information.

_AU-level graph representations:_ As a distinctive type, AU-level graphs
provide certain semantics of facial affects by representing each AU and its
co-occurrence dependency. The measurement criteria of AU correlations are
versatile but not general. Most AU-label graphs rely on the label
distributions of one or multiple given databases. Nevertheless, AU labelling
requires annotators with professional certificates and is a time-consuming
task that causes existing databases with AU annotations to be usually small-
scale. Therefore, the distribution from limited samples may not reflect the
true dependencies of individual AUs, and its impact on FAA still needs to be
assessed.

_Sample-level graph representations:_ Sample-level graphs are an appealing
field that introduces latent relationships in data distributions. Such
characteristic makes it convenient to integrate with existing FAA methods.
However, it also puts forward higher requirements for the diversity and
balance of samples. On the other hand, to the best of our knowledge, there is
no work combining sample-level and other in-face graphs to construct a joint
representation, which we think is a good topic.

##  4 Affective Graph Relational Reasoning

Generally, graph relational reasoning can be considered a two-step process,
i.e., understanding the structure from a certain group of entities and making
inferences of the system as a whole or the property within [124]. However,
things are slightly different in the case of graph-based FAA. Depending on
what kind of affective graph representation is exploited, the contribution of
graph relational reasoning can be either merged before the decision level with
other affective features or reflected as a collaborative way in the level of
feature learning.

In this Section, we review relational reasoning methods designed for affective
graph representations in four categories: _Dynamic BNs (DBNs)_ , classical
deep models, _Graph Neural Networks (GNNs)_ and non-deep machine learning
techniques.

###  4.1 Dynamic Bayesian Networks

_DBNs_ are often used to reason about relationships among facial displays like
AUs [125] and, of course, for AU-label graph representations. The _BN_ is a
_DAG_ that reflects a joint probability distribution among a set of variables.
In the work of [92, 36], a _DAG_ was manually initialized according to prior
knowledge, and than large databases were used to perform structure learning to
find the optimal probability graph structure. After that, the probabilities of
different AUs were inferred by learning the _DBN_. Following this idea, [93]
additionally integrated _DBN_ to a multi-task feature learning framework and
made the AU inference by calculating the joint probability of each category
node. Sometimes _DBN_ is also combined with some statistical methods to
explore different graph structures [98, 24], such as _Hidden Markov Models_
[126]. Another advanced research of _DBN_ is [41] that modeled the inherent
relationships between category labels and property labels. Its parameters were
utilized to denote the conditional probability distribution of each AU given
the facial affect. The wrong labels could be corrected by leveraging the
dependencies after the structure optimization.

###  4.2 Adjustments of Classical Deep Models

Before _GNNs_ are widely employed, many studies have adopted conventional
_Deep Neural Networks (DNNs)_ to process affective representations with the
graph structure. These deep models are not explicitly designed but can conduct
standard operations on structural graph data by adjusting the internal
architecture or applying an additional transformation to the input graph
representation. Fig. 9 shows examples of classical deep models for graph
relational reasoning.

Figure 9: Classical deep models for graph relational reasoning. (a) RNN [95];
(b) CNN [35]; (c) MLP [39]. Zoom in for better view.

####  4.2.1 Recurrent neural networks

The _Recurrent Neural Networks (RNNs)_ variant is one of the successfully
extended model types for handling graph structural inputs. Similar to random
walk, [21] applied a _Bidirectional RNN_ to deal with its landmark-level
spatial graph representation in a rigid order. The Gabor features of each
graph node was updated by multiplying with the average of the connected edges
to incorporate the structural information. Subsequently, the nodes were
iterated by the RNN with learnable parameters in forwarding and the backward
direction. In [95], the authors built a structure inference module to capture
AU relationships from an AU-map graph representation. Based on a collection of
interconnected recurrent structure inference units and a parameter sharing
_RNN_ , the mutual relationship between two nodes could be updated by
replicating an iterative message passing mechanism with the control of a
gating strategy (see Fig. 9a). Following the sequential idea of RNNs,[22]
exploited a _Gated Graph Neural Network (GGNN)_ [127] that calculated the
hidden state of the next time-step by jointly considering the current hidden
state of each node and adjacent nodes. The relational reasoning could be done
through the iterative update of _GGNN_ over its AU-map graph.

####  4.2.2 Convolutional neural networks

Unlike the sequential networks, [35] utilized a variant _CNN_ to process the
landmark-level spatial affective graph. Compared to standard convolution
architectures, the convolution layer in this study convolved over the diagonal
of a particular adjacency matrix to aggregate the information from multiple
nodes. Then a list of the diagonal convolution outputs was further processed
by three 1D sequential convolution layers. The corresponding pooling processes
were performed behind convolution operations to integrate feature sets (see
Fig. 9b). Another attempt for landmark-level spatial graph representations is
the _Graph Temporal Convolutional Networks (Graph-TCN)_ [76]. It followed the
idea of _TCNs_ that consisted of residual convolution, dilated causal
convolution, and weight normalization [128]. By using different dilation
factors, TCNs were applied to convolve the elements inside one node sequence
and from multiple node sequences. Thus, the _TCN_ for a node and _TCN_ for an
edge could be trained respectively to extract node feature and edge feature
simultaneously. Besides, [37] exploited a Semantic Correspondence Convolution
module to model the correlation among its region-level spatial graph. Based on
the assumption that the channels of co-occurring AUs might be activated
simultaneously, the _Dynamic Graph CNN (DG-CNN)_ [129] was applied on the
edges of the constructed _KNN_ graph to connect feature maps sharing similar
visual patterns. After the aggregation function, affective features were
obtained to estimate AU intensities.

####  4.2.3 Multilayer perceptron networks

As a vanilla architecture, _Multilayer Perceptrons (MLPs)_ has also been
explored. [39] employed a hierarchical _Auto-Encoder (AE)_ based on _MLPs_ to
capture relationships from a landmark-level spatial graph. Specifically, the
first stage learned the texture variations from the extracted _HOG_ features
for each node. In contrast, the second stage accumulated features of multiple
nodes whose appearance changes were closely related and computed the
confidence scores as the triangle-wise weights over edges. Finally, a _Random
Forest (RF)_ was used for facial affect classification and AU detection
simultaneously. In [98], a hybrid graph network composed of different dynamic
_MLPs_ performed multiple types of message passing, which provided more
complementary information for reasoning the positive and negative dependencies
among AU nodes (see Fig. 9c).

###  4.3 Graph Neural Networks

Unlike conventional deep learning frameworks mentioned in Sec. 4.2, _GNNs_ are
proposed to extend the ’depth’ from 2D image to graph structure and establish
an end-to-end learning framework instead of additional architecture adjustment
or data transformation [130]. Several types of _GNNs_ have successfully
addressed the relational reasoning of affective graph representations in FAA
methods. Fig. 10 illustrates several _GNN_ architectures for graph relational
reasoning.

Figure 10: GNNs for graph relational reasoning. (a) GCN as an auxiliary module
[96]; (b) GCN as a collaborative framework [24]; (C) STGCN [88] (d) Spectral
GCN [80]; (e) GAT [88]. Zoom in for better view.

####  4.3.1 Graph convolutional networks

_Graph Convolutional Networks (GCNs)_ , especially the spatial _GCN_ [131],
are the most popular _GNN_ in graph-based FAA research. Practically, _GCNs_
can be set as an auxiliary module [94, 73] or part of the collaborative
feature learning framework [24].

For the auxiliary module, _GCNs_ are applied immediately after the graph
representation. However, the outputs of relational reasoning are not directly
used for facial affect classification or AU detection but are later combined
with other deep features as a weighting factor (see Fig. 10a). [96] employed a
two-layer _GCN_ for message passing among different nodes in its AU-level
graph. Both the dependency of positive and negative samples were considered
and used to infer a link condition between any two nodes. The output of _GCN_
was formulated as a weight matrix of the pre-trained AU classifiers. Besides,
_GCNs_ can also be utilized following the above manner to execute relational
reasoning on atypical graph representations, such as multi-target graph [84],
distribution graph [102], and cross-domain graph [85].

For the collaborative framework, _GCNs_ usually inherit the previous node
feature learning model progressively (see Fig. 10b). Like in [79], a _GCN_
-based multi-label encoder was proposed to update features of each node over a
region-level spatial graph representation. The reasoning process was the same
as that in the auxiliary framework. Similar studies also include [23] and
[100]. In addition, to incorporate the dynamic in spatio-temporal graphs, [90]
set _GCNs_ as an imitation of attention mechanism or weighting mechanism to
share the most contributing features to explore the dependencies among frames.
After training, the structure helped nodes update features based on messages
from the peak frame and emphasize the concerned facial region. A more feasible
way is to apply _Spatial Temporal GCN (STGCN)_ [132] on spatio-temporal graphs
[86, 87, 40, 88] (see Fig. 10c). In their relational reasoning, features of
each node were generated with its neighbor nodes in the current frame and
consecutive frames by using spatial graph convolution and temporal
convolution, respectively.

Alternatively, the approach of spectral _GCN_ [133] has also been studied
[109]. [80] devised a lightweight GCN following the _Message Passing Neural
Network_ [134]. A learnable adjacency matrix was adapted to infer the spatial
dependencies of ROI nodes in different facial affects. [91] extended in
_Inception_ idea from standard _CNNs_ to spectral _GCNs_ that captured emotion
dynamics at multiple temporal scales. The yielded embeddings of different
dimensions were jointly learned over a classification loss and a graph
learning loss for the optimal graph structure.

####  4.3.2 Graph attention networks

_Graph Attention Networks (GATs)_ aim to strengthen the node connections with
high contribution and offer a more flexible way to process the graph structure
[135]. [97] introduced an uncertain _GNN_ with _GAT_ as the backbone. The goal
is to select valuable edges, depress noisy edges, and learn AU dependencies on
its AU-map graph. In addition, the underlying uncertainties were considered in
a probabilistic way, close to the idea of _Bayesian_ methods in _GNN_ [136],
to alleviate the data imbalance by weighting the loss function. On the other
hand, _GAT_ collaboratively worked with _GCN_ in [71] to deal with two-stream
graph inputs. Compared to applying _GAT_ directly, [38] proposed a GNN that
added a self-attention graph pooling layer after three sequential _GCN_
layers. A similar block was done in [111] which revised the _GCN_ block with
channel and node attention. It improved the reasoning process on graph
representations because only important nodes would be aggregated, including
affective information and facial topology. To make nodes interact more
dynamically instead of using a constant graph structure, [88] applied a set of
learnable edge attention masks to the _STGCN_ for subtle adjustments of the
defined spatio-temporal graph representation (see Fig. 10e).

###  4.4 Non-deep Machine Learning Methods

Although refining deep features extracted by parameterized neural networks and
gradient-based methods is the mainstream, they require numerous training
samples for effective learning. Due to the insufficient data in the early
years or the purpose of efficient computation, many non-deep machine learning
techniques have been applied for affective graph relational reasoning. Graph
structure learning is one of the widely used approaches. In [72], the
reasoning of its spatial graph representation was conducted by _LT_ learning.
Parameters update and graph-edit of _LT_ structure were performed iteratively
to maximize the marginal _log_ -likelihood of a set of training data. [99]
employed _CRF_ to infer AU dependencies in an AU-map graph. The use of
_copula_ functions allowed it to model non-linear dependencies among nodes
easily. At the same time, an iterative balanced batch learning strategy was
introduced to optimize the most representative graph structure by updating
each set of parameters with batches. Approaches of graph feature selection are
also exploited in this part, such as _Graph Sparse Coding (GSC)_ [81, 101] and
_Elastic Graph Matching (EGM)_ [83]. These methods have provided a more
diverse concept for graph relational reasoning.

###  4.5 Discussion

Although all the methods above can achieve affective graph relational
reasoning, the choice has a causal relationship with the type of graph
representation (see Table II).

TABLE II: Causal Relationships between Graphs and Reasoning methods Category | Spatial | Spatio-Temporal | AU-level | Sample-level
---|---|---|---|---
DBNs |  |  | $\surd$ |
DNNs | $\surd$ | $\surd$ | $\surd$ |
GNNs | $\surd$ | $\surd$ | $\surd$ | $\surd$
Non-deep | $\surd$ | $\surd$ | $\surd$ | $\surd$

_Dynamic Bayesian network_ : Nearly half of AU-label graph representations
employ _DBNs_ as their relational reasoning model. However, the representation
quality highly relies on the available training data that need balanced label
distribution in positive-negative samples and categories. This strong
assumption will limit the effectiveness of node dependencies learned by
_DBNs_. Another problem is that _DBNs_ can only be combined with facial
features as a relatively independent module and are hard to integrate into an
end-to-end learning framework.

_Classical deep model_ : Standard deep models, including _CNNs_ , _RNNs_ and
_MLPs_ , have been explored to conduct graph relational reasoning before the
emergence of _GNNs_. Even if they are suitable for more graph representations
than _DBNs_ , these grid models focus more on local features. The additional
adjustments in input format or/and network architecture cause losses of node
information or let node messages only pass and update in a specific sequence,
which suppresses the global property represented by the graph. Thus, we think
the specifically designed networks like _GNNs_ will become dominant in this
part.

_Graph neural network_ : _GNNs_ are developing techniques that make full
advantages of graphs. Architectures with different focuses have been proposed
but have their flaws as well. For instance, _GCNs_ cannot handle directed
edges well (e.g., AU-level graphs), while _GATs_ only use the node links
without considering edge attributes (e.g., spatial graphs). Besides, due to
the low dimension of the nodes in affective graphs, too deep _GNNs_ may be
counterproductive. In addition, being an auxiliary block or part of the whole
framework will influence the construction of _GNNs_. Therefore, managing graph
representation and relational reasoning using _GNNs_ still need to be
explored.

_Non-deep methods_ : Non-deep machine learning has a place in early studies
and is even applied in recent work because no training is required. They
partly inspire advanced techniques like _DBNs_ and _GNNs_. Nevertheless, one
of the reasons they have been replaced is that these approaches need to be
designed separately to cope with different graph representations, similar to
hand-crafted feature extraction. Hence, it is not easy to form a general
framework. On the other hand, more training data and richer computing
resources allow deep models to perform more effective and higher-level
relational reasoning on affective graphs.

##  5 Applications and Performance

According to different description models of facial affects, the FAA can be
subdivided into multiple applications. The typical output of FAA systems is
the label of a basic facial affect or AUs. Recent research also extends the
goal to predict micro-expression or affective intensity labels or continuous
affects. This section compares and discusses graph-based FAA methods from four
main application categories: facial expression recognition, AU detection,
micro-expression recognition, and a few special applications. Due to page
limitation, we select most relevant and representative papers following these
standards: published in more well-known forums in the past five years; or
belonging to distinct branches of graph representation and reasoning for
diversity consideration.

###  5.1 Databases

Most FAA studies apply public databases of facial affect as validation
material. A comprehensive overview is presented in Table III. The
characteristics of these databases are listed from four aspects: samples,
attributes, graph-related properties, and certain contents. Fig. 11 exhibits
several examples of facial affects under different conditions. In addition,
for better interpreting the graph-based FAA, we summarize corresponding
elements (e.g., landmark coordinates, AU labels) self-carried by databases,
which are rarely considered in previous related surveys.

TABLE III: An Overview of Facial Affect Databases

Database ’year | Samples | Attributes |  Graph-based Properties3 | Special Contents4
---|---|---|---|---
Data Type | Subjects | Number |  Eli. & Sou.1 |  Affects2 |  B.B.1 | LM | AU | Dynamic
CK(+) ’10 | Sequences | 97(123) | 486(593) | P & Lab | 6B+N(+C) | $\bullet$ | $\bullet$ |  $\bullet$ \+ I | $\bullet$ | /
MMI ’10 | Images/Videos | 75 | 740/2900 | P & Lab | 6B+N | $\circ$ | $\circ$ | $\bullet$ |  $\bullet$ \+ D | Head pose
Oulu-CASIA ’11 | Sequences | 80 | 2880 | P & Lab | 6B | $\bullet$ | $\circ$ | $\circ$ | $\bullet$ | NIR
DISFA ’13 | Sequences | 27 | 130,000 | S & Lab | 6B+N | $\bullet$ | $\bullet$ |  $\bullet$ \+ I |  $\bullet$ \+ F | /
FER-2013 ’13 | Images | / | 35887 | S & Web | 6B+N | $\bullet$ | $\circ$ | $\circ$ | $\circ$ | Wild
SFEW 2.0 ’15 | Images | / | 1766 | S & Movie | 6B+N | $\bullet$ | $\bullet$ | $\circ$ | $\circ$ | Wild
AFEW 7.0 ’17 | Videos | / | 1809 | S & Movie | 6B+N | $\bullet$ | $\bullet$ | $\circ$ | $\bullet$ | Audio, Wild
BU-3DFE ’06 | Images | 100 | 2500 | P & Lab | 6B+N | $\bullet$ | $\bullet$ | $\circ$ | $\circ$ | 3D, Multi-view, I
BU-4DFE ’08 | Videos | 101 | 606 | P & Lab | 6B+N | $\bullet$ | $\bullet$ |  $\bullet$ \+ I | $\bullet$ | 3D, Multi-view
BP4D ’14 | Videos | 41 | 328 | S & Lab | 6B+E+P | $\bullet$ | $\bullet$ |  $\bullet$ \+ I |  $\bullet$ \+ F | 3D, Head pose
SMIC ’13 | Sequences | 16 | 164 | S & Lab |  3B†+N | $\bullet$ | $\circ$ | $\circ$ | $\bullet$ | Micro., NIR
CASME II ’14 | Sequences | 35 | 247 | S & Lab |  3B‡+R+O | $\bullet$ | $\circ$ | $\bullet$ |  $\bullet$ \+ D | Micro.
SAMM ’18 | Sequences | 32 | 159 | S & Lab | 6B+C | $\bullet$ | $\circ$ | $\bullet$ |  $\bullet$ \+ D | Micro.
CAS(ME)2 ’18 | Sequences | 22 | 300+57 | S & Lab |  3B†+O | $\bullet$ | $\circ$ | $\bullet$ |  $\bullet$ \+ D | Macro. & Micro.
EmotioNet ’16 | Images | / | 950,000 | S & Web | 6B+17Comp. | $\bullet$ | $\bullet$ |  $\bullet$ \+ I | $\circ$ | Wild
ExpW ’18 | Images | / | 91,793 | S & Web | 6B+N | $\bullet$ | $\circ$ | $\circ$ | $\circ$ | Multi-sub., Wild
RAF-DB ’19 | Images | / | 29672 | S & Web | 6B+N+11Comp. | $\bullet$ | $\bullet$ | $\circ$ | $\circ$ | Wild
AffectNet ’19 | Images | / | 420,299 | S & Web | 6B+N+C+O | $\bullet$ | $\bullet$ | $\circ$ | $\circ$ | V&A, Wild
EMOTIC ’19 | Images | / | 23,571 | S & Web | 6B+N+19Comp. | $\bullet$ | $\circ$ | $\circ$ | $\circ$ | V&A, Multi-sub., Wild

  * 1

Eli.: elicitation; Sou.: source; P: posed; S: spontaneous; B.B.: bounding
boxes; LM: landmarks; $\bullet$ = Yes, $\circ$ = No.

  * 2

6B: six basic affects; N: neutral; C: contempt; E: embarrassment; P: pain; O:
others; R: repression; 3B†: three basic affects (positive, negative,
surprise); 3B‡: three basic affects (happiness, disgust, surprise); Comp.:
compound affects.

  * 3

I: intensity annotation; D: onset-apex-offset annotation; F: frame-level
annotation.

  * 4

NIR: near-infrared; Multi-sub.: multiple subjects per image; Micro.: micro-
expression; Macro.: macro-expression; Wild: in-the-wild; V&A: valence and
arousal.

Figure 11: Facial affect databases. (a) Oulu-CASIA contains posed facial
affects; (b) SFEW 2.0 has facial affects under in-the-wild scenarios; (c) BP4D
provides 3D affective face images; (d) EMOTIC, multiple faces appear per image
with VAD annotations; (e) The SMIC collects images of spontaneous micro facial
affects in visual light and near infrared light; (f) DISFA offers frame-level
AU intensity labels.

Databases containing posed facial affects, including Extended Cohn-Kanade
Dataset (CK+) [137, 70], M&M Initiative Facial Expression Database (MMI)
[138], and Oulu-CASIA NIR&VIS Facial Expression Database (Oulu-CASIA) [139],
are chosen by early FAA methods. More challenging databases, such as FER-2013
[140], Static Facial Expression in the Wild (SFEW) 2.0 [141], and Acted Facial
Expression in the Wild (AFEW) 7.0 [142], tend to acquire spontaneous affective
data from complex and wild environments.

Some databases also contain intensity labels of facial affects and even AUs,
e.g., Denver Intensity of Spontaneous Facial Action Database (DISFA) [143],
Binghamton University 3D/4D Facial Expression Database (BU-3DFE/4DFE) [144,
145] and Binghamton-Pittsburgh 3D Dynamic Spontaneous Facial Expression
Database (BP4D) [146].

Another type of database is for micro-expressions. Participants are required
to keep a neutral face while watching videos associated with induction of
specific affects [147]. Following this setting, Spontaneous Micro Facial
Expression Database (SMIC) [148], Improved Chinese Academy of Sciences Micro-
Expression Database (CASME II) [149], Spontaneous Micro-Facial Movement
Database (SAMM) [150], Chinese Academy of Sciences Macro-Expression and Micro-
Expression Database (CAS(ME)2) [151] have been released. However, it is hard
to collect and annotate large-scale micro-expression data with uncontrolled
scenarios due to its subtle, rapid, and involuntary nature.

Recently, several large-scale databases have been developed to provide massive
data with spontaneous facial affects and in-the-wild conditions, such as Real-
World Affective Face Database (RAF-DB) [152], Large-Scale Face Expression in-
the-Wild dataset (ExpW) [10], EMOTIC [153], AffectNet [154], and EmotioNet
[155].

Concerning graph-based FAA, it is available to find and select suitable
databases with corresponding metadata, such as landmarks, AU labels, and
dynamics, for different graph representation purposes. However, existing
databases also have some shortcomings. On the one hand, not enough accurate AU
annotations are provided by in-the-wild databases, limiting AUs’ role in FAA.
On the other hand, there is still a blank in the dynamic large-scale affective
database field, so it is hard to use temporal information to generate
affective graph representation. Finally, databases about natural and
spontaneous facial affects in a continuous domain need more attention instead
of discrete categories.

###  5.2 Facial Expression Recognition

Facial expression recognition (FER), or macro-expression recognition, has been
working on basic facial affects classification. An inevitable trend of FER is
that the research focus has shifted from the early posed facial affects in
controlled conditions to the recent spontaneous facial affects in real
scenarios. In other words, recognizing the former is considered a solved
problem for FAA methods, including graph-based FER, which can be corroborated
from the results in Table IV. For example, the performance on the CK+ database
is very close to $100\%$ [78, 89, 90, 111].

TABLE IV: Performance summary of representative graph-based FER methods

References |  Prep.1 |  Representation2 | Reasoning |  Posed Database3 |  Wild Database | Validation4
---|---|---|---|---|---|---
B.B. | LM | Category | Node | Edge | Model | Classifier |  |
Kotsia and Pitas [78] | $\circ$ | 104 | $\mathcal{S:L}$ | $\mathbb{C}$ | $\bigtriangleup$ | _Tracking_ | _SVM_ |  CK _ar_ : 0.997 |  / | LO CV
Zafeiriou and Pitas [83] | $\bullet$ | / | $\mathcal{S:R}$ | _NM_ | _L_ | _Tracking_ | _EGM_ |  CK _ar_ : 0.971 |  / | LO CV
Mohseni et al. [74] | $\circ$ | 50 | $\mathcal{S:L}$ | $\mathbb{C}$ |  _M_ +$\mathbb{E}$ | _Tracking_ | _Adaboost_ |  MMI _ar_ : 0.877 |  / | 10F CV
Rivera and Chae [89] | $\bullet$ | / | $\mathcal{ST:R}$ | _Histograms_ | _DNG_ | _Tracking_ | _SVM_ |  CK+ _ar_ : 1; MMI _ar_ : 0.976; Oulu _ar_ : 0.984 |  / | 10F CV
Yao et al. [82] | $\bullet$ | / | $\mathcal{S:R}$ | _LBP_ | _L_ | / |  _SVM_ _CNN_ |  / |  SFEW _ar_ : 0.5538; AFEW _ar_ : 0.5380; | HO
Dapogny et al. [39] | $\bullet$ | 49 | $\mathcal{S:L}$ |  $\mathbb{C}$+_HOG_ | $\bigtriangleup$ | _AE_ | _RF_ |  CK+ _ar_ : 0.934; BU-4DFE _ar_ : 0.750 |  SFEW _ar_ : 0.371 |  10F-SI CV /HO
Zhong et al. [21] | $\bullet$ | 46 | $\mathcal{S:L}$ | _Gabor_ |  _F_ +$\mathbb{E}$ | _RNN_ | _Softmax_ |  CK+ _ar_ : 0.9827; MMI _ar_ : 0.9444; Oulu _ar_ : 0.9306 |  / | 10F-SI CV
Chen et al. [42] | $\bullet$ | / | $\mathcal{AU:B}$ | $\Phi$ | _KNN_ | / | _Softmax_ |  CK+ _ar_ : 0.9308; MMI _ar_ : 0.7049; Oulu _ar_ : 0.6385 |  SFEW _ar_ : 0.5650; RAF _ar_ : 0.8553; AffNet: _ar_ : 0.5935 |  CD (AffNet+RAF)
Liu et al. [35] | $\bullet$ | 68 | $\mathcal{S:L}$ |  $\mathbb{C}$+_HOG_ | $\bigtriangleup$ | _CNN_ | _Softmax_ |  CK+ _ar_ : 0.9767; MMI _ar_ : 0.8011 |  SFEW _ar_ : 0.5536 |  10F-SI CV /HO
Xie et al. [85] | $\bullet$ | / | $\mathcal{S:R}$ | _ResNet_ | _K-means_ | _GCN_ | _Softmax_ |  CK+ _ar_ : 0.8527; JAFEE [156] _ar_ : 0.6150 |  SFEW _ar_ : 0.5643; FER2013 _ar_ : 0.5895 ExpW _ar_ : 68.50% | CD (RAF)
Zhou et al. [86] | $\bullet$ | 34 | $\mathcal{ST:L}$ |  $\mathbb{C}$+_HOG_ |  _M_ +$\mathbb{H}$ | _GCN_ | _Softmax_ |  CK+ _ar_ : 0.9863; Oulu _ar_ : 0.8723 |  / | 10F-SI CV
Liu et al. [90] | $\bullet$ | / | $\mathcal{ST:F}$ | _VGG_ | _F_ | _GCN_ | _LSTM_ |  CK+ _ar_ : 0.9954; MMI _ar_ : 0.8589; Oulu _ar_ : 0.9104 |  / | 10F-SI CV
Zhou et al. [40] | $\bullet$ | 44 | $\mathcal{ST:L}$ |  $\mathbb{C}$+_HOG_ |  _L_ +$\mathbb{H}$ | _GCN_ | _Softmax_ |  CK+ _ar_ : 0.9892; Oulu _ar_ : 0.8750 |  AFEW _ar_ : 0.4512 |  10F-SI CV /HO
Cui et al. [36] | $\circ$ | / | $\mathcal{AU:B}$ | _CNN_ | $\Phi$ | _DBN_ | _Softmax_ |  CK+ _ar_ : 0.9759; BP4D _ar_ : 0.8382; MMI _ar_ : 0.8490 |  EmotioNet _ar_ : 0.9555 | 5F-SI CV
Liu et al. [111] | $\bullet$ | 40 | $\mathcal{S:L}$ |  $\mathbb{C}$+_HOG_ +_Gabor_ /$\mathbb{C}$+CNN |  _L_ +$\mathbb{H}$ /_L_ +$\mathbb{E}$ | _GCN_ | _Softmax_ |  CK+ _ar_ : 0.9923; MMI _ar_ : 0.8575; Oulu _ar_ : 0.9088 |  SFEW _ar_ : 0.5742 RAF _ar_ : 0.8713 |  10F-SI CV /HO
Chen et al. [101] | $\circ$ | / | $\mathcal{X}$ | _HOG_ | _KNN_ | _GSC_ | _SVM_ |  CK+ (JAFEE) _ar_ : 0.7171; JAFEE (CK+) _ar_ : 0.5667; Oulu (CK+) _ar_ : 0.4834; CK+ (Oulu) _ar_ : 0.7756 |  / | CD
Shirian et al. [91] | $\circ$ | 68 | $\mathcal{ST:F}$ | $\mathbb{C}$ | _F_ | _GCN_ | _Softmax_ |  RML [157] _ar_ : 0.9411; eNTERFACE [158] _ar_ : 0.8749; RAVDESS [159] _ar_ : 0.8565 |  / | 10F CV
Jin et al. [80] | $\bullet$ | 68 | $\mathcal{S:R}$ | _LBP/AE_ | _F/M_ | _GCN_ | _Softmax_ |  CK+ _ar_ : 0.9432; Oulu _ar_ : 0.7328 |  RAF _ar_ : 0.5825 |  10F-SI CV /HO
Zhao et al. [73] | $\bullet$ | 68 | $\mathcal{S:L}$ |  $\mathbb{C}$+_CNN_ | _M_ | _GCN_ | _Softmax_ |  CK+ (RAF) _ar_ : 0.9320; MMI (RAF) _ar_ : 0.7439; Oulu (RAF) _ar_ : 0.6302 |  SFEW _ar_ : 0.5711 RAF _ar_ : 0.8752 |  10F-SI CV /HO
Rao et al. [109] | $\bullet$ | 68 | $\mathcal{S:L}$ | $\mathbb{C}$ | _M_ | _GCN_ | _Softmax_ |  CK+ _ar_ : 0.9868; JAFEE _ar_ : 0.9664 |  FER2013 _ar_ : 0.7806 RAF _ar_ : 0.8695 |  10F/LO CV /HO

  * 1

Prep.: processing; B.B.: bounding boxes; $\bullet$ = Yes, $\circ$ = No; LM:
landmarks.

  * 2

$\mathcal{S/ST/AU/X}$: {spatial; spatio-temporal; AU-level; sample-level}
representation; $\mathcal{:L/R/F}$: {landmark; region; frame}-level graph;
$\mathbb{C}$: landmark coordinates; $\Phi$: label distributions;
$\bigtriangleup$: triangulation; _L/M/F_ : {learning; manual; full}
connections; $\mathbb{E}$: Euclidean distance; $\mathbb{H}$: Hop distance.

  * 3

_ar_ : average accuracy rate; DB1 (DB2): train on database 2, test on database
1.

  * 4

CV: cross validation; LO: leave-one-subject-out; HO: holdout validation; 10F:
10-flods; SI: subject independent; CD (DB): cross database validation
(training database).

From the view of the representation, spatial graphs and spatio-temporal graphs
are dominant. Specifically, hand-crafted features (e.g., _LBP_ [82, 80],
_Gabor_ [21, 111], _HOG_ [35, 86, 101]) or deep-based features (e.g., _CNN_
[111, 73], _VGG_ [90], _ResNet_ [38]) are employed to enhance the node
representation similar to many non-graph FER methods [19, 30]. For reasoning
approaches, early studies prefer to capture the relations of an individual
node from predefined graph structures using tracking strategies [78, 83, 74,
89] or general machine learning models (e.g., _RF_ [39], _RNN_ [21], _CNN_
[35]). In the latest work, _GCNs_ become one of the mainstream choices in the
latest work and show state-of-the-art performances on posed and in-the-wild
databases [38, 111, 80, 73, 109]. Another observation is that the framework of
combining the spatio-temporal graph representation and _GNNs_ is getting more
attention in FER studies [86, 40, 90, 91].

Although many graph-based studies have shown improvements in predicting facial
affects, FER still has some potential topics. One thing is that the goal of
existing methods stays on classifying basic facial affects. No study of graph-
based methods to recognize compound affects (or mixture affects), whose labels
are provided by recent databases like RAF-DB and EmotioNet, is reported. One
possible solution is introducing AU-level graph representations that can
describe fine-grained macro-expressions with closer inter-class distances. The
other topic is practical graph-based representations due to the big gap
between the performance of current methods and the acceptable result in
practice when analyzing in-the-wild facial affects. In addition, since
existing databases lack sufficient dynamic annotated samples, the evaluation
of spatio-temporal graphs in large-scale conditions remains explored.

###  5.3 Action Unit Detection

The AU detection (AUD) facilitates a comprehensive analysis of the facial
affect and is typically formulated as a multi-task problem that learns a two-
class classification model for each AU. It can expand the recognition
categories of macro-expressions through the AU combination [72] and can be
used as a pre-step to enhance the recognition of micro-expressions [94].
Compared with graph-based FER, the wide usage of graph structures has a long
history in AUD [33] and has played a more dominant role. Table V summarizes
graph-based AUD methods including the performance comparison.

TABLE V: Performance summary of representative graph-based AUD methods

References |  Prep.1 |  Representation2 | Reasoning |  Database3 | Validation4
---|---|---|---|---|---
B.B. | LM | Category | Node | Edge | Model | Output |
Tong et al. [92] | $\bullet$ | / | $\mathcal{AU:B}$ | _Gabor_ | $\Phi$ | _DBN_ | _Adabst_ |  CK _ar_ : 0.9933; MMI (CK) _ar_ : 0.939 |  LO CV /CD
Zhu et al. [93] | $\bullet$ | 49 | $\mathcal{AU:B}$ |  $\mathbb{C}$+_Gabor_ | $\Phi$ | _DBN_ | _MPE_ |  CK+ _ar_ : 90.48%, _f_ 1: 0.7072; DISFA _ar_ : 93.56%, _f_ 1: 0.7095 | 2F, 10F CV
Kaltwang et al. [72] | $\circ$ | 66 | $\mathcal{S:L}$ | $\mathbb{C}$ | _L_ | _LT_ |  DISFA _corr_ : 0.43, _mse_ : 0.39, _icc_ : 0.36 | 9F CV
Walecki et al. [99] | $\circ$ | / | $\mathcal{AU:M}$ | _CNN_ | _L_ | _CRF_ | _ Ordinal Regression _ |  FERA 2015 [160] _icc_ : 0.63, _mae_ : 1.23; DISFA _icc_ : 0.45, _mae_ : 0.61 | HO
Corneanu et al. [95] | $\bullet$ | / | $\mathcal{AU:M}$ | _VGG_ | $\Phi$ | _RNN_ | _Sigmoid_ |  BP4D _f_ 1: 0.617; DISFA _f_ 1: 0.567 | 3F-SI CV
Dapogny et al. [39] | $\bullet$ | 49 | $\mathcal{S:L}$ |  $\mathbb{C}$+_HOG_ | $\bigtriangleup$ | _AE_ | _RF_ |  CK+ _auc_ : 0.953, _f_ 1: 0.788, _nf_ 1: 0.865; BP4D: _auc_ : 0.727, _f_ 1: 0.557, _nf_ 1: 0.636; DISFA _auc_ : 0.824, _f_ 1: 0.491 | 10F-SI CV
Li et al. [22] | $\bullet$ | 20 | $\mathcal{AU:M}$ |  $\mathbb{C}$+_VGG_ | _M+L_ | _GGNN_ | _FC_ |  BP4D _auc_ : 0.741, _f_ 1: 0.629; DISFA _auc_ : 0.807, _f_ 1: 0.559 | No report
Niu et al. [96] | $\bullet$ | / | $\mathcal{AU:M}$ | $\mathbb{W}$ | _L_ | _GCN_ | _FC_ |  BP4D _f_ 1: 0.598; EmotioNet _f_ 1: 0.681 | 3F-SI CV/ HO
Liu et al. [79] | $\bullet$ | 19 | $\mathcal{S:R}$ |  _CNN_ +_AE_ | _M_ | _GCN_ | _FC_ |  BP4D _auc_ : 0.873, _f_ 1: 0.628; DISFA _auc_ : 0.746, _f_ 1: 0.550 | 3F-SI CV
Fan et al. [37] | $\bullet$ | 20 | $\mathcal{S:R}$ | _ResNet_ |  KNN+$\mathbb{E}$ | _DG-CNN_ | _ Heatmap Regression _ |  BP4D _icc_ : 0.72, _mae_ : 0.58; DISFA _icc_ : 0.47, _mae_ : 0.20 | 3F-SI CV
Liu et al. [35] | $\bullet$ | 68 | $\mathcal{S:L}$ |  $\mathbb{C}$+_HOG_ | $\bigtriangleup$ | _CNN_ | _FC_ |  CK+ _auc_ : 0.929 | 10F-SI CV
Zhang et al. [23] | $\bullet$ | 18 | $\mathcal{S:R}$ | _HRN_ | _L_ | _GCN_ | _ Heatmap Regression _ |  BP4D _f_ 1: 0.635; DISFA _f_ 1: 0.620 | 3F-SI CV
Cui et al. [41] | $\circ$ | 51 | $\mathcal{AU:B}$ | _LBP_ | $\Phi$ | _DBN_ |  _LR_ _/CNN_ _/SVM_ |  CK+ _f_ 1: 0.830; BP4D _f_ 1: 0.687; EmotionNet _f_ 1: 0.626; MMI (CK+) _f_ 1: 0.532 |  5F-SI CV /CD
Cui et al. [36] | $\circ$ | / | $\mathcal{AU:B}$ | _VGG_ | $\Phi$ | _DBN_ | _FC_ |  CK+ _f_ 1: 0.74; BP4D _f_ 1: 0.57; MMI _f_ 1: 0.58 | 5F-SI CV
Song et al. [97] | $\bullet$ | / | $\mathcal{AU:M}$ | _ResNet_ | _Mask_ | _GAT_ | _Softmax_ |  BP4D _ar_ : 0.782, _f_ 1: 0.633; DISFA _ar_ : 0.934, _f_ 1: 0.600 | 3F-SI CV
Song et al. [98] | $\bullet$ | / | $\mathcal{AU:M}$ | _ResNet_ | $\Phi$ | _Hybrid GNN_ | _Softmax_ |  BP4D _f_ 1: 0.634; DISFA _f_ 1: 0.610 | 3F-SI CV
Song et al. [24] | $\bullet$ | / | $\mathcal{AU:M}$ | _ResNet_ | $\Phi$ | _GCN+LSTM_ | _FC_ |  FERA 2015 _icc_ : 0.72, _mae_ : 0.57; DISFA _icc_ : 0.56, _mae_ : 0.22 | 3F-SI CV
Chen et al. [88] | $\bullet$ | 68 | $\mathcal{ST:L}$ | _DCT+CNN_ | _M+Mask_ | _GCN_ | _Softmax_ |  BP4D _f_ 1: 0.649; DISFA _f_ 1: 0.658 | 3F-SI CV

  * 1

Prep.: processing; B.B.: bounding boxes; $\bullet$ = Yes, $\circ$ = No; LM:
landmarks.

  * 2

$\mathcal{S/ST/AU}$: {spatial; spatio-temporal; AU-level} representation;
$\mathcal{:L/R/B/M}$: {landmark; region; label; map}-level graph;
$\mathbb{C}$: landmark coordinates; $\Phi$: label distributions;
$\bigtriangleup$: triangulation; _L/M_ : {learning; manual} connections;
$\mathbb{E}$: Euclidean distance.

  * 3

_ar_ : average accuracy rate; _f_ 1: F1 score; _nf_ 1: F1-norm score; _corr_ :
Pearson correlation coefficient; _mae_ : mean absolute error; _mes_ : mean
squared error; _icc_ : intra-class correlation coefficient, ICC(3,1); _auc_ :
area under the receiver operating characteristic curve; DB1 (DB2): train on
database 2, test on database 1.

  * 4

CV: cross validation; LO: leave-one-subject-out; (K)F: k-folds; SI: subject
independent; HO: holdout validation; CD: cross database validation.

Specifically, spatial graphs and AU-level graphs are equally popular in the
representation part of AUD. Interestingly, no matter landmark-level or region-
level, all the spatial graphs constructed in the listed AUD methods employed
facial landmarks [72, 39, 79, 37, 35, 23], even for the spatio-temporal graph
[88]. The possible reason is that the landmark information is helpful and
practical for locating the facial areas where AUs may occur. In this setting,
their node representations were close to that in spatial graphs of FER
methods, which usually combined geometric coordinates with appearance features
(e.g., _HOG_ [39, 35]). Although some AUD methods using AU-level graphs also
exploited traditional features (e.g., _Gabor_ [92, 93], _LBP_ [41]) or deep
features (e.g., _VGG_ [95]) to introduce appearance information, their graph
representations were initialized from the AU label distribution of the
training set. Thus, the _DBN_ model has become popular in the relational
reasoning stage [92, 93, 41]. Another similar trend to graph-based FER is that
_GNNs_ have been widely utilized to learn the latent dependency among
individual AUs in recent studies, such as _GCN_ [96, 79, 23, 88], _GAT_ [97],
_GGNN_ [22], and _DG-CNN_ [37]. But the difference is that _fully-connected
(FC)_ layers [22, 96, 79, 35] or regression models [99, 37, 23] are often
applied for predicting labels instead of _softmax_ classifier [97, 88].

A particular line of AUD research analyzes the facial affects by estimating
the AU intensities, which could have greater information value in
understanding complex affective states [161]. Even though a few attempts in
estimating AU intensities based on graph structures have existed [72, 95, 37],
the study of using the latest spatio-temporal graph representations and _GNNs_
has not been reported. Another big challenge in AUD is few and imbalanced
samples. Recent graph-based methods using transfer learning [96, 41] or
uncertainty learning [97] were proposed to address this problem. They showed
an advantage of the graph-based method in this topic and are helpful to
implement AUD in large-scale unlabeled data.

###  5.4 Micro-Expression Recognition

Micro-expressions are fleeting and involuntary facial affects that people
usually exhibit in high stake situations when attempting to conceal or mask
their true feelings [147]. The earliest well-known studies came from [162] as
well as [163]. Generally, a micro-expression lasts only 1/25 to 1/2 seconds
long and is too subtle and fleeting for an untrained person to perceive.
Therefore, developing an automatic micro-expression recognition (MER) system
is valuable in reading human hidden affective states. Besides the short
duration, low intensity and localization characteristics also make it
challenging.

To this end, graph-based MER methods have been designed to address the above
challenges and have become appealing in the past two years [81], especially in
2020 [76, 38]. Table VI lists the reported performance of a few representative
recent studies of graph-based MER. These methods fall into the landmark-level
spatial graph [81, 76] and the AU-level graph [38] in terms of representation
types. For the former, their idea is to use landmarks to locate and analyze
specific facial areas to deal with the local response and the subtleness of
micro-expressions. The latter aims to infer the AU relationship to improve the
final performance. The difference in processing ideas is also reflected in the
reasoning procedure. Approaches like _GSC_ [81] and variant _CNNs_ [76] are
exploited in the landmark-level graph to integrate the individual node feature
representations. In comparison, _GCNs_ are employed to learn an optimal graph
structure of the AU dependency knowledge from training data and make
predictions. Nevertheless, one common thing is that all the methods consider
the local appearance in a spatio-temporal way by using _optical-flow_ or
_DNNs_.

TABLE VI: Performance summary of representative graph-based MER methods

References |  Prep.1 |  Representation2 | Reasoning |  Database3 | Validation4
---|---|---|---|---|---
B.B. | LM | Category | Node | Edge | Model | Classifier |
Liu et al. [81] | $\bullet$ | 66 | $\mathcal{S:R}$ | _Optical-flow_ | _KNN_ | _ GSC _ | _SVM_ |  SMIC (3 cl.) _ar_ : 0.6795, _f_ 1: 0.6844; CASME I [164] (4 cl.) _ar_ : 0.7219, _f_ 1: 0.7236; CASME II (5 cl.) _ar_ : 0.6356, _f_ 1: 0.6364 | LO CV
Lei et al. [76] | $\bullet$ | 28 | $\mathcal{S:L}$ | _TCN_ | _F_ | _Graph-TCN_ | _Softmax_ |  CASME II (5 cl.) _ar_ : 0.7398, _f_ 1: 0.7246; SAMM (5 cl.) _ar_ : 0.7500, _f_ 1: 0.6985; SAMM (4 cl.) _ar_ : 0.8050, _f_ 1: 0.7657 | LO CV
Xie et al. [38] | $\bullet$ | / | $\mathcal{AU:M}$ | _CNN_ | _L_ | _GCN_ | _Softmax_ |  CASME II (3 cl.) _ar_ : 0.712, _f_ 1: 0.355; CASME II (7 cl.) _ar_ : 0.561, _f_ 1: 0.394; SAMM (3 cl.) _ar_ : 0.702, _f_ 1: 0.433; SAMM (8 cl.) _ar_ : 0.523, _f_ 1: 0.357 SMIC (CASME II) (3 cl.) _ar_ : 0.344, _f_ 1: 0.319; SMIC (SAMM) (3 cl.) _ar_ : 0.451, _f_ 1: 0.309 |  LO CV /CD
Kumar and Bhanu [71] | $\bullet$ | 51 | $\mathcal{S:L}$ |  $\mathbb{C}$+_Optical-flow_ | _M+L_ | _GCN+GAT_ | _Softmax_ |  CASME II (3 cl.) _ar_ : 0.8966, _f_ 1: 0.8695; CASME II (5 cl.) _ar_ : 0.8130, _f_ 1: 0.7090; SAMM (3 cl.) _ar_ : 0.8872, _f_ 1: 0.8118; SAMM (5 cl.) _ar_ : 0.8824, _f_ 1: 0.8279 | LO CV
Lei et al. [94] | $\bullet$ | 30 | $\mathcal{AU:B}$ | _Embedding_ | $\Phi$ | _GCN_ | _Softmax_ |  CASME II (4 cl.) _ar_ : 0.8080, _f_ 1: 0.7871; CASME II (5 cl.) _ar_ : 0.7427, _f_ 1: 0.7047; SAMM (4 cl.) _ar_ : 0.8239, _f_ 1: 0.7735; SAMM (5 cl.) _ar_ : 0.7426, _f_ 1: 0.7045 | LO CV

  * 1

Prep.: processing; B.B.: bounding boxes; $\bullet$ = Yes, $\circ$ = No; LM:
landmarks.

  * 2

$\mathcal{S/AU}$: {spatial; AU-level} representation; $\mathcal{:L/B/M}$:
{landmark; label; map}-level graph; $\mathbb{C}$: landmark coordinates;
$\Phi$: label distributions; _L/F_ : {learning; fully} connections;
$\mathbb{E}$: Euclidean distance; $\mathbb{G}$: Gradient using slope equation.

  * 3

_ar_ : average accuracy rate; _f_ 1: F1 score; _war_ :weighted average recall;
_wf_ 1: weighted F1 score; _uar_ : unweighted average recall; (N) cl.: (N)
affective classes; DB1 (DB2): train on database 2, test on database 1.

  * 4

CV: cross validation; LO: leave-one-subject-out; CD: cross database
validation.

A problem in graph-based MER is the lack of large-scale in-the-wild data. The
small sample size limits the AU-level graph representation that relies on
initializing the AU relationship from the AU label distribution of the
training set. The lab-controlled data make it difficult to follow the trend in
FER studies, which generalizes the graph-based FAA methods in real-world
scenarios. However, the analysis of uncontrolled micro-expressions is
fundamental because micro-expressions and macro-expressions can co-occur in
many real cases. For example, the furrowing on the forehead slightly and
quickly when smiling indicates the true feeling [163]. Since the evolutionary
appearance information is crucial for the micro-expression analysis, building
a spatio-temporal graph representation that can model the duration and the
dynamic of micro-expressions is also a helpful but unexplored topic.

###  5.5 Special Tasks

The graph-based methods also play a vital role in several special FAA tasks,
such as pain detection [72], non-basic affect recognition [84, 102], occluded
FER [39, 40], and multi-modal affect recognition [87, 100]. Table VII
summarizes the latest graph-based FAA methods for special tasks. Their node
representations and edge initialization strategies for graph constructions in
this field are similar to those in graph-based FER, MER, and AUD methods.
While for the reasoning step, _GCN_ is the top-1 option. This observation
implies that the framework of the graph-based method discussed in this paper
can be easily extended to many other FAA tasks and promote performance
improvement.

TABLE VII: Performance summary of graph-based methods for special FAA tasks

References |  Prep.1 |  Representation2 | Reasoning |  Database3 | Validation4
---|---|---|---|---|---
B.B. | LM | Category | Node | Edge | Model | Output |
Kaltwang et al. [72] | $\circ$ | 66 | $\mathcal{S:L}$ | $\mathbb{C}$ | _L_ | _LT_ |  ShoulderPain [165] _corr_ : 0.23, _mse_ : 0.60 | 8F CV
Dapogny et al. [39] | $\bullet$ | 49 | $\mathcal{S:L}$ |  $\mathbb{C}$+_HOG_ | $\bigtriangleup$ | _AE_ | _RF_ |  CK+ (eyes occluded) _ar_ : 0.879; CK+ (mouth occluded) _ar_ : 0.727 | 10F-SI CV
Zhang et al. [84] | $\bullet$ | / | $\mathcal{S:R}$ | _VGG_ | _L_ | _GCN_ |  _Softmax_ /_FC_ |  EMOTIC (26 cl.) _prc_ : 0.2842; EMOTIC (VAD) _er_ : 0.9 | HO
Chen et al. [87] | $\bullet$ | 68 | $\mathcal{ST:L}$ | $\mathbb{C}$ |  $\bigtriangleup$+_M_ | _GCN_ | _FC_ |  CES [17] _val_ccc_ : 0.515, _aro_ccc_ : 0.513 | HO
He and Jin [102] | $\circ$ | / | $\mathcal{X}$ | _Embedding_ | _M+L_ | _GCN_ | _Softmax_ |  FlickLDL [166] _ar_ : 0.691; TwiterLDL [167] _ar_ : 0.758 | HO
Zhou et al. [40] | $\bullet$ | 44 | $\mathcal{ST:L}$ |  $\mathbb{C}$+_HOG_ |  _L_ +$\mathbb{H}$ | _GCN_ | _SoftMax_ |  CK+ (random occlusion) _ar_ : 0.9551; Oulu (random occlusion) _ar_ : 0.8121; AFEW (random occlusion) _ar_ : 0.4047 |  10F-SI CV /HO
Chien et al. [100] | $\circ$ | / | $\mathcal{X}$ | _CNN_ | _ Transfer Knowledge _ | _GCN_ | _FC_ |  Amigos [168] (Ascertain) _val_uar_ : 0.798, _aro_uar_ : 0.679; Ascertain [169] (Amigos) _val_uar_ : 0.704, _aro_uar_ : 0.569 | CD

  * 1

Prep.: processing; B.B.: bounding boxes; $\bullet$ = Yes, $\circ$ = No; LM:
landmarks.

  * 2

$\mathcal{S/ST/X}$: {spatial; spatio-temporal; sample-level} representation;
$\mathcal{:L/R}$: {landmark; region}-level graph; $\mathbb{C}$: landmark
coordinates; $\Phi$: label distributions; $\bigtriangleup$: triangulation;
_M/L_ : {manual/learning} connections.

  * 3

_corr_ : Pearson correlation coefficient; _mes_ : mean squared error; _prc_ :
area under the precision recall curves; _er_ : average error rate; _val/aro_ :
valance/arousal; _ccc_ : concordance correlation coefficient; _uar_ :
unweighted average recall; (N) cl.: (N) affective classes; VAD: valence,
arousal, dominance; DB1 (DB2): train on database 2, test on database 1; _ar_ :
average accuracy rate; _f_ 1: F1 score; _mif_ 1: micro F1 score; _maf_ 1:
macro F1 score.

  * 4

CV: cross validation; (K)F: k-folds; SI: subject independent; HO: holdout
validation; CD: cross database validation.

##  6 Open Directions

Graph-based FAA methods have been dissected into fundamental components for
elaboration and discussion in this review. When encoding facial affect into
graphs, strategies vary according to node and edge elements. Relational
reasoning approaches infer latent relationships or inherent dependencies of
graph nodes in terms of space, time, and semantics. The category of graph
representations will affect the technique choice of relational reasoning to a
certain extent.

Despite significant advances and numerous work, the graph-based FAA is still
an appealing field with many open directions. Due to advantages in modeling
and reasoning latent relationships of facial affects, graph-based methods may
provide complementary information to help solve some challenges that non-
graph-based approaches face. Also, the graph-based method has natural
advantages or unexplored research potential in other topics.

###  6.1 In-the-wild Scenarios

Although many efforts have been made for graph-based FAA in natural conditions
[82, 39, 42, 35, 38, 40, 96, 41], even the state-of-the-art performance is far
from actual applications. Factors like illumination, head pose, and part
occlusion are challenging in constructing an effective graph representation.
For one thing, significant illumination changes and head pose variations will
impair the accuracy of face detection and registration, which is vital for
establishing landmark-level graphs. ROI graphs without landmarks or NPI graphs
[82, 85] should be a possible direction to avoid this problem. Also, missing
face parts resulting from camera view or context occlusion make it challenging
to encode enough facial information and obtain meaningful connections in an
affective graph. Pilot work [39, 40] has tried to exploit a sub-graph without
masked facial parts or generate adaptive edge links to alleviate the
influence. Unfortunately, there has still been a considerable performance
decrease compared to normal conditions. Proposing more effective spatio-
temporal graphs can account for these problems based on evolutional affective
information.

###  6.2 3D and 4D Facial Affects

Using 3D and 4D face images might be another good topic because the 3D face
shape provides additional depth information and dynamically contains subtle
facial deformations. They are intuitively insensitive to pose and light
changes. Some studies have transformed 3D faces into 2D images and generated
graph representations [39, 95, 22, 37], but they have not fully taken
advantage of the 3D data. Alternatively, non-graph-based [170] and graph-based
methods [77] have been explored to conduct FAA directly on 3D or 4D faces.
Since the 3D face mesh structure is naturally close to the graph structure,
employing the graph representation and reasoning to handle 3D face images will
promote the improvement of in-the-wild FAA. Besides, there is also a potential
topic of using 3D and 4D data with graph-based methods, especially landmark-
level graphs and _GNNs_ , in micro-expression recognition.

###  6.3 Valence and Arousal

Estimating the continuous dimension is a rising topic in FAA. Unlike discrete
labels, Valence-Arousal (V-A) annotations describe a wider range of facial
affects that are consistent with those in the real world. Large-scale FAA
databases (Aff-Wild I [14], II [171]) containing V-A annotations have been
released to support the continuous FAA. Existing graph-based methods mainly
perform the V-A measurement [72, 99, 37] on lab-controlled databases except
for a few studies like [84]. Recent graph-based methods have studied multi-
label learning according to intrinsic mappings between facial affect
categories and other annotations [22, 41]. Such underlying assumptions can
also be extended to the V-A measurement task, where AU-level graphs and _DBNs_
, as well as sample-level graphs, are potential directions.

###  6.4 Context and Multi-modality

Most current FAA methods only consider a single face in one image or sequence.
However, people usually have affective behaviors, including facial
expressions, body gestures, and emotionally speaking in real cases [172].
These facial affective displays are highly associated with context
surroundings that include but are not limited to the affective behavior of
other people in social interactions or inanimate objects. Existing studies
like [84] and [85] have employed graph reasoning to infer relationships
between the target face and other objects in the same image. Facial affects
and other helpful contexts can be combined in a graph representation to
perform the analysis on a fuller scope, such as gesture [173, 174]. Another
valuable topic is to introduce additional data channels that are multi-
modality. Sample-level and spatio-temporal have also been successfully
extended to process multi-modal affect analysis tasks with audio [87] and
physiological signal [100], respectively, which shows a good research
prospect.

###  6.5 Cross-database and Transfer Learning

Insufficient annotations and imbalanced labels are two problems that limit the
development of FAA research. One possible solution is to use graph-based
transfer learning. Efforts like [96, 41, 97] have exploited the graph
structure to solve this challenge in semi-supervision, label correction,
generation, or uncertainty measurement. On the other hand, the performance of
affective features extracted using graph-based representation and reasoning
has been proved through cross-database validation in all FER [42, 85], AUD
[92, 41], MER [38], and cross-corpus analysis [101, 100]. Specifically, the
strength of distribution modeling of AU-level and sample-level graphs is
valuable in improving the generalization capability of affective features.

## Acknowledgments

Funding: This work was supported by the China Scholarship Council [CSC,
No.202006150091], and the Ministry of Education and Culture of Finland for AI
forum project.

The authors would like to thank Muzammil Behzad and Tuomas Varanka for
providing materials and suggestions for the figures used in this paper.

## References

  * Mehrabian et al. [1971] A. Mehrabian _et al._ , _Silent messages_. Wadsworth Belmont, CA, 1971, vol. 8, no. 152.
  * Mehrabian and Russell [1974] A. Mehrabian and J. A. Russell, _An approach to environmental psychology._ the MIT Press, 1974.
  * Darwin and Prodger [1998] C. Darwin and P. Prodger, _The expression of the emotions in man and animals_. Oxford University Press, USA, 1998.
  * Gazzaniga et al. [2014] M. S. Gazzaniga, R. B. Ivry, and G. Mangun, _Cognitive Neuroscience. The Biology of the Mind, (2014)_. Norton: New York, 2014.
  * Picard et al. [2001] R. W. Picard, E. Vyzas, and J. Healey, “Toward machine emotional intelligence: Analysis of affective physiological state,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 23, no. 10, pp. 1175–1191, 2001\.
  * Jack et al. [2012] R. E. Jack, O. G. Garrod, H. Yu, R. Caldara, and P. G. Schyns, “Facial expressions of emotion are not culturally universal,” _National Academy of Sciences_ , vol. 109, no. 19, pp. 7241–7244, 2012.
  * Sariyanidi et al. [2014] E. Sariyanidi, H. Gunes, and A. Cavallaro, “Automatic analysis of facial affect: A survey of registration, representation, and recognition,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 37, no. 6, pp. 1113–1133, 2014.
  * Calvo et al. [2018] M. G. Calvo, A. Gutiérrez-García, and M. Del Líbano, “What makes a smiling face look happy? visual saliency, distinctiveness, and affect,” _Psychological Research_ , vol. 82, no. 2, pp. 296–309, 2018.
  * Tavakolian and Hadid [2019] M. Tavakolian and A. Hadid, “A spatiotemporal convolutional neural network for automatic pain intensity estimation from facial dynamics,” _International Journal of Computer Vision_ , vol. 127, no. 10, pp. 1413–1425, 2019.
  * Zhang et al. [2018a] Z. Zhang, P. Luo, C. C. Loy, and X. Tang, “From facial expression recognition to interpersonal relation prediction,” _International Journal of Computer Vision_ , vol. 126, no. 5, pp. 550–569, 2018.
  * Zhao et al. [2020] L. Zhao, X. Peng, Y. Tian, M. Kapadia, and D. N. Metaxas, “Towards image-to-video translation: A structure-aware approach via multi-stage generative adversarial networks,” _International Journal of Computer Vision_ , vol. 128, no. 10, pp. 2514–2533, 2020.
  * Valstar et al. [2017] M. F. Valstar, E. Sánchez-Lozano, J. F. Cohn, L. A. Jeni, J. M. Girard, Z. Zhang, L. Yin, and M. Pantic, “Fera 2017-addressing head pose in the third facial expression recognition and analysis challenge,” in _IEEE International Conference on Automatic Face & Gesture Recognition (FG)_. IEEE, 2017, pp. 839–847.
  * Dhall et al. [2020] A. Dhall, G. Sharma, R. Goecke, and T. Gedeon, “Emotiw 2020: Driver gaze, group emotion, student engagement and physiological signal based challenges,” in _International Conference on Multimodal Interaction (ICMI)_ , 2020, pp. 784–789.
  * Zafeiriou et al. [2017] S. Zafeiriou, D. Kollias, M. A. Nicolaou, A. Papaioannou, G. Zhao, and I. Kotsia, “Aff-wild: valence and arousal’in-the-wild’challenge,” in _IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , 2017, pp. 34–41.
  * Kollias et al. [2021] D. Kollias, I. Kotsia, E. Hajiyev, and S. Zafeiriou, “Analysing affective behavior in the second abaw2 competition,” _arXiv preprint arXiv:2106.15318_ , 2021.
  * Benitez-Quiroz et al. [2017] C. F. Benitez-Quiroz, R. Srinivasan, Q. Feng, Y. Wang, and A. M. Martinez, “Emotionet challenge: Recognition of facial expressions of emotion in the wild,” _arXiv preprint arXiv:1703.01210_ , 2017.
  * Ringeval et al. [2019] F. Ringeval, B. Schuller, M. Valstar, N. Cummins, R. Cowie, L. Tavabi, M. Schmitt, S. Alisamir, S. Amiriparian, E.-M. Messner _et al._ , “Avec 2019 workshop and challenge: state-of-mind, detecting depression with ai, and cross-cultural affect recognition,” in _International on Audio/Visual Emotion Challenge and Workshop_ , 2019, pp. 3–12.
  * Stappen et al. [2020] L. Stappen, A. Baird, G. Rizos, P. Tzirakis, X. Du, F. Hafner, L. Schumann, A. Mallol-Ragolta, B. W. Schuller, I. Lefter _et al._ , “Muse 2020 challenge and workshop: Multimodal sentiment analysis, emotion-target engagement and trustworthiness detection in real-life media: Emotional car reviews in-the-wild,” in _International on Multimodal Sentiment Analysis in Real-life Media Challenge and Workshop_ , 2020, pp. 35–44.
  * Zhao and Pietikainen [2007] G. Zhao and M. Pietikainen, “Dynamic texture recognition using local binary patterns with an application to facial expressions,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 29, no. 6, pp. 915–928, 2007\.
  * Friesen and Ekman [1978] E. Friesen and P. Ekman, “Facial action coding system: a technique for the measurement of facial movement,” _Palo Alto_ , vol. 3, no. 2, p. 5, 1978\.
  * Zhong et al. [2019] L. Zhong, C. Bai, J. Li, T. Chen, S. Li, and Y. Liu, “A graph-structured representation with brnn for static-based facial expression recognition,” in _IEEE International Conference on Automatic Face and Gesture Recognition (FG)_. IEEE, 2019, pp. 1–5.
  * Li et al. [2019a] G. Li, X. Zhu, Y. Zeng, Q. Wang, and L. Lin, “Semantic relationships guided representation learning for facial action unit recognition,” in _AAAI Conference on Artificial Intelligence (AAAI)_ , vol. 33, no. 01, 2019, pp. 8594–8601.
  * Zhang et al. [2020] Z. Zhang, T. Wang, and L. Yin, “Region of interest based graph convolution: A heatmap regression approach for action unit detection,” in _ACM International Conference on Multimedia (ACM MM)_ , 2020, pp. 2890–2898.
  * Song et al. [2021a] T. Song, Z. Cui, Y. Wang, W. Zheng, and Q. Ji, “Dynamic probabilistic graph convolution for facial action unit intensity estimation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 4845–4854.
  * Ekman [1994] P. Ekman, “Strong evidence for universals in facial expressions: a reply to russell’s mistaken critique.” _Psychological Bulletin_ , vol. 115, no. 2, pp. 268–287, 1994.
  * Cohn and De la Torre [2015] J. F. Cohn and F. De la Torre, _Automated face analysis for affective computing._ Oxford University Press, 2015\.
  * Zarins and Kondrats [2015] U. Zarins and S. Kondrats, _Anatomy for sculptors: understanding the human figure_. Anatomy Next, Incorporated, 2015\.
  * Bimler and Paramei [2006] D. L. Bimler and G. V. Paramei, “Facial-expression affective attributes and their configural correlates: components and categories,” _Spanish Journal of Psychology_ , vol. 9, no. 1, p. 19, 2006.
  * Ekman [2002] P. Ekman, “Facial action coding system (facs),” _A Human Face_ , 2002.
  * Zhang et al. [2017] K. Zhang, Y. Huang, Y. Du, and L. Wang, “Facial expression recognition based on deep evolutional spatial-temporal networks,” _IEEE Transactions on Image Processing_ , vol. 26, no. 9, pp. 4193–4203, 2017.
  * Li et al. [2020] Y. Li, G. Lu, J. Li, Z. Zhang, and D. Zhang, “Facial expression recognition in the wild using multi-level features and attention mechanisms,” _IEEE Transactions on Affective Computing_ , 2020.
  * Jacob and Stenger [2021] G. M. Jacob and B. Stenger, “Facial action unit detection with transformers,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 7680–7689.
  * Martinez et al. [2017] B. Martinez, M. F. Valstar, B. Jiang, and M. Pantic, “Automatic analysis of facial actions: A survey,” _IEEE Transactions on Affective Computing_ , vol. 10, no. 3, pp. 325–347, 2017.
  * Barrett [2017] L. F. Barrett, _How emotions are made: The secret life of the brain_. Houghton Mifflin Harcourt, 2017\.
  * Liu et al. [2019] Y. Liu, X. Zhang, Y. Lin, and H. Wang, “Facial expression recognition via deep action units graph network based on psychological mechanism,” _IEEE Transactions on Cognitive and Developmental Systems_ , vol. 12, no. 2, pp. 311–322, 2019.
  * Cui et al. [2020a] Z. Cui, T. Song, Y. Wang, and Q. Ji, “Knowledge augmented deep neural networks for joint facial expression and action unit recognition,” _Advances in Neural Information Processing Systems (NeurIPS)_ , vol. 33, 2020.
  * Fan et al. [2020] Y. Fan, J. Lam, and V. Li, “Facial action unit intensity estimation via semantic correspondence learning with dynamic graph convolution,” in _AAAI Conference on Artificial Intelligence (AAAI)_ , vol. 34, no. 07, 2020, pp. 12 701–12 708.
  * Xie et al. [2020a] H.-X. Xie, L. Lo, H.-H. Shuai, and W.-H. Cheng, “Au-assisted graph attention convolutional network for micro-expression recognition,” in _ACM International Conference on Multimedia (ACM MM)_ , 2020, pp. 2871–2880.
  * Dapogny et al. [2018] A. Dapogny, K. Bailly, and S. Dubuisson, “Confidence-weighted local expression predictions for occlusion handling in expression recognition and action unit detection,” _International Journal of Computer Vision_ , vol. 126, no. 2, pp. 255–271, 2018.
  * Zhou et al. [2020a] J. Zhou, X. Zhang, and Y. Liu, “Learning the connectivity: Situational graph convolution network for facial expression recognition,” in _IEEE International Conference on Visual Communications and Image Processing_. IEEE, 2020, pp. 230–234.
  * Cui et al. [2020b] Z. Cui, Y. Zhang, and Q. Ji, “Label error correction and generation through label relationships,” in _AAAI Conference on Artificial Intelligence (AAAI)_ , vol. 34, no. 04, 2020, pp. 3693–3700.
  * Chen et al. [2020] S. Chen, J. Wang, Y. Chen, Z. Shi, X. Geng, and Y. Rui, “Label distribution learning on auxiliary label space graphs for facial expression recognition,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 13 984–13 993.
  * Corneanu et al. [2016] C. A. Corneanu, M. O. Simón, J. F. Cohn, and S. E. Guerrero, “Survey on rgb, 3d, thermal, and multimodal approaches for facial expression recognition: History, trends, and affect-related applications,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 38, no. 8, pp. 1548–1568, 2016.
  * Kollias et al. [2019] D. Kollias, P. Tzirakis, M. A. Nicolaou, A. Papaioannou, G. Zhao, B. Schuller, I. Kotsia, and S. Zafeiriou, “Deep affect prediction in-the-wild: Aff-wild database and challenge, deep architectures, and beyond,” _International Journal of Computer Vision_ , vol. 127, no. 6, pp. 907–929, 2019.
  * Li and Deng [2020] S. Li and W. Deng, “Deep facial expression recognition: A survey,” _IEEE Transactions on Affective Computing_ , 2020.
  * Goh et al. [2020] K. M. Goh, C. H. Ng, L. L. Lim, and U. U. Sheikh, “Micro-expression recognition: an updated review of current trends, challenges and solutions,” _The Visual Computer_ , vol. 36, no. 3, pp. 445–468, 2020.
  * Zhang et al. [2018b] L. Zhang, B. Verma, D. Tjondronegoro, and V. Chandran, “Facial expression analysis under partial occlusion: A survey,” _ACM Computing Surveys_ , vol. 51, no. 2, pp. 1–49, 2018.
  * Rouast et al. [2019] P. V. Rouast, M. Adam, and R. Chiong, “Deep learning for human affect recognition: Insights and new developments,” _IEEE Transactions on Affective Computing_ , 2019.
  * Ben et al. [2021] X. Ben, Y. Ren, J. Zhang, S.-J. Wang, K. Kpalma, W. Meng, and Y.-J. Liu, “Video-based facial micro-expression analysis: A survey of datasets, features and algorithms,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2021.
  * Ekman and Friesen [1971] P. Ekman and W. V. Friesen, “Constants across cultures in the face and emotion.” _Journal of Personality and Social Psychology_ , vol. 17, no. 2, pp. 124–129, 1971.
  * Du et al. [2014] S. Du, Y. Tao, and A. M. Martinez, “Compound facial expressions of emotion,” _Proceedings of the National Academy of Sciences_ , vol. 111, no. 15, pp. E1454–E1462, 2014.
  * Sethu et al. [2019] V. Sethu, E. M. Provost, J. Epps, C. Busso, N. Cummins, and S. Narayanan, “The ambiguous world of emotion representation,” _arXiv preprint arXiv:1909.00360_ , 2019.
  * Mehrabian [1995] A. Mehrabian, “Framework for a comprehensive description and measurement of emotional states.” _Genetic, social, and general psychology monographs_ , 1995.
  * Plutchik [1980] R. Plutchik, “A general psychoevolutionary theory of emotion,” in _Theories of Emotion_. Elsevier, 1980, pp. 3–33.
  * Greenwald et al. [1989] M. K. Greenwald, E. W. Cook, and P. J. Lang, “Affective judgment and psychophysiological response: dimensional covariation in the evaluation of pictorial stimuli.” _Journal of Psychophysiology_ , vol. 3, no. 1, pp. 51–64, 1989.
  * Russell [1978] J. A. Russell, “Evidence of convergent validity on the dimensions of affect.” _Journal of Personality and Social Psychology_ , vol. 36, no. 10, p. 1152, 1978.
  * Soleymani et al. [2015] M. Soleymani, S. Asghari-Esfeden, Y. Fu, and M. Pantic, “Analysis of eeg signals and facial expressions for continuous emotion detection,” _IEEE Transactions on Affective Computing_ , vol. 7, no. 1, pp. 17–28, 2015.
  * Yin et al. [2008] L. Yin, X. Chen, Y. Sun, T. Worm, and M. Reale, “A high-resolution 3d dynamic facial expression database,” in _IEEE International Conference on Automatic Face and Gesture Recognition (FG)_ , 2008, pp. 1–6.
  * Baltrušaitis et al. [2016] T. Baltrušaitis, P. Robinson, and L.-P. Morency, “Openface: an open source facial behavior analysis toolkit,” in _2016 IEEE Winter Conference on Applications of Computer Vision (WACV)_. IEEE, 2016, pp. 1–10.
  * Baltrusaitis et al. [2018] T. Baltrusaitis, A. Zadeh, Y. C. Lim, and L.-P. Morency, “Openface 2.0: Facial behavior analysis toolkit,” in _2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018)_. IEEE, 2018, pp. 59–66.
  * Viola and Jones [2001] P. Viola and M. Jones, “Rapid object detection using a boosted cascade of simple features,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , vol. 1. IEEE, 2001, pp. I–I.
  * Zhu and Ramanan [2012] X. Zhu and D. Ramanan, “Face detection, pose estimation, and landmark localization in the wild,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, 2012, pp. 2879–2886.
  * Cootes et al. [2001] T. F. Cootes, G. J. Edwards, and C. J. Taylor, “Active appearance models,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 23, no. 6, pp. 681–685, 2001.
  * Zhang et al. [2016] K. Zhang, Z. Zhang, Z. Li, and Y. Qiao, “Joint face detection and alignment using multitask cascaded convolutional networks,” _IEEE Signal Processing Letters_ , vol. 23, no. 10, pp. 1499–1503, 2016.
  * Ranjan et al. [2017] R. Ranjan, V. M. Patel, and R. Chellappa, “Hyperface: A deep multi-task learning framework for face detection, landmark localization, pose estimation, and gender recognition,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 41, no. 1, pp. 121–135, 2017.
  * Dong et al. [2020] X. Dong, Y. Yang, S.-E. Wei, X. Weng, Y. Sheikh, and S.-I. Yu, “Supervision by registration and triangulation for landmark detection,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2020.
  * Bulat and Tzimiropoulos [2017] A. Bulat and G. Tzimiropoulos, “How far are we from solving the 2d and 3d face alignment problem?(and a dataset of 230,000 3d facial landmarks),” in _IEEE International Conference on Computer Vision (ICCV)_ , 2017, pp. 1021–1030.
  * Masi et al. [2018] I. Masi, Y. Wu, T. Hassner, and P. Natarajan, “Deep face recognition: A survey,” in _SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI)_. IEEE, 2018, pp. 471–478.
  * Wang et al. [2018] N. Wang, X. Gao, D. Tao, H. Yang, and X. Li, “Facial feature point detection: A comprehensive survey,” _Neurocomputing_ , vol. 275, pp. 50–65, 2018.
  * Lucey et al. [2010] P. Lucey, J. F. Cohn, T. Kanade, J. Saragih, Z. Ambadar, and I. Matthews, “The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression,” in _IEEE Conference on Computer Vision and Pattern Recognition-Workshops_. IEEE, 2010, pp. 94–101.
  * Kumar and Bhanu [2021] A. J. R. Kumar and B. Bhanu, “Micro-expression classification based on landmark relations with graph attention convolutional network,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 1511–1520.
  * Kaltwang et al. [2015] S. Kaltwang, S. Todorovic, and M. Pantic, “Latent trees for estimating intensity of facial action units,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2015, pp. 296–304.
  * Zhao et al. [2021] R. Zhao, T. Liu, Z. Huang, D. P.-K. Lun, and K. K. Lam, “Geometry-aware facial expression recognition via attentive graph convolutional networks,” _IEEE Transactions on Affective Computing_ , 2021.
  * Mohseni et al. [2014] S. Mohseni, N. Zarei, and S. Ramazani, “Facial expression recognition using anatomy based facial graph,” in _IEEE International Conference on Systems, Man, and Cybernetics (SMC)_. IEEE, 2014, pp. 3715–3719.
  * Liu et al. [2015] Y.-J. Liu, J.-K. Zhang, W.-J. Yan, S.-J. Wang, G. Zhao, and X. Fu, “A main directional mean optical flow feature for spontaneous micro-expression recognition,” _IEEE Transactions on Affective Computing_ , vol. 7, no. 4, pp. 299–310, 2015.
  * Lei et al. [2020] L. Lei, J. Li, T. Chen, and S. Li, “A novel graph-tcn with a graph structured representation for micro-expression recognition,” in _ACM International Conference on Multimedia (ACM MM)_ , 2020, pp. 2237–2245.
  * Pei and Zha [2009] Y. Pei and H. Zha, “3d facial expression editing based on the dynamic graph model,” in _IEEE International Conference on Multimedia and Expo (ICME)_. IEEE, 2009, pp. 1354–1357.
  * Kotsia and Pitas [2006] I. Kotsia and I. Pitas, “Facial expression recognition in image sequences using geometric deformation features and support vector machines,” _IEEE Transactions on Image Processing_ , vol. 16, no. 1, pp. 172–187, 2006\.
  * Liu et al. [2020a] Z. Liu, J. Dong, C. Zhang, L. Wang, and J. Dang, “Relation modeling with graph convolutional networks for facial action unit detection,” in _International Conference on Multimedia Modeling (MMM)_. Springer, 2020, pp. 489–501.
  * Jin et al. [2021] X. Jin, Z. Lai, and Z. Jin, “Learning dynamic relationships for facial expression recognition based on graph convolutional network,” _IEEE Transactions on Image Processing_ , vol. 30, pp. 7143–7155, 2021.
  * Liu et al. [2021a] Y.-J. Liu, B.-J. Li, and Y.-K. Lai, “Sparse mdmo: Learning a discriminative feature for spontaneous micro-expression recognition,” _IEEE Transactions on Affective Computing_ , vol. 12, no. 1, pp. 254–261, 2021.
  * Yao et al. [2015] A. Yao, J. Shao, N. Ma, and Y. Chen, “Capturing au-aware facial features and their latent relations for emotion recognition in the wild,” in _ACM on International Conference on Multimodal Interaction (ICMI)_ , 2015, pp. 451–458.
  * Zafeiriou and Pitas [2008] S. Zafeiriou and I. Pitas, “Discriminant graph structures for facial expression recognition,” _IEEE Transactions on Multimedia_ , vol. 10, no. 8, pp. 1528–1540, 2008.
  * Zhang et al. [2019a] M. Zhang, Y. Liang, and H. Ma, “Context-aware affective graph reasoning for emotion recognition,” in _IEEE International Conference on Multimedia and Expo (ICME)_. IEEE, 2019, pp. 151–156.
  * Xie et al. [2020b] Y. Xie, T. Chen, T. Pu, H. Wu, and L. Lin, “Adversarial graph representation adaptation for cross-domain facial expression recognition,” in _ACM International Conference on Multimedia (ACM MM)_ , 2020, pp. 1255–1264.
  * Zhou et al. [2020b] J. Zhou, X. Zhang, Y. Liu, and X. Lan, “Facial expression recognition using spatial-temporal semantic graph network,” in _IEEE International Conference on Image Processing (ICIP)_. IEEE, 2020, pp. 1961–1965.
  * Chen et al. [2019] H. Chen, Y. Deng, S. Cheng, Y. Wang, D. Jiang, and H. Sahli, “Efficient spatial temporal convolutional features for audiovisual continuous affect recognition,” in _International on Audio/Visual Emotion Challenge and Workshop (AVEC)_ , 2019, pp. 19–26.
  * Chen et al. [2021a] Y. Chen, D. Chen, Y. Wang, T. Wang, and Y. Liang, “Cafgraph: Context-aware facial multi-graph representation for facial action unit recognition,” in _Proceedings of the 29th ACM International Conference on Multimedia (ACM MM)_ , 2021, pp. 1029–1037.
  * Rivera and Chae [2015] A. R. Rivera and O. Chae, “Spatiotemporal directional number transitional graph for dynamic texture recognition,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 37, no. 10, pp. 2146–2152, 2015.
  * Liu et al. [2021b] D. Liu, H. Zhang, and P. Zhou, “Video-based facial expression recognition using graph convolutional networks,” in _International Conference on Pattern Recognition (ICPR)_. IEEE, 2021, pp. 607–614.
  * Shirian et al. [2021] A. Shirian, S. Tripathi, and T. Guha, “Dynamic emotion modeling with learnable graphs and graph inception network,” _IEEE Transactions on Multimedia_ , 2021\.
  * Tong et al. [2007] Y. Tong, W. Liao, and Q. Ji, “Facial action unit recognition by exploiting their dynamic and semantic relationships,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 29, no. 10, pp. 1683–1699, 2007\.
  * Zhu et al. [2014] Y. Zhu, S. Wang, L. Yue, and Q. Ji, “Multiple-facial action unit recognition by shared feature learning and semantic relation modeling,” in _International Conference on Pattern Recognition (ICPR)_. IEEE, 2014, pp. 1663–1668.
  * Lei et al. [2021] L. Lei, T. Chen, S. Li, and J. Li, “Micro-expression recognition based on facial graph representation learning and facial action unit fusion,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 1571–1580.
  * Corneanu et al. [2018] C. Corneanu, M. Madadi, and S. Escalera, “Deep structure inference network for facial action unit recognition,” in _European Conference on Computer Vision (ECCV)_ , 2018, pp. 298–313.
  * Niu et al. [2019] X. Niu, H. Han, S. Shan, and X. Chen, “Multi-label co-regularization for semi-supervised facial action unit recognition,” in _International Conference on Neural Information Processing Systems (NeurIPS)_ , vol. 32, 2019, pp. 1–11.
  * Song et al. [2021b] T. Song, L. Chen, W. Zheng, and Q. Ji, “Uncertain graph neural networks for facial action unit detection,” in _AAAI Conference on Artificial Intelligence (AAAI)_ , 2021, pp. 1–10.
  * Song et al. [2021c] T. Song, Z. Cui, W. Zheng, and Q. Ji, “Hybrid message passing with performance-driven structures for facial action unit detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 6267–6276.
  * Walecki et al. [2017] R. Walecki, O. Rudovic, V. Pavlovic, B. Schuller, and M. Pantic, “Deep structured learning for facial expression intensity estimation,” _Image and Vision Computting_ , vol. 259, pp. 143–154, 2017.
  * Chien et al. [2020] W.-S. Chien, H.-C. Yang, and C.-C. Lee, “Cross corpus physiological-based emotion recognition using a learnable visual semantic graph convolutional network,” in _ACM International Conference on Multimedia (ACM MM)_ , 2020, pp. 2999–3006.
  * Chen et al. [2021b] D. Chen, P. Song, and W. Zheng, “Learning transferable sparse representations for cross-corpus facial expression recognition,” _IEEE Transactions on Affective Computing_ , 2021.
  * He and Jin [2019] T. He and X. Jin, “Image emotion distribution learning with graph convolutional networks,” in _International Conference on Multimedia Retrieval (ICMR)_ , 2019, pp. 382–390.
  * Zhao et al. [2016] K. Zhao, W.-S. Chu, and H. Zhang, “Deep region and multi-label learning for facial action unit detection,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016, pp. 3391–3399.
  * Oh et al. [2018] Y.-H. Oh, J. See, A. C. Le Ngo, R. C.-W. Phan, and V. M. Baskaran, “A survey of automatic facial micro-expression analysis: databases, methods, and challenges,” _Frontiers in Psychology_ , vol. 9, p. 1128, 2018.
  * Zhang et al. [2014a] Z. Zhang, P. Luo, C. C. Loy, and X. Tang, “Facial landmark detection by deep multi-task learning,” in _European conference on computer vision (ECCV)_. Springer, 2014, pp. 94–108.
  * Kakumanu and Bourbakis [2006] P. Kakumanu and N. Bourbakis, “A local-global graph approach for facial expression recognition,” in _IEEE International Conference on Tools with Artificial Intelligence (ICTAI)_. IEEE, 2006, pp. 685–692.
  * Kazemi and Sullivan [2014] V. Kazemi and J. Sullivan, “One millisecond face alignment with an ensemble of regression trees,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2014, pp. 1867–1874.
  * Dalal and Triggs [2005] N. Dalal and B. Triggs, “Histograms of oriented gradients for human detection,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , vol. 1. Ieee, 2005, pp. 886–893.
  * Rao et al. [2021] T. Rao, J. Li, X. Wang, Y. Sun, and H. Chen, “Facial expression recognition with multi-sale graph convolutional networks,” _IEEE MultiMedia_ , 2021.
  * Liu and Wechsler [2003] C. Liu and H. Wechsler, “Independent component analysis of gabor features for face recognition,” _IEEE Transactions on Neural Networks_ , vol. 14, no. 4, pp. 919–928, 2003.
  * Liu et al. [2021c] Y. Liu, X. Zhang, J. Zhou, and L. Fu, “Sg-dsn: A semantic graph-based dual-stream network for facial expression recognition,” _Neurocomputing_ , vol. 462, pp. 320–330, 2021.
  * Afzal et al. [2009] S. Afzal, T. M. Sezgin, Y. Gao, and P. Robinson, “Perception of emotional expressions in different representations using facial feature points,” in _International Conference on Affective Computing and Intelligent Interaction and Workshops_. IEEE, 2009, pp. 1–6.
  * Baltrušaitis et al. [2010] T. Baltrušaitis, L. D. Riek, and P. Robinson, “Synthesizing expressions using facial feature point tracking: how emotion is conveyed,” in _Proceedings of the 3rd international workshop on Affective interaction in natural environments_ , 2010, pp. 27–32.
  * Lee et al. [2019] J. Lee, S. Kim, S. Kim, J. Park, and K. Sohn, “Context-aware emotion recognition networks,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (CVPR)_ , 2019, pp. 10 143–10 152.
  * Wang et al. [2020a] K. Wang, X. Peng, J. Yang, D. Meng, and Y. Qiao, “Region attention networks for pose and occlusion robust facial expression recognition,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 4057–4069, 2020.
  * Wang et al. [2020b] J. Wang, K. Sun, T. Cheng, B. Jiang, C. Deng, Y. Zhao, D. Liu, Y. Mu, M. Tan, X. Wang _et al._ , “Deep high-resolution representation learning for visual recognition,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2020.
  * He et al. [2016] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016, pp. 770–778.
  * Ren et al. [2016] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: towards real-time object detection with region proposal networks,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 39, no. 6, pp. 1137–1149, 2016\.
  * Simonyan and Zisserman [2014] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint arXiv:1409.1556_ , 2014.
  * Liu et al. [2020b] Q. Liu _et al._ , “Phase space reconstruction driven spatio-temporal feature learning for dynamic facial expression recognition,” _IEEE Transactions on Affective Computing_ , 2020.
  * Mikolov et al. [2013] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” in _Advances in neural information processing systems (NeurIPS)_ , 2013, pp. 3111–3119.
  * Tian et al. [2001] Y.-I. Tian, T. Kanade, and J. F. Cohn, “Recognizing action units for facial expression analysis,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 23, no. 2, pp. 97–115, 2001.
  * Berkes et al. [2008] P. Berkes, F. Wood, and J. Pillow, “Characterizing neural dependencies with copula models,” in _International Conference on Neural Information Processing Systems (NeurIPS)_ , vol. 21, 2008, pp. 129–136.
  * Kemp and Tenenbaum [2008] C. Kemp and J. B. Tenenbaum, “The discovery of structural form,” _National Academy of Sciences_ , vol. 105, no. 31, pp. 10 687–10 692, 2008\.
  * El Kaliouby and Robinson [2005] R. El Kaliouby and P. Robinson, “Real-time inference of complex mental states from facial expressions and head gestures,” in _Real-time vision for human-computer interaction_. Springer, 2005, pp. 181–200.
  * Baltrušaitis et al. [2011] T. Baltrušaitis, D. McDuff, N. Banda, M. Mahmoud, R. El Kaliouby, P. Robinson, and R. Picard, “Real-time inference of mental states from facial expressions and upper body gestures,” in _2011 IEEE International Conference on Automatic Face & Gesture Recognition (FG)_. IEEE, 2011, pp. 909–914.
  * Li et al. [2016] Y. Li, D. Tarlow, M. Brockschmidt, and R. Zemel, “Gated graph sequence neural networks,” in _International Conference on Learning Representations (ICLR)_ , 2016, pp. 1–20.
  * Bai et al. [2018] S. Bai, J. Z. Kolter, and V. Koltun, “An empirical evaluation of generic convolutional and recurrent networks for sequence modeling,” _arXiv preprint arXiv:1803.01271_ , 2018.
  * Wang et al. [2019] Y. Wang, Y. Sun, Z. Liu, S. E. Sarma, M. M. Bronstein, and J. M. Solomon, “Dynamic graph cnn for learning on point clouds,” _ACM Transactions On Graphics_ , vol. 38, no. 5, pp. 1–12, 2019.
  * Li and Gupta [2018] Y. Li and A. Gupta, “Beyond grids: Learning graph representations for visual recognition,” in _International Conference on Neural Information Processing Systems (NeurIPS)_ , 2018, pp. 9245–9255.
  * Kipf and Welling [2017] T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” in _International Conference on Learning Representations (ICLR)_ , 2017.
  * Li et al. [2019b] B. Li, X. Li, Z. Zhang, and F. Wu, “Spatio-temporal graph routing for skeleton-based action recognition,” in _AAAI Conference on Artificial Intelligence (AAAI)_ , vol. 33, no. 01, 2019, pp. 8561–8568.
  * Defferrard et al. [2016] M. Defferrard, X. Bresson, and P. Vandergheynst, “Convolutional neural networks on graphs with fast localized spectral filtering,” in _International Conference on Neural Information Processing Systems (NeurIPS)_ , 2016, pp. 3837–3845.
  * Gilmer et al. [2017] J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl, “Neural message passing for quantum chemistry,” in _International Conference on Machine Learning (ICML)_. PMLR, 2017, pp. 1263–1272.
  * Veličković et al. [2018] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio, “Graph attention networks,” in _International Conference on Learning Representations_ , 2018.
  * Zhang et al. [2019b] Y. Zhang, S. Pal, M. Coates, and D. Ustebay, “Bayesian graph convolutional neural networks for semi-supervised classification,” in _AAAI Conference on Artificial Intelligence (AAAI)_ , vol. 33, no. 01, 2019, pp. 5829–5836.
  * Kanade et al. [2000] T. Kanade, J. F. Cohn, and Y. Tian, “Comprehensive database for facial expression analysis,” in _IEEE International Conference on Automatic Face and Gesture Recognition (FG)_. IEEE, 2000, pp. 46–53.
  * Valstar and Pantic [2010] M. Valstar and M. Pantic, “Induced disgust, happiness and surprise: an addition to the mmi facial expression database,” in _International Workshop on EMOTION: Corpora for Research on Emotion and Affect_. Paris, France, 2010, p. 65.
  * Zhao et al. [2011] G. Zhao, X. Huang, M. Taini, S. Z. Li, and M. PietikäInen, “Facial expression recognition from near-infrared videos,” _Image and Vision Computing_ , vol. 29, no. 9, pp. 607–619, 2011.
  * Goodfellow et al. [2015] I. J. Goodfellow, D. Erhan, P. L. Carrier, A. Courville, M. Mirza, B. Hamner, W. Cukierski, Y. Tang, D. Thaler, D.-H. Lee _et al._ , “Challenges in representation learning: A report on three machine learning contests,” _Neural Networks_ , vol. 64, pp. 59–63, 2015.
  * Dhall et al. [2015] A. Dhall, O. Ramana Murthy, R. Goecke, J. Joshi, and T. Gedeon, “Video and image based emotion recognition challenges in the wild: Emotiw 2015,” in _ACM on International Conference on Multimodal Interaction (ICMI)_ , 2015, pp. 423–426.
  * Dhall et al. [2017] A. Dhall, R. Goecke, S. Ghosh, J. Joshi, J. Hoey, and T. Gedeon, “From individual to group-level emotion recognition: Emotiw 5.0,” in _ACM International Conference on Multimodal Interaction (ICMI)_ , 2017, pp. 524–528.
  * Mavadati et al. [2013] S. M. Mavadati, M. H. Mahoor, K. Bartlett, P. Trinh, and J. F. Cohn, “Disfa: A spontaneous facial action intensity database,” _IEEE Transactions on Affective Computing_ , vol. 4, no. 2, pp. 151–160, 2013.
  * Yin et al. [2006] L. Yin, X. Wei, Y. Sun, J. Wang, and M. J. Rosato, “A 3d facial expression database for facial behavior research,” in _International Conference on Automatic Face and Gesture Recognition (FG)_. IEEE, 2006, pp. 211–216.
  * Zhang et al. [2013] X. Zhang, L. Yin, J. F. Cohn, S. Canavan, M. Reale, A. Horowitz, and P. Liu, “A high-resolution spontaneous 3d dynamic facial expression database,” in _IEEE International Conference and workshops on Automatic Face and Gesture Recognition (FG)_. IEEE, 2013, pp. 1–6.
  * Zhang et al. [2014b] X. Zhang, L. Yin, J. F. Cohn, S. Canavan, M. Reale, A. Horowitz, P. Liu, and J. M. Girard, “Bp4d-spontaneous: a high-resolution spontaneous 3d dynamic facial expression database,” _Image and Vision Computing_ , vol. 32, no. 10, pp. 692–706, 2014.
  * Zhao and Li [2019] G. Zhao and X. Li, “Automatic micro-expression analysis: open challenges,” _Frontiers in Psychology_ , vol. 10, p. 1833, 2019.
  * Li et al. [2013] X. Li, T. Pfister, X. Huang, G. Zhao, and M. Pietikäinen, “A spontaneous micro-expression database: Inducement, collection and baseline,” in _IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG)_. IEEE, 2013, pp. 1–6.
  * Yan et al. [2014] W.-J. Yan, X. Li, S.-J. Wang, G. Zhao, Y.-J. Liu, Y.-H. Chen, and X. Fu, “Casme ii: An improved spontaneous micro-expression database and the baseline evaluation,” _PloS One_ , vol. 9, no. 1, p. e86041, 2014.
  * Davison et al. [2018] A. K. Davison, C. Lansley, N. Costen, K. Tan, and M. H. Yap, “Samm: A spontaneous micro-facial movement dataset,” _IEEE Transactions on Affective Computing_ , vol. 9, no. 1, pp. 116–129, 2018.
  * Qu et al. [2018] F. Qu, S.-J. Wang, W.-J. Yan, H. Li, S. Wu, and X. Fu, “Cas(me)2 : A database for spontaneous macro-expression and micro-expression spotting and recognition,” _IEEE Transactions on Affective Computing_ , vol. 9, no. 4, pp. 424–436, 2018.
  * Li and Deng [2019] S. Li and W. Deng, “Reliable crowdsourcing and deep locality-preserving learning for unconstrained facial expression recognition,” _IEEE Transactions on Image Processing_ , vol. 28, no. 1, pp. 356–370, 2019.
  * Kosti et al. [2019] R. Kosti, J. M. Alvarez, A. Recasens, and A. Lapedriza, “Context based emotion recognition using emotic dataset,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 42, no. 11, pp. 2755–2766, 2019.
  * Mollahosseini et al. [2019] A. Mollahosseini, B. Hasani, and M. H. Mahoor, “Affectnet: A database for facial expression, valence, and arousal computing in the wild,” _IEEE Transactions on Affective Computing_ , vol. 10, no. 1, pp. 18–31, 2019.
  * Fabian Benitez-Quiroz et al. [2016] C. Fabian Benitez-Quiroz, R. Srinivasan, and A. M. Martinez, “Emotionet: An accurate, real-time algorithm for the automatic annotation of a million facial expressions in the wild,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016, pp. 5562–5570.
  * Lyons et al. [1999] M. J. Lyons, J. Budynek, and S. Akamatsu, “Automatic classification of single facial images,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 21, no. 12, pp. 1357–1362, 1999.
  * Wang and Guan [2008] Y. Wang and L. Guan, “Recognizing human emotional state from audiovisual signals,” _IEEE transactions on multimedia_ , vol. 10, no. 5, pp. 936–946, 2008.
  * Martin et al. [2006] O. Martin, I. Kotsia, B. Macq, and I. Pitas, “The enterface’05 audio-visual emotion database,” in _International Conference on Data Engineering Workshops_. IEEE, 2006, pp. 8–8.
  * Livingstone and Russo [2018] S. R. Livingstone and F. A. Russo, “The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english,” _PloS one_ , vol. 13, no. 5, p. e0196391, 2018.
  * Valstar et al. [2015] M. F. Valstar, T. Almaev, J. M. Girard, G. McKeown, M. Mehu, L. Yin, M. Pantic, and J. F. Cohn, “Fera 2015-second facial expression recognition and analysis challenge,” in _IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG)_ , vol. 6. IEEE, 2015, pp. 1–8.
  * Zhi et al. [2020] R. Zhi, M. Liu, and D. Zhang, “A comprehensive survey on automatic facial action unit analysis,” _The Visual Computer_ , vol. 36, no. 5, pp. 1067–1093, 2020.
  * Haggard and Isaacs [1966] E. A. Haggard and K. S. Isaacs, “Micromomentary facial expressions as indicators of ego mechanisms in psychotherapy,” in _Methods of Research in Psychotherapy_. Springer, 1966, pp. 154–165.
  * Ekman and Friesen [1969] P. Ekman and W. V. Friesen, “Nonverbal leakage and clues to deception,” _Psychiatry_ , vol. 32, no. 1, pp. 88–106, 1969.
  * Yan et al. [2013] W.-J. Yan, Q. Wu, Y.-J. Liu, S.-J. Wang, and X. Fu, “Casme database: a dataset of spontaneous micro-expressions collected from neutralized faces,” in _IEEE International Conference and workshops on Automatic Face and Gesture Recognition (FG)_. IEEE, 2013, pp. 1–7.
  * Lucey et al. [2011] P. Lucey, J. F. Cohn, K. M. Prkachin, P. E. Solomon, and I. Matthews, “Painful data: The unbc-mcmaster shoulder pain expression archive database,” in _IEEE International Conference on Automatic Face and Gesture Recognition (FG)_. IEEE, 2011, pp. 57–64.
  * Borth et al. [2013] D. Borth, R. Ji, T. Chen, T. Breuel, and S.-F. Chang, “Large-scale visual sentiment ontology and detectors using adjective noun pairs,” in _Proceedings of the ACM international conference on Multimedia (ACM MM)_ , 2013, pp. 223–232.
  * Yang et al. [2017] J. Yang, D. She, and M. Sun, “Joint image emotion classification and distribution learning via deep convolutional neural network.” in _Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI)_ , 2017, pp. 3266–3272.
  * Correa et al. [2018] J. A. M. Correa, M. K. Abadi, N. Sebe, and I. Patras, “Amigos: A dataset for affect, personality and mood research on individuals and groups,” _IEEE Transactions on Affective Computing_ , 2018.
  * Subramanian et al. [2016] R. Subramanian, J. Wache, M. K. Abadi, R. L. Vieriu, S. Winkler, and N. Sebe, “Ascertain: Emotion and personality recognition using commercial sensors,” _IEEE Transactions on Affective Computing_ , vol. 9, no. 2, pp. 147–160, 2016\.
  * Behzad et al. [2020] M. Behzad, N. Vo, X. Li, and G. Zhao, “Landmarks-assisted collaborative deep framework for automatic 4d facial expression recognition,” in _IEEE International Conference on Automatic Face and Gesture Recognition (FG)_. IEEE, 2020, pp. 1–5.
  * Kollias et al. [2020] D. Kollias, A. Schulc, E. Hajiyev, and S. Zafeiriou, “Analysing affective behavior in the first abaw 2020 competition,” in _IEEE International Conference on Automatic Face and Gesture Recognition (FG)_. IEEE Computer Society, 2020, pp. 794–800.
  * Huang et al. [2018] X. Huang, A. Dhall, R. Goecke, M. Pietikäinen, and G. Zhao, “Multimodal framework for analyzing the affect of a group of people,” _IEEE Transactions on Multimedia_ , vol. 20, no. 10, pp. 2706–2721, 2018.
  * Kaliouby and Robinson [2005] R. e. Kaliouby and P. Robinson, “Real-time inference of complex mental states from facial expressions and head gestures,” in _Real-time vision for human-computer interaction_. Springer, 2005, pp. 181–200.
  * Liu et al. [2021d] X. Liu, H. Shi, H. Chen, Z. Yu, X. Li, and G. Zhao, “imigue: An identity-free video dataset for micro-gesture understanding and emotion analysis,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 10 631–10 642.

|  Yang Liu reveived his Ph.D. degree in computer sciench and technology from
the South China University of Technology, in 2021. He is currently a
PostDoctoral researcher at the Center for Machine Vision and Signal Analysis,
University of Oulu, Finland. His current research interests include facial
expression recognition, affective computing, and deep learning.
---|---
|  Xingming Zhang is currently a Professor with the School of Computer Science
and Engineering, South China University of Technology, Guangzhou, China. He is
a member of the Standing Committee of the Education Specialized Committee of
China Computer Federation and the Standing director of the University Computer
Education Research Association of China. His research focuses on video
processing, big data, video surveillance, and face recognition.
---|---
|  Yante Li received her M.S. degree in computer science and engineering from
the China University of Petroleum (East China), in 2017. She is currently
pursuing a Ph.D. degree with the University of Oulu, Finland. Her current
research interests include micro-expression analysis and facial action unit
detection.
---|---
|  Jinzhao Zhou received his M.S. degree in computer technology from the South
China University of Technology, in 2021. He is currently pursuing a Ph.D. at
the University of Technology Sydney, Australia. His research interests include
affective computing, reinforcement learning, and machine learning.
---|---
|  Xin Li received the M.S. degree in computer science and engineering from
the South China University of Technology, in 2021. He is currently pursuing
his Ph.D. at the Department of Electrical and Computer Engineering, Rutgers
University, United States. His research interests include mobile computing and
sensing.
---|---
|  Guoying Zhao is currently an Academy Professor with the University of Oulu,
IEEE Fellow, IAPR Fellow, and AAIA Fellow. She has authored or co-authored
more than 260 papers in journals and conferences with 17900+ citations in
Google Scholar and h-index 65. She is a co-program chair for ICMI 2021, has
served as area chair for several conferences, and is an associate editor for
Pattern Recognition, IEEE TCSVT, and Image and Vision Computing Journals. Her
current research interests include image and video descriptors, facial-
expression and micro-expression recognition, emotional gesture analysis,
affective computing, and biometrics.
---|---