Efficient Neural Architecture Search for Emotion Recognition

Monu Verma a (monuverma.cv@gmail.com), Murari Mandal b, Satish Kumar Reddy c,
Yashwanth Reddy Meedimale d, Santosh Kumar Vipparthie

a Electrical and Computer Engineering, University of Miami, Florida,
USA,(33146)
b School of Computer Engineering, Kalinga Institute of Industrial Technology,
Bhubaneswar, Odisha, India, (751024)
cJP Morgan Chase & Co, Hyderabad, Telangana, India, (500081)
dWalmart Global Tech, Hyderabad, Telangana, India, (560103)
eCVPR Lab, Dept. of Electrical Engineering, Indian Institute of Technology,
Ropar, Roopnagar, India (140001)

Corresponding Author:
Santosh Kumar Vipparthi
CVPR Lab, Dept. of Electrical Engineering, Indian Institute of Technology,
Ropar, Roopnagar, India (140001)
Tel: +91-9549658135
<EMAIL_ADDRESS>
###### Abstract

Automated human emotion recognition from facial expressions is a well-studied
problem and still remains a very challenging task. Some efficient or accurate
deep learning models have been presented in the literature. However, it is
quite difficult to design a model that is both efficient and accurate at the
same time. Moreover, identifying the minute feature variations in facial
regions for both macro and micro-expressions requires expertise in network
design. In this paper, we proposed to search for a highly efficient and robust
neural architecture for both macro and micro-level facial expression
recognition. To the best of our knowledge, this is the first attempt to design
a NAS-based solution for both macro and micro-expression recognition. We
produce lightweight models with a gradient-based architecture search
algorithm. To maintain consistency between macro and micro-expressions, we
utilize dynamic imaging and convert micro-expression sequences into a single
frame, preserving the spatiotemporal features in the facial regions. The
EmoNAS has evaluated over 13 datasets (7 macro expression datasets: CK+,
DISFA, MUG, ISED, OULU-VIS CASIA, FER2013, RAF-DB, and 6 micro-expression
datasets: CASME-I, CASME-II, CAS(ME)2̂, SAMM, SMIC, MEGC2019 challenge). The
proposed models outperform the existing state-of-the-art methods and perform
very well in terms of speed and space complexity.

###### keywords:

Human emotion, micro-expression, macro-expression, neural architecture search
(NAS), deep learning

††journal: Expert Systems with Applications

##  1 Introduction

Human emotion recognition from visual content has attracted significant
attention in the last few decades. Facial appearances are one of the most
discriminative features of emotional responses Ekman & Friesen [1971].
Therefore, researchers have primarily focused on developing robust facial
expression recognition (FER) systems for emotion recognition. It has numerous
applications in human-computer interaction, behavior profiling, and smart
healthcare solutions.

Figure 1: The training and evaluation procedure of the proposed EmoNAS. In the
training phase, the regular and reduction cells are optimized. In evaluation,
the optimized cells are stacked to classify the input image. d-s: depthwise-
separable, dil: dilated

Facial expressions can be classified into two types: macro and micro-
expressions. Macro expressions are considered as a prevailing display of
emotions and usually last up to 3 to 4 minutesEkman [2003]. Whereas, micro-
expressions are generated when someone tries to hide their actual feelings.
Micro-expression appears on the facial region for a fraction of time
(1/3-1/22) secondsVerma et al. [2019b]. The existing state-of-the-art
convolutional neural network (CNN) based methods have primarily improved upon
the FER performance and few others have focused on improving the efficiency
through smaller network design Verma et al. [2019a]. However, state-of-the-art
approaches require the substantial effort of human expertise to design an
effective neural network architecture. Due to the human efforts involved in
deep network design, there is room for improvement in both efficiency and
accuracy. Moreover, the subtle variations in certain facial regions lead to
changes in emotion class. There is a need for developing robust and
lightweight FER models for real-world applications.

Recently, there has been a growing interest in developing algorithmic
solutions to automatically design an architecture for deep learning models.
These algorithms are known as neural architecture search (NAS). Inspired by
the differentiable architecture search algorithm Liu et al. [2018b], in this
paper, we present EmoNAS to search for the most robust and efficient neural
architecture for FER. The training and evaluation procedure of the proposed
EmoNAS is shown in Figure 1. As discussed earlier, macro and micro expressions
are different in nature, so it is challenging to design a network that can
efficiently work for both types of expressions. (more detailed differences
between macro and micro expressions are discussed in the supplementary
document). Therefore, solutions for both MaEs and MEs are non-identical in the
literature. Therefore, Most of the existing FER algorithms are either designed
for macro or micro expression classification. Some of the concepts utilize the
macro adaptive features for micro-expression classification Sun et al. [2020],
Yang et al. [2021]. However, there is no single algorithm to solve macro and
micro-expression recognition. We made the first attempt to introduce a
universal NAS algorithm to solve the generic macro facial expressions as well
as micro facial expressions. Usually, a single instance of an image is
sufficient for macro expression recognition analysis whereas, MER requires
spatiotemporal data due to its fleeting and short-lived nature. Therefore, to
maintain a uniform experimental setting, we adopted the dynamic imaging
concept to extract a single instance feature map consisting of the
spatiotemporal features from the temporal sequence of frames. Therefore, this
paper aims to provide a uniform search and training framework to target the
challenges in FER and MER using an EmoNAS. Our proposed framework EmoNAS has
the following contributions:

  1. 1.

We propose a differentiable architecture search-based algorithm named EmoNAS
to handle the challenges in macro and micro-expression with the optimized CNN
models. We propose to employ a uniform number of cells in the architecture
design as used in the search phase. Moreover, the search space is restricted
to a shallower structure to obtain lightweight models for real-time
applications.

  2. 2.

The EmoNAS achieves remarkable efficiency improvement over the existing deep
learning models in terms of the final model computational complexity.

  3. 3.

The effectiveness of EmoNAS is validated on 13 datasets: 7 macro facial
expression recognition (FER) and 6 micro-expression recognition (MER)
datasets. It achieves highly competitive results and outperforms most of all
the existing state-of-the-art methods in these 13 datasets. The impact of
selecting a different number of cells and nodes is extensively studied by
conducting 9 ablation experiments on CK+, DISFSA, CASME-I, and CAS(ME)2̂
datasets.

##  2 Related Work

###  2.1 Macro-expression

Macro-expression recognition, commonly referred to as facial expression
recognition (FER), has been widely studied in the literature. The deep
learning algorithms for FER Pons & Masip [2017], Xie et al. [2020] have far
outperformed the traditional hard-crafted approaches Mandal et al. [2019],
Iqbal et al. [2020]. Researchers have developed numerous CNN Verma et al.
[2019a], Zhang et al. [2016] for FER. Fan et al. Fan et al. [2020] proposed a
two-stage attention network to detect posed and spontaneous expressions.
Similarly, Li et al. Li et al. [2020b] used attention at the patch and whole-
face level for effective FER. Li and Deng Li & Deng [2020] conducted an
extensive experiment to delve into the bias across different FER datasets and
also explore the intrinsic causes of the dataset discrepancy. Wang et al. Wang
et al. [2020b] studied the effect of ambiguous annotations, low-quality
expression images, and obscure facial expressions over deep learning networks.
A more detailed study of the deep learning-based FER methods is given in Li &
Deng [2020].

###  2.2 Micro-expression

Micro-expression recognition (MER) has received much attention in past few
years. The earlier MER works focused on the spatiotemporal feature descriptors
and optical flow Liu et al. [2018c], Xu et al. [2017], Xia et al. [2020]. The
recent works have used the CNN, RNN, GANs, 3D-CNN and other deep learning
techniques to improve the performance Verma et al. [2020b], Sun et al. [2020],
Li et al. [2020a], Verma et al. [2020a], Wang et al. [2020a], Van Quang et al.
[2019], Liong et al. [2019a]. Liong et al. Liong et al. [2019a] proposed a
triple stream CNN by using three optical flow features (optical strain,
horizontal and vertical optical flow) computed between the onset and apex
frames of each sequence. Furthermore, Wang et al. Wang et al. [2020a] proposed
an attention-based residual network to guide the CNN towards the micro-
expression regions. More recently, a composite database for MER was presented
along with the benchmark experimental evaluation protocol in Zhang et al.
[2020]. Subsequently, Xia et al. Xia et al. [2020] studied the influence of
learning complexity on the composite database. They conclude that the low-
resolution input and shallower architecture are more suitable for composite-
dataset problems in comparison to the deep architectures. A more detailed
study of various MER methods can be found in Zhou et al. [2020].

###  2.3 Neural Architecture Search

Recently NAS algorithms intended for automatically searching and designing CNN
architectures have achieved very competitive performance in computer vision
tasks such as image classification Real et al. [2019] and object detection
Zoph et al. [2018]. Despite their remarkable performance, the earlier NAS
algorithms were computationally expensive and demands huge memory footprints.
The reinforcement learning (RL)Zoph et al. [2018] NAS requires 2000 GPU days
to get existing architecture for CIFAR-10 and ImageNet. Similarly, another NAS
algorithm Real et al. [2019] needs 3150 GPU days of evaluation. Furthermore,
various NAS algorithms Bender et al. [2018], Baker et al. [2017], Liu et al.
[2018a] have been introduced to speed up the search process. The RL,
evolution, MCTS Negrinho & Gordon [2017], SMBO Liu et al. [2018a] based NAS
algorithms consider the architecture search as a black-box optimization
problem over a discrete space, which leads to a huge number of architecture
assessments. Therefore,Liu et al. [2018b] relaxed the search space to be
continuous to optimize the architecture through gradient descent with respect
to its validation set performance. This is the most effective solution to
accelerate the search process. Therefore, we use this optimization technique
in our work. Recently Yu et al. Yu et al. [2020b, a, c] proposed three
different NAS based algorithm for facial anti-spoofing.The darts-based
approaches have focused on searching the inner cell structure, only. Li et al
Li et al. [2021] introduced a NAS-based algorithm Auto-FERNet to automatically
search a CNN model on a macro expression dataset.

The network design process for the existing deep learning FER and MER methods
requires a lot of manual effort. Usually, the designed network performs well
over a few expression datasets but does not transfer well to other expression
datasets. Moreover, the macro- and micro-expression recognition tasks require
quite different algorithmic approaches to obtain robust performance. Thus, it
is difficult to design a single network to effectively solve both the FER and
MER tasks. The NAS algorithms offer an alternative approach to universally
solving both the FER and MER problems by automatically searching for efficient
and robust architectures. Motivated by this, we propose the EmoNAS algorithm
and experimentally show its effectiveness in FER and MER. To the best of our
knowledge, this is the first attempt to solve both macro- and micro-expression
tasks with neural architecture search.

(a) Video sequence (b) Dynamic image

Figure 2: Sample visual representation of the dynamic image response for
micro-expression sequences. The dynamic image captures the spatiotemporal
features for subtle changes in facial regions.

##  3 Dynamic Imaging

The main aim of this paper is to provide a uniform search and training
framework to address the challenges in FER and MER using a NAS method.
Usually, a single instance of the image if sufficient for FER analysis
whereas, MER requires spatiotemporal data due to its fleeting and short-lived
nature. Therefore, to maintain a uniform experimental setting, we adopted the
dynamic imaging concept to extract a single instance feature map consisting of
the spatiotemporal features from the temporal sequence of frames. Most of the
existing MER approaches Van Quang et al. [2019], Li et al. [2020a], Liong et
al. [2019b], Liu et al. [2019b] rely only on the apex frame for the analysis.
However, some studies emphasize the importance of dynamic aspects for
detecting the subtle changes Ambadar et al. [2005] and their effect on the
performance of MER. In a micro-expression (MEs) video, each frame has its own
significance in the identification of the emotion class. Some recent
approaches Reddy Sai Prasanna Teja et al. [2019], Li et al. [2019a] utilized
video sequences to design an effective MER system. However, all publicly
available MER datasets hold videos with a variant number of frames. In order
to normalize the video sequences, some approaches have utilized time
interpolation algorithms. This may lose or alter the domain knowledge of
micro-expressions by shearing or filling holes in between the frames. To
overcome the above-mentioned issues and embed the subtle facial changes, we
have utilized the dynamic imaging concept Bilen et al. [2016] to generate a
single instance of an image. Dynamic imaging represents the video information
into a single instance by conserving high stake active dynamics of MEs. It
also ensures uniform search and training architecture for both macro and
micro-expression recognition.

The dynamic imaging technique has been effectively used in the recent
literature for micro-expression recognition Verma et al. [2019b]. The dynamic
image interprets the content of the video by aiming at the facial moving
regions and compresses that into a single instance. The dynamic image is an
RGB response holding the spatiotemporal features of a video sequence. The
dynamic image $d^{*}$ is calculated by Eq. 1-Eq. 5.

$d^{*}=\sum_{p=1}^{N}{\mathbb{Z}_{p}}$ (1)

where $\mathbb{Z}_{p}$ and $N\in q$ represent the $p^{th}$ intermediate motion
image and a total number of frames in a video $V$, respectively. The
intermediate motion image is calculated by using Eq. 2.

${\mathbb{Z}_{p}}=V_{p}\times F(p)$ (2)

where $F(p)$ implies the frame weight and is calculated by using Eq. 3.

$F(p)=\sum_{l=p}^{q}\mathbb{R}(1,l)$ (3)

where $V_{p}\in V$ represents the $p^{th}$ frame of the video V, $p=1,2,...,q$
and $\mathbb{R}$ is a ranking function which upgrade the rank of frames as
follows:

$\displaystyle\mathbb{R}(1,l)=\frac{2\times\mathbb{I}(1,l)-q}{\mathbb{I}(1,l)}$
(4) $\displaystyle\mathbb{I}(1,l)=(l,l+1,l+2,...q)$ (5)

where $q$ implies the total frames in video sequences. $l$ is a index of the
frames in range $\left[1,q\right]$. We have shown sample dynamic responses in
Figure 2. It can be observed that both the uniform and non-uniform features of
a video stream are embedded in a single frame. We use these dynamic images to
search and train the NAS models for micro-expression recognition.

##  4 EmoNAS

The proposed EmoNAS algorithm for architecture search is depicted in Figure 1.
We search for a computationally viable cell as the building block of the final
architecture. The cell structure is shown in Figure 3. The objective is to
discover the optimal cell structure that can lead to the best architecture
design for the specific dataset. We use two types of cells: regular cells and
reduction cells. Regular cells carry the feature maps to the next cell by
maintaining spatial resolution, while reduction cells imply the down-sampling.
Further, the learned cell is stacked to form a shallower convolutional
network. In the existing method Liu et al. [2018b], the structures discovered
through the search process are not necessarily optimal for FER/MER evaluation
due to the deep dense final architecture. In literature Verma et al. [2019b],
Pasupa & Sunhem [2016], it is well proven that deep networks fail to achieve
adequate performance over small-sized datasets. Most of the existing benchmark
FER datasets have a lesser number of image/video samples. Also, the features
extracted in the search step might get lost or distorted by more cell stacking
while training due to repetitive cross-correlation and pooling operations. We
are also motivated by the fact that the properties of deep and shallow
networks are quite different which has been studied in the literature
Srivastava et al. [2015]. Therefore, we stacked search cells (5 cells only)
within the premise of shallower networks to ensure better efficiency for real-
time applications. Through EmoNAS, we obtain lightweight networks with a
minimum number of cells without compromising the quantitative performance.

Figure 3: An overview of the cell architecture. Initially, the operations on
the edges are unknown. The operations are selected from 7 candidate
operations.

###  4.1 Architecture Search Space and Optimization

Our goal is to discover a robust cell topology and use it to develop a
convolutional network of cells. A cell is characterized as a directed acyclic
graph having $N$ nodes as shown in Figure 3. The nodes represent the feature
maps in convolutional networks and the directed edge $(m,n)$ represents an
operation $o^{(m,n)}$ that transforms input $x^{(m)}$ into response feature
maps. Each cell has 2 input nodes and 1 output node. The cell outputs from the
previous layers are used as input nodes. Eventually, the cell structure is
formed by learning the set of operations on its edges. A node $k$ in a cell
$c$ can be defined by using 5-tuples $(P_{1},P_{2},Q_{1},Q_{2},S)$ where,
$P_{1},P_{2}\in\mathbb{P}_{k}^{c}$, $Q_{1},Q_{2}\in\mathbb{Q}$ and $S$
represent selection of input tensors, selection of employed layers over
selected input tensors and method used to form an output tensor $OT_{l}^{C}$.
Further, the output tensor $OT^{C}$ of a cell can be calculated by using Eq.
6.

$OT^{C}=OT_{1}^{C}||OT_{2}^{C}||...||OT_{B}^{C}$ (6)

where $||$ indicate the concatenation operation.

The set of candidate operations are defined by $\kappa$, where each operation
is denoted by $o(\cdot)$. The continuous search space is used by relaxing the
choice of a particular operation to a softmax over all possible operations as
given in Eq. 7.

$\tilde{o}^{(m,n)}(x)=\sum_{o\in\kappa}\frac{exp(\alpha^{(m,n)}_{o})}{\sum_{{o}^{\prime}\in\kappa}exp(\alpha_{o^{\prime}}^{(m,n)})}o(x)$
(7)

where the operation mixing weights for a pair of nodes $(m,n)$ are learned by
a vector $\alpha^{(m,n)}$ of dimension $|\kappa|$. After the completion of
search, a discrete architecture is retrieved by replacing each mixed operation
with the most likely operation. We leverage the approximation scheme given in
Liu et al. [2018b] to perform the optimization.

Let $\lambda_{train}$ and $\lambda_{val}$ represent the training and the
validation loss, respectively. These losses are calculated based on both the
architecture $\alpha$ and weights $w$ in the network. The goal for EmoNAS is
to discover $\alpha^{*}$ that minimizes the validation loss
$\lambda_{val}(w^{*},\alpha^{*})$. The weights $w^{*}$ allied with the
architecture are computed by minimizing the training loss
$w∗=argmin_{w}\;\lambda_{train}(w,\alpha^{*})$. Thus, the bilevel optimization
problem with $\alpha$ as the upper-level and $w$ as the lower-level variable
is represented in Eq. 8 and Eq. 9.

$\min_{\alpha}\;\;\lambda_{val}(w^{*}(\alpha),\alpha)$ (8)
$s.t.\;\;\;\;w^{*}(\alpha)=argmin_{w}\;\;\lambda_{train}(w,\alpha)$ (9)

The detailed procedure and approximation technique is outlined in Liu et al.
[2018b].

###  4.2 Network Configurations

The candidate operation space $\kappa$ consists of the following set of 7
operations: $3\times 3$ depthwise-separable convolution, $5\times 5$
depthwise-separable convolution, identity, $3\times 3$ average pooling,
$3\times 3$ max pooling, $3\times 3$ dilated convolution, $5\times 5$ dilated
convolution and skip. connection. The network search and evaluation process is
discussed below.

####  4.2.1 Architecture Search

We use 5 cells with each containing 7 nodes. We intentionally select lower
number of cells to ensure discovery of lightweight and efficient networks.
Similarly, the search space is also limited to only 7 operations. These design
decisions helped us to create very lightweight architectures for FER over
different datasets. We were also motivated by the success of shallower CNN
models in FER to search the shallower architectures through NAS. To compose
the nodes in the discrete architecture, we retain the top-2 strongest
operations (from distinct nodes) among all the candidate operations. The
results analysis discussed in Section 5 validate that search space with 5
cells achieve better performance as compared to DARTS with significant
performance improvement over all the FER datasets.

####  4.2.2 Architecture Evaluation

In order to evaluate the discovered cell, we stack same 5 copies of the cells
(searched by architecture search), but untied weights with using either stride
1 or stride 2, as shown in Figure 1 to generate a network. The network is
generated by stacking reduction cells at 1/3 and 2/3 locations, the rest of
locations are stacked with regular cell in proposed EmoNAS. The number of
cells can be adjusted to improve the accuracy/efficiency of the network. We
analyze the impact of different parameter changes through ablation study in
Section 5.4. Each cell consists of two input nodes and one output node, where
input nodes are previous cell outputs and output node is concatenation on all
intermediate nodes. At the end of the network, we use global average pooling,
followed by a softmax classification layer. Furthermore, we trained the model
developed by stacking the cells, on the relevant dataset. More implementation
details are presented in Section 5.1.1.

Figure 4: Illustration of different expression samples of macro datasets with
challenging tasks a) CK+: posed expressions, b) MUG: posed expressions with
enriched resolution, c) DISFA: spontaneous expressions, d) ISED: spontaneous
expressions, e) OULU: illumination variations, f) FER-13: Real-time wild
expressions, and g) RAF-DB: in-the-wild expressions.

(a)

(b)

(c)

(d)

Figure 5: The final regular and reduction cell structures discovered by EmoNAS
on (a,b) CK+ (6-Exp), and (c,d) MUG (6-Exp).

##  5 Experiments and Results

###  5.1 Experimental Details

This section discusses the implementation details, dataset, and evaluation
strategies. Further, a comparative study of the proposed EmoNAS and state-of-
the-art approaches is presented. We carry out the ablation study and
computational analysis in the following subsection. document.

####  5.1.1 Implementation Details

Initially, to search for the best possible cell structure, we used 50 epochs
and 16 for batch size due to GPU memory constraints. The main goal of the cell
search is to learn the best hyper-parameter $\alpha$ that defines the best
possible operations in the cell structure. We used stochastic gradient descent
(SGD) optimizer with a momentum of 0.9, a cosine learning rate of 0.007, and a
weight decay of $3e^{-4}$. Further, we stacked 5 sets of discovered cells with
7 hidden nodes to design the final EmoNAS architecture. For training a model,
similar settings of searching like SGD optimizer with an initial learning rate
of 0.007, weight decay $3e^{-4}$, and momentum 0.9 are initialized. The batch
size is set to 16, and epochs are set to 70 for training a model. The cross-
entropy loss function is used for loss optimization. We implement our model
with Pytorch 0.3.1 and run all experiments on an NVIDIA RTX 2080Ti GPU. For
each tested dataset, the architecture search is conducted. The same NAS code
and hyper-parameters (different image sizes) are used for all the datasets.

Table 1: Comparative results of the proposed EmoNAS and existing state-of-the-
art FER methods on CK+ dataset.

|  | CK+
---|---|---
Method | Pub-Yr | 6 Exp | 7 Exp
RADAP Mandal et al. [2019] | IP-19 | 88.48 | 83.72
sLSP+LB Iqbal et al. [2020] | TAFF-20 | 96.77 | 95.13
VGG16 12 Simonyan & Zisserman [2014] | Arxiv-15 | 91.31 | 88.18
VGG19 12 Simonyan & Zisserman [2014] | Arxiv-15 | 89.98 | 78.71
ResNet50 12 He et al. [2015] | CVPR-16 | 89.32 | 87.31
HiNet Verma et al. [2019a] | LCS-19 | 91.40 | 88.60
DCMA-CNN Xie & Hu [2019] | TMM-19 | 93.46 | N/A
DLP-CNN Li et al. [2017] | CVPR-17 | 95.78 | N/A
Lopes Lopes et al. [2017] | PR-17 | 96.76 | 95.75
IA-gen Yang et al. [2018] | FG-18 | 96.57 | N/A
Khor-Net Khorrami et al. [2015] | ICCVW-15 | 95.70 | N/A
IF-GAN Cai et al. [2019] | Arxiv-19 | 95.90 | N/A
DARTS * Liu et al. [2018b] | ICML-19 | 91.50 | 95.01
P-DART * Chen et al. [2019a] | ICCV-19 | 85.93 | 86.85
Auto-FERNet Li et al. [2021] | TNSE-21 | N/A | 98.89
ViT-SE Aouayeb et al. [2021] | Arxiv-21 | N/A | 99.80
Squeeze ViT Kim et al. [2022] | Sensors-21 | N/A | 99.54
EmoNAS | Ours | 97.13 | 96.30

  * Here, * and $A^{a}[b]$ implies for re-implemented results and a: results taken from, b: reference of the method. While, Pub-Yr, 6 Exp, and 7 Exp represent Publication-Year, 6 expression, and 7 expression classes.

(a)

(b)

(c)

(d)

Figure 6: The final regular and reduction cell structures discovered by EmoNAS
on (a,b) CASME-II and (c,d) SAMM.

####  5.1.2 Datasets and Evaluation Settings

To evaluate the proposed EmoNAS, we conduct experiments over 7 macro-
expression datasets CK+ Lucey et al. [2010], MUG Aifanti et al. [2010], ISED
S. L. Happy et al. [2017], DISFA Mavadati et al. [2013], OULU-CASIA Zhao et
al. [2011], FER2013 Kaggle.com [2019 (accessed December 3, 2019], and RAF-DB
Li et al. [2017]. The sample facial expressions from these datasets are
depicted in Figure 4. We adopted a very strict experiment setting of
subject/person independent evaluation. The models are tested over completely
unseen subjects. This makes the achieved results very reliable. For FER2013,
we follow the standard train-test scheme presented in the literatureHappy &
Routray [2017]. The input image size of $120\times 120$ is used for CK+, MUG,
ISED and DISFA. For OULU-VIS-Strong and FER2013, we use the image size of
$128\times 128$ and $48\times 48$, respectively. The final cell structures
discovered by EmoNAS for CK+ (6-exp), MUG (6-exp) are depicted in Figure 5.

We also conduct experiments on 6 micro-expression datasets CASME-I Yan et al.
[2013], CASME-II Yan et al. [2014], CAS(ME)2̂ Qu et al. [2017], SAMM Davison
et al. [2016], SMIC Li et al. [2013], Composite (MEGC2019 challenge) dataset
See et al. [2019]. The video sequences in these datasets are preprocessed to
create dynamic image Bilen et al. [2016]. We use input size of $120\times 120$
for all 6 datasets. The final cell structures discovered by EmoNAS for
CASME-I, CASME-II, CAS(ME)2̂, and SAMM are depicted in Figure 6. Similar to
macro-expression, the models are evaluated on completely unseen subjects in
leave-one-person-out manner.

Table 2: Comparative results of the proposed EmoNAS and existing state-of-the-
art FER methods on MUG, ISED, DISFA dataset

|  | MUG | ISED | DISFA
---|---|---|---|---
Method | Pub-Yr | 6 Exp | 7 Exp | 4 Exp | 5 Exp | 6 Exp | 7 Exp
RADAP Mandal et al. [2019] | IP-19 | 82.65 | 78.57 | 67.05 | N/A | 62.38 | 58.71
sLSP+LB Iqbal et al. [2020] | TAFF-2- | N/A | N/A | 78.03 | N/A | N/A | N/A
VGG16 12 Simonyan & Zisserman [2014] | Arxiv-15 | 85.14 | 84.67 | 73.09 | 59.32 | 56.76 | 57.42
VGG19 12 Simonyan & Zisserman [2014] | Arxiv-15 | 85.22 | 85.12 | 70.24 | 69.20 | 59.98 | 53.66
ResNet50 12 He et al. [2015] | CVPR-16 | 86.88 | 85.58 | 68.10 | 63.47 | 62.48 | 54.01
HiNet Verma et al. [2019a] | LCS-19 | 87.80 | 87.20 | N/A | N/A | N/A | N/A
DARTS * Liu et al. [2018b] | ICML-19 | 94.39 | 93.70 | 76.87 | 47.29 | 66.55 | 49.98
P-DART * Chen et al. [2019a] | ICCV-19 | 67.19 | 91.16 | 71.43 | 53.50 | 45.14 | 33.58
EmoNAS | Ours | 95.60 | 96.55 | 79.93 | 65.88 | 69.18 | 61.2

###  5.2 Macro-Expression Results Analysis

We compute the classification accuracy to evaluate the proposed EmoNAS on the
macro-expression datasets CK+, MUG, ISED, DISFA, OULO-VIS-Strong, FER2013, and
RAF-DB. The results on all the datasets are shown in Table 1, Table 2, Table
3, Table 4, Table 5. The results for VGG16, VGG19, ResNet50 and MobileNet were
collected from Mandal et al. [2019] and Verma et al. [2019a].

####  5.2.1 CK+

The proposed method is compared with existing deep learning methods Xie & Hu
[2019], Li et al. [2017], Lopes et al. [2017], Yang et al. [2018], Liu et al.
[2014], Khorrami et al. [2015], Cai et al. [2019], Verma et al. [2019a] in
Table 1. The architecture discovered by the proposed EmoNAS outperforms the
existing state-of-the-art deep learning methods. The proposed NAS-based
approach leads to the discovery of lightweight architectures as compared to
the existing FER methods (refer Table 14). From Table 14 we can observe that
our model consists of lesser number of parameters ($\sim$0.55 M) in comparison
to manually designed lightweight networks such as MobileNet ($\sim$3.2 M) and
HiNet Verma et al. [2019a] ($\sim$1 M). In addition, it outperforms the
existing methods in quantitative performance as well. EmoNAS obtains 5.73% and
7.7% higher recognition accuracy than HiNet in 6-class and 7-class FER,
respectively. Similarly, it also outperforms the existing NAS based method
DARTS Liu et al. [2018b] (by 5.63%, 1.29%) and P-DARTS Chen et al. [2019b] (by
11.2%, 9.45%) in 6- and 7-class problems, respectively. The results show that
straightforward use of NAS algorithms (designed for generic classification
problems) is not sufficient to obtain good performance in FER problems.
However, a well-designed NAS method, keeping the challenges in FER in mind,
can achieve significantly better performance than the traditional CNN methods.
from the results, we can observe that the Auto-FERNet is gaining more accuracy
as compared to the proposed EmoNAS for CK+ dataset. However, Auto-FER requires
approximately double computational power as compared to the proposed EmoNAS as
tabulated in Table 14. Also, recent vision transformer-based models Aouayeb et
al. [2021], Kim et al. [2022] are achieving better performance on 7 expression
class problems. Still, vision transformer-based models Kim et al. [2022] need
a huge computational power than compared to the proposed EmoNAS as reported in
Table 14

Table 3: Comparative results of the proposed EmoNAS and existing state-of-the-art methods on OULU-VIS Strong dataset |  | OULU-VIS Strong
---|---|---
Method | Pub-Yr | 6 Exp | 7 Exp
RADAP Mandal et al. [2019] | IP-19 | 75.83 | 74.11
VGG16 12 Simonyan & Zisserman [2014] | Arxiv-15 | 73.40 | 70.70
VGG19 12 Simonyan & Zisserman [2014] | Arxiv-15 | 71.4 | 70.5
ResNet5 12 He et al. [2015] | CVPR-16 | 73.1 | 65.4
MobileNet 7 Howard et al. [2017] | Arxiv-17 | 60.9 | 60.4
HiNet Verma et al. [2019a] | LCS-19 | 70.3 | 72.0
EmoNAS | Ours | 93.54 | 90.71
Table 4: Comparative results of the proposed EmoNAS and existing state-of-the-art FER methods on FER2013 dataset Method | Pub-Yr | 7 Exp (pr)
---|---|---
Bag of Words Ionescu et al. [2013] | ICML-13 | 67.4
Mollahosseini Mollahosseini et al. [2016] | WACV-16 | 66.4
CNN-Ensemble Liu et al. [2016] | CW-16 | 65.0
VGG+SVM Georgescu et al. [2018] | IA-18 | 66.3
GoogleNet Giannopoulos et al. [2018] | AHIM-18 | 65.2
Fa-Net Wang et al. [2019] | Arxiv-19 | 62.3
Auto-FERNet Li et al. [2021] | TNSE-21 | 73.1
EmoNAS | Ours | 67.9
Table 5: Comparative results of the proposed EmoNAS and existing methods on
RAF-DB dataset

Method | Pub-Yr | Angry | Disgust | Fear | Happy | Sad | Surprise | Neutral | Acc.
---|---|---|---|---|---|---|---|---|---
baseDCNN Li et al. [2017] | CVPR-17 | 70.99 | 52.50 | 50.00 | 92.91 | 77.82 | 79.64 | 83.09 | 82.66
Center Loss Li et al. [2017] | CVPR-17 | 68.52 | 53.13 | 54.05 | 93.08 | 78.45 | 79.63 | 83.24 | 82.86
DLP-CNN Li et al. [2017] | CVPR-17 | 71.60 | 52.15 | 62.16 | 92.83 | 80.13 | 81.16 | 80.29 | 82.74
VGG-FACE Fan et al. [2018] | ICANN-18 | 82.19 | 56.62 | 55.41 | 86.38 | 79.52 | 83.93 | 71.18 | 79.16
MRE-VGG Fan et al. [2018] | ICANN-18 | 83.95 | 57.50 | 60.81 | 88.78 | 79.92 | 86.02 | 80.15 | 82.63
Kuo et al. Kuo et al. [2018] | CVPRW-18 | 82.07 | 44.59 | 41.25 | 81.01 | 44.14 | 90.12 | 75.44 | 72.21
FSN Zhao et al. [2018] | BMVC-18 | 72.80 | 46.90 | 56.80 | 90.50 | 81.60 | 81.80 | 76.90 | 81.14
PG-CNN Li et al. [2019b] | TIP-19 | - | - | - | - | - | - | - | 83.27
SPWFA-SE Li et al. [2020b] | TAFFC-20 | - | - | - | - | - | - | - | 86.31
DSAN-VGG Fan et al. [2020] | TAFFC-20 | 82.71 | 56.25 | 58.11 | 94.01 | 83.89 | 89.06 | 80.00 | 85.37
DSAN-RES Fan et al. [2020] | TAFFC-20 | 82.10 | 62.50 | 55.41 | 93.00 | 84.94 | 79.94 | 83.97 | 85.27
ViT-SE Aouayeb et al. [2021] | Arxiv-21 | - | - | - | - | - | - | - | 87.22
Squeeze ViT Kim et al. [2022] | Sensors-21 | - | - | - | - | - | - | - | 88.90
DAN Wen et al. [2021] | Arxiv-21 | - | - | - | - | - | - | - | 85.32
DPOSTER Zheng et al. [2022] | Arxiv-21 | - | - | - | - | - | - | - | 86.03
EmoNAS | Ours | 83.78 | 86.02 | 70.62 | 77.77 | 83.05 | 94.85 | 82.37 | 87.28

####  5.2.2 MUG, ISED and DISFA

We further evaluate the quantitative results for ISED, DISFA, and MUG in Table
2. The MUG consists of posed facial expressions whereas, ISED and DISFA
consist of spontaneous expressions. for MUG, the proposed EmoNAS beats the
other NAS approaches DARTS Liu et al. [2018b] (by 1.21%, 2.85%) and P-DARTS
Chen et al. [2019b] (by 28.41%, 5.39%) in 6- and 7-class FER, respectively. In
ISED, our method substantially improves upon the DARTS Liu et al. [2018b] (by
2.76% and 18.59%) and P-DARTS Chen et al. [2019b](by 8.5% and 12.38%) in 4-
and 5-class problems, respectively. Furthermore, in DISFA, the proposed EmoNAS
outperforms DARTS Liu et al. [2018b] (by margins of 2.63% and 11.22%) and
P-DARTS Chen et al. [2019b] (by margins of 24.04% and 27.62%) in 6- and
7-class problems, respectively. In comparison to the traditional designed CNN
networks, the proposed NAS based models show improvement in both the
quantitative performance (Table 2) and computational complexity (Table 14).
The results also show that the proposed method is effective for both posed and
spontaneous FER problems.

####  5.2.3 OULU-VIS-Strong and FER 2013

The quantitative results in OULU-VIS-Strong are shown in Table 3. From Table
3, it is evident that the proposed models derived from EmoNAS obtain superior
performance over existing approaches. Our model achieves 23.24%, 18.71%
improvement over HiNet in 6- and 7-class FER, respectively. We also evaluate
our model over the private test set of FER2013 in the 7-class setting in Table
4. The FER 2013 is a challenging benchmark FER dataset. The previous methods
Mollahosseini et al. [2016], Liu et al. [2016], Wang et al. [2019],
Giannopoulos et al. [2018], Georgescu et al. [2018], Ionescu et al. [2013]
have used the traditional CNN networks to improve the performance. Our NAS-
based approach clearly outperforms the existing state-of-the-art approaches
Mollahosseini et al. [2016], Liu et al. [2016], Wang et al. [2019],
Giannopoulos et al. [2018], Georgescu et al. [2018], Ionescu et al. [2013].
This further shows the robustness of the proposed EmoNAS to a variety of FER
datasets collected from a diverse set of scenarios.

####  5.2.4 RAF-DB

We also conduct experiments on the in-the-wild RAF-DB dataset. We search and
train the EmoNAS with the seven expression labels provided in the RAF-DB.
Table 5 shows the class-wise accuracy and the overall accuracy obtained by
EmoNAS for RAF-DB. The class-wise accuracy for anger, disgust, fear, happy,
sad, surprise, and neutral (83.78%, 86.02%, 70.62%, 77.77%, 83.05%, 94.85%,
82.37%) show that each specific expression is correctly recognized. We also
compare our results with the existing state-of-the-art approaches. The
proposed EmoNAS (87.28% overall accuracy) outperforms the all most all other
methods in the literature. More specifically, it obtains 2.01%, 1.91%, 0.97%,
4.01% improvement over the recent works published in TIP-19 (PG-CNN Li et al.
[2019b]), TAFFC-20 (SPWFA-SE Li et al. [2020b]), TAFFC-20 (DSAN-VGG Fan et al.
[2020]), TAFFC-20 (DSAN-RES Fan et al. [2020]), respectively. Furthermore, the
results reveal that the EmoNAS attains the highest accuracy when identifying
the category of disgust, fear, and surprise. It also maintains good accuracy
in the remaining categories, leading to overall accuracy, better than the
existing state-of-the-art. Therefore, the features learned and chosen by
EmoNAS should contain more discriminative information for FER. This
illustrates the effectiveness and superiority of the NAS-based search for the
most robust architecture over the traditional CNN models.

Method | Pub-Yr | Task | CASME-I | CASME-II
---|---|---|---|---
MDMO Yong-Jin et al. [2015] | TAFFC-15 | [P, N, S, O] | 56.29 | 51.69
SparseMDMO Liu et al. [2018c] | TAFFC-18 | [P, N, S, O] | 74.83 | 66.95
CNN-LSTM Wang et al. [2018b] | Neuro-18 | [H, S, D, R, O] | 60.98 | N/A
3D-Flow Li et al. [2019a] | PAA-19 | [H, S, D, R, T] | 55.44 | 59.11
Spatio-Temp Kim et al. [2017] | TAFFC-17 | [H, S, D, R, O] | N/A | 60.98
FuseNet *Reddy Sai Prasanna Teja et al. [2019] | IJCNN-18 | [P, N, S, O] | 54.84 | 45.11
3D-ResNet34 * He et al. [2015] | CVPR-16 | [P, N, S, O] | 55.21 | 35.21
3D-ResNet50 * He et al. [2015] | CVPR-16 | [P, N, S, O] | 56.26 | 35.78
OrigiNet Verma et al. [2020b] | IJCNN-20 | [P, N, S, O] | 66.30 | 61.58
Aff.Net Verma et al. [2020a] | IEEEMM-21 | [P, N, S, O] | 66.99 | 61.58
DARTS * Liu et al. [2018b] | ICLR-19 | [P, N, S, O] | 73.72 | 59.63
P-DART * Chen et al. [2019a] | ICCV-19 | [P, N, S, O] | 57.03 | 64.38
AutoMER Verma et al. [2021] | TNNLS-21 | [P, N, S, O] | 77.58 | 74.15
ADL_Class | Ablation | [P, N, S, O] | 76.04 | 75.35
EmoNAS | Ours | [P, N, S, O] | 80.00 | 68.42

  * H: Happy, S: Surprise, D: Disgust, R: Regression, O: Others, P: Positive, N: Negative.

Table 6: Comparative results of the proposed EmoNAS and existing approaches on
CASME-I and CASME-II

Method | Pub-Yr | Task | CAS(ME)2̂ | SAMM
---|---|---|---|---
OrigiNet Verma et al. [2020b] | IJCNN-20 | 3/4 EXP | 52.85 | 47.46
Aff.Net Verma et al. [2020a] | IEEEMM-21 | 3/4 EXP | 54.56 | 34.89
FuseNet * Reddy Sai Prasanna Teja et al. [2019] | IJCNN-19 | 3/4 EXP | 50.77 | 46.72
3D-ResNet34 * He et al. [2015] | CVPR-16 | 3/4 EXP | 44.59 | 47.13
3D-ResNet50 * He et al. [2015] | CVPR-16 | 3/4 EXP | 43.09 | 49.89
DARTS * Liu et al. [2018b] | ICLR-19 | 3/4 EXP | 58.20 | 76.98
P-DART * Chen et al. [2019a] | ICCV-19 | 3/4 EXP | 58.19 | 72.88
AutoMER Verma et al. [2021] | TNNLS-21 | 3/4 EXP | 78.08 | 72.45
ADL_Class | Ablation | 3/4 EXP | 70.78 | 75.11
EmoNAS | Ours | 3/4 EXP | 71.15 | 77.04

Table 7: Comparative results of the proposed EmoNAS and existing approaches on
CASME2̂ and SAMM.

###  5.3 Micro-Expression Results Analysis

Usually, the macro expression recognition algorithms rely on static images for
analysis. Whereas, the micro-expression (MEs) recognition algorithms require a
video sequence as input. For micro-expression, we use dynamic imaging to
generate a single instance of an image micro-expressions. DI ensures uniform
search and training architecture for both macro and micro-expression
recognition problems. We use the generated dynamic images to search and train
the NAS models for micro-expression.

####  5.3.1 CASME-I and CASME-II

For micro-expression, the effectiveness is examined by testing in terms of
average classification accuracy. CASME-I dataset consists of 195 video clips
of 35 subjects of Chinese ethnicity. We cataloged the micro-expressions into 4
groups: positive-12, negative-50, surprise-21 and others-106 as given in Verma
et al. [2019b]. Similarly, CASME-II videos are grouped into positive-12,
negative-50, surprise-21, and others-106. Table 6 shows the classification
accuracy results for CASME-I and CASME-II. We compare our results with both
handcrafted and deep learning methods in the literature. A recently published
work in Xia et al. [2020] re-evaluated several previous methods over the
leave-one-subject-out (LOSO) protocol for a fair comparison. In our
experiments, we follow a similar setup and compare our proposed method with
the comparative results as presented in Xia et al. [2020]. The proposed EmoNAS
significantly outperforms the existing methods in recognition accuracy. It
also improves upon the existing NAS methods DARTS Liu et al. [2018b] (by
6.28%, 8.79%) and P-DARTS Chen et al. [2019b] (by 22.97%, 4.04%) in CASME-I
and CASME-II, respectively.

Table 8: Recognition accuracy and unweighted average recall (UAR) comparison
on SMIC and Composite dataset (MEGC2019 Challenge)

Method | Pub-Year | SMIC | COMPOSITE
---|---|---|---
Acc. | UAR | UAR
LBP-TOP Guoying & Pietikainen [2007] | A-Tran-07 | N/A | 52.80 | 57.85
MDMO Yong-Jin et al. [2015] | TAFFC-15 | 64.00 | 56.91 | N/A
OffApexNet Gan et al. [2019] | SP-IC-19 | 67.68 | N/A | N/A
CNN-LSTM Wang et al. [2018a] | Ar-18 | N/A | 41.25 | 39.24
TSCNN Song et al. [2019] | Access-19 | N/A | 54.12 | 59.48
STSTNet Liong et al. [2019a] | FG-19 | N/A | 59.95 | 67.24
NMER Liu et al. [2019b] | FG-19 | N/A | 55.55 | 59.36
DualInc Zhou et al. [2019] | FG-19 | N/A | 61.49 | 68.58
CapsuleNet Van Quang et al. [2019] | FG-19 | N/A | 58.77 | 65.06
DCN-DB Xia et al. [2019] | ICBEA-19 | N/A | 59.79 | 60.37
DSSN Khor et al. [2019] | ICIP-19 | 63.41 | N/A | N/A
RCN-W Xia et al. [2020] | TIP-20 | N/A | 66.00 | 71.00
LGCconD Li et al. [2020a] | TIP-20 | 63.41 | N/A | N/A
AutoMER Verma et al. [2021] | TNNLS | 81.20 | N/A | N/A
ADL_Class | Ablation | 71.39 | 58.54 | 78.76
EmoNAS | Ours | 77.09 | 69.03 | 73.22

####  5.3.2 CAS(ME)2̂ and SAMM

CAS(ME)2̂ consists of 339 sequences with three emotion classes: anger-101,
happy-149, and disgust-88. SAMM includes 159 spontaneous micro-expressions of
29 subjects, recorded at 200 fps. Similar to CASME-I and CASME-II, we combine
similar emotion classes to augment 4 groups: positive-27, negative-80,
surprise-15, and others-37. Table 7 shows the classification accuracy results
for CAS(ME)2̂ and SAMM. Our method outperforms the existing methods. More
specifically, it outperforms the FuseNet Reddy Sai Prasanna Teja et al. [2019]
by 20.38% and 30.32% in CAS(ME)2̂ and SAMM, respectively. It also improves
upon DARTS Liu et al. [2018b] (by a margin of 12.95% and 0.06%) and PDARTS
Chen et al. [2019b] (by a margin of 12.96% and 4.16%) in CAS(ME)2̂ and SAMM,
respectively.

Table 9: EmoNAS ablation analysis on CK+ with 6 expressions #Cells | #Nodes
---|---
6 | 7 | 8
5 | 95.67 | 97.13 | 95.96
8 | 84.93 | 85.23 | 83.88
10 | 96.15 | 96.04 | 95.84
15 | 94.78 | 96.11 | 93.42

####  5.3.3 SMIC and Composite Dataset

We also conduct experiments on SMIC and the Composite dataset See et al.
[2019]. SMIC consists of 164 emotion sequences from 16 subjects. The emotions
are grouped into positive-51, negative-70, and surprise-43. The Composite
dataset is collected by combining three emotion classes (negative, positive,
and surprise) of three datasets CASME-II, SAMM, and SMIC as given in MEGC-2019
See et al. [2019] challenge. The quantitative results for SMIC and Composite
datasets are given in Table 8. In SMIC, the proposed EmoNAS obtains superior
recognition accuracy and unweighted average recall (UAR) over the existing
methods. It outperforms the most recent method LGCconD Li et al. [2020a]
(TIP-20) in accuracy by a margin of 13.68%. Similarly, it outperforms RCN-W
Xia et al. [2020] (TIP-20) in UAR by 3.03% margin. In the Composite dataset,
our EmoNAS achieves 2.22% higher UAR as compared to the current state-of-the-
art method RCN-W Xia et al. [2020] (TIP-20).

Table 10: EmoNAS ablation analysis on DISFA with 6 expressions #Cells | #Nodes
---|---
6 | 7 | 8
5 | 63.89 | 69.18 | 72.57
8 | 68.84 | 73.28 | 71.54
10 | 64.63 | 69.89 | 73.37
15 | 65.68 | 67.17 | 71.03

###  5.4 Ablation Study

We study the effect of multiple components of EmoNAS through ablation analysis
to understand its behavior for the task of FER. We conduct experiments on two
macro-expression datasets CK+, DISFA, and two micro-expression datasets. The
ablation results are shown in Table 9, Table 10, Table 11, and Table 12. We
change the number of nodes within each cell to 6, 7, and 8 nodes. We stack
these cells with 5, 8, 10, and 15 repetitions to obtain 12 different CNN
structures. Thus, for each dataset, we conduct 12 ablation experiments.

From Table 9 and Table 10 we can see that increasing the number nodes to 8 or
decreasing to 6 resulted in marginal decrease or increase in the accuracy for
CK+ and DISFA. Similarly, increasing the number of cells to 10/15 doesn’t
improve the performance substantially in both datasets. Moreover, a higher
number of cells would increase the network complexity. Therefore, in most
cases, we select 5 cells with each having 7 nodes to maintain a good trade-off
between accuracy and model efficiency. The micro-expression ablations results
are given in Table 11 and Table 12. The best performance is achieved with 5
cells and 7 nodes. This again shows that the combination of 5 cells and 7
nodes are the most suitable parameters for FER datasets. These results also
prove the literature Verma et al. [2019b], Pasupa & Sunhem [2016], Srivastava
et al. [2015] deep networks fail to achieve adequate performance over small-
sized datasets. In addition, we conducted an ablation study with the
architecture search of the autodeeplab Liu et al. [2019a] for the
classification (ADL_Class). The experiments are conducted over the five micro
expression datasets CASME-I, CASME-II, CAS(ME)2̂, SMIC, SAMM along with
Composite dataset and the results are included in Table 6, 7 and 8.

Table 11: EmoNAS ablation analysis on CASME-I #Cells | #Nodes
---|---
6 | 7 | 8
5 | 69.50 | 80.00 | 70.65
8 | 79.29 | 74.31 | 73.46
10 | 73.37 | 77.19 | 76.17
15 | 75.74 | 72.11 | 74.52
Table 12: EmoNAS ablation analysis on CAS(ME)2̂ #Cells | #Nodes
---|---
6 | 7 | 8
5 | 67.90 | 71.15 | 69.30
8 | 73.25 | 74.95 | 69.82
10 | 70.44 | 67.57 | 62.16
15 | 70.63 | 65.56 | 60.56
Table 13: Memory, speed, and complexity analysis of EmoNAS models discovered on macro: CK$+$, MUG, ISED, DISFA, OULU-VIS-Strong, FER2013 and micro: CASME-I, CASME-II, CAS(ME)2̂, SAMM datasets. Dataset | Input Size | Size(mb) | Speed(fps) | Params(M)
---|---|---|---|---
CK$+$ | $120\times 120$ | 15.45 | 60.30 | 0.90
MUG | $120\times 120$ | 13.75 | 17.15 | 0.49
ISED | $120\times 120$ | 13.93 | 56.25 | 0.51
DISFA | $120\times 120$ | 14.15 | 30.90 | 0.58
OULU-VIS-Strong | $128\times 128$ | 14.00 | 54.50 | 0.59
FER2013 | $48\times 48$ | 7.43 | 200.80 | 1.36
CASME-I | $128\times 128$ | 14.63 | 42.45 | 0.70
CASME-II | $180\times 180$ | 16.43 | 32.63 | 0.82
CAS(ME)2̂ | $180\times 180$ | 14.94 | 31.68 | 0.53
SAMM | $120\times 120$ | 13.64 | 40.56 | 0.6
Table 14: Comparative analysis of the complexity of the EmoNAS with existing
methods.

Method | Pub-Year | Task | Size (mb) | Speed (fps) | Params (M)
---|---|---|---|---|---
OrigiNet Verma et al. [2020b] | IJCNN-20 | MER | 14.3 |  $\sim$0.22 | 1.8
AffectiveNet Verma et al. [2020a] | Mult-20 | MER | 8.3 | 0.12 | 2.2
MobileNet Verma et al. [2019a] | LCS-19 | MER | 25.3 | 0.08 | 4.2
DARTS Liu et al. [2018b] | ICLR-19 | MER |  $\sim$16 |  $\sim$35 |  $\sim$0.9
P-DARTS Chen et al. [2019b] | ICCV-19 | MER | 52 |  $\sim$5 |  $\sim$3.6
EmoNAS | Ours | MER |  $\sim$14 |  $\sim$35 |  $\sim$0.60
MobileNet Verma et al. [2019a] | LCS-19 | FER | 23.8 | NA | 3.2
HiNet Verma et al. [2019a] | LCS-19 | FER | 5.3 | NA | 1.0
DARTS Liu et al. [2018b] | ICLR-19 | FER | 17 |  $\sim$35 |  $\sim$1.0
P-DARTS Chen et al. [2019b] | ICCV-19 | FER | 51 |  $\sim$5 |  $\sim$3.5
DARTS Liu et al. [2018b] | ICLR-19 | FER |  $\sim$17 |  $\sim$35 |  $\sim$1.0
Auto-FERNet Li et al. [2021] | TNSE-21 | FER | N/A | N/A |  $\sim$2.1
ViT Kim et al. [2022] | Sensors-22 | FER | N/A | N/A |  $\sim$86.86
Squeeze ViT Kim et al. [2022] | Sensors-22 | FER | N/A | N/A |  $\sim$11.96
EmoNAS | Ours | FER |  $\sim$14 |  $\sim$45 |  $\sim$0.55

###  5.5 Speed, Memory and Complexity analysis

The models generated through EmoNAS search are very lightweight and fast.
Table 13 shows the memory, speed, and computational complexity of the models
generated by EmoNAS in different macro and micro-expression datasets. The
proposed models require trainable parameters of approximately $\approx$ 0.5
Million (M) except for FER2013 and CK+. This is substantially lower than
MobileNet (3.2M), VGG16 (138M), VGG19 (144M) ResNet50 (31M). It is even lower
than HiNet (1M) with much better accuracy over the same. Also, our models use
less memory space ( 13-15MB) as compared to MobileNet (23.8 MB), VGG16 (500.3
MB), VGG19 (520.4 MB), and ResNet (88.4 MB). Hence, it can be useful for
deployment in embedded devices. The speed over GPU shows that the proposed
models are remarkably fast (up to 60 fps), which makes them suitable for real-
time applications.

We further compare our work with the existing methods for both MER and FER in
Table 14. The EmoNAS models require the lowest number of operations for
inference with $\sim$0.55-$\sim$0.60 million parameters in the model. Our
model size is $\sim$38 MB less than the existing NAS method (P-DARTS).
Similarly, the total number of parameters for EmoNAS is 6$\times$ less than
the P-DARTS. The inference speed in MER and FER is $\sim$35 and $\sim$45,
respectively. Moreover, our proposed EmoNAS is also more efficient as compared
to recent vision transformer based FER models as shown in Table 14.
Particularly, proposed EmoNAS is 155$\times$ and 20$\times$ less than the ViT
and Squeeze viT Kim et al. [2022] FER models, respectively. Thus, the
optimized networks discovered by EmoNAS are suitable for embedded devices used
in real-time applications.

##  6 Conclusions

To the best of our knowledge, this paper presents the first attempt at a NAS-
based approach for the task of FER in both macro and micro-level facial
expressions. We proposed EmoNAS which is based on the optimization techniques
presented in DARTS to expedite the searching process. The design decisions for
EmoNAS are the result of careful analysis of the FER domain-specific
challenges. The same is validated by its superior performance over DART and
P-DART methods. The architecture search also led to the formulation of
shallower and lightweight CNNs. The resulting models achieve better
performance compared to the existing state-of-the-art FER methods in 13
benchmark (7 FER and 6 MER) datasets. The searched models obtain higher
accuracy with a fraction of the computational cost and very high inference
speed. In the future, shared parameters-based schemes can be investigated in
the search process to discover even better performance-aware architectures.
Furthermore, the number of skip connections can be optimized separately for a
better trade-off between complexity and performance. The decision of selecting
the number of cells in evaluation could also be included as a parameter for
optimization while searching. The proposed framework is designed to work with
single image input. This requires pre-processing of the video data into a
single instance feature map for MER analysis. In the future, we plan to design
a framework to handle the image and raw video data input for both macro- and
micro-expression problems.

## References

  * Aifanti et al. [2010] Aifanti, N., Papachristou, C., & Delopoulos, A. (2010).  The mug facial expression database.  In 11th International Workshop on Image Analysis for Multimedia Interactive Services WIAMIS 10 (pp. 1–4).  IEEE.
  * Ambadar et al. [2005] Ambadar, Z., Schooler, J. W., & Cohn, J. F. (2005).  Deciphering the enigmatic face: The importance of facial dynamics in interpreting subtle facial expressions.  Psychological science, 16, 403–410.
  * Aouayeb et al. [2021] Aouayeb, M., Hamidouche, W., Soladie, C., Kpalma, K., & Seguier, R. (2021).  Learning vision transformer with squeeze and excitation for facial expression recognition.  arXiv preprint arXiv:2107.03107, .
  * Baker et al. [2017] Baker, B., Gupta, O., Raskar, R., & Naik, N. (2017).  Accelerating neural architecture search using performance prediction.  arXiv preprint arXiv:1705.10823, .
  * Bender et al. [2018] Bender, G., Kindermans, P.-J., Zoph, B., Vasudevan, V., & Le, Q. V. (2018).  Understanding and simplifying one-shot architecture search.  In ICML.
  * Bilen et al. [2016] Bilen, H., Fernando, B., Gavves, E., Vedaldi, A., & Gould, S. (2016).  Dynamic image networks for action recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 3034–3042).
  * Cai et al. [2019] Cai, J., Meng, Z., Khan, A.-S., Li, Z., O’Reilly, J., & Tong, Y. (2019).  Identity-free facial expression recognition using conditional generative adversarial network.  ArXiv, abs/1903.08051.
  * Chen et al. [2019a] Chen, X., Xie, L., Wu, J., & Tian, Q. (2019a).  Progressive differentiable architecture search: Bridging the depth gap between search and evaluation.  In Proceedings of the IEEE International Conference on Computer Vision (pp. 1294–1303).
  * Chen et al. [2019b] Chen, X., Xie, L., Wu, J., & Tian, Q. (2019b).  Progressive differentiable architecture search: Bridging the depth gap between search and evaluation.  In Proceedings of the IEEE International Conference on Computer Vision, (pp. 1294–1303).
  * Davison et al. [2016] Davison, A. K., Lansley, C., Costen, N., Tan, K., & Yap, M. H. (2016).  Samm: A spontaneous micro-facial movement dataset.  IEEE Transactions on Affective Computing, 9, 116–129.
  * Ekman [2003] Ekman, P. (2003).  Darwin, deception, and facial expression.  Annals of the New York Academy of Sciences, 1000, 205–221.
  * Ekman & Friesen [1971] Ekman, P., & Friesen, W. V. (1971).  Constants across cultures in the face and emotion.  Journal of personality and social psychology, 17, 124.
  * Fan et al. [2018] Fan, Y., Lam, J. C., & Li, V. O. (2018).  Multi-region ensemble convolutional neural network for facial expression recognition.  In International Conference on Artificial Neural Networks (pp. 84–94).  Springer.
  * Fan et al. [2020] Fan, Y., Li, V., & Lam, J. C. (2020).  Facial expression recognition with deeply-supervised attention network.  IEEE Transactions on Affective Computing, .
  * Gan et al. [2019] Gan, Y., Liong, S.-T., Yau, W.-C., Huang, Y.-C., & Tan, L.-K. (2019).  Off-apexnet on micro-expression recognition system.  Signal Processing: Image Communication, 74, 129–139.
  * Georgescu et al. [2018] Georgescu, M.-I., Ionescu, R. T., & Popescu, M. (2018).  Local learning with deep and handcrafted features for facial expression recognition.  IEEE Access, 7, 64827–64836.
  * Giannopoulos et al. [2018] Giannopoulos, P., Perikos, I., & Hatzilygeroudis, I. (2018).  Deep learning approaches for facial emotion recognition: A case study on fer-2013.  In Advances in hybridization of intelligent methods (pp. 1–16).  Springer.
  * Guoying & Pietikainen [2007] Guoying, Z., & Pietikainen, M. (2007).  Dynamic texture recognition using local binary patterns with an application to facial expressions.  In IEEE transactions on pattern analysis and machine intelligence (pp. 915–928).  volume 29.
  * Happy & Routray [2017] Happy, S. L., & Routray, A. (2017).  Fuzzy histogram of optical flow orientations for micro-expression recognition.  In IEEE Transactions on Affective Computing.
  * He et al. [2015] He, K., Zhang, X., Ren, S., & Sun, J. (2015).  Deep residual learning for image recognition.  2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (pp. 770–778).
  * Howard et al. [2017] Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017).  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  ArXiv, abs/1704.04861.
  * Ionescu et al. [2013] Ionescu, R. T., Popescu, M., & Grozea, C. (2013).  Local learning to improve bag of visual words model for facial expression recognition.  In Workshop on challenges in representation learning, ICML.
  * Iqbal et al. [2020] Iqbal, M. T. B., Ryu, B., Ramirez Rivera, A., Makhmudkhujaev, F., Chae, O., & Bae, S. (2020).  Facial expression recognition with active local shape pattern and learned-size block representations.  IEEE Transactions on Affective Computing, (pp. 1–1). doi:10.1109/TAFFC.2020.2995432.
  * Kaggle.com [2019 (accessed December 3, 2019] Kaggle.com (2019 (accessed December 3, 2019)).  Challenges in Representation Learning: Facial Expression Recognition Challenge.  URL: https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge/.
  * Khor et al. [2019] Khor, H.-Q., See, J., Liong, S.-T., Phan, R. C., & Lin, W. (2019).  Dual-stream shallow networks for facial micro-expression recognition.  In 2019 IEEE International Conference on Image Processing (ICIP) (pp. 36–40).  IEEE.
  * Khorrami et al. [2015] Khorrami, P., Paine, T. L., & Huang, T. S. (2015).  Do deep neural networks learn facial action units when doing expression recognition?  2015 IEEE International Conference on Computer Vision Workshop (ICCVW), (pp. 19–27).
  * Kim et al. [2017] Kim, D. H., Baddar, W. J., Jang, J., & Ro, Y. M. (2017).  Multi-objective based spatio-temporal feature representation learning robust to expression intensity variations for facial expression recognition.  In IEEE Transactions on Affective Computing (pp. 223–236).  volume 10.
  * Kim et al. [2022] Kim, S., Nam, J., & Ko, B. C. (2022).  Facial expression recognition based on squeeze vision transformer.  Sensors, 22, 3729\.
  * Kuo et al. [2018] Kuo, C.-M., Lai, S.-H., & Sarkis, M. (2018).  A compact deep learning model for robust facial expression recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition workshops (pp. 2121–2129).
  * Li et al. [2019a] Li, J., Wang, Y., See, J., & Liu, W. (2019a).  Micro-expression recognition based on 3d flow convolutional neural network.  In Pattern Analysis and Applications (pp. 1331–1339).  volume 22.
  * Li & Deng [2020] Li, S., & Deng, W. (2020).  Deep facial expression recognition: A survey.  IEEE Transactions on Affective Computing, .
  * Li & Deng [2020] Li, S., & Deng, W. (2020).  A deeper look at facial expression dataset bias.  IEEE Transactions on Affective Computing, (pp. 1–1). doi:10.1109/TAFFC.2020.2973158.
  * Li et al. [2017] Li, S., Deng, W., & Du, J. (2017).  Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild.  In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2852–2861).
  * Li et al. [2021] Li, S., Li, W., Wen, S., Shi, K., Yang, Y., Zhou, P., & Huang, T. (2021).  Auto-fernet: A facial expression recognition network with architecture search.  IEEE Transactions on Network Science and Engineering, 8, 2213–2222.
  * Li et al. [2013] Li, X., Pfister, T., Huang, X., Zhao, G., & Pietikäinen, M. (2013).  A spontaneous micro-expression database: Inducement, collection and baseline.  In 2013 10th IEEE International Conference and Workshops on Automatic face and gesture recognition (fg) (pp. 1–6).  IEEE.
  * Li et al. [2020a] Li, Y., Huang, X., & Zhao, G. (2020a).  Joint local and global information learning with single apex frame detection for micro-expression recognition.  IEEE Transactions on Image Processing, 30, 249–263.
  * Li et al. [2020b] Li, Y., Lu, G., Li, J., Zhang, Z., & Zhang, D. (2020b).  Facial expression recognition in the wild using multi-level features and attention mechanisms.  IEEE Transactions on Affective Computing, .
  * Li et al. [2019b] Li, Y., Zeng, J., Shan, S., & Chen, X. (2019b).  Occlusion aware facial expression recognition using cnn with attention mechanism.  IEEE Transactions on Image Processing, 28, 2439–2450.
  * Liong et al. [2019a] Liong, S.-T., Gan, Y., See, J., Khor, H.-Q., & Huang, Y.-C. (2019a).  Shallow triple stream three-dimensional cnn (ststnet) for micro-expression recognition.  In 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019) (pp. 1–5).  IEEE.
  * Liong et al. [2019b] Liong, S.-T., Gan, Y. S., John See, H.-Q. K., & Huang, Y.-C. (2019b).  Shallow triple stream three-dimensional cnn (ststnet) for micro-expression recognition.  In In 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (pp. 1–5).  IEEE.
  * Liu et al. [2019a] Liu, C., Chen, L.-C., Schroff, F., Adam, H., Hua, W., Yuille, A. L., & Fei-Fei, L. (2019a).  Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 82–92).
  * Liu et al. [2018a] Liu, C., Zoph, B., Neumann, M., Shlens, J., Hua, W., Li, L.-J., Fei-Fei, L., Yuille, A., Huang, J., & Murphy, K. (2018a).  Progressive neural architecture search.  In Proceedings of the European Conference on Computer Vision (ECCV) (pp. 19–34).
  * Liu et al. [2018b] Liu, H., Simonyan, K., & Yang, Y. (2018b).  Darts: Differentiable architecture search.  arXiv preprint arXiv:1806.09055, .
  * Liu et al. [2016] Liu, K., Zhang, M., & Pan, Z. (2016).  Facial expression recognition with cnn ensemble.  2016 International Conference on Cyberworlds (CW), (pp. 163–166).
  * Liu et al. [2014] Liu, P., Han, S., Meng, Z., & Tong, Y. (2014).  Facial expression recognition via a boosted deep belief network.  2014 IEEE Conference on Computer Vision and Pattern Recognition, (pp. 1805–1812).
  * Liu et al. [2019b] Liu, Y., Du, H., Zheng, L., & Gedeon, T. (2019b).  A neural micro-expression recognizer.  In 2019 14th IEEE international conference on automatic face & gesture recognition (FG 2019) (pp. 1–4).  IEEE.
  * Liu et al. [2018c] Liu, Y.-J., Li, B.-J., , & Lai, Y.-K. (2018c).  Sparse mdmo: Learning a discriminative feature for spontaneous micro-expression recognition.  In IEEE Transactions on Affective Computing.
  * Lopes et al. [2017] Lopes, A. T., de Aguiar, E., de Souza, A. F., & Oliveira-Santos, T. (2017).  Facial expression recognition with convolutional neural networks: Coping with few data and the training sample order.  Pattern Recognition, 61, 610–628.
  * Lucey et al. [2010] Lucey, P., Cohn, J. F., Kanade, T., Saragih, J., Ambadar, Z., & Matthews, I. (2010).  The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression.  In 2010 ieee computer society conference on computer vision and pattern recognition-workshops (pp. 94–101).  IEEE.
  * Mandal et al. [2019] Mandal, M., Verma, M., Mathur, S., Vipparthi, S. K., Murala, S., & Deveerasetty, K. K. (2019).  Regional adaptive affinitive patterns (radap) with logical operators for facial expression recognition.  IET Image Processing, 13, 850–861.
  * Mavadati et al. [2013] Mavadati, S. M., Mahoor, M. H., Bartlett, K., Trinh, P., & Cohn, J. F. (2013).  Disfa: A spontaneous facial action intensity database.  IEEE Transactions on Affective Computing, 4, 151–160.
  * Mollahosseini et al. [2016] Mollahosseini, A., Chan, D., & Mahoor, M. H. (2016).  Going deeper in facial expression recognition using deep neural networks.  2016 IEEE Winter Conference on Applications of Computer Vision (WACV), (pp. 1–10).
  * Negrinho & Gordon [2017] Negrinho, R., & Gordon, G. (2017).  Deeparchitect: Automatically designing and training deep architectures.  arXiv preprint arXiv:1704.08792, .
  * Pasupa & Sunhem [2016] Pasupa, K., & Sunhem, W. (2016).  A comparison between shallow and deep architecture classifiers on small dataset.  In 2016 8th International Conference on Information Technology and Electrical Engineering (ICITEE) (pp. 1–6).  IEEE.
  * Pons & Masip [2017] Pons, G., & Masip, D. (2017).  Supervised committee of convolutional neural networks in automated facial expression analysis.  IEEE Transactions on Affective Computing, 9, 343–350.
  * Qu et al. [2017] Qu, F., Wang, S.-J., Yan, W.-J., Li, H., Wu, S., & Fu, X. (2017).  Cas(me)2̂: A database for spontaneous macro-expression and micro-expression spotting and recognition.  IEEE Transactions on Affective Computing, 9, 424–436.
  * Real et al. [2019] Real, E., Aggarwal, A., Huang, Y., & Le, Q. V. (2019).  Regularized evolution for image classifier architecture search.  In Proceedings of the aaai conference on artificial intelligence (pp. 4780–4789).  volume 33.
  * Reddy Sai Prasanna Teja et al. [2019] Reddy Sai Prasanna Teja, Surya Teja Karri, Shiv Ram Dubey, & Snehasis Mukherjee (2019).  Spontaneous facial micro-expression recognition using 3d spatiotemporal convolutional neural networks.  In In 2019 International Joint Conference on Neural Networks (IJCNN) (pp. 1–8).  IEEE.
  * S. L. Happy et al. [2017] S. L. Happy, Priyadarshi Patnaik, Aurobinda Routray, & Rajlakshmi Guha (2017).  The indian spontaneous expression database for emotion recognition.  IEEE Transactions on Affective Computing, 8, 131–142.
  * See et al. [2019] See, J., Yap, M. H., Li, J., Hong, X., & Wang, S.-J. (2019).  Megc 2019–the second facial micro-expressions grand challenge.  In 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019) (pp. 1–5).  IEEE.
  * Simonyan & Zisserman [2014] Simonyan, K., & Zisserman, A. (2014).  Very deep convolutional networks for large-scale image recognition.  CoRR, abs/1409.1556.
  * Song et al. [2019] Song, B., Li, K., Zong, Y., Zhu, J., Zheng, W., Shi, J., & Zhao, L. (2019).  Recognizing spontaneous micro-expression using a three-stream convolutional neural network.  IEEE Access, 7, 184537–184551.
  * Srivastava et al. [2015] Srivastava, R. K., Greff, K., & Schmidhuber, J. (2015).  Training very deep networks.  In Advances in neural information processing systems (pp. 2377–2385).
  * Sun et al. [2020] Sun, B., Cao, S., Li, D., He, J., & Yu, L. (2020).  Dynamic micro-expression recognition using knowledge distillation.  IEEE Transactions on Affective Computing, (pp. 1–1). doi:10.1109/TAFFC.2020.2986962.
  * Van Quang et al. [2019] Van Quang, N., Chun, J., & Tokuyama, T. (2019).  Capsulenet for micro-expression recognition.  In 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019) (pp. 1–7).  IEEE.
  * Verma et al. [2021] Verma, M., Reddy, M. S. K., Meedimale, Y. R., Mandal, M., & Vipparthi, S. K. (2021).  Automer: Spatiotemporal neural architecture search for microexpression recognition.  IEEE Transactions on Neural Networks and Learning Systems, .
  * Verma et al. [2019a] Verma, M., Vipparthi, S. K., & Singh, G. (2019a).  Hinet: Hybrid inherited feature learning network for facial expression recognition.  IEEE Letters of the Computer Society, 2, 36–39.
  * Verma et al. [2020a] Verma, M., Vipparthi, S. K., & Singh, G. (2020a).  Affectivenet: Affective-motion feature learning for micro expression recognition.  IEEE MultiMedia, (pp. 1–1). doi:10.1109/MMUL.2020.3021659.
  * Verma et al. [2020b] Verma, M., Vipparthi, S. K., & Singh, G. (2020b).  Non-linearities improve originet based on active imaging for micro expression recognition.  In 2020 International Joint Conference on Neural Networks (IJCNN) (pp. 1–8).  doi:10.1109/IJCNN48605.2020.9207718.
  * Verma et al. [2019b] Verma, M., Vipparthi, S. K., Singh, G., & Murala, S. (2019b).  Learnet: Dynamic imaging network for micro expression recognition.  IEEE Transactions on Image Processing, 29, 1618–1627.
  * Wang et al. [2018a] Wang, C., Peng, M., Bi, T., & Chen, T. (2018a).  Micro-attention for micro-expression recognition.  arXiv preprint arXiv:1811.02360, .
  * Wang et al. [2020a] Wang, C., Peng, M., Bi, T., & Chen, T. (2020a).  Micro-attention for micro-expression recognition.  Neurocomputing, 410, 354–362.
  * Wang et al. [2020b] Wang, K., Peng, X., Yang, J., Lu, S., & Qiao, Y. (2020b).  Suppressing uncertainties for large-scale facial expression recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  * Wang et al. [2018b] Wang, S.-J., Li, B.-J., Liu, Y.-J., Yan, W.-J., Ou, X., Huang, X., Xu, F., & Fu, X. (2018b).  Micro-expression recognition with small sample size by transferring long-term convolutional neural network.  In Neurocomputing 312 (pp. 251–262).
  * Wang et al. [2019] Wang, W., Sun, Q., Chen, T., Cao, C., Zheng, Z., Xu, G., Qiu, H., & Fu, Y. (2019).  A fine-grained facial expression database for end-to-end multi-pose facial expression recognition.  ArXiv, abs/1907.10838.
  * Wen et al. [2021] Wen, Z., Lin, W., Wang, T., & Xu, G. (2021).  Distract your attention: multi-head cross attention network for facial expression recognition.  arXiv preprint arXiv:2109.07270, .
  * Xia et al. [2019] Xia, Z., Liang, H., Hong, X., & Feng, X. (2019).  Cross-database micro-expression recognition with deep convolutional networks.  In Proceedings of the 2019 3rd International Conference on Biometric Engineering and Applications (pp. 56–60).
  * Xia et al. [2020] Xia, Z., Peng, W., Khor, H.-Q., Feng, X., & Zhao, G. (2020).  Revealing the invisible with model and data shrinking for composite-database micro-expression recognition.  IEEE Transactions on Image Processing, 29, 8590–8605.
  * Xie & Hu [2019] Xie, S., & Hu, H. (2019).  Facial expression recognition using hierarchical features with deep comprehensive multipatches aggregation convolutional neural networks.  IEEE Transactions on Multimedia, 21, 211–220.
  * Xie et al. [2020] Xie, Y., Chen, T., Pu, T., Wu, H., & Lin, L. (2020).  Adversarial graph representation adaptation for cross-domain facial expression recognition.  In Proceedings of the 28th ACM international conference on Multimedia (pp. 1255–1264).
  * Xu et al. [2017] Xu, F., Zhang, J., , & Wang, J. Z. (2017).  Microexpression identification and categorization using a facial dynamics map.  In IEEE Transactions on Affective Computing (pp. 254–267).  volume 8.
  * Yan et al. [2014] Yan, W.-J., Li, X., Wang, S.-J., Zhao, G., Liu, Y.-J., Chen, Y.-H., & Fu, X. (2014).  Casme ii: An improved spontaneous micro-expression database and the baseline evaluation.  PloS one, 9, e86041.
  * Yan et al. [2013] Yan, W.-J., Wu, Q., Liu, Y.-J., Wang, S.-J., & Fu, X. (2013).  Casme database: a dataset of spontaneous micro-expressions collected from neutralized faces.  In 2013 10th IEEE international conference and workshops on automatic face and gesture recognition (FG) (pp. 1–7).  IEEE.
  * Yang et al. [2021] Yang, B., Cheng, J., Yang, Y., Zhang, B., & Li, J. (2021).  Merta: micro-expression recognition with ternary attentions.  Multimedia Tools and Applications, 80, 1–16.
  * Yang et al. [2018] Yang, H., Zhang, Z., & Yin, L. (2018).  Identity-adaptive facial expression recognition through expression regeneration using conditional generative adversarial networks.  2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), (pp. 294–301).
  * Yong-Jin et al. [2015] Yong-Jin, L., Jin-Kai, Z., Wen-Jing, Y., Su-Jing, W., Guoying, Z., & Xiaolan, F. (2015).  A main directional mean optical flow feature for spontaneous micro-expression recognition.  IEEE Transactions on Affective Computing, 7, 299–310.
  * Yu et al. [2020a] Yu, Z., Qin, Y., Xu, X., Zhao, C., Wang, Z., Lei, Z., & Zhao, G. (2020a).  Auto-fas: Searching lightweight networks for face anti-spoofing.  In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 996–1000).  IEEE.
  * Yu et al. [2020b] Yu, Z., Wan, J., Qin, Y., Li, X., Li, S. Z., & Zhao, G. (2020b).  Nas-fas: Static-dynamic central difference network search for face anti-spoofing.  arXiv preprint arXiv:2011.02062, .
  * Yu et al. [2020c] Yu, Z., Zhao, C., Wang, Z., Qin, Y., Su, Z., Li, X., Zhou, F., & Zhao, G. (2020c).  Searching central difference convolutional networks for face anti-spoofing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 5295–5305).
  * Zhang et al. [2016] Zhang, T., Zheng, W., Cui, Z., Zong, Y., Yan, J., & Yan, K. (2016).  A deep neural network-driven feature learning method for multi-view facial expression recognition.  IEEE Transactions on Multimedia, 18, 2528–2536.
  * Zhang et al. [2020] Zhang, T., Zong, Y., Zheng, W., Chen, C. L. P., Hong, X., Tang, C., Cui, Z., & Zhao, G. (2020).  Cross-database micro-expression recognition: A benchmark.  IEEE Transactions on Knowledge and Data Engineering, (pp. 1–1). doi:10.1109/TKDE.2020.2985365.
  * Zhao et al. [2011] Zhao, G., Huang, X., Taini, M., Li, S. Z., & PietikäInen, M. (2011).  Facial expression recognition from near-infrared videos.  Image and Vision Computing, 29, 607–619.
  * Zhao et al. [2018] Zhao, S., Cai, H., Liu, H., Zhang, J., & Chen, S. (2018).  Feature selection mechanism in cnns for facial expression recognition.  In BMVC (p. 317).
  * Zheng et al. [2022] Zheng, C., Mendieta, M., & Chen, C. (2022).  Poster: A pyramid cross-fusion transformer network for facial expression recognition.  arXiv preprint arXiv:2204.04083, .
  * Zhou et al. [2019] Zhou, L., Mao, Q., & Xue, L. (2019).  Dual-inception network for cross-database micro-expression recognition.  In 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019) (pp. 1–5).  IEEE.
  * Zhou et al. [2020] Zhou, L., Shao, X., & Mao, Q. (2020).  A survey of micro-expression recognition.  Image and Vision Computing, (p. 104043).
  * Zoph et al. [2018] Zoph, B., Vasudevan, V., Shlens, J., & Le, Q. V. (2018).  Learning transferable architectures for scalable image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 8697–8710).