# Complex Facial Expression Recognition Using Deep Knowledge Distillation of
Basic Features

Angus Maiden and Bahareh Nakisa A. Maiden is with the School of Information
Technology, Deakin University, Geelong, VIC 3220 Australia.B. Nakisa is with
the School of Information Technology, Deakin University, Geelong, VIC 3220
Australia.

###### Abstract

Complex emotion recognition is a cognitive task that has so far eluded the
same excellent performance of other tasks that are at or above the level of
human cognition. Emotion recognition through facial expressions is
particularly difficult due to the complexity of emotions expressed by the
human face. For a machine to approach the same level of performance in complex
facial expression recognition as a human, it may need to synthesise knowledge
and understand new concepts in real-time, as humans do. Humans are able to
learn new concepts using only few examples by distilling important information
from memories. Inspired by human cognition and learning, we propose a novel
continual learning method for complex facial expression recognition that can
accurately recognise new compound expression classes using few training
samples, by building on and retaining its knowledge of basic expression
classes. In this work, we also use GradCAM visualisations to demonstrate the
relationship between basic and compound facial expressions. Our method
leverages this relationship through knowledge distillation and a novel
Predictive Sorting Memory Replay, to achieve the current state-of-the-art in
continual learning for complex facial expression recognition, with 74.28%
Overall Accuracy on new classes. We also demonstrate that using continual
learning for complex facial expression recognition achieves far better
performance than non-continual learning methods, improving on state-of-the-art
non-continual learning methods by 13.95%. Our work is also the first to apply
few-shot learning to complex facial expression recognition, achieving the
state-of-the-art with 100% accuracy using only a single training sample per
class.

Facial expressions are one of the most powerful signals for humans to convey
emotional states, comprising over 55% of our emotional communication. By
developing AI systems that can accurately recognise facial expressions at
human performance level, they could be trusted to assist with functions and
services that demand emotional communication, such as in healthcare and
customer service. We propose a novel method of complex facial expression
recognition using continual learning and few-shot learning, inspired by human
cognition and learning. The method uses knowledge distillation of basic
expressions and a novel Predictive Sorting Memory Replay to reduce the
catastrophic forgetting associated with continual learning, achieving state-
of-the-art performance compared with other methods. Our method demonstrates
the increased performance of neural networks when recognising complex concepts
by retaining and distilling the knowledge of basic concepts. This will pave
the way for further research and applications in other domains using our
method.

Deep learning, Knowledge transfer, Neural networks, Convolutional neural
networks, Human-centred artificial intelligence, Multi-task learning

##  1 Introduction

We are now entering the fourth industrial age, where artificial intelligence
(AI) plays a crucial role in many of our activities and endeavours. By
equalling or outperforming humans in cognitive tasks such as image recognition
and natural language processing, and by completing physical and processing
tasks with far greater precision and speed, AI can increasingly be used to
assist with complex functions and services that were once the exclusive domain
of humans, such as automobile driving, medical diagnosis, and customer service
operations. However, many of these services rely not only on technical
accuracy and precision, but on the very human elements of communication,
empathy and compassion.

Communication between humans is fundamental to our ability to learn, work and
build structures and societies, allowing us to survive, adapt, progress and
prosper. According to Darwin [1], facial expressions are one of the most
powerful signals for humans to convey emotional states and intentions, with
over 55% of emotional communication being conveyed through facial expressions
[2]. Facial expression recognition (FER) is thus a crucial factor in our
ability to operate in more complex and nuanced roles such as healthcare and
customer service. By developing AI systems that can accurately recognise human
emotional states at or above the level of human performance, they could be
trusted to assist with these more complex functions in order to enhance these
services.

In order for an AI system to recognise complex facial expressions at or above
the level of human cognition, it may need to learn in a similar way to humans,
learning new concepts from only a few examples by synthesising those concepts
with existing knowledge. Continual learning and few-shot learning are two
approaches to machine learning that are inspired by human cognition and
learning patterns. Continual learning is an approach for incrementally
learning new classes using a model that was previously trained on other
classes. It uses techniques such as knowledge distillation and memory replay
to retain the knowledge of known classes, such that it performs well in
recognising both old and new classes. Few-shot learning involves training
machine learning models using only a few examples. High-performing few-shot
learning methods use novel feature extraction and data augmentation techniques
to achieve high recognition accuracy using fewer training samples. Applying
these methods of continual learning and few-shot learning to complex FER could
lead to human-like performance on this challenging task.

This work investigates the hypothesis that by retaining the knowledge of basic
facial expression features, a machine learning model will achieve better
performance when learning new complex facial expression labels which share
those features. The work is comprised of three phases. In the initial Basic
FER Phase, we design and build a neural network model that can achieve high
classification accuracy on six basic expression classes. In the Continual
Learning Phase, new complex expression classes are iteratively learned by
using knowledge distillation of basic expressions as well as a unique
Predictive Sorting Memory Replay. In the Few-shot Learning Phase, we
demonstrate how new complex expression classes can be learned to high accuracy
with a very small number of training samples using knowledge distillation of
basic expressions.

The main contributions of this work are:

  * •

We propose a novel method of complex facial expression recognition using
continual learning and few-shot learning. The method uses data augmentation,
knowledge distillation of basic expressions, and a novel Predictive Sorting
Memory Replay to reduce catastrophic forgetting and improve performance using
few training examples.

  * •

We demonstrate that using continual learning for complex facial expression
recognition achieves far better performance than non-continual learning
methods, improving on the state-of-the-art in non-continual learning methods
by 13.95%.

  * •

We achieve the current state-of-the-art in continual learning for complex
facial expression recognition, with 74.28% Overall Accuracy on new classes (an
improvement of 0.67%) in a comprehensive analysis using experiments that
compare with other continual learning methods.

  * •

Our method is the first to apply few-shot learning to complex facial
expression recognition to the best of our knowledge, achieving the state-of-
the-art with 100% accuracy using a single training sample for each expression
class.

##  2 Literature Review

###  2.1 Background and Scope

The majority of prior research on FER uses a categorical expression labelling
system based on [3], who defined six basic emotions expressed on the human
face: anger, disgust, fear, happiness, sadness, and surprise, with contempt
later added by [4]. These basic expressions are apparently conveyed and
perceived similarly across different cultures [5]. High recognition
performance in recent years indicates that FER using basic expression labels
is essentially solved (see Table 1).

Method | Dataset | Classes | Acc.
---|---|---|---
Basic FER Methods
Deep learning using CNN and inception blocks [6] | CK+ | 7 | 0.9320
Manual feature extraction using Local Binary Patterns [7] | CK+ | 7 | 0.9626
Deep neural network using multi-step pre-processing and feature extraction [8] | CK+ | 7 | 0.9680
Deep network with joint fine-tuning [9] | CK+ | 7 | 0.9725
Boosted deep belief network [10] | CK+ | 7 | 0.9670
Complex FER Methods
Deep CNN using Optical Flow feature extraction [11] | CMED | 18 | 07018
Deep CNN using knowledge distillation of AUs and SVM classifier [12] | CASME | 5 | 0.8180
Continual learning using knowledge distillation [13] | CFEE | 21 | 0.7382
Incremental active learning from sparsely annotated data [14] | CFEE | 21 | 0.8502
SVM classifier using extracted shape and appearance features [15] | CFEE | 21 | 76.91
Table 1: State-of-the-art methods for basic and complex FER

However, human beings express a wide range of emotions through facial
expressions that do not fit into predefined categories, and there is evidence
that no such basic, prototypical emotion categories exist [16]. Instead, FER
develops naturally over time in humans, who are able to identify new, complex
emotions on the fly as they appear [1, 13]. To approach human-like FER
performance, a machine should be able to recognise complex expressions of
emotion such as happily disgusted, and distinguish them from similar emotions
like happy, disgusted, or happily surprised. These compound expressions are
more than the sum of their parts; they are distinct concepts which express a
unique emotion [15]. To humans, such synthesising of known concepts to form
new ones comes relatively naturally, and we are able to learn, process and
recognise new compound expressions using very little data. However, the state-
of-the-art in FER still has difficulty with such cognitive complexity, due to
the similarity of features across both basic and compound expressions. AI
performs significantly worse at complex FER compared to basic FER, as seen in
Table 1.

Figure 1: Examples from CFEE database [15]. Compound expressions such as
happily disgusted are more than the sum of their parts (happy and disgusted).

###  2.2 FER System Architectures and Design

Complex facial expressions can be represented in a number of ways, such as
combinations of basic expressions like happily surprised (called compound
expressions), combinations of expressions in sequence such as surprise
becoming anger, or as new expressions formed from adding a temporal dimension
to basic ones, such as depression, which can be categorised as a persistent
sadness over time [17]. The Facial Action Coding System (FACS) [3] defines 44
action units (AUs), each one representing the activation of particular facial
muscles groups. For example, a happy expression may be composed of AUs 12 (lip
corner puller) and 25 (lips part) since this expression often involves the
activation of these muscle groups, whereas a disgusted expression may be
composed of AUs 10 (upper lip raiser) and 17 (chin raiser). The compound
expression happily disgusted predominantly uses AUs 10, 12 and 25, which is
the intersection of AUs found in each of the basic expressions happy and
disgusted [15]. Our work focuses solely on compound FER as a form of complex
FER.

Some traditional FER methods use manual feature extraction techniques such as
local binary patterns [7, 18], however deep learning is currently the most
popular method, enabling automatic feature extraction through gradient descent
backpropagation, and achieving the current state-of-the-art in FER (see Table
1). High-performing deep learning architectures for computer vision such as
ResNet [19] and Xception [20] can be used by pre-training a model on a large
general image database like ImageNet [21], with fine-tuning to adapt to the
FER task. This technique is used in a few state-of-the-art models such as [12]
but is currently underutilised.

There are a number of common problems in attaining high accuracy for complex
FER. Machine learning, particularly deep learning, requires a huge amount of
training data to avoid over-fitting, however currently facial expression
databases are not sufficient for this task compared with those used for
general image recognition or object detection, such as ImageNet [21]. Subject
identity bias (differences in personal attributes of the subjects used for
sample images, such as age, gender, ethnic background and level of
expressiveness) reduces generalisation performance since there is increased
bias towards recognising specific subjects or features (e.g. skin colour or
wrinkles) in the training data. Many state-of-the-art FER methods employ
various pre-processing methods to reduce the effect of these issues and
increase performance.

Face detection and alignment can increase FER performance by creating
consistency in training images. For example, the eyes are generally in the
same region for images processed by face detection, allowing spatial models
such as Convolutional Neural Networks (CNNs) to more easily learn these
features. Some common facial detection and alignment techniques include the
Viola-Jones face detector [22] as used by [18, 23], generative object
detection [24] as used by [25], supervised descent [26] as used by [6, 18],
and the RetinaFace face detector [27].

Data augmentation applies random transformations to the training image data,
such as rotation, flipping and contrast adjustment, generating new images with
the same semantic content but different data values. This alleviates the
problem created by using smaller FER datasets for deep learning and improves
generalisation performance, as the model is trained on more varied data. Data
augmentation is used in some FER methods, such as [6, 9, 14], but is not used
in many state-of-the-art methods, limiting their performance.

###  2.3 Few-shot Learning and Continual Learning

As discussed in Section 2.1, to improve the performance of complex FER
systems, a novel approach is required which can incrementally learn new
complex expressions as they appear, in the same way that a human might induce
the expression happily surprised from its prior knowledge of happy and
surprised, using only a few new examples.

Continual learning is a group of methods which focuses on the problem of
incrementally adding classes to a trained model. Most of these methods suffer
from catastrophic forgetting, which is a reduction in performance on
previously learned classes due to substantial weight changes when learning new
ones [13]. Recent advances in continual learning applied to other domains have
partially solved the problem of catastrophic forgetting using such methods as
memory replay [28] and knowledge distillation [29]. Some pioneering methods of
complex FER such as [13, 14, 30] also use continual learning.

Few-shot learning is a research area focused on training machine learning
models with a very small amount of training data, often down to a single
sample, making them more adaptable to real-world applications such as
streaming video data, security footage where the subject appears only briefly,
or learning from a single passport or identification photo supplied by a user.
Applying few-shot learning to the domain of complex FER could have a number of
applications in areas such as human-computer interaction.

For example, a model that is able to recognise new complex expressions from
only a few labelled images could learn a human’s emotional state that it
hasn’t been trained on previously, if they state what they’re feeling whilst
looking at the machine’s camera input. By capturing the facial expression
images with some additional processing such as speech recognition, a machine
that achieves high performance using few-shot learning could learn this new
expression on-the-fly and recognise it the next time it sees it. This approach
could enable AI systems to perform better in applications of human-computer
interaction such as AI assistants and robotic nurses, which would require
human-like levels of emotional intelligence in communication.

The main problem with few-shot learning is under-fitting, whereby the model
does not have enough varied training examples to generalise well on
recognising unknown samples. A number of techniques are used to increase few-
shot learning performance in state-of-the-art methods. [31] achieves high
accuracy in one-shot learning for facial identity recognition using supervised
auto-encoders to augment a single training sample, producing variations in
illumination, expression, occlusion and pose. [32] uses a hybrid feature
enhancement network to increase the importance of low-level features such as
image textures for semantic segmentation tasks. [33] uses a pseudo-Siamese
network to learn classes from a new domain by training one model branch on the
original images, and another branch on augmented sketch map images produced by
extracting the images’ contour features. This method also cites as inspiration
the ability of humans to build on prior knowledge when learning new tasks.
Common themes amongst these methods of few-shot learning are data augmentation
and enhanced feature extraction.

###  2.4 Knowledge Distillation

Knowledge distillation [29] is a method for transferring knowledge from a
teacher model to a student model. The student uses the predicted probabilities
from the teacher’s output as a soft target in place of the usual hard target
(ground truth labels) when calculating the loss between predicted and target
outputs. Knowledge is distilled by the student model through gradient descent
back-propagation of the distillation loss between the predictions and soft
teacher targets. The student thus learns a mapping from its inputs to all the
probable label outputs as learned by the teacher, enabling it to quickly form
a rich representation of the teacher’s knowledge.

Figure 2: Softmax output of ’Surprised’ image with different temperatures.

The distillation loss is a modified cross-entropy loss whereby the teacher
model’s prediction output is modified by a temperature $T$ to attain the soft
prediction output $y^{soft}$ given by (1), where $k$ is the number of classes
and $z\in\mathbb{R}^{k}$ is a vector of logits (the output prior to the
softmax activation layer). As seen in Figure 2, higher temperatures smooth the
probability distribution of the soft outputs, giving increased weight to
smaller probabilities and decreased weight to the highest probability, whilst
retaining the probabilities’ relative ranking order.

$\hat{y}^{soft}=\frac{e^{\frac{z_{j}}{T}}}{\sum_{j=1}^{k}e^{\frac{z_{j}}{T}}}$
(1)

(2) demonstrates the categorical cross-entropy loss $\mathcal{L}$ used during
standard model training, whilst (3) is the distillation loss
$\mathcal{L}_{dist}$ used for knowledge distillation, whereby $y$ is the true
label (one-hot encoded), $\hat{y}$ is the prediction output, and
$\hat{y}^{t-soft}$ and $\hat{y}^{soft}$ are the soft teacher prediction and
soft student prediction outputs, respectively.

$\mathcal{L}\left(y,\hat{y}\right)=-\sum_{j=1}^{k}y_{j}\log{y_{j}}$ (2)

$\mathcal{L}_{dist}\left(\hat{y}^{t-soft},\hat{y}^{soft}\right)=-\sum_{j=1}^{k}\hat{y}^{t-soft}_{j}\log\hat{y}^{soft}_{j}$
(3)

Knowledge distillation can reduce the effects of catastrophic forgetting in
continual learning by reinforcing a model’s knowledge of known classes when
learning new classes. [13] uses knowledge distillation with continual learning
for FER through an indicator loss function which is the weighted sum of the
distillation loss and hard loss. An indicator function allows the model to
treat each new training example differently depending on whether it is
reinforcing prior knowledge or gaining new knowledge. For new classes, the
hard loss is weighted more and for old classes the distillation loss is
weighted more. FER accuracy with this method is reduced significantly less
with each new class learned, compared to other methods.

Figure 3: System architecture and method

###  2.5 Representative Memory Replay

Memory replay techniques are used in continual learning to store a subset of
labelled training examples for known classes, which are added to the new class
training data. This helps to prevent catastrophic forgetting by reinforcing
the knowledge of known classes when learning new classes. It can also be used
in conjunction with knowledge distillation, as in [13, 28, 34]. Some memory
replay methods use specific sample selection techniques with the aim of
selecting the most representative samples of each label, such as [28] and
[34]. Other methods such as [13] instead use a randomly selected
representative memory, stating that randomising the order of inputs when
training a model improves generalisation performance. However, a major
limitation of this work is that it is not true continual learning, as it draws
from the full set of prior training examples using its random selection policy
for memory replay. This effectively makes the results comparable to training a
new model from the beginning with each new class. Our method aims to improve
on [13] whilst adhering to the true principle of continual learning, that is
in not having access to the prior training dataset except for a subset of data
that is set aside and stored in a representative memory. We use a novel
Predictive Sorting Memory Replay to select the samples which are most
representative of their class. This enhances the knowledge distillation,
thereby greatly reducing catastrophic forgetting and improving continual
learning performance.

One of the advantages of continual learning is the potential for training a
model to learn new classes in real-time without progressively using more
resources. To this end, the representative memory is often kept to a fixed
number of samples, $K$, over each continual learning iteration, as in [13] and
[34]. In this way $m=\frac{K}{k}$ samples are retained for each new class,
where $k$ is the number of observed classes so far. However, $k$ increases
with each continual learning iteration which progressively reduces the number
of samples of each known class in the representative memory, and can
reintroduce the effect of catastrophic forgetting. If instead $K$ is allowed
to increase, the representative memory can keep a constant number of samples
for each label, reducing the effect of catastrophic forgetting at the expense
of an ever-growing memory size. [28] evaluates both these methods and compares
the results, which as expected shows a decrease in performance using a fixed
$K$ compared with an increasing $K$.

##  3 Research Design and Methodology

In this section, the main components of a novel system architecture and method
for compound FER are described. The code to reproduce this method is available
at https://github.com/AngusMaiden/complex-FER. The proposed method has three
phases:

  * •

A Basic FER Phase in which a Basic FER Model learns to recognise six basic
expression classes from a dataset of labelled static images of facial
expressions.

  * •

A Continual Learning Phase in which the trained model from the Basic FER Phase
is used to learn new compound expression classes sequentially, by
incrementally adding new classes until all expressions have been learned.

  * •

A Few-shot Learning Phase in which the trained model from the Basic FER Phase
is used to learn new compound expression classes with only a very small number
of training samples from the new class. In this phase, each new class is
trained and tested as a separate experiment, re-initialising the model each
time based on the model from the Basic FER Phase.

The re-use of the trained Basic FER model in the Continual Learning and Few-
shot Learning Phases is intended to somewhat mimic the human pattern of
learning as we age, by learning complex concepts like compound facial
expressions only after the basic concepts, i.e. basic expressions, are
understood.

###  3.1 Basic FER Phase

Figure 4: Basic FER model architecture

In this initial phase, a Basic FER Model learns the mapping between input
images $X_{basic}$ and their ground truth expression labels
$y_{basic}\in\mathbb{R}^{k_{basic}}$. Each image is labelled with one of
$k_{basic}$ facial expression labels. The model has two main components:

  1. 1.

A feature extractor $FE$ using a base residual network, ResNet50V2 [35],
learns feature mappings of images through stacked convolutional blocks with
residual connections and dense layers on top. The network is pre-trained on
ImageNet [21] to extract common image features such as shapes and lines, then
fine tuned on the FER dataset. The network’s output is a feature vector
$FE(X_{basic})\in\mathbb{R}^{k_{FE}}$ where $k_{FE}$ is the number of output
nodes of $FE$.

  2. 2.

A classification layer $CL_{basic}$ with $k_{basic}$ output nodes. This layer
takes as input the output of $FE(X_{basic})$, and outputs a logit vector
$z\in\mathbb{R}^{k_{basic}}$, where $k_{basic}$ is the number of basic
expressions from the Basic FER Phase.

A forward pass of the Basic FER Model from inputs $X_{basic}$ to logit outputs
$z$ is given by (4). A standard softmax activation function as in (1), where
$T=1$, is applied to the logit vector to produce the probability vector
$\hat{y}\in\mathbb{R}^{k_{basic}}$.

$z=CL_{basic}(FE(X_{basic}))$ (4)

A categorical cross-entropy loss $\mathcal{L}_{cat}$ calculates the error
between the predicted labels $\hat{y}$ and ground truth labels $y_{basic}$ for
each input image, and is calculated according to (2).

The main aim of the Basic FER Phase is to train a model to achieve high FER
accuracy with robust feature mappings from images of basic facial expressions.
The feature mappings are then transferred using knowledge distillation to the
models in the Continual Learning and Few-shot Learning Phases. It is
hypothesised that compound FER accuracy can be improved using minimal new
training examples through knowledge distillation of basic expressions, as
compound and basic facial expressions share some basic features.

In order to achieve a high recognition accuracy in the Basic FER Phase, and
subsequently in the Continual Learning and Few-shot Learning Phases, a good
model architecture is needed. Based on empirical knowledge from state-of-the-
art research in computer vision, the architecture of the Basic FER Model is
designed as displayed in Figure 4.

The input pipeline for the Basic FER model takes 3-channel (RGB) images of
size 224 x 224 x 3 in batches. Each image is labelled with one of $k_{basic}$
basic expression classes. Images are pre-processed using the RetinaFace face
detection algorithm [27]. Each image is then normalised such that their pixel
values lie between -1 and 1. Data augmentation of the training set is also
used to generate random transformations including horizontal flipping,
translation and zooming for each image.

Algorithm 1 Basic FER Phase

0: $X_{basic}\leftarrow$ training images

0: $y_{basic}\leftarrow$ labels for images in $X_{basic}$

0: $X_{basic}^{test}$ test images

0: $y_{basic}^{test}$ labels for images in $X_{basic}$

0: $FE\leftarrow ResNet50V2$ (pre-trained on ImageNet)

0: $CL_{basic}\leftarrow$ Classification layer

1: for epoch in no. of epochs do

2: for batch in Batch($X_{basic}$,$y_{basic}$) do

3: for x, y in batch do

4: $x\leftarrow FaceDetection(x)$

5: $x\leftarrow Normalize(x)$

6: $x\leftarrow Augment(x)$

7: $z\leftarrow CL_{basic}(FE(x))$

8: $\hat{y}\leftarrow Softmax(z)$

9: $\mathcal{L}_{cat}\leftarrow CrossEntropy(\hat{y},y)$

10: end for

11: $\mathcal{L}_{cat}\leftarrow Average(\mathcal{L}_{cat})$ over $batch$

12: Update model weights (back-propagation of $\mathcal{L}_{cat}$)

13: end for

14: for $x^{test}$, $y^{test}$ in $X_{basic}^{test}$, $y_{basic}^{test}$ do

15: $\hat{y}^{test}\leftarrow Softmax(CL_{basic}(FE(x^{test}))$

16: $acc\leftarrow Accuracy(\hat{y}^{test},y^{test})$

17: end for

18: Accuracy = $Average(acc)$

19: end for

###  3.2 Continual Learning Phase

The Continual Learning Phase is an iterative cycle whereby labelled images
from each complex expression class are added to the existing dataset and
trained sequentially. At each iteration $i$, one new complex expression class
is selected, which is comprised of images $X_{new_{i}}$ and associated true
labels $y_{new_{i}}\in\mathbb{R}^{k_{i}}$, where $k_{i}$ is the total number
of expression classes available at this iteration ($k_{i}$=$k_{basic}$+$i$).
The total number of known (trained) classes at the beginning of each iteration
is $k_{i-1}$. The phase runs for $k_{compound}$ iterations, whereby
$k_{compound}$ is the number of compound expression classes available. For
each iteration $i$, a new node is added to the classification layer of the
previous model, $CL_{i-1}$, with the new layer denoted as $CL_{i}$. This new
node’s weights are randomly initialised using a Glorot Uniform distribution,
whilst the other $k_{i-1}$ nodes inherit their weights from the corresponding
$CL_{i-1}$ layer of the trained model from the previous iteration $i-1$. In
the first iteration, whereby $i=1$, the previous model is the trained model
from the Basic FER Phase, such that $CL_{0}$=$CL_{basic}$. The structure of
the feature extractor $FE$ does not change at each iteration and is reused as-
is. Figure 3 provides a visualisation of the continual learning method and
process flow.

A representative memory $X_{mem_{i}}$ is used to store a portion of training
samples from the previous iteration $i-1$ together with the new class training
samples $X_{new_{i}}$. A number of training samples $X_{select_{i}}$, and
their associated labels $y_{select_{i}}$, are selected according to the
Predictive Sorting Memory Replay (PSMR) selection policy. The pseudo-code for
this policy is outlined in Algorithm 2.

Algorithm 2 Predictive Sorting Memory Replay (PSMR)

0: $K\leftarrow$ number of samples in representative memory

0: $k_{i}\leftarrow$ number of classes at iteration i

0: $m\leftarrow\frac{K}{k_{i-1}}$

0: $X_{mem_{i-1}},y_{mem_{i-1}}\leftarrow$ representative memory of previous
iteration

1: if $i=1$ then

2: $X_{select_{i}},y_{select_{i}}\leftarrow$ randomly select $K$ $X_{basic}$
images and associated labels

3: else

4: $\hat{y}\leftarrow Softmax(CL_{i}(FE(X_{mem_{i-1}}))$

5: for each $\hat{y}_{j}$ in $j$ classes where $j=1,...,k_{i}$ do

6: $Sort(\hat{y}_{j})$ in order of prediction probability

7: $Append(y_{select_{i}})\leftarrow$ Select top $m$ sorted labels
$\hat{y}_{j}$

8: $Append(X_{select_{i}})\leftarrow$ images for $y_{select_{i}}$ labels

9: end for

10: end if

11: return $X_{select_{i}},y_{select_{i}}$

Using this policy, the representative memory is therefore comprised of images
that are the most representative of their respective class. When learning new
classes, these representative memory samples are trained alongside the new
class samples, which reinforces the knowledge of previous classes and reduces
the effect of catastrophic forgetting. In a similar way, humans retain
memories of only the most pronounced moments of an experience, which
efficiently enables recognition and classification of the entire experience.
Once initialised, the representative memory does not acquire any new samples
from the Basic FER Phase data, to emulate human learning whereby the raw data
from prior experiences is no longer available, and only memories are retained.
This also ensures the method is aligned with practical applications that may
have limiting memory and computation requirements, such as mobile computing,
IoT and robotics.

A forward pass of the continual learning model at iteration $i$ from inputs
$X_{mem_{i}}$ to output logits $z_{i}$ is given in (5). A softmax activation
function as in (1), where $T=1$, is applied to the logit vector to produce the
prediction vector $\hat{y}_{i}\in\mathbb{R}^{k_{i}}$.

$z_{i}=CL_{i}(FE(X_{mem_{i}}))$ (5)

###  3.3 Distillation Loss

The loss function of a neural network calculates the difference between the
target output, usually the ground truth labels, and the predicted output of a
model. By minimising the loss through gradient descent optimisation, we aim to
minimise this difference, bringing the predicted outputs closer to the real
outputs with each gradient descent step. In this phase, the loss $\mathcal{L}$
is the weighted sum of a standard categorical cross-entropy loss
$\mathcal{L}_{cat}$ and distillation loss $\mathcal{L}_{dist}$, with a
distillation weighting factor, $\gamma:0\leq\gamma\leq 1$, as shown in (6).

$\mathcal{L}(y_{i},\hat{y}_{i})=(\gamma-1)\cdot\mathcal{L}_{cat}(y_{i},\hat{y}_{i})+\gamma\cdot\mathcal{L}_{dist}(\hat{y}_{i}^{t-soft},\hat{y}_{i}^{soft})$
(6)

The cross-entropy loss $\mathcal{L}_{cat}$ calculates the error between the
predicted labels $\hat{y}_{i}$ and ground truth labels $y_{i}$ for each input
image, as in (2). The distillation loss $\mathcal{L}_{dist}$ calculates the
error between $\hat{y}_{i}^{t-soft}$and $\hat{y}_{i}^{soft}$, as in (3). Our
method demonstrates a unique handling of the distillation loss whereby the
teacher model is a static copy of the trained model attained in the Basic FER
Phase. The weights of this teacher model are never updated in the Continual
Learning Phase. A forward pass of the teacher model at iteration $i$ from
inputs $X_{mem_{i}}$ to teacher logits $z_{i}^{t}$ is given in (7). A softmax
activation function with temperature $T$, as in (1), is applied to the logits
$z_{i}$ from (5) to produce student predictions $\hat{y}_{i}^{soft}$ and to
the teacher logits $z_{i}^{t}$ from (7) to produce teacher predictions
$\hat{y}_{i}^{t-soft}$, which are used in the calculation of the distillation
loss $\mathcal{L}_{dist}$ (6).

$z_{i}^{t}=CL_{basic}(FE_{basic}(X_{mem_{i}}))$ (7)

With each iteration, $\mathcal{L}_{dist}$ naturally increases, adding a higher
penalty to the overall loss $\mathcal{L}$. This is because
$\mathcal{L}_{dist}$ is calculated between the vectors
$\hat{y}_{t-soft}\in\mathbb{R}^{k_{basic}}$ (zero-padded) and
$\hat{y}_{i}^{soft}\in\mathbb{R}^{k_{i}}$. Due to the zero-padding, there will
be $i$ constant zero values in $\hat{y}$, causing $\mathcal{L}_{dist}$ to
increase with each step $i$. To counteract this effect, a distillation weight
decay is used, as in (8), which decreases the value of $\gamma$ with each
iteration $i$, thereby reducing the weight of $\mathcal{L}_{dist}$ in the
overall loss $\mathcal{L}$ (6). This function approaches but never equals the
asymptote at $\gamma=0$, ensuring there is always some amount of distilled
knowledge of basic expressions contributing to the overall loss.

$\gamma_{i}=\gamma_{i-1}e^{\frac{-1}{1+e}}$ (8)

After each epoch of training, the accuracy of the continual learning model is
tested using a held-out test dataset. Training is stopped when the test
accuracy is no longer improving over previous epochs. We then record the test
accuracy for iteration $i$ for all $k_{i}$ trained expression classes, as well
as the single class test accuracy for the newest facial expression. Evaluating
this method of knowledge distillation in the Continual Learning and Few-shot
Learning Phases tests the hypothesis that compound FER accuracy can be
improved using few new training examples through knowledge distillation of
basic expressions.

Algorithm 3 Continual Learning Phase

0: $X_{basic}\leftarrow$ basic training images

0: $y_{basic}\leftarrow$ labels for images in $X_{basic}$

0: $X_{new}\leftarrow$ compound training images

0: $y_{new}\leftarrow$ labels for images in $X_{new}$

0: $X_{new}^{test}\leftarrow$ compound test images

0: $y_{new}^{test}\leftarrow$ labels for images in $X_{new}$

0: $FE\leftarrow ResNet50V2$ (trained in Basic FER Phase)

0: $CL_{0}\leftarrow CL_{basic}$ (from Basic FER Phase)

0: $T\leftarrow$ Softmax Temperature for Distillation Loss

0: $\gamma\leftarrow$ distillation weighting factor

1: for i in list of $k_{compound}$ facial expressions do

2: $X_{new_{i}},y_{new_{i}}\leftarrow$ select images, labels of next
expression in list

3: $y_{select_{i}},X_{select_{i}}\leftarrow PSMR$

4: $X_{mem_{i}}\leftarrow Concatenate(X_{select_{i}},X_{new_{i}})$

5: $y_{mem_{i}}\leftarrow Concatenate(y_{select_{i}},y_{new_{i}})$

6: $CL_{i}\leftarrow$ Add one output node to $CL_{i-1}$

7: for epoch in no. of epochs do

8: for $batch$ in Batch($X_{mem_{i}}$, $y_{mem_{i}}$) do

9: for $x$, $y$ in $batch$ do

10: $x\leftarrow FaceDetection(x)$

11: $x\leftarrow Normalize(x)$

12: $x\leftarrow Augment(x)$

13: $z\leftarrow CL_{i}(FE(x))$

14: $\hat{y}\leftarrow Softmax(z)$

15: $z^{t}\leftarrow CL_{basic}(FE_{basic}(x))$

16: $\hat{y}^{t-soft}\leftarrow Softmax(z^{t},T)$

17: $\hat{y}^{soft}\leftarrow Softmax(z,T)$

18: $\mathcal{L}_{cat}\leftarrow CrossEntropy(\hat{y},y)$

19: $\mathcal{L}_{dist}\leftarrow
CrossEntropy(\hat{y}^{t-soft},\hat{y}^{soft})$

20:
$\mathcal{L}\leftarrow(\gamma-1)\cdot\mathcal{L}_{cat}+\gamma\cdot\mathcal{L}_{dist}$

21: end for

22: $\mathcal{L}\leftarrow Average(\mathcal{L})$ over $batch$

23: Update model weights (back-propagation of $\mathcal{L}$)

24: end for

25: for $x^{test}$, $y^{test}$ in $X_{basic}^{test}$, $y_{basic}^{test}$ do

26: $\hat{y}^{test}\leftarrow Softmax(CL_{i}(FE(x^{test}))$

27: $acc\leftarrow Accuracy(\hat{y}^{test},y^{test})$

28: end for

29: Accuracy = $Average(acc)$

30: end for

31: Update weight decay $\gamma$ according to (8)

32: end for

###  3.4 Few-shot Learning Phase

In this phase, the hypothesis that better complex FER performance can be
achieved through knowledge distillation of basic expressions is tested in the
context of few-shot learning, whereby a very limited number of training
examples are used. We run one experiment for each of the $k_{compound}$
compound facial expressions, with repeat trials of 5, 3 and 1 training
examples. To train these expressions, the same system architecture and
methodology is used as in the Continual Learning Phase, with the exception of
the Representative Memory Replay component, which is excluded from this phase
as it is specific to continual learning. Each few-shot learning experiment is
equivalent to one iteration of the continual learning experiment. The model’s
parameters are reset after each experiment, before a new class is chosen.
These experiments test the model’s ability to learn each separate complex
expression class using very few training examples through distilling the
knowledge of basic facial expressions. The aggregate results of these separate
experiments allows us to generalise about the model’s capabilities for few-
shot learning with compound FER.

For this phase, at the beginning of each experiment
$j\in[1,\ldots,k_{compound}]$, one new compound expression class is selected
which is comprised of images $X_{new_{j}}$ and associated true labels
$y_{new_{j}}\in\mathbb{R}^{k_{fewshot}}$, where $k_{fewshot}=k_{basic}+1$. As
with the first iteration of the Continual Learning Phase, the classification
layer $CL_{fewshot}$ is comprised of a new node added to the classification
layer of the Basic FER model $CL_{basic}$. For each experiment $j$, a forward
pass of the few-shot learning model produces the output logits $z_{j}$ as in
(9), whilst a forward pass of the teacher model produces the output logits
$z_{j}^{t}$ as in (10). A softmax activation function is applied to the logit
vectors $z_{j}$ and $z_{j}^{t}$ to produce the prediction vectors
$\hat{y}_{j}^{soft}$ and $\hat{y}_{j}^{t-soft}$, respectively, as in (1). The
cross-entropy loss $\mathcal{L}_{cat}$ calculates the error between the
predicted labels $\hat{y}_{j}$ and ground truth labels $y_{j}$ for each input
image, as in (2). The distillation loss $\mathcal{L}_{dist}$ calculates the
error between $\hat{y}_{j}^{t-soft}$ and $y_{j}^{soft}$, as in (3). The loss
function is given in (6) and calculated as in the Continual Learning Phase.

$z_{j}=CL_{fewshot}(FE(X_{new_{j}}))$ (9)

$z_{j}^{t}=CL_{basic}(FE_{basic}(X_{new_{j}}))$ (10)

After each epoch of training, the accuracy of the few-shot learning model is
tested using the same held-out test dataset from the Continual Learning Phase.
Training is stopped when the test accuracy is no longer improving over
previous epochs. Few-shot learning experiments are conducted using 5, 3 and 1
training examples per class, with the single class test accuracy and number of
training steps recorded for each $j\in\mathbb{R}^{k_{compound}}$ compound
expression classes.

##  4 Grad-CAM Visualisation of Basic and Compound Features

This experimental setup was also used to visualise the features of basic
expressions which are distilled into the model when learning new complex
expressions. The Grad-CAM [36] method was applied to visualise the areas of
each image that are most activated for particular expression classes. Using
this method, we can visualise the features of basic expressions that are
common to those compound expressions derived from them. Initially, Grad-CAM
heat maps were produced from the Basic FER model for the $k_{basic}$ basic
expressions. Similarly to the Few-Shot Learning Phase, here we use just one
iteration of the Continual Learning Phase, without the representative memory,
training the model for $j\in\mathbb{R}^{k_{compound}}$ compound expressions
and resetting the weights each time, but using all available training examples
in order to produce the strongest feature activations. The trained models were
then used to produce Grad-CAM feature maps for each expression which
highlights the areas of the image which are most activated when predicting
that expression. Connections between the basic and compound expressions were
drawn by identifying semantic relationships between them and their Grad-CAM
feature maps in order to highlight facial expression features which are common
to these expressions.

For example, Figure 5 demonstrates the Grad-CAM visualisation of the basic
features angry, disgusted and fearful in relation to the compound expressions
angrily disgusted and fearfully angry. In this visualisation, we can clearly
see the features which are common to both the basic and compound expressions,
such as in the furrowed brow which is common to the fearful, angry and
fearfully angry expressions. We can also see how the angrily disgusted
expression shares the down-turned mouth with the disgusted expression and the
furrowed brow with the angry expression. These common features also represent
the knowledge which is transferred from the Basic FER model to the continual
learning model using knowledge distillation. This visualisation, together with
the improved performance in the Continual Learning and Few-Shot Learning
Phases of the models which use knowledge distillation of basic features,
supports the hypothesis that knowledge of basic features can improve the
recognition of compound facial expression labels which share those features.

Figure 5: Grad-CAM Visualisation of basic features in angrily disgusted and
fearfully angry compound expressions Figure 6: Grad-CAM visualisation of basic
features in angrily surprised disgustedly surprised and compound expressions
Figure 7: Grad-CAM visualisation of basic features in disgustedly surprised
and angrily surprised compound expressions

##  5 Empirical Evaluation and Results

The dataset used for evaluation is the Compound Facial Expressions of Emotion
(CFEE) database [15]. It contains 5044 images of 230 different subjects with
acted facial expressions in a controlled lab environment. The dataset is
labelled with 21 different facial expressions.

For evaluation, a subject-independent k-fold cross validation method was used
whereby the dataset is partitioned into 10 folds of 23 subjects each. 1 fold
is held out as the test set, with the training data comprising the remaining 9
folds. The fold split points are chosen according to subject, such that the
test data contains images of completely different subjects than the training
data. This aims to reduce the effect of subject identity bias and improve the
generalisation performance of the trained model, whilst also simulating the
model’s performance with real subjects that it has not seen before.

In the Basic FER Phase, the model is trained to completion, with the accuracy
evaluated using the test dataset at the end of each epoch. The next test fold
is then selected, with the training data comprising the remaining 9 folds, and
the model is again trained and evaluated. This process is iterated over 10
times for the 10 possible test and training set combinations. The final
evaluation results are aggregated to produce a maximum, mean and standard
deviation for Basic FER accuracy.

A set of six basic expression classes were chosen in line with [3] (happy,
sad, angry, surprised, disgusted, fearful), making $k_{basic}=6$. These basic
expression images comprise the $X_{basic}$ images used in the Basic FER Phase,
and their corresponding labels are the true labels $y_{basic}$. The compound
expression classes are the remaining set of 15 classes in the CFEE [15]
dataset (happily surprised, happily disgusted, sadly angry, angrily disgusted,
appalled, hatred, angrily surprised, sadly surprised, disgustedly surprised,
fearfully surprised, awed, sadly fearful, fearfully disgusted, fearfully
angry, and sadly disgusted), making $k_{compound}=15$. The total number of
classes available in the dataset is $k_{total}=k_{basic}+k_{compound}$.

A ResNet50V2 model [36] is used as the base model for the $FE$ network and
pre-trained on ImageNet [21] with fine-tuning. Initially, the base model is
frozen to train only the top dense layers on the FER task. Early-stopping is
used by monitoring the test accuracy during training. Once test accuracy has
ceased increasing, training stops and the best model weights are restored.
Next, the layers of the base ResNet model are unfrozen and the model is
trained again to fine-tune these weights to the FER task. During training in
the Continual Learning and Few-shot Learning Phases, the layers of the first
two convolutional blocks in the base model are frozen, since knowledge encoded
in these weights relates to the fundamentals of image recognition such as
lines and shapes. Freezing these layers prevents the weights from being
destroyed when learning new expression classes, and also saves computational
power and training time, making the model more practically applicable.

A initial range of good hyperparameters for the Basic FER Model were chosen
based on empirical knowledge of good deep learning architectures. These
hyperparameters form a multi-dimensional search space which can be searched to
optimise the model. The Hyperband optimisation algorithm [37] was chosen as
the search algorithm and implemented using KerasTuner [38]. Hyperband uses an
infinite-armed bandit method combined with successive halving to explore the
search space and converge to an optimal set of hyperparameters. Following this
tuning process, the optimal hyperparameters were selected as in Table 2.

Basic FER Phase
---
Initial Training epochs | 1000 (early stopping)
Fine Tuning Training epochs | 1000 (early stopping)
Batch size | 32
Optimisation function | Adam
Initial Learning Rate | 1e-4
Fine Tuning Learning Rate | 1e-6
Continual Learning and Few-shot Learning Phases
Training epochs | 1000 (early stopping)
Batch size | 32
Optimisation function | Adam
Learning Rate | 1e-5
Temperature (T) | 3
Distillation Weight ($\gamma$) | 0.1
Table 2: Hyperparameters

Evaluating the model in the Basic FER Phase using Subject-Independent K-Fold
Cross Validation, a maximum, mean and standard deviation FER accuracy are
obtained as shown in Table 3. The best results are comparable with state-of-
the-art basic FER methods (see Table 1).

Max Acc. | Mean Acc. | SD Acc.
---|---|---
0.9624 | 0.8525 | 0.061
Table 3: Accuracy of Basic FER Model using Subject-Independent K-fold Cross
Validation

For the Continual Learning Phase, the best performing test set from the Basic
FER Phase cross validation was used. The continual learning model was
evaluated using the method and metrics developed by [13]. Using this method,
the sequence of new classes learned in continual learning are randomised in
order to test the model’s invariance to the sequence order of expression
classes. For each randomised list of complex expressions, out of $C$ total
lists (where $C=10$ for our experiments), the accuracy at each continual
learning step $i$, over $N$ number of test samples, is recorded. The average
step accuracy, $aveSA_{i}$ is the average accuracy at step $i$ over all lists
$C$, as shown in (11).

$aveSA_{i}=\frac{1}{C}\sum_{j=1}^{C}\frac{1}{|y^{j}|}\sum_{n=1}^{N}H(y_{n}^{j},\hat{y}_{n}^{j})$
(11)

Where $H(y_{n}^{j},\hat{y}_{n}^{j})$ is an indicator function that returns $1$
if $y_{n}^{j}=\hat{y}_{n}^{j}$ and $0$ if $y_{n}^{j}\neq\hat{y}_{n}^{j}$.
Furthermore, the $OverallAccuracy$ is calculated as the average $aveSA$ over
all continual learning steps $i$, as demonstrated in (12).

$OverallAccuracy=\frac{1}{k_{total}}\sum_{i=1}^{k_{total}}aveSA_{i}$ (12)

The $OverallAccuracy$ results obtained in the Continual Learning Phase are
displayed in Table 4, along with comparable baseline results as reported in
[13]. Furthermore, out of the 10 randomised complex expression lists, Figures
8 and 9 demonstrate the performance of the best, worst, and nearest to
$OverallAccuracy$ in terms of the average accuracy over each step $i$.

Figure 8: Best, worst and near-average accuracy at each continual learning step $i$ Figure 9: Best, worst and near-average accuracy at each continual learning step $i$ (excl. singular labels*) Method | Overall Accuracy
---|---
Tree-CNN [39] | (0.5107, 0.6437)
Fine-tuning [40] | (0.6837, 0.7418)
LwF [41] | (0.5373, 0.6638)
TOPIC [42] | (0.5268, 0.7168)
Deep SLDA [43] | (0.5387, 0.7478)
REMIND [44] | (0.5398, 0.7463)
Lucir-CNN [45] | (0.5698, 0.7639)
PODNet-CNN [46] | (0.5991, 0.8163)
Lucir [45] /w AANets [47] | (0.6414, 0.8598)
PODNet-CNN [46] /w AANets [47] | (0.6781, 0.8697)
iCaRL [48] | (0.7138, 0.8327)
DCLEER [13] | (0.7361, 0.8904)
Our Method | (0.7428, 0.7327)
Our Method (excl. singular labels*) | (0.8232, 0.7810)
Table 4: Overall Accuracy of our method compared with other continual learning
methods (unknown class only, all classes)

*Additionally, a second experiment was conducted using only compound facial expressions whose labels are composed of two basic expression labels, such that the expressions hatred, appalled, and awed are omitted and $k_{compound}=12$. The semantic relationship between these singular expressions and the six basic facial expressions is not as clear as with expressions like happily surprised, which is clearly related to happy and surprised. By removing these singular expressions, we can focus on FER for compound expressions and more thoroughly test the hypothesis that higher accuracy can be achieved for compound FER through knowledge distillation of the features of basic expressions which are common to related compound expressions.

To further evaluate the use of continual learning for compound FER over other
methods, the baseline metrics reported by [13] are again used. Here, a number
of state-of-of-the-art non-continual learning methods are used to evaluate the
21 emotion labels of the CFEE [15] database. The results are compared with our
method as evaluated after the final continual learning iteration with 21
labels, and are displayed in Table 5.

Method | Accuracy
---|---
AlexNet [49] | 0.5637
VGG-16 [50] | 0.5018
VGG-19 [50] | 0.4971
Inception-v3 [51] | 0.4288
DenseNet-201 [52] | 0.4862
SCN [53] | 0.4621
PSR [54] | 0.5591
ESRs [55] | 0.5781
Our Method | 0.7176
Our Method (excl. singular labels*) | 0.7182
Table 5: AveSA at final iteration of our method and non-continual learning
methods

To evaluate the Few-shot Learning Phase, the best performing test set from the
Basic FER Phase was used. The same hyperparameters were used as in the
Continual Learning Phase. One experiment was conducted for each of the 15
compound expression classes in the CFEE [15] dataset, making
$k_{compound}=15$. Each of these 15 experiments were repeated in trials
consisting of 5, 3 and 1 training samples. The test accuracy and number of
steps to train the model to convergence (using early stopping) were recorded
at the end of each experiment. The results are displayed in Table 6.

Expression | 5-shot | 3-shot | 1-shot
---|---|---|---
| Acc. | Steps | Acc. | Steps | Acc. | Steps
Happily surprised | 1.0 | 3840 | 1.0 | 4800 | 1.0 | 5536
Happily disgusted | 1.0 | 800 | 1.0 | 960 | 1.0 | 1344
Sadly angry | 1.0 | 480 | 1.0 | 384 | 1.0 | 608
Angrily disgusted | 1.0 | 640 | 1.0 | 864 | 1.0 | 1056
Appalled | 1.0 | 320 | 1.0 | 288 | 1.0 | 320
Hatred | 1.0 | 640 | 1.0 | 672 | 1.0 | 896
Angrily surprised | 1.0 | 640 | 1.0 | 768 | 1.0 | 1536
Sadly surprised | 1.0 | 640 | 1.0 | 576 | 1.0 | 768
Disgustedly surprised | 1.0 | 800 | 1.0 | 960 | 1.0 | 1504
Fearfully surprised | 1.0 | 480 | 1.0 | 384 | 1.0 | 736
Awed | 1.0 | 800 | 1.0 | 1056 | 1.0 | 1184
Sadly fearful | 1.0 | 480 | 1.0 | 480 | 1.0 | 480
Fearfully disgusted | 1.0 | 640 | 1.0 | 768 | 1.0 | 928
Fearfully angry | 1.0 | 480 | 1.0 | 384 | 1.0 | 448
Sadly disgusted | 1.0 | 1280 | 1.0 | 1248 | 1.0 | 1472
Table 6: Accuracy and training time for few-shot learning experiments

##  6 Conclusions

A novel method for compound facial expression recognition was developed by
distilling the knowledge of basic expressions when learning new compound
expressions. Two main experiments were conducted in continual learning and
few-shot learning.

We demonstrate improvements in continual learning for complex FER through our
novel knowledge distillation and Predictive Sorting Memory Replay techniques,
achieving the state-of-the-art with 74.28% Overall Accuracy on new classes
only (an improvement of 0.67%). The Overall Accuracy on all classes is 73.27%
which is comparable to baseline results and indicates a reduction in the
effects of catastrophic forgetting as the accuracy on known classes is not
greatly impacted when learning new classes. Results could potentially be
further improved by tuning the Temperature ($T$) and Distillation Weight
($\gamma$) hyperparameters.

Our method also demonstrates an improvement in accuracy over other state-of-
the-art non-continual learning methods for facial expression recognition by
13.95%. This demonstrates the benefits of our approach to learning facial
expressions through continual learning, by first learning to recognise basic
facial expressions and then synthesising that knowledge to learn new complex
facial expressions in a similar way to humans.

Our method achieves 100% accuracy in all classes using only 5, 3 or 1 samples
in few-shot learning, which is the state-of-the-art in the facial expression
recognition domain to the best of our knowledge. These results also
demonstrate the benefits of learning to recognise basic facial expressions
prior to learning complex facial expressions, as the model was able to very
quickly learn the new expression classes using only a limited number of image
samples through knowledge distillation of basic features.

By visually inspecting the Grad-CAM heatmaps of basic expression images with
those of compound expressions, a strong correlation was found between the
activations of features in basic expressions and that of features in compound
expressions. The activated areas also appear to align with the facial action
encoding system [3]. In future works, the use of action units as an additional
feature extraction component may also be able to increase the accuracy of the
model for both basic facial expression recognition and complex facial
expression recognition.

One limitation of this work is in evaluating the model using only one dataset.
By evaluating on multiple datasets, we could get a better picture of the
model’s generalisation performance with a wider variety of subjects and image
conditions. Evaluating the model on in-the-wild complex emotion datasets such
as EmotioNet [56] and AffectNet [57] would also prove useful in assessing the
model’s generalisation and suitability for practical applications such as
human-computer interaction, as these datasets more closely resemble real
facial expressions.

## References

  * [1] C. Darwin, _The Expression of the Emotions in Man and Animals_. Oxford University Press, 1872.
  * [2] A. Mehrabian, _Communication without words_. Routledge, 2017, pp. 193–200.
  * [3] P. Ekman, “Facial action coding system (facs),” _A human face_ , 2002.
  * [4] D. Matsumoto, “More evidence for the universality of a contempt expression,” _Motivation and Emotion_ , vol. 16, no. 4, pp. 363–368, 1992.
  * [5] P. Ekman and W. V. Friesen, “Constants across cultures in the face and emotion,” _Journal of personality and social psychology_ , vol. 17, no. 2, p. 124, 1971.
  * [6] A. Mollahosseini, D. Chan, and M. H. Mahoor, “Going deeper in facial expression recognition using deep neural networks,” in _2016 IEEE Winter conference on applications of computer vision (WACV)_. IEEE, 2016, Conference Proceedings, pp. 1–10.
  * [7] G. Zhao and M. Pietikainen, “Dynamic texture recognition using local binary patterns with an application to facial expressions,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 29, no. 6, pp. 915–928, 2007\.
  * [8] J. Li and E. Y. Lam, “Facial expression recognition using deep neural networks,” in _2015 IEEE International Conference on Imaging Systems and Techniques (IST)_ , 2015, Conference Proceedings, pp. 1–6.
  * [9] H. Jung, S. Lee, J. Yim, S. Park, and J. Kim, “Joint fine-tuning in deep neural networks for facial expression recognition,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, Conference Proceedings, pp. 2983–2991.
  * [10] P. Liu, S. Han, Z. Meng, and Y. Tong, “Facial expression recognition via a boosted deep belief network,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2014, Conference Proceedings, pp. 1805–1812.
  * [11] Y. Zhao and J. Xu, “A convolutional neural network for compound micro-expression recognition,” _Sensors_ , vol. 19, no. 24, p. 5553, 2019\. [Online]. Available: https://www.mdpi.com/1424-8220/19/24/5553
  * [12] B. Sun, S. Cao, D. Li, J. He, and L. Yu, “Dynamic micro-expression recognition using knowledge distillation,” _IEEE Transactions on Affective Computing_ , pp. 1–1, 2020.
  * [13] S. Thuseethan, S. Rajasegarar, and J. Yearwood, “Deep continual learning for emerging emotion recognition,” _IEEE Transactions on Multimedia_ , pp. 1–1, 2021.
  * [14] ——, “Complex emotion profiling: An incremental active learning based approach with sparse annotations,” _IEEE Access_ , vol. 8, pp. 147 711–147 727, 2020.
  * [15] S. Du, Y. Tao, and A. M. Martinez, “Compound facial expressions of emotion,” _Proceedings of the National Academy of Sciences_ , vol. 111, no. 15, pp. E1454–E1462, 2014.
  * [16] J.-M. Fernández-Dols and C. Crivelli, “Emotion and expression: Naturalistic studies,” _Emotion Review_ , vol. 5, no. 1, pp. 24–29, 2013.
  * [17] The National Institute of Mental Health, “Depression,” 2021. [Online]. Available: https://www.nimh.nih.gov/health/publications/depression
  * [18] F. Z. Salmam, A. Madani, and M. Kissi, “Emotion recognition from facial expression based on fiducial points detection and using neural network,” _International Journal of Electrical and Computer Engineering_ , vol. 8, no. 1, p. 52, 2018.
  * [19] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, Conference Proceedings, pp. 770–778.
  * [20] F. Chollet, “Xception: Deep learning with depthwise separable convolutions,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, Conference Proceedings, pp. 1251–1258.
  * [21] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, “Imagenet large scale visual recognition challenge,” _International Journal of Computer Vision_ , vol. 115, no. 3, pp. 211–252, 2015. [Online]. Available: https://doi.org/10.1007/s11263-015-0816-y
  * [22] P. Viola and M. Jones, “Rapid object detection using a boosted cascade of simple features,” in _Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001_ , vol. 1, 2001, Conference Proceedings, pp. I–I.
  * [23] H.-W. Ng, V. D. Nguyen, V. Vonikakis, and S. Winkler, “Deep learning for emotion recognition on small datasets using transfer learning,” p. 443–449, 2015. [Online]. Available: https://doi-org.ezproxy-f.deakin.edu.au/10.1145/2818346.2830593
  * [24] I. Fasel, B. Fortenberry, and J. Movellan, “A generative framework for real time object detection and classification,” _Computer Vision and Image Understanding_ , vol. 98, no. 1, pp. 182–210, 2005.
  * [25] M. F. Valstar and M. Pantic, “Fully automatic recognition of the temporal phases of facial actions,” _IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics)_ , vol. 42, no. 1, pp. 28–43, 2012.
  * [26] X. Xiong and F. De la Torre, “Supervised descent method and its applications to face alignment,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2013, Conference Proceedings, pp. 532–539.
  * [27] J. Deng, J. Guo, E. Ververas, I. Kotsia, and S. Zafeiriou, “Retinaface: Single-shot multi-level face localisation in the wild,” in _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, Conference Proceedings, pp. 5202–5211.
  * [28] F. M. Castro, M. J. Marín-Jiménez, N. Guil, C. Schmid, and K. Alahari, “End-to-end incremental learning,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, Conference Proceedings, pp. 233–248.
  * [29] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” 2015.
  * [30] M. U. Ahmed, K. J. Woo, K. Y. Hyeon, M. R. Bashar, and P. K. Rhee, “Wild facial expression recognition based on incremental active learning,” _Cognitive Systems Research_ , vol. 52, pp. 212–222, 2018. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1389041718301189
  * [31] S. Gao, Y. Zhang, K. Jia, J. Lu, and Y. Zhang, “Single sample face recognition via learning deep supervised autoencoders,” _IEEE Transactions on Information Forensics and Security_ , vol. 10, no. 10, pp. 2108–2118, 2015.
  * [32] H. Min, Y. Zhang, Y. Zhao, W. Jia, Y. Lei, and C. Fan, “Hybrid feature enhancement network for few-shot semantic segmentation,” _Pattern Recognition_ , vol. 137, 2023.
  * [33] Y. Gong, Y. Yue, W. Ji, and G. Zhou, “Cross-domain few-shot learning based on pseudo-siamese neural network,” _Sci Rep_ , vol. 13, no. 1, p. 1427, 2023\. [Online]. Available: https://www.ncbi.nlm.nih.gov/pubmed/36697442
  * [34] S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert, “icarl: Incremental classifier and representation learning,” in _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , 2017, Conference Proceedings, pp. 2001–2010.
  * [35] K. He, X. Zhang, S. Ren, and J. Sun, “Identity mappings in deep residual networks,” in _European conference on computer vision_. Springer, 2016, Conference Proceedings, pp. 630–645.
  * [36] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-cam: Visual explanations from deep networks via gradient-based localization,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, Conference Proceedings, pp. 618–626.
  * [37] L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar, “Hyperband: A novel bandit-based approach to hyperparameter optimization,” _The Journal of Machine Learning Research_ , vol. 18, no. 1, pp. 6765–6816, 2017.
  * [38] T. O’Malley, E. Bursztein, J. Long, F. Chollet, H. Jin, and L. Invernizzi, “Kerastuner,” 2019. [Online]. Available: https://github.com/keras-team/keras-tuner
  * [39] D. Roy, P. Panda, and K. Roy, “Tree-cnn: a hierarchical deep convolutional neural network for incremental learning,” _Neural Networks_ , vol. 121, pp. 148–160, 2020.
  * [40] R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2014, pp. 580–587.
  * [41] Z. Li and D. Hoiem, “Learning without forgetting,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 40, no. 12, pp. 2935–2947, 2017\.
  * [42] X. Tao, X. Hong, X. Chang, S. Dong, X. Wei, and Y. Gong, “Few-shot class-incremental learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 12 183–12 192.
  * [43] T. L. Hayes and C. Kanan, “Lifelong machine learning with deep streaming linear discriminant analysis,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops_ , 2020, pp. 220–221.
  * [44] T. L. Hayes, K. Kafle, R. Shrestha, M. Acharya, and C. Kanan, “Remind your neural network to prevent catastrophic forgetting,” in _European Conference on Computer Vision_. Springer, 2020, pp. 466–483.
  * [45] S. Hou, X. Pan, C. C. Loy, Z. Wang, and D. Lin, “Learning a unified classifier incrementally via rebalancing,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 831–839.
  * [46] A. Douillard, M. Cord, C. Ollion, T. Robert, and E. Valle, “Podnet: Pooled outputs distillation for small-tasks incremental learning,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XX 16_. Springer, 2020, pp. 86–102.
  * [47] Y. Liu, B. Schiele, and Q. Sun, “Adaptive aggregation networks for class-incremental learning,” in _Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition_ , 2021, pp. 2544–2553.
  * [48] S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert, “icarl: Incremental classifier and representation learning,” in _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , 2017, pp. 2001–2010.
  * [49] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” _Advances in neural information processing systems_ , vol. 25, 2012.
  * [50] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint arXiv:1409.1556_ , 2014.
  * [51] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 2818–2826.
  * [52] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 4700–4708.
  * [53] K. Wang, X. Peng, J. Yang, S. Lu, and Y. Qiao, “Suppressing uncertainties for large-scale facial expression recognition,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 6897–6906.
  * [54] T.-H. Vo, G.-S. Lee, H.-J. Yang, and S.-H. Kim, “Pyramid with super resolution for in-the-wild facial expression recognition,” _IEEE Access_ , vol. 8, pp. 131 988–132 001, 2020.
  * [55] H. Siqueira, S. Magg, and S. Wermter, “Efficient facial feature learning with wide ensemble-based convolutional neural networks,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 34, no. 04, 2020, pp. 5800–5809.
  * [56] C. F. Benitez-Quiroz, R. Srinivasan, Q. Feng, Y. Wang, and A. M. Martinez, “Emotionet challenge: Recognition of facial expressions of emotion in the wild,” _arXiv preprint arXiv:1703.01210_ , 2017.
  * [57] A. Mollahosseini, B. Hasani, and M. H. Mahoor, “Affectnet: A database for facial expression, valence, and arousal computing in the wild,” _IEEE Transactions on Affective Computing_ , vol. 10, no. 1, pp. 18–31, 2017.

[]Angus Maiden completed the Master of Applied Artificial Intelligence
(Professional) from Deakin University in July 2022, with a Weighted Average
Mark of 91.7% and a GPA of 4.0/4, for which he received the Deakin Scholarship
for Excellence (Postgraduate), the Student Excellence Award (Research) for
best paper, and the Student Excellence Award (Postgraduate) for achieving the
highest overall weighted average mark in his cohort. He is also a member of
the Golden Key International Honour Society.

Angus has been appointed as a Visiting Researcher at Deakin University since
October 2022. His research interests include artificial intelligence, deep
learning, machine learning, natural language processing, computer vision and
emotion recognition. He is currently employed as a Data Scientist Technical
Lead at a startup software company where he builds innovative health-tech
products using machine learning and artificial intelligence.

[]Bahareh Nakisa (Member, IEEE) received the Bachelor of Science degree in
Software Engineering from Iran in 2008, the Master of Computer Science degree
from the National University of Malaysia in 2014, and the Ph.D. degree in
Computer Science (Artificial Intelligence) from the Queensland University of
Technology (QUT), Australia in 2019.

Bahareh started working in the industry as an AI Scientist and the Lead AI
Scientist. She then joined the School of Information Technology, Deakin
University, as a Lecturer of applied AI, in 2019. She is currently a Lecturer
of applied AI and the Course Director of applied AI at the School of
Information Technology, Deakin University.

Bahareh’s research interests include artificial intelligence, machine
learning, and deep learning. She has applied analytic and algorithmic tools
from these fields to solve real-world problems related to diverse domains,
especially in health and affective computing.