# Masked Student Dataset of Expressions

Sridhar Sola  Sri Sathya Sai Institute of Higher
LearningBengaluruKarnatakaIndia<EMAIL_ADDRESS>and  Darshan Gera  Sri
Sathya Sai Institute of Higher LearningBengaluruKarnatakaIndia
<EMAIL_ADDRESS>
(2022; Date: )

###### Abstract.

Facial expression recognition (FER) algorithms work well in constrained
environments with little or no occlusion of the face. However, real-world face
occlusion is prevalent, most notably with the need to use a face mask in the
current Covid-19 scenario. While there are works on the problem of occlusion
in FER, little has been done before on the particular face mask scenario.
Moreover, the few works in this area largely use synthetically created masked
FER datasets. Motivated by these challenges posed by the pandemic to FER, we
present a novel dataset, the Masked Student Dataset of Expressions or MSD-E,
consisting of 1,960 real-world non-masked and masked facial expression images
collected from 142 individuals. Along with the issue of obfuscated facial
features, we illustrate how other subtler issues in masked FER are represented
in our dataset. We then provide baseline results using ResNet-18, finding that
its performance dips in the non-masked case when trained for FER in the
presence of masks. To tackle this, we test two training paradigms: contrastive
learning and knowledge distillation, and find that they increase the model’s
performance in the masked scenario while maintaining its non-masked
performance. We further visualise our results using t-SNE plots and Grad-CAM,
demonstrating that these paradigms capitalise on the limited features
available in the masked scenario. Finally, we benchmark SOTA methods on MSD-E.
The dataset is available at https://github.com/SridharSola/MSD-E.

Human-computer interaction, Computer vision, Facial expression recognition,
Covid-19

††journalyear: 2022††copyright: acmcopyright††conference: Proceedings of the
Thirteenth Indian Conference on Computer Vision, Graphics and Image
Processing; December 8–10, 2022; Gandhinagar, India††booktitle: Proceedings of
the Thirteenth Indian Conference on Computer Vision, Graphics and Image
Processing (ICVGIP’22), December 8–10, 2022, Gandhinagar, India††price:
15.00††doi: 10.1145/3571600.3571608††isbn: 978-1-4503-9822-0/22/12††ccs:
Human-centered computing HCI design and evaluation methods

##  1\. Introduction

Facial expressions give hints about others’ thoughts, emotions, and reactions.
We humans modify our actions and words based on these facial signals.
Evidently, facial expressions are a cornerstone of human-human interaction. It
is not surprising then that enabling computers to recognize different facial
expressions will be crucial in developing seamless human-computer interaction.
Automatic Facial Expression Recognition (FER) aims to do just that.

By using Deep Learning (DL) and Convolutional Neural Networks, FER has reached
a state where models are ready to be deployed in the real world (Saleem et
al., 2021). However, despite the advantages of DL, FER has been plagued by
issues like occlusion, pose variation, and noisy annotations. The amount of
occlusion, in particular, has been exacerbated by the Covid-19 pandemic. The
necessity to wear a face mask means that FER systems lose around half of
facial features as masks cover the face and nose regions. Crucially, these
areas have critical features needed to discriminate between emotions (Kotsia
et al., 2008).

Figure 1. Challenges in FER: Variety of masks depicted in the right-hand
columns of (a) and (b), and (a)stonger expressions are easier for FER while
(b)subtler ones are hard.

While FER in the presence of masks is challenging enough, the issue of data
adds to the difficulty; Datasets for occlusion are few in number and small in
size. Further, none are specific to the case of face masks. While researchers
are working on handling occlusion in FER, little has been done in the past to
develop occlusion (masked) datasets. Wang et al. (Wang et al., 2020)
segregated occlusion images from three popular FER datasets: FERPlus (Barsoum
et al., 2016), RAFDB (Li et al., 2017; Li and Deng, 2019) and and AffectNet
(Mollahosseini et al., 2017), to form Occlusion-FERPlus, Occlusion-RAFDB, and
Occlusion-AffectNet. FEDRO (Li et al., 2019) is another recently released FER
dataset with real-world occlusion. The above datasets have occlusion by
sunglasses, medical masks, hands, and hair. While the number of images in
these datasets is small, mask occluded images are present only in FEDRO
dataset. Table 1 presents the statistics of the occlusion datasets mentioned.
Towards masked FER, Yang et al. (Bo et al., 2020) manually crawled 562 masked
FER images from the internet to obtain M-FER-T dataset comprising masked
images. However, the dataset is annotated with three labels - positive,
negative, and neutral - rather than the seven basic emotions used in popular
datasets like RAFDB (Li et al., 2017; Li and Deng, 2019) and AffectNet
(Mollahosseini et al., 2017).

Figure 2. Sample images from (a)MSD-E and (b)MSD-PE. Non-masked images are on
the left and masked are on the right.

Currently, work on masked FER is benchmarked on synthetically created
datasets. In (Bo et al., 2020), 10,487 images from LFW dataset are annotated
with three expression classes as before to get M-LFW-FER. They also created
M-KDDI-FER – a dataset with 17,236 images. Gera et al. (Gera and S, 2021)
synthetically applied face masks using a publicly available tool
111https://github.com/X-zhangyang/Real-World-Masked-Face-Dataset on FERPlus
(Barsoum et al., 2016) and RAFDB (Li et al., 2017; Li and Deng, 2019) to
obtain Masked-FERPlus, which has 28,709 images for training and 3589 images
for testing, and Masked-RAFDB, which has 12,271 and 3,008 images for training
and testing respectively.

Apart from the above-mentioned real-world occlusion datasets and the synthetic
masked datasets, we could not find any other study of masked FER datasets.
This striking lack of data has motivated us to propose the Masked Student
Dataset of Expressions (MSD-E). The dataset consists of 1,960 images annotated
with the 7 basic emotions: Surprise, fear, disgust, happy, sad, anger, and
neutral. 142 subjects provided non-masked and masked images of these
expressions. To ensure that the labels are reliable, we ask volunteers to
verify whether the expression label given during image collection matches the
expression performed. Based on the observations, we relabel any images with a
disparity.

We then benchmark our dataset using ResNet-18 (Barsoum et al., 2016) and two
SOTA methods for occlusion in FER. To further tackle the case of occlusion by
masks in FER, we employ two training paradigms on our dataset: Contrastive
Learning (CL) and Knowledge Distillation (KD). These methods are motivated by
work done in the field of masked face recognition (Neto et al., 2021;
Georgescu et al., 2022; Huber et al., 2021). As both paradigms require pairs
of images – one non-masked and one masked, we test them on a subset of the
expression dataset, MSD-E, called Masked Student Dataset of Paired Expressions
(MSD-PE) that consists of pairs of non-masked and masked images. Our
experiments show that these training paradigms significantly increase FER
accuracy in the masked case while dampening the drop in performance in the
non-masked case. They also increase the model’s discrimination power in the
case of hard emotions like disgust and fear. Significantly, the dataset has
numerous hard images, i.e., images where the expression is subtle or a mixture
of emotions as depicted in Figure 1.

Table 1. No. of Images in Challenging Subsets of Benchmark Datasets Dataset | Occluded Images | Masked Images
---|---|---
FERPlus (Barsoum et al., 2016) | 111 | 0
RAFDB (Li et al., 2017; Li and Deng, 2019) | 123 | 0
Affectnet (Mollahosseini et al., 2017) | 100 | 0
FEDRO (Li et al., 2019) | 234 | 10

To the best of our knowledge, the expression dataset MSD-E is the first FER
dataset collected specifically for the masked scenario and the first FER
dataset of Indian ethnicity. The following sections discuss the dataset in
detail and provide the specifics of experiments using the dataset.

##  2\. Masked Student Dataset of Expressions

###  2.1. Image Data

The basic facial expressions of 142 volunteers were recorded using two
devices. The first was a DSLR camera with 31.7 megapixels and a $36\times 24$
mm sensor, while the second was a mobile phone camera with a 50 megapixels
(f/1.8) primary camera and two 2 megapixels (f/2.4) cameras. We do not
experiment with images collected from different cameras separately. The
experimenter instructed each participant to perform the seven basic emotions:
Surprise, fear, disgust, happy, sad, anger, and neutral, first without a mask
and then with one. As a result, we obtained 14 images from each participant –
7 non-masked and 7 masked. We anticipated that some images would have to be
either removed due to camera artifact or relabeled while verifying the given
labels. To avoid loss of data, we asked 14 participants with the ability to
emote the expressions well to provide an additional set of images. The
experimenter intentionally requested them to perform ”tough” expressions such
as fear, disgust, and anger. As a result, we obtained 1,007 pairs of images or
2,014 images in total before the verification process.

The participants were all male, except for two, and their ages varied between
10 and 40. All the participants were of Indian ethnicity. Significantly, we
had participants from all six physiographic divisions of India. In addition,
the participant used their own mask. Some were the common surgical mask, while
many used masks with printed designs, providing real variety in the type of
mask in the images. Further, there is significant intra-class variation as
different participants emoted the same expression differently. We discuss the
challenges in the dataset in Section 2.3. Since the dataset comprises
primarily of student subjects, we name the dataset Masked Student Dataset of
Expessions or MSD-E.

Table 2. Statistics of MSD-E and its subset MSD-PE. Expression | MSD-E | MSD-PE(Pairs)
---|---|---
Non-masked | Masked | Total
Surprise | 128 | 121 | 249 | 113
Fear | 121 | 128 | 249 | 108
Disgust | 114 | 127 | 241 | 106
Happy | 193 | 164 | 357 | 155
Sad | 123 | 142 | 256 | 119
Angry | 104 | 143 | 247 | 100
Neutral | 181 | 171 | 352 | 156
Total | 964 | 996 | 1960 | 857

###  2.2. Validating Expression Labels

A major limitation of posed facial expression data is that the expression
requested may not always match the expression performed. This is because the
subject may be unable to act out an otherwise spontaneous facial expression.
Assuming these expression labels to be the ground truth is highly unreliable.
Taking a leaf from the work of Lucey et al. (Lucey et al., 2010) we determined
whether (1) the image matched the given label, (2) whether the image needed to
be relabelled or, (3) whether it needed to be removed. The verification
process was as follows:

  1. (1)

7 volunteers used perceptual judgment to determine whether an image matched
its expected expression class. Working independently, they visually inspected
the images and flagged any image they judged did not lie in its given class.
If possible, they marked another class the image could belong to, giving it a
new label. If an image was not flagged by the volunteer, its new label was its
original label.

  2. (2)

After collating the volunteers’ remarks, majority voting based on the new
label determined the action performed on the image. Images not flagged by
anyone were trivially reassigned to their original label. 1,719 images were
not flagged by any participant. For the remaining 288 images flagged by some
number of participants, we noted the new label with the highest number of
endorsements. The image was assigned that class if at least half the
participants approved of that label and no more than 25 percent endorsed
another class. The images with their new labels made it to the Mask Student
Dataset of Expressions (MSD-E).

  3. (3)

To obtain the Mask Student Dataset of Paired Expressions (MSD-PE), we had to
ensure that the non-masked image and its corresponding masked image had the
same new label. To this end, we removed all pairs of images that had a
mismatch. 150 pairs were removed, leaving us 857 pairs of non-masked and
masked images.

The inventory of the above validation process for the two datasets is given in
Table 2 and examples of image pairs are depicted in Figure 2.

###  2.3. Challenges in the Dataset

FER in the real world will need to be nuanced. Machines will need to be robust
to the presence of eyewear, like spectacles and sunglasses. For this reason,
we did not ask subjects to remove their glasses during data collection. Apart
from this, the variety of masks and their styles, and the manner in which
expressions are emoted contribute to the complexity of real-world FER. We
discuss how our dataset incorporates these challenges below.

Variety of masks: The masked example images in Figure 2 depict the variety of
masks used by the volunteers while posing for the photos. A sizeable number
used the common light blue surgical mask; another substantial set of
participants brought with them an N95 mask that was either white, blue, or
black; many more participants used a cloth mask. The cloth masks contribute
the most variety to the dataset as they come in different colours and designs.
Our dataset reflects the real-world scenario concerning masks used in images.
But this is not the only way our dataset exhibits real-world attributes.

Intra-class variation: As discussed earlier, facial expressions reflect our
thoughts and feelings. There cannot be discrete levels or combinations of
states of being. Consequently, the representations of these states through
facial expressions cannot be disconnected from one another; they must be
combinational. For precisely this reason, datasets like RAFDB (Li et al.,
2017; Li and Deng, 2019) employ compound emotion labelling where each image is
labeled with multiple emotions to represent the expression better. For MSD-E,
we have chosen to label each image with the dominant expression. However, this
does not take away from the fact that many images have more than one emotion
present. For instance, some images may be best described as ”fearfully
surprised” or ”angrily disgusted” and would have been labeled as surprise and
disgust, respectively. To add to this, it may be easy to classify some images
compared to others. Images where the dominant expression is ”harsh” and ”loud”
are easy for a human and a machine to distinguish. On the other hand, the
expression in other images is subtler and may be confusing for even humans to
discern. However, such expressions arise from the sundry of facial
expressions, making it imperative that computer vision models learn to perform
on easy and difficult expression images to enable a more nuanced human-
computer interaction. Figure 1 depicts examples of easy and difficult images
in our dataset.

With such variety, our dataset presents various challenges. But these
challenges provide opportunities to fine-tune FER models to perform well in a
real-world environment. Along with our extensive validation process mentioned
in Section 2.2, we present a novel dataset with real-world subtleties and
characteristics. The following section discusses experiments we have performed
using MSD-E.

##  3\. Experiments

For robust FER in the current post-pandemic scenario, we need models that
perform well in the non-masked and masked case simultaneously. While FER
architectures have been developed to be occlusion-invariant (Gera and S, 2021;
Wang et al., 2020; Li et al., 2019; Ding et al., 2020), their performance has
not been tested on real-world masked expression data. For these reasons, we
perform the following experiments:

  * •

We find the performance of ResNet-18 (He et al., 2016) on MSD-E and MSD-PE to
act as a baseline to compare to other models.

  * •

We evaluate the performance of two training paradigms: Contrastive learning
(Neto et al., 2021) and knowledge distillation (Georgescu et al., 2022; Huber
et al., 2021), that have both shown promising results in masked face
recognition. As these paradigms require pairs of images, we evaluate their
performance on MSD-PE.

  * •

Finally, we evaluate the performances of state-of-the-art (SOTA) methods on
MSD-E and MSD-PE, and compare them with that of contrastive learning and
knowledge distillation. These are particularly approaches that have been
developed to tackle the problem of occlusion in FER.

###  3.1. Experimental Setup

Training Protocol:  While performing experiments on MSD-E or MSD-PE, we use
10-fold cross validation strategy to make use of all the data. To this end,
the images are separated into 10 identity-independent folds. One fold is set
as the validation set while the remaining are used for training. The model’s
accuracy on the validation set is noted. This is done 10 times, once with each
fold as the validation set. We then report the average validation set
accuracy. As the number of images per subject is not the same, the number of
images per fold varies slightly as well.

Implementation Details:  ResNet-18 is used as the backbone network for all
experiments, except when mentioned otherwise. Pre-trained weights of ResNet-18
trained on large scale face recognition dataset MS-Celeb-1M (Guo et al., 2016)
are loaded onto the backbone. We use Adam optimizer and Cross Entropy loss for
classification. The models are trained for 25 epochs with an initial learning
rate of 0.0001 and 0.001 for the backbone and classification layer
respectively which is reduced by a factor of 10 at epochs 8, 14, and 20. All
the experiments are implemented in PyTorch and are run on a Tesla P100 GPU.

Figure 3. Architecture of Contrastive Learning-based FER on MSD-PE. Two
feature vectors are generated by passing a non-masked and masked image to the
model. Their difference is given by $L_{\mathrm{MSE}}$. The error in
expression classification is given by $L_{\mathrm{NM}}$ and $L_{\mathrm{M}}$.
The architecture is based on the work of Neto et al.[15].

###  3.2. Baseline Results

In order to establish the baseline performance on MSD-E, we trained and tested
ResNet-18 (He et al., 2016) on three subsets of MSD-E using the protocol
mentioned in the previous subsection. The first was the Non-masked MSD-E
(represented as NM), comprising of only non-masked images; the second was
Masked MSD-E (represented as M), comprising of only masked images; the third
was the whole MSD-E (represented as Mix). The classification accuracies
obtained are reported in Table 3 for NM and Table 4 for M. The model acheives
65.35% accuracy when trained and tested on NM, and 45.85 % when trained and
tested on M using the training protocol mentioned. However, when trained on
Mix, the model’s performance on both non-masked (55.88%) and masked images
(44.15%) dips significantly. These results indicate that the model is unable
to learn FER for both non-masked and masked images concurrently when trained
routinuely on the whole set of images as it ”forgets” FER for the non-masked
case. To avoid this ”forgetting”, we require paradigms that assist the model
to improve its masked accuracy while retaining its non-masked performance. The
next subsection discusses two approaches that intend to accomplish this. For
comparison of these methods, we compute the baseline on MSD-PE which is
reported in Table 5.

###  3.3. Experiments on MSD-PE

As mentioned earlier, we need mask-invariant models that perform well on non-
masked images as well. The results in Section 3.2 tell us that routinue
training does not achieve this. Two methods that have shown promising results
in masked face recognition literature are Contrastive Learning (CL) (Neto et
al., 2021) and Knowledge Distillation (KD) (Georgescu et al., 2022; Huber et
al., 2021). We evaluate their performance for FER using the MSD-PE dataset. We
note here that we use the whole MSD-PE dataset, i.e., both non-masked and
masked images at the same time. First, we briefly describe the approaches.
Both CL and KD aim to promote the model to produce similar feature vectors for
both non-masked and masked images. The intuition is that this will force the
model to focus on non-occluded regions of the face. However, CL and KD do this
in different ways. We discuss the details below. As far as we are aware, this
is the first study on the use of CL and KD for FER in the face mask scenario.

Contrastive Learning:  In the case of contrastive learning, two images are
passed to the model: one is a non-masked image of subject A with expression E,
the other is the corresponding masked image, i.e., the image of subject A
expressing E with a mask on. The model produces two feature embeddings of size
$7\times 7\times 512$, one for each image. The model is trained to produce
non-masked and masked feature vectors that are closer to each other using Mean
Square Error loss given below.

(1)
$\mathbf{L_{\mathrm{MSE}}}=\frac{1}{B}\frac{1}{F}\Sigma_{i=1}^{B}{\Sigma_{j=1}^{F}{(xn_{\mathrm{ij}}-xm_{\mathrm{ij}})^{2}}}$

Here, B represents the mini-batch size, F is the feature size,
$xn_{\mathrm{ij}}$ is the value of the non-masked feature vector i at position
j, while $xm_{\mathrm{ij}}$ represents the same but for the masked image
feature vector. To train the network for FER, standard cross entropy loss is
used. In Eq. 2, M stands for the number of emotion classes, $y_{c}$ is the
binary indicator 1 if class label is class c and 0 otherwise, while $p_{o,c}$
is the predicted probability that observation o belongs to class c.

(2) $\mathbf{L_{\mathrm{CE}}}=-\sum_{c=1}^{M}y_{o,c}\log(p_{o,c})$

We use two $L_{\mathrm{CE}}$ losses: one for non-masked images, and another
for masked images denoted as $L_{\mathrm{NM}}$ and $L_{\mathrm{M}}$
respectively. The total loss is combination of the three losses mentioned
above as given in Eq. 3. Here, $\lambda$ and $\alpha$ control how much each
loss contributes to training and we set them to 0.2 and 1.0 respectively. The
architecture is depicted in Figure 3.

(3)
$\mathbf{L_{\mathrm{Total}}^{\text{CL}}}=\lambda\mathbf{L_{\mathrm{MSE}}}+\alpha(L_{\mathrm{NM}}+L_{\mathrm{M}})$

Using the experimental setup and training protocol mentioned before, we train
ResNet-18 (He et al., 2016), employing CL, on MSD-PE and report the non-masked
and masked results obtained. These results are presented in Table 5 and the
mean diagonal values of the confusion plots are available in Table 6. From
these results, we find that CL achieves its primary objective of bettering
masked accuracy with an increase of 3.09% compared to the baseline. In
addition, non-masked accuracy increased by 4.13% compared to the baseline. It
is worth noting that this has been achieved with no increase in the number of
trainable parameters.

Table 3. FER accuracy (%) on NM (non-masked images of MSD-E) Method | Non-Masked | No. of parameters
---|---|---
Baseline on NM | 65.35 |  $\sim$11M
RAN (Wang et al., 2020) | 61.55 |  $\sim$23M
SCAN (Gera and S, 2021) | 66.41 |  $\sim$66M
Table 4. FER accuracy (%) on M (masked images of MSD-E) Method | Accuracy
---|---
Baseline | 45.854
RAN (Wang et al., 2020) | 39.41
SCAN (Gera and S, 2021) | 49.29
Table 5. FER accuracy (%) on MSD-PE. Models are trained on whole of MSD-PE. Results are given for different subsets. Method | Non-Masked | Masked | Average
---|---|---|---
Baseline | 56.25 | 43.71 | 49.98
CL | 60.01 | 48.06 | 54.03
KD | 57.63 | 45.86 | 51.74
RAN (Wang et al., 2020) | 60.89 | 41.54 | 51.42
SCAN (Gera and S, 2021) | 65.01 | 57.29 | 61.15
Table 6. Mean diagonal value of confusion matrices. Method | Expression |  |
---|---|---|---
Surprise | Fear | Disgust | Happy | Sad | Angry | Neutral
Train on NM | 0.78 | 0.46 | 0.57 | 0.81 | 0.50 | 0.59 | 0.95
Train on M | 0.73 | 0.15 | 0.32 | 0.47 | 0.34 | 0.43 | 0.56
Train on Mix | 0.72 | 0.27 | 0.39 | 0.68 | 0.45 | 0.48 | 0.66
CL | 0.61 | 0.17 | 0.55 | 0.56 | 0.3 | 0.43 | 0.77
KD | 0.67 | 0.16 | 0.52 | 0.53 | 0.48 | 0.38 | 0.62
Figure 4. Architecture of Knowledge Distillation-based FER on MSD-PE. A non-
masked or masked image is randomly passed to the model. The corresponding non-
masked image is passed to the teacher model. $L_{\mathrm{MSE}}$ gives the
difference between the feature vectors obtained. $L_{\mathrm{CE}}$ gives the
classification error of the student model. The student model is trained using
$L_{\mathrm{Total}}$.

Knowledge Distillation:  While their objectives are similar, knowledge
distillation differs from contrastive learning as it makes use of a teacher
network that is already trained on non-masked images for FER. A student
network ”learns” masked FER from the teacher network by moving the feature
vectors of the student model closer to those of the teacher model. During
training, a non-masked image is fed to the teacher model. On the other hand,
the student model is randomly given, with a probability threshold ’p’, either
the same non-masked image, or its corresponding masked image. This is done to
ensure that the student model performs well on both non-masked and masked
images. We set ’p’ as 0.5. The feature vectors produced by the student model
and the teacher model are pushed closer using Mean Square Error given in Eq 1.
The student model is trained for FER using the Cross Entropy Loss given in Eq
2. The total loss is formally defined in Eq 4 where $\lambda$ controls the
amount of knowledge distillation from the teacher model. We set $\lambda$ to
100. A diagrammatic overview of the proposed training framework is presented
in Figure 4.

(4)
$\mathbf{L_{\mathrm{Total}}^{\text{KD}}}=\lambda\mathbf{L_{\mathrm{MSE}}}+L_{\mathrm{CE}}$

The result of using KD for FER are presented in Table 5 and the mean diagonal
values of confusion plots, indicating the specific discrimination ability of
the model for each expression, are given in Table 6. KD performs above the
baseline but does not perform better than CL. Its accuracy is 2.15% better
than the baseline in the masked scenario and 1.38% better in the non-masked
case. The number of trainable parameters in KD is still the same: 11M. The
only addition are the frozen weights of the teacher model, meaning there are
still 11M parameters during inference.

###  3.4. Visualizations

To further evaluate the impact of the techniques used above, we use two types
of visualizations.

t-SNE plots: We first use t-SNE plots (Van der Maaten and Hinton, 2008) to
visualise the distribution of the non-masked and masked facial expression
features . As seen in Figure 5(a), the non-masked features produced by the
baseline are not well-separated. In particular, the features of the anger and
fear classes are mixed with other classes. On the other hand, the features
produced by CL and KD are more sparse. Coming to masked features, CL provides
the most discriminative facial expression features, with evident intra-class
similarity as seen in Figure 5(b). KD, however, does not give better-separated
features compared to the baseline in this case.

Grad-CAM: To test our hypothesis that CL and KD push the model to focus on
non-occluded regions of the face, we employ Grad-CAM(Selvaraju et al., 2017).
Figure 6 depicts the result of applying Grad-CAM to non-masked and masked
images. In the masked case, we see that when the model is trained on both
types of images routinely, it focuses on occluded areas at times. As seen in
Figure 6(a), a majority of images have a considerable portion of the mask
highlighted, indicating it was used for FER. In the case of CL and KD (refer
Figure 6(b) and (c)), however, we find that the model prioritizes the eye and
forehead region for decision making, and the focus on the mask is
comparitively less. It follows that CL and KD make better use of the limited
features available in the masked FER scenario, justifying our choice to use
them. In the non-masked scenario, we find that the models use relevant
features from the entire face; the mouth region seems to be an important
feature for detecting happy, surprise, disgust and neutral expressions,
establishing the premise for this paper: masks occlude vital facial features
for FER.

Figure 5. t-SNE plots of (a) Non-masked features and (b) Masked features for
different experiments. Best viewed in colour. Figure 6. GradCam Visualization
on (a) Baseline for MSD-PE, and using (b) Contrastive Learning and (c)
Knowledge Distillation. Non-masked images are on the left and masked images on
the right. Green and red boxes around images indicate whether the model got
the prediction right or wrong respectively.The heatmap is superimposed over
the image, where areas highlighted in red contributed the most toward
classification. Best viewed in colour. Figure 7. Confusion matrices of cross-
dataset study. (a)RAFDB (Li et al., 2017; Li and Deng, 2019) to non-masked
images in MSD-E. (b)Synthetically masked RAFDB to masked images of MSD-E.
Predicted labels are on the vertical axis and true labels are on the
horizontal axis.

###  3.5. Performance of state-of-the-art methods

In this section, we report the performance of occlusion-robust SOTA FER
methods, RAN (Wang et al., 2020) and SCAN (Gera and S, 2021), on MSD-E and
MSD-PE. Region attention network (RAN) selects image crops from the input
image independent of landmark points. A relation-attention module relates
local and global context and gives the expression representation for
classification. Our experiments with RAN (Wang et al., 2020) use ResNet-50 (He
et al., 2016) as the backbone with fixed position cropping of images,
resulting in six crops per image. Spatio-channel Attention Net Branch (SCAN)
(Gera and S, 2021) computes attention weight for each channel and spatial
location within the channel across all local patches to make the model robust
to occlusion. We use the publically available implementation of
SCAN222https://github.com/1980x/SCAN-CCI-FER for the experiments. We note here
that we used weights pretrained on AffectNet for the backbone of SCAN. As
before, we use 10-fold cross validation while evaluating these methods as
well.

MSD-E. Table 3 and Table 4 presents the FER accuracy of RAN (Wang et al.,
2020) and SCAN (Gera and S, 2021) on MSD-E along with the number of trainable
parameters for each method. SCAN achieves 3.44 % above the baseline on M. It
also surpasses the baseline on NM by 1.06%. RAN (Wang et al., 2020), however
does not surpass the baseline in either case.

MSD-PE. Table 5 presents the comparison on MSD-PE. Here too SCAN massively
outperforms the baseline and CL and KD as well. It is worth noting that SCAN
achieves 55% on masked images, i.e., 7.4% higher than CL but has 6 times more
parameters and is loaded with AffectNet pretrained weights. On the other hand,
RAN produces better results on non-masked images than CL and KD while falling
short on masked images.

###  3.6. Cross-Dataset Study

To explore the universality of the expressions in our dataset, we conducted a
RAFDB to MSD-E cross-dataset study. We used images of RAFDB for training and
those of MSD-E as testing. In particular, we trained ResNet-18 on RAFDB and
then tested it on the non-masked subset of MSD-E. We then, trained ResNet-18
on synthetically masked RAFDB images and tested them on the masked images in
MSD-E. Figure 7 depicts the confusion matrices of these two experiments.
Surprise, disgust, happiness, and neutral were easier to recognise in the non-
masked case, while fear was confused with surprise, as shown in Figure 7(a).
In the masked case, depicted in Figure 7(b), however, only surprise and
neutral were recognised easily and fear was confused with surprise again. This
indicates that training on synthetically masked images for FER is not
reliable, further emphasising the need for real-world masked FER data.

##  4\. Conclusion

As society emerges from the Covid-19 pandemic, we have grown accustomed to
wearing masks in public. FER systems, however, find expression recognition in
this new normal an arduous task. This challenge is increased by the variety of
masks people use and the high intra-class variation. In this paper, we have
proposed a novel FER dataset pertaining to the face mask scenario called MSD-E
with a subset consisting of pairs of images called MSD-PE. MSD-E contains
1,960 images and MSD-PE has 857 pairs of images. Each image is labeled for
different facial expressions. We used contrastive learning and knowledge
distillation to develop FER models that are mask-invariant and demonstrate how
they make the model focus on visible areas in the face mask scenario. Using
these training paradigms on larger networks can perhaps learn more
discriminative features for FER. Finally, we tested SOTA methods on the
dataset and performed a cross-dataset study. By releasing this dataset for
research, we hope that more researchers will be encouraged to address the hard
problem of drastic occlusion by masks in FER. Our aspiration is that the
dataset will be a useful benchmark resource to compare FER solutions in this
challenging condition.

## References

  * (1)
  * Barsoum et al. (2016) Emad Barsoum, Cha Zhang, Cristian Ferrer, and Zhengyou Zhang. 2016\.  Training Deep Networks for Facial Expression Recognition with Crowd-Sourced Label Distribution. 279–283.  https://doi.org/10.1145/2993148.2993165
  * Bo et al. (2020) Yang Bo, Jianming Wu, and Gen Hattori. 2020.  Facial Expression Recognition with the advent of Face Masks.  https://doi.org/10.1145/3428361.3432075
  * Ding et al. (2020) Hui Ding, Peng Zhou, and Rama Chellappa. 2020.  Occlusion-adaptive deep network for robust facial expression recognition. In _2020 IEEE International Joint Conference on Biometrics (IJCB)_. IEEE, 1–9.
  * Georgescu et al. (2022) Mariana-Iuliana Georgescu, Georgian-Emilian Duţǎ, and Radu Tudor Ionescu. 2022.  Teacher–student training and triplet loss to reduce the effect of drastic face occlusion.  _Machine Vision and Applications_ 33, 1 (2022), 1–19.
  * Gera and S (2021) Darshan Gera and Balasubramanian S. 2021.  Landmark Guidance Independent Spatio-Channel Attention and Complementary Context Information based Facial Expression Recognition.  _Pattern Recognition Letters_ 145 (02 2021).  https://doi.org/10.1016/j.patrec.2021.01.029
  * Guo et al. (2016) Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He, and Jianfeng Gao. 2016.  Ms-celeb-1m: A dataset and benchmark for large-scale face recognition. In _European conference on computer vision_. Springer, 87–102.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016\.  Deep Residual Learning for Image Recognition. 770–778.  https://doi.org/10.1109/CVPR.2016.90
  * Huber et al. (2021) Marco Huber, Fadi Boutros, Florian Kirchbuchner, and Naser Damer. 2021.  Mask-invariant face recognition through template-level knowledge distillation. In _FG 2021_. IEEE, 1–8.
  * Kotsia et al. (2008) Irene Kotsia, Ioan Buciu, and Ioannis Pitas. 2008.  An analysis of facial expression recognition under partial facial image occlusion.  _Image and Vision Computing_ 26, 7 (2008), 1052–1067.  https://doi.org/10.1016/j.imavis.2007.11.004
  * Li and Deng (2019) Shan Li and Weihong Deng. 2019.  Reliable Crowdsourcing and Deep Locality-Preserving Learning for Unconstrained Facial Expression Recognition.  _IEEE Transactions on Image Processing_ 28, 1 (2019), 356–370.
  * Li et al. (2017) Shan Li, Weihong Deng, and JunPing Du. 2017.  Reliable Crowdsourcing and Deep Locality-Preserving Learning for Expression Recognition in the Wild. In _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, 2584–2593.
  * Li et al. (2019) Yong Li, Jiabei Zeng, Shiguang Shan, and Xilin Chen. 2019\.  Occlusion Aware Facial Expression Recognition Using CNN With Attention Mechanism.  _IEEE Transactions on Image Processing_ 28, 5 (2019), 2439–2450.  https://doi.org/10.1109/TIP.2018.2886767
  * Lucey et al. (2010) Patrick Lucey, Jeffrey F. Cohn, Takeo Kanade, Jason Saragih, Zara Ambadar, and Iain Matthews. 2010\.  The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression. In _2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops_. 94–101.  https://doi.org/10.1109/CVPRW.2010.5543262
  * Mollahosseini et al. (2017) Ali Mollahosseini, Behzad Hasani, and Mohammad Mahoor. 2017\.  AffectNet: A New Database for Facial Expression, Valence, and Arousal Computation in the Wild.  _IEEE Transactions on Affective Computing_ (2017).
  * Neto et al. (2021) Pedro C Neto, Fadi Boutros, João Ribeiro Pinto, Naser Darner, Ana F Sequeira, and Jaime S Cardoso. 2021.  FocusFace: Multi-task contrastive learning for masked face recognition. In _2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021)_. IEEE, 01–08.
  * Saleem et al. (2021) Sharmeen Saleem, Subhi Zeebaree, and Maiwan Abdulrazzaq. 2021\.  Real-life Dynamic Facial Expression Recognition: A Review.  _Journal of Physics: Conference Series_ 1963 (07 2021), 012010\.  https://doi.org/10.1088/1742-6596/1963/1/012010
  * Selvaraju et al. (2017) Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2017.  Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. In _2017 IEEE International Conference on Computer Vision (ICCV)_. 618–626.  https://doi.org/10.1109/ICCV.2017.74
  * Van der Maaten and Hinton (2008) Laurens Van der Maaten and Geoffrey Hinton. 2008.  Visualizing data using t-SNE.  _Journal of machine learning research_ 9, 11 (2008).
  * Wang et al. (2020) K. Wang, Xiaojiang Peng, Jianfei Yang, Debin Meng, and Yu Qiao. 2020.  Region Attention Networks for Pose and Occlusion Robust Facial Expression Recognition.  _IEEE Transactions on Image Processing_ 29 (2020), 4057–4069.