# Exploring the Boundaries of Semi-Supervised Facial Expression Recognition:
Learning from In-Distribution, Out-of-Distribution, and Unconstrained Data

Shuvendu Roy, and Ali Etemad S. Roy and A. Etemad are with the Department of
ECE and Ingenuity Labs Research Institute, Queen’s University, Kingston,
Canada.
E-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Deep learning-based methods have been the key driving force behind much of the
recent success of facial expression recognition (FER) systems. However, the
need for large amounts of labelled data remains a challenge. Semi-supervised
learning offers a way to overcome this limitation, allowing models to learn
from a small amount of labelled data along with a large unlabelled dataset.
While semi-supervised learning has shown promise in FER, most current methods
from general computer vision literature have not been explored in the context
of FER. In this work, we present a comprehensive study on 11 of the most
recent semi-supervised methods, in the context of FER, namely Pi-model,
Pseudo-label, Mean Teacher, VAT, UDA, MixMatch, ReMixMatch, FlexMatch,
CoMatch, and CCSSL. Our investigation covers semi-supervised learning from in-
distribution, out-of-distribution, unconstrained, and very small unlabelled
data. Our evaluation includes five FER datasets plus one large face dataset
for unconstrained learning. Our results demonstrate that FixMatch consistently
achieves better performance on in-distribution unlabelled data, while
ReMixMatch stands out among all methods for out-of-distribution,
unconstrained, and scarce unlabelled data scenarios. Another significant
observation is that semi-supervised learning produces a reasonable improvement
over supervised learning, regardless of whether in-distribution, out-of-
distribution, or unconstrained data is utilized as the unlabelled set. We also
conduct sensitivity analyses on critical hyper-parameters for the two best
methods of each setting.

###### Index Terms:

Facial expression recognition, semi-supervised learning, in-distribution, out-
of-distribution, unconstrained.

##  1 Introduction

Facial Expression Recognition (FER) [1, 2] is a critical application of
computer vision that enables computers to identify and understand human
expressions, with applications ranging from health care [3, 4, 5] to
intelligent vehicles [6]. Deep learning methods have been the driving force
behind most of the recent successes in FER. However, one of the major barriers
to further improvement of deep learning-based FER is the need for large-scale
labelled data. To this end, Semi-supervised learning has shown immense promise
as a solution for improving performance while leveraging minimal supervision.

The basic concept of semi-supervised learning involves using large amounts of
unlabelled data to learn better representations from a small labelled set.
While a few studies have focused on semi-supervised learning in FER [7, 8],
many recent prominent methods originally proposed in general computer vision
literature have not been explored in this context. In our recent work [9], we
investigated and benchmarked the performance of the several popular semi-
supervised learning methods for FER. However, our previous study only focused
on semi-supervised FER with in-distribution (ID) unlabelled sets. Yet, real-
world scenarios often require dealing with unlabelled sets that are out-of-
distribution (OOD) and diverse.

Figure 1: Semi-supervised FER under ID, OOD, and unconstrained unlabelled data
scenarios.

In this study, we aim to extend our previous study [9] by exploring semi-
supervised FER under more realistic data settings. First, we study semi-
supervised learning with ‘OOD’ unlabelled data, where the images in the
unlabelled set contain images of the same classes as the labelled set but come
from a different source and, therefore, have a different data distribution.
Next, we study semi-supervised FER for ‘unconstrained’ unlabelled data, where
the unlabelled data come from a different distribution and do not contain
images of the same classes as those in the labelled set. This setting is
considerably harder than both ID and OOD learning. Figure 1 depicts an
overview of our study on semi-supervised learning from different unlabelled
set configurations. Finally, we study the effectiveness of semi-supervised
learning on very small datasets. We also expand the scope of our study by
including a few of the more recent semi-supervised methods. More specifically,
we study 11 recent semi-supervised approaches, namely Pi-model [10], Pseudo-
label [11], Mean Teacher [12], VAT [13], UDA [14], MixMatch [15], ReMixMatch
[16], FlexMatch [17], CoMatch [18], and CCSSL [19]. We conduct extensive
experiments for different unlabelled set configurations using five public FER
datasets, namely FER13 [20], RAF-DB [21], AffectNet [22], KDEF [23], and DDCF
[24], along with a non-FER dataset for the unconstrained data, namely CelebA
[25]. Findings from these experiments suggest that FixMatch performs the best
among the 11 semi-supervised methods for conventional ID semi-supervised
learning, but it performs poorly in other challenging settings. For both OOD
and unconstrained unlabelled data, ReMixMatch exhibits the best performance.
ReMixMatch also outperforms other methods in the low-data scenario. We also
show hyper-parameter sensitivity studies for each of these semi-supervised
settings to further boost performance.

Our contributions in this work can be summarized as follow:

  * •

We present a comprehensive and extensive study on 11 recent semi-supervised
methods for FER and their comparison to fully supervised learning using six
public datasets.

  * •

We compare the performance of all methods under various data scenarios where
the unlabelled data is ID, OOD, unconstrained, or very small.

  * •

Our study finds that FixMatch consistently exhibits the best performance for
ID unlabelled data, while ReMixMatch is the top performing approach for OOD,
unconstrained, and scarce unlabelled data. We also find that semi-supervised
learning improves performance over supervised learning in all the tested
scenarios.

Figure 2: Overview of semi-supervised learning for FER. Figure 3: Overview of
the semi-supervised learning methods explored in this study.

##  2 Related Work

In this section, we review the existing literature on semi-supervised learning
from two main perspectives that are relevant to our work: (a) with ID data,
(b) with OOD data, and (c) semi-supervised methods used specifically for FER.

###  2.1 ID Semi-supervised Learning

In recent years, there has been an increasing interest in the literature on
semi-supervised learning due to its potential for training large models with
small amounts of labelled data. An overview of the workflow of a semi-
supervised method is shown in Figure 2. Most of these methods have been
demonstrated to perform well on unlabelled data that come from the same
distribution as the labelled set. The existing literature on semi-supervised
learning can broadly be divided into two categories: entropy minimization [11,
26] and consistency regularization [12, 14, 13].

Entropy minimization-based methods learn by predicting the pseudo-labels of
unlabelled samples with low entropy. Pseudo-label [11] was one of the first
works in this direction, where the pseudo-labels for unlabelled samples are
predicted and added to the labelled set if their entropy is low. Noisy-Student
[26] generates these pseudo-labels with a pre-trained encoder, while Meta
Pseudo-label [27] uses a teacher network to update the pseudo-labels based on
the student network’s performance. In contrast, consistency regularization-
based methods aim to generate consistent predictions for different
perturbations of the same input. Pi-model [10] was one of the first works in
this direction, where two augmentations of an unlabelled image are forced to
generate the same class prediction. Virtual adversarial training (VAT) [13]
replaces the augmentation with adversarial perturbations and enforces
consistency on the predictions. Unsupervised domain adaptation (UDA) [14]
shows that replacing simple augmentations with hard augmentations [28, 29]
could bring significant improvement to semi-supervised methods. Most of the
semi-supervised methods since then use some form of hard augmentation in their
pipeline.

Another line of work combines these two prominent approaches into the same
framework. For example, MixMatch [15] enforces consistency on two
perturbations (generated with MixUp [30]) and optimizes for lower entropy in
the predictions. ReMixMatch [16] improves upon MixMatch by introducing two new
concepts: augmentation anchoring and distribution alignment. Another hybrid
method is FixMatch [31], which has gained tremendous success because of its
simplicity while achieving state-of-the-art results in various domains.
FixMatch learns by predicting the pseudo-labels for unlabelled images from its
weak augmentation and uses them as ground truth for the hard augmentation of
the same image if the confidence of the prediction is higher than a threshold.
Several improvements have been made to FixMatch since its introduction, such
as FlexMatch [17], which introduces the curriculum concept to adjust the
threshold for different classes dynamically, and CoMatch [18], which
introduces an extra contrastive loss term guided by the predicted pseudo-
labels.

###  2.2 OOD Semi-supervised Learning

Semi-supervised learning remains largely unexplored when it comes to
unlabelled datasets collected from a different source (OOD data) and without
constraints. Previous works have indicated that the presence of OOD data in
the unlabelled dataset can negatively impact the overall performance of semi-
supervised learning [32, 33]. Despite this, some recent studies have proposed
semi-supervised methods for OOD unlabelled data [34, 35]. The main idea behind
most of these methods is to incorporate an OOD detection module to identify
and remove those samples from the unlabelled set, effectively focusing on
learning the ID labelled data. For instance, [34] learns to recognize and
discard samples belonging to unknown classes. Similarly, in [35], a new
scoring method is proposed to detect and remove OOD samples. Class-aware
contrastive semi-Supervised learning (CCSSL) [19] uses class information along
with contrastive learning in this setting.

###  2.3 Semi-supervised FER

Besides our previous work, which focused on benchmarking the most commonly
used semi-supervised methods for FER [9], there are few other studies in this
area. For instance, [8] investigated the use of Deep Belief Networks for semi-
supervised FER and found that they produced a relative improvement over
supervised methods. In [36], a Bayesian network was proposed for semi-
supervised FER. More recently, [37] proposed an entropy-minimization method
for semi-supervised FER, which introduced an adaptive confidence margin
concept to partition the unlabelled data based on the confidence of pseudo-
labels. The method then trained on low and high-confidence predictions
separately. Furthermore, [37] explored the use of multi-modal data to learn
from audio-visual signals in a semi-supervised setting.

##  3 Method

In this section, we first discuss the problem setup for semi-supervised
learning. This is followed by an overview of 11 semi-supervised methods
explored in this study.

###  3.1 Problem Setup for Semi-supervised Learning

Let be given a small labelled set
$D_{l}=\\{(x_{i}^{l},y_{i}^{l})\\}^{N}_{i=1}$, where $N$ is the total number
of samples and their corresponding class labels, and a large unlabelled set
$D_{u}=\\{(x_{i}^{u})\\}^{M}_{i=1}$, where $M$ is the total number of
unlabelled samples and $M\gg N$. Accordingly, semi-supervised learning aims to
learn from labelled set $D_{l}$ in a supervised setting while utilizing the
unlabelled set $D_{u}$ in an unsupervised setting to learn a better
representation of the data. The performance is validated on a separate
validation set $D_{v}=\\{(x_{i},y_{i})\\}^{V}_{i=1}$. The is no overlap
between the sample in labelled, unlabelled, and validation sets, i.e.,
$D_{l}\cap D_{u}\cap D_{v}=\emptyset$.

###  3.2 Semi-Supervised Methods

####  3.2.1 Pi-Model

Pi-Model [10] is one of the earliest and most popular consistency
regularization-based semi-supervised methods. While learning from the labelled
samples in a supervised setting, Pi-model learns from the unlabelled set with
a consistency regularization term. More specifically, it applies two
augmentations on an unlabelled image and forces their predictions to be
similar. Pi-Model also uses dropout and random max-pooling to add stochastic
behaviour to the predictions. A schematic diagram of the Pi-Model is presented
in Figure 3 (a). The regularization loss of Pi-Model is represented as

$\mathbb{E}_{x\in
D_{l}}\mathcal{R}(f(\theta,\tau_{1}(x)),f(\theta,\tau_{2}(x))),$ (1)

where $\tau_{1}$ and $\tau_{2}$ are two sample transformations applied to an
unlabelled sample $x$, $f$ is the model, and $\theta$ represents the
parameters of $f$.

####  3.2.2 Mean Teacher

Mean Teacher [12] is another consistency regularization-based semi-supervised
method that is built upon Pi-model. However, Mean-teacher differs in the way
it generates embeddings of two augmented samples. Instead of utilizing the
online encoder (trainable encoder) to make the prediction for both images,
Mean-teacher uses an exponential moving average (EMA) encoder to generate a
prediction for the second image. This EMA of the student model is referred to
as the teacher model, while the online encoder is called the student model.
So, the Mean-teacher learns from unlabelled data by enforcing consistency
between the predictions of the teacher and student models on two augmentations
of the same sample through a regularization loss. A diagram demonstrating the
Mean teacher method is presented in Figure 3 (b). The loss function for Mean
teacher can be expressed as:

$\mathbb{E}_{x\in
D_{u}}\mathcal{R}(f(\theta,\tau_{1}(x)),f(EMA(\theta),\tau_{2}(x))),$ (2)

where $EMA(\theta)$ is the parameters of the teacher model. Finally, the
weight update operation of the teacher model with an exponential moving
average is formulated as:

$\theta^{\prime}=m\theta^{\prime}+(1-m)\theta,$ (3)

where $m$ is a momentum coefficient.

Figure 4: Sample images from three main FER datasets: FER13, RAF-DB, and
AffectNet.

####  3.2.3 UDA

UDA [14] is also based on the consistency regularization concept. The basic
idea of UDA is similar to Pi-model but shows a large improvement in
performance, only replacing the augmentation module. UDA replaces the usual
augmentation module with advanced augmentation methods like AutoAugment [38]
and RandAugment [28], resulting in dynamic and diverse sample augmentations.
Figure 3 (c) shows an overview of the UDA method. The UDA loss function can be
expressed as:

$\mathbb{E}_{x\in D_{u}}\mathcal{R}(f(\theta,x),f(\theta,\tau(x))),$ (4)

where $\tau$ represents hard augmentations.

####  3.2.4 VAT

VAT [13] is similar to the Pi-model and UDA in terms of its regularization
concept. However, instead of regularizing the embeddings of two augmented
versions of the same sample, VAT uses adversarial perturbation as a different
form of augmentation of the input sample. The overview of VAT is depicted in
Figure 3 (d). The loss function for VAT can be expressed as follows:

$\mathbb{E}_{x\in D_{u}}\mathcal{R}(f(\theta,x),f(\theta,\gamma^{adv}(x))),$
(5)

where $\gamma^{adv}$ is the adversarial perturbation operation.

####  3.2.5 Pseudo-label

The Pseudo-label method, introduced in [11], is an entropy minimization-based
method that presents a simple yet effective semi-supervised solution. Pseudo-
label involves predicting the class probabilities for each of the unlabelled
samples, which act as pseudo-labels for those images. If the confidence of the
prediction is high (low entropy), the pseudo-label of the unlabelled sample is
treated as a label to train alongside labelled data. This pseudo-labelling
concept has been used as a basis for several of the current state-of-the-art
methods. A visual illustration of the Pseudo-label method is depicted in
Figure 3 (e). The loss function of the Pseudo-label method is expressed as:

$\mathcal{L}=\mathcal{L}(y_{i}^{l},f(\theta,x_{i}^{l}))+\lambda\mathcal{L}(y_{i}^{u},f(\theta,x_{i}^{u})),$
(6)

where $y_{i}^{u}$ is the predicted pseudo-label for an unlabelled sample
$x_{i}^{u}$, and $\lambda$ is a coefficient to balance the weight of two loss
terms.

####  3.2.6 MixMatch

MixMatch [15] is a semi-supervised method of the hybrid category that combines
the concept of consistency regularization and entropy minimization. Similar to
the entropy-based methods, MixMatch aims to generate low entropy predictions
on the unlabelled data and also enforces consistency in its predictions
similar to consistency-based methods. The novel component of MixMatch is the
MixUp operation (an interpolation function) on labelled and unlabelled samples
to generate mixed samples. Both entropy minimization and consistency
regularization operations are applied to the mixed samples. The MixUp
operation can be represented as:

$x^{\prime}=\alpha x_{l}+(1-\alpha)x_{u},$ (7)

where $x_{l}$ and $x_{u}$ are input samples from the labelled and unlabelled
set, and $\alpha$ is the weight factor that balances the labelled and
unlabelled components in the generated sample. MixMatch uses a beta
distribution to sample the value for alpha randomly.

Another key innovation of MixMatch is the use of multiple augmentations on the
unlabelled set and averaging the predictions on the augmented samples to
generate the final prediction for that unlabelled sample. Let $d_{l}^{\prime}$
and $d_{u}^{\prime}$ be a batch of labelled and unlabelled samples after the
MixUp operation. The MixMatch loss can be represented as:

$\displaystyle\mathcal{L}_{l}$
$\displaystyle=\frac{1}{|d_{l}^{\prime}|}\sum_{x,y\in
d_{l}^{\prime}}H(y,f(x,\theta)),$ (8) $\displaystyle\mathcal{L}_{u}$
$\displaystyle=\frac{1}{C|d_{u}^{\prime}|}\sum_{x^{\prime},y^{\prime}\in
d_{u}^{\prime}}\|y\prime-f(x,\theta)\|_{2}^{2},$ (9)
$\displaystyle\mathcal{L}$
$\displaystyle=\mathcal{L}_{l}+\lambda\mathcal{L}_{u}.$ (10)

where $H(.)$ is the cross-entropy loss, $C$ is the total number of classes,
and $\lambda$ is the weight factor between the supervised and unsupervised
loss terms. Figure 3 (f) shows a visual illustration of the MixMatch method.

####  3.2.7 ReMixMatch

ReMixMatch [16] is a modified version of MixMatch that incorporates two new
concepts: distribution alignment and augmentation anchoring. The former aims
to ensure that the predictions made on the unlabelled data align with the
distribution of the predictions on the labelled data. Augmentation anchoring
replaces the consistency regularization of MixMatch and focuses on making the
representation of strongly augmented samples similar to those of weakly
augmented samples. This technique compares one weakly augmented sample against
multiple strongly augmented samples. ReMixMatch also introduces a new strong
augmentation method called CTAugment, which is more suitable in semi-
supervised learning settings. Figure 3 (g) provides a visual representation of
the ReMixMatch method.

####  3.2.8 FixMatch

FixMatch [31] is another hybrid semi-supervised learning method that shows
impressive performance in many applications. For an unlabelled sample,
FixMatch first applies weak augmentations and generates a prediction. FixMatch
then considers this as a pseudo-label for a hard augmentation of the same
sample if the confidence of this pseudo-label is beyond a threshold. Standard
shift and flip augmentations are utilized for the weak augmentation module of
FixMatch. FixMatch explores RandAugment [28] and CTAugment [16] as the hard
augmentation module. The unsupervised loss of FixMatch can be represented as:

$\mathcal{L}_{u}=\frac{1}{|d_{u}|}\sum_{x\in
d_{u}}\mathbf{1}(max(q)\geq\tau)\leavevmode\nobreak\
\mathcal{H}(\hat{q},f(\mathcal{A}(x),\theta))$ (11)

where, $\tau$ is the threshold, $q$ is the prediction on the weakly augmented
sample, and $\hat{q}=arg\leavevmode\nobreak\ max(q)$. A visual illustration of
FixMatch is depicted in Figure 3 (h).

####  3.2.9 FlexMatch

FlexMatch [17] proposes an improvement over FixMatch with a curriculum
learning concept for the threshold parameter. Rather than using a fixed $\tau$
for all classes, FlexMatch updates a class-specific threshold based on the
learning status of that class. FlexMatch uses per-class accuracy as an
indicator of the learning status of that class, which is calculated as:

$\alpha(c)=\sum_{n}\mathbf{1}(max(q)\geq\tau)\leavevmode\nobreak\
\mathbf{1}(argmax(q))=c),$ (12)

where $n$ is the number of samples. A schematic diagram of FlexMatch is shown
in Figure 3 (i).

####  3.2.10 CoMatch

CoMatch [18] introduces a graph contrastive learning concept over Fixmatch.
This approach basically applies a contrastive loss that is directed by the
predicted pseudo-labels. The Concept of CoMatch is depicted in Figure 3 (j).

####  3.2.11 CCSSL

Finally, CCSSL [19] is also built over FixMatch, specializing in OOD learning.
It learns from ID data like FixMatch, but for OOD data, it learns by instance
discrimination with contrastive loss. The CCSSL method is illustrated in
Figure 3 (k).

##  4 Experiments

In this section, we first provide details of the datasets used in this study.
Following this, we present the experimental setup employed for the semi-
supervised FER experiments. Finally, we present the main results for all the
semi-supervised settings.

(a) Random Crop

(b) Rotation

(c) Flip

(d) Color Jitter

(e) ShearX

(f) Gaussian Blur

(g) CutOut

(h) Multiple

Figure 5: Examples of different augmentations applied to a facial image.

###  4.1 Datasets

In this study, we utilize a total of six datasets to conduct the Facial
Expression Recognition (FER) experiments. For the main results, we use three
datasets: FER13 [20], RAF-DB [39], and AffectNet [22]. In addition, for the
experiments on unconstrained FER, we use the CelebA [25] dataset. Finally, for
the FER experiments on limited data, we use the KDEF [23] and DDCF [24]
datasets. Below, we provide a brief description of each dataset used in this
study. Some samples from these datasets are shown in Figure 4.

FER13 [20] is a widely used dataset for FER that contains over 28.7K images of
7 emotions (anger, disgust, fear, happiness, sadness, surprise, and neutral).
The images have been collected from the internet and resized to 48$\times$48
pixels.

RAF-DB [39] is a dataset for FER that contains around 15K images, with 12K
images for training and 3K used for testing. This dataset has been annotated
by 315 annotators, with each image annotated by around 40 annotators.

AffectNet [22] is a large-scale dataset for FER that contains around 284K
images of 8 emotions (anger, contempt, disgust, fear, happiness, sadness,
surprise, and neutral). This dataset has also been collected from the
internet, and the images are of relatively high resolution.

CelebA [25] is a large-scale dataset of faces. This dataset does not
necessarily contain images of expressive faces, and the images are also
collected from the internet and are of relatively high resolution.

KDEF [23] is a small dataset for FER that contains images of 7 emotions, which
have been captured in lab conditions. This dataset contains a total of
approximately 5K images. The images are of relatively high resolution and have
been captured from different angles.

DDCF [24] is another small dataset for FER that contains images of 8 emotions,
which have been captured in lab conditions. There are nearly 6.5K images in
this dataset. The images are of relatively high resolution and have been
collected from different angles.

###  4.2 Implementation details

For a fair comparison between all the methods, we use the same encoder and
training protocol. For the encoder, we use ResNet-50 [40]. All the experiments
are reported for different number of labelled samples. For this, we randomly
sample $N=n\times C$ images and their corresponding labels from the labelled
set $D_{l}$, where $C$ is the number of classes, and $n$ is the number of
samples per class. Specifically, we present all results for
$n\in\\{10,25,100,250\\}$.

We follow the implementation details of the original methods for method-
specific parameters. For instance, we use a confidence cut-off value of 0.95
for FixMatch, a moving average weight of 0.999 for Mean-teacher, and a
temperature value of 0.5 for methods that utilize sharpening distribution. For
training, we use $2^{20}$ iterations with a batch size of 64 and SGD optimizer
with a learning rate of 0.03, a momentum of 0.9, and a cosine learning rate
decay scheduler.

In many recent semi-supervised methods, data augmentations, particularly hard
augmentations, are proven to be an essential component. Among the various hard
augmentation modules mentioned in the literature, RandAugment [28] is the most
commonly used one. This technique involves defining a set of augmentations
that can be applied to an image, such as random crops, flips, and colour
changes. Then, a random subset of these augmentations is selected and applied
to the image. We provide some examples of the augmentations used in
RandAugment in Figure 5.

Table I: The performance of different semi-supervised methods with ID unlabelled data on FER13, RAF-DB, and AffectNet, when 10, 25, 100, and 250 labelled samples per class are used for training. | FER13 | RAF-DB | AffectNet | Avg. Acc
---|---|---|---|---
Method / $m$ | 10 labels | 25 labels | 100 labels | 250 labels | 10 labels | 25 labels | 100 labels | 250 labels | 10 labels | 25 labels | 100 labels | 250 labels |
$\Pi$-model [10] | 37.09 | 40.87 | 50.66 | 56.42 | 39.86 | 50.97 | 63.98 | 71.15 | 24.17 | 25.37 | 31.24 | 32.40 | 43.68
Pseudo-label [11] | 32.79 | 36.04 | 49.21 | 54.88 | 58.31 | 39.11 | 54.07 | 67.40 | 18.00 | 21.05 | 33.05 | 37.37 | 41.77
Mean Teacher [12] | 45.21 | 55.14 | 52.17 | 58.06 | 62.05 | 45.17 | 45.57 | 76.85 | 19.54 | 20.21 | 20.80 | 44.05 | 45.40
VAT [13] | 24.95 | 55.22 | 51.55 | 55.64 | 63.10 | 45.82 | 62.05 | 59.45 | 17.68 | 35.02 | 37.68 | 37.92 | 45.50
UDA [14] | 46.72 | 49.89 | 50.62 | 60.68 | 46.87 | 53.15 | 58.86 | 60.82 | 27.42 | 32.16 | 37.25 | 37.64 | 46.84
MixMatch [15] | 45.69 | 46.41 | 55.73 | 58.27 | 36.34 | 43.12 | 64.14 | 73.66 | 30.80 | 32.40 | 39.77 | 48.31 | 47.88
ReMixMatch [16] | 41.07 | 43.25 | 44.62 | 57.49 | 37.35 | 42.56 | 42.86 | 61.70 | 29.28 | 33.54 | 41.60 | 46.51 | 43.48
FixMatch [31] | 47.88 | 49.90 | 59.46 | 62.20 | 63.25 | 52.44 | 64.34 | 75.51 | 30.08 | 38.31 | 46.37 | 51.25 | 53.41
FlexMatch [17] | 39.77 | 42.88 | 51.14 | 56.06 | 40.51 | 42.67 | 50.75 | 61.70 | 17.20 | 19.80 | 22.34 | 29.83 | 39.55
CoMatch [18] | 40.24 | 49.04 | 54.97 | 59.47 | 40.04 | 52.59 | 68.05 | 73.46 | 21.23 | 23.54 | 27.45 | 30.31 | 45.03
CCSSL [19] | 40.23 | 45.36 | 57.01 | 61.77 | 50.59 | 51.30 | 63.79 | 74.93 | 16.89 | 21.34 | 24.46 | 28.94 | 44.71

(a) Accuracy vs. $p_{\text{cutoff}}$ for FixMatch.

(b) Accuracy vs. $\lambda$ for FixMatch.

(c) Accuracy vs. $\alpha$ for MixMatch.

(d) Accuracy vs. $\lambda$ for MixMatch.

Figure 6: Sensitivity study of various parameters for two of the best semi-
supervised methods on ID unlabelled data.

###  4.3 Semi-supervised FER with ID unlabelled data

####  4.3.1 Setup

This section presents the main result of 11 semi-supervised learning
approaches on the FER13, RAF-DB, and AffectNet datasets for ID FER. As
previously mentioned, the results are presented for 10, 25, 100, and 250
labelled samples for each emotion class. The remaining samples from each
dataset are treated as unlabelled sets.

####  4.3.2 Performance

The main results for ID semi-supervised learning are presented in Table I,
which also includes the average accuracy across all settings (4 data splits of
3 datasets) for an overall understanding of the performance of each method. In
summary, FixMatch appears to be the most successful semi-supervised method for
ID data, as it outperforms other methods on 7 out of 12 settings and achieves
the second-best result on other two settings. FixMatch achieves an average
accuracy of 53.41% across all settings, while the second-best method,
MixMatch, achieves an average accuracy of 47.88%. The second-best method is
around 6% behind FixMatch and only obtains the best results on one setting.
Therefore, we can conclude that FixMatch is the most robust semi-supervised
method for FER with ID unlabelled data.

####  4.3.3 Sensitivity study

All the experiments shown in the table above were conducted using default
parameters as reported in the original papers for each algorithm. In this
subsection, we present a sensitivity study on the key hyper-parameters of the
two best-performing methods in order to improve FER performance further.
Figure 6 displays this study on the $P_{\text{cutoff}}$ and $\lambda$ values
for FixMatch, as well as the $\alpha$ and $\lambda$ values for MixMatch. In
the FixMatch, the $P_{\text{cutoff}}$ value determines the confidence
threshold at which a predicted pseudo-label is considered as the label for an
unlabelled image. The results shown in Figure 6(a) indicate that the best
accuracy is achieved for a $P_{\text{cutoff}}$ value of 0.95 for all datasets,
which is consistent with the original FixMatch method. The $\lambda$ value
balances the weight of supervised and unsupervised loss in FixMatch. We
conduct an experiment on $\lambda$ (Figure 6(b)) and find that the optimal
value varies for different datasets. While AffectNet and RAF-DB show better
results for relatively smaller values of $\lambda$ (1.0 and 0.5,
respectively), FER13 achieves the best performance with higher values of
$\lambda$, specifically 5.0. In the MixMatch method, $\alpha$ is the mixing
coefficient used in the MixUp operation. The experiment on $\alpha$ (Figure
6(c)) indicates that higher values of $\alpha$ generally produce better
results for all datasets, with the best performance obtained with
$\alpha=0.9$. Finally, in the experiment on the $\lambda$ value of MixMatch
(Figure 6(d)), we observe improvements for larger values of $\lambda$, with
the best accuracy achieved when $\lambda$ is set to 100 for all three
datasets.

Table II: Comparison between fully-supervised learning and semi-supervised learning with ID unlabelled data. | Fully-sup. | Semi-sup. (best)
---|---|---
Dataset | All data | 250/class | 250/class
FER13 | 64.57 | 53.58 |  62.41 (FixMatch)
RAF-DB | 80.47 | 65.87 |  76.85 (MeanTeacher)
AffectNet | 54.91 | 40.28 |  51.25 (FixMatch)
Table III: The performance of different semi-supervised methods with OOD unlabelled data on FER13 and RAF-DB, when 10, 25, 100, and 250 labelled samples per class are used for training. | FER13 | RAF-DB | Avg. Acc.
---|---|---|---
Method / $m$ | 10 labels | 25 labels | 100 labels | 250 labels | 10 labels | 25 labels | 100 labels | 250 labels |
Pi-Model [10] | 24.71 | 29.02 | 43.63 | 53.58 | 38.62 | 39.73 | 46.81 | 66.53 | 42.83
Mean Teacher [12] | 25.16 | 27.61 | 47.9 | 54.82 | 38.62 | 37.65 | 52.09 | 65.45 | 43.66
VAT [13] | 23.95 | 28.49 | 43.37 | 52.98 | 38.62 | 38.85 | 49.58 | 61.6 | 42.18
Pseudo-label [11] | 24.09 | 36.04 | 47.2 | 53.23 | 38.62 | 38.62 | 49.84 | 63.17 | 43.85
UDA [14] | 27.6 | 38.2 | 52.16 | 56.13 | 39.37 | 41.79 | 60.95 | 65.71 | 47.74
MixMatch [15] | 31.64 | 37.66 | 49.68 | 56.39 | 43.02 | 51.14 | 63.04 | 70.37 | 50.37
ReMixMatch [16] | 33.31 | 44.15 | 53.15 | 58.07 | 48.37 | 58.47 | 67.47 | 74.87 | 54.73
FixMatch [31] | 29.6 | 38.98 | 52.44 | 57.4 | 23.99 | 45.86 | 61.15 | 64.8 | 46.78
FlexMatch [17] | 31.89 | 41.0 | 50.43 | 56.17 | 46.87 | 53.16 | 69.59 | 72.59 | 52.71
CCSSL [19] | 30.05 | 39.09 | 53.32 | 57.77 | 47.82 | 56.52 | 69.33 | 69.07 | 52.87

(a) Accuracy vs. ${\alpha}$ for ReMixMatch.

(b) Accuracy vs. $\lambda$ for ReMixMatch.

(c) Accuracy vs. $P_{\text{cutoff}}$ for CCSSL.

(d) Accuracy vs. $\lambda$ for CCSSL.

Figure 7: Sensitivity study of various parameters for two of the best semi-supervised methods on OOD unlabelled data. Table IV: Comparison between fully-supervised learning and semi-supervised learning with OOD unlabelled data. | Fully-sup. | Semi-sup.
---|---|---
Dataset | All data | 250/class | ID unlab. | OOD unlab.
FER13 | 64.57 | 53.58 | 62.20 | 58.48
RAF-DB | 80.47 | 65.87 | 76.85 | 74.87

####  4.3.4 Discussion

Table II provides a summary of the best results obtained for each dataset
using 250 labels per class and compares the performances with fully supervised
training with an equal amount of labelled samples. Additionally, the table
shows the results obtained with fully supervised learning using all the data
including their labels from the original dataset. For FER13, the semi-
supervised method achieves an 8.83% improvement over fully supervised training
with the same amount of labelled data (250 labels per class, for a total of
1750 out of 28K images) and is only 2.16% lower than fully supervised training
with all labelled data (28K images). Similarly, for RAF-DB, the semi-
supervised method obtains a 10.98% improvement over fully supervised training
with an equal amount of labelled samples (250 labels per class, for a total of
1750 out of 12k images) and is only 3.62% lower than fully supervised training
with all samples being labelled (12K images). Finally, for AffectNet, the
semi-supervised method achieves a 10.98% improvement over the fully supervised
training with the same amount of labelled data (250 labels per class, for a
total of, 1750 out of 284k images) and is only 3.66% lower than the fully
supervised training with all labelled data. Based on this summary, we can
conclude that semi-supervised methods are able to achieve a significant
improvement over fully supervised training with the same amount of labelled
data, and can achieve comparable performance to fully supervised training on
large amounts of labelled samples.

Table V: The performance of different semi-supervised methods with unconstrained unlabelled data on FER13, RAF-DB, and AffectNet, when 10, 25, 100, and 250 labelled samples per class are used for training | FER13 | RAF-DB | AffectNet | Avg. Acc
---|---|---|---|---
Method / $m$ | 10 labels | 25 labels | 100 labels | 250 labels | 10 labels | 25 labels | 100 labels | 250 labels | 10 labels | 25 labels | 100 labels | 250 labels |
Pi-Model [10] | 22.28 | 27.03 | 44.96 | 54.79 | 38.62 | 38.62 | 52.18 | 65.74 | 18.00 | 18.74 | 19.66 | 35.57 | 36.35
Mean Teacher [12] | 24.88 | 29.77 | 43.66 | 53.22 | 36.18 | 38.62 | 50.59 | 66.49 | 17.69 | 19.11 | 21.34 | 38.34 | 36.66
VAT [13] | 21.94 | 30.05 | 43.56 | 52.42 | 38.62 | 41.17 | 51.76 | 62.32 | 17.86 | 19.23 | 20.89 | 26.34 | 35.51
Pseudo-label [11] | 23.36 | 33.51 | 47.70 | 53.57 | 38.62 | 37.45 | 49.54 | 63.40 | 17.29 | 19.54 | 21.06 | 24.03 | 35.76
UDA [14] | 24.88 | 33.99 | 49.40 | 52.97 | 27.05 | 43.94 | 52.41 | 62.87 | 16.91 | 17.60 | 28.31 | 33.14 | 36.96
MixMatch [15] | 31.43 | 39.44 | 51.99 | 56.26 | 42.37 | 50.52 | 60.01 | 68.84 | 25.14 | 26.26 | 33.40 | 39.23 | 43.74
ReMixMatch [16] | 32.49 | 42.05 | 52.83 | 58.04 | 49.15 | 54.60 | 64.50 | 70.70 | 26.29 | 29.23 | 37.31 | 40.40 | 46.47
FixMatch [31] | 26.58 | 35.16 | 49.29 | 54.30 | 33.41 | 39.99 | 52.93 | 60.59 | 15.77 | 17.23 | 28.46 | 31.23 | 37.08
FlexMatch [17] | 25.42 | 37.21 | 50.21 | 54.83 | 36.86 | 49.45 | 59.62 | 64.05 | 24.40 | 28.06 | 32.26 | 36.74 | 41.59
CoMatch [18] | 33.01 | 40.90 | 50.67 | 55.83 | 27.95 | 32.53 | 64.11 | 68.17 | 23.40 | 27.50 | 32.33 | 35.79 | 41.02
CCSSL [19] | 24.69 | 39.51 | 51.14 | 55.63 | 31.32 | 47.26 | 55.25 | 61.90 | 16.51 | 25.11 | 25.00 | 33.71 | 38.92

(a) Accuracy vs. ${\alpha}$ for ReMixMatch.

(b) Accuracy vs. $\lambda$ for ReMixMatch.

(c) Accuracy vs. $\alpha$ for MixMatch.

(d) Accuracy vs. $\lambda$ for MixMatch.

Figure 8: Sensitivity study of various parameters for two of the best semi-supervised methods on unconstrained unlabelled data. Table VI: Comparison between fully-supervised learning and semi-supervised learning with unconstrained unlabelled data. | Fully-sup. | Semi-sup.
---|---|---
Dataset | All data | 250/class | ID unlab. | Uncon. unlab.
FER13 | 64.57 | 53.58 | 62.20 | 58.04
RAF-DB | 80.47 | 65.87 | 76.85 | 70.70
AffectNet | 54.91 | 40.28 | 51.25 | 40.40

###  4.4 Semi-supervised FER with OOD unlabelled data

####  4.4.1 Setup

In this section, we present the results of using OOD unlabelled data for semi-
supervised learning. In this setting, the unlabelled data consists of images
belonging to the same expression categories but originating from different
sources and therefore having a different distribution. Specifically, we use
AffectNet as the unlabelled dataset, which is the largest of the three
datasets, and FER13 and RAF-DB as the labelled datasets. Similar to the
previous experiments, we report the results for 10, 25, 100, and 250 labelled
samples per category as the labelled set and all available images from
AffectNet as the unlabelled set.

####  4.4.2 Performance

The performance of different semi-supervised methods with OOD unlabelled data
are presented in Table III. We can draw two key observations from this table.
Firstly, we can observe a significant drop in the performance of all methods
compared to when ID unlabelled data are used. These findings are consistent
with previous works in the OOD semi-supervised literature [32, 33]. For
instance, in our study, we find that FixMatch achieves a 62.20% accuracy on
FER-13 (250 labels) with ID unlabelled data, but this drops to 57.4% when OOD
data are utilized. The performance drop is even more substantial with smaller
labelled set sizes. For example, when only 10 samples per class are available
on FER-13, the FixMatch performance drops from 47.88% to only 29.6%. The
second observation is that the best-performing method is different for OOD
semi-supervised learning. While FixMatch and MixMatch were the top two methods
for ID semi-supervised learning, ReMixMatch, and CCSSL are the top two methods
for OOD semi-supervised learning. Both of these methods perform significantly
better than FixMatch.

Table VII: The performance of different semi-supervised methods with small unlabelled data on KDEF and DDCF, when 10, 25, 100, and 250 labelled samples per class are used for training | KDEF | DDCF | Avg. Acc
---|---|---|---
Method / $m$ | 10 labels | 25 labels | 100 labels | 250 labels | 10 labels | 25 labels | 100 labels | 250 labels |
Pi-Model [10] | 31.9 | 58.9 | 85.28 | 94.27 | 19.81 | 30.82 | 77.2 | 88.99 | 60.9
Mean Teacher [12] | 29.24 | 50.31 | 82.82 | 91.62 | 18.4 | 30.03 | 74.37 | 89.94 | 58.34
VAT [13] | 24.54 | 44.79 | 79.75 | 91.21 | 23.58 | 40.57 | 72.01 | 88.99 | 58.18
Pseudo-label [11] | 31.08 | 53.37 | 80.98 | 93.25 | 29.4 | 41.19 | 77.83 | 86.79 | 61.74
UDA [14] | 36.81 | 46.42 | 88.14 | 97.55 | 30.35 | 80.82 | 86.79 | 93.71 | 70.07
MixMatch [15] | 28.83 | 63.39 | 91.21 | 93.87 | 14.78 | 84.28 | 89.15 | 93.4 | 69.86
ReMixMatch [16] | 29.86 | 67.08 | 93.46 | 97.75 | 25.79 | 85.53 | 89.62 | 95.6 | 73.09
FixMatch [31] | 33.54 | 54.6 | 89.16 | 97.55 | 38.52 | 79.4 | 86.64 | 94.65 | 71.76
FlexMatch [17] | 27.61 | 40.08 | 93.05 | 96.32 | 39.62 | 24.37 | 86.01 | 93.71 | 62.6
CoMatch [18] | 21.08 | 52.12 | 85.05 | 92.16 | 44.94 | 63.52 | 87.18 | 90.7 | 67.09
CCSSL [19] | 19.43 | 37.83 | 86.5 | 94.27 | 14.15 | 55.5 | 86.01 | 94.65 | 61.04

(a) Accuracy vs. ${\alpha}$ for ReMixMatch.

(b) Accuracy vs. $\lambda$ for ReMixMatch.

(c) Accuracy vs. $P_{\text{cutoff}}$ for FixMatch.

(d) Accuracy vs. $\lambda$ for FixMatch.

Figure 9: Sensitivity study of various parameters for two of the best semi-
supervised methods on small unlabelled data..

####  4.4.3 Sensitivity study

Next, we conduct a sensitivity analysis on the two best-performing methods,
ReMixMatch and CCSSL, for OOD semi-supervised learning. Since these methods
were originally designed for ID semi-supervised learning in general computer
vision applications, it is crucial to explore their performance for different
hyperparameters under the OOD setting. Figure 7 presents the results of this
study. We examine the impact of $\alpha$ and $\lambda$ for ReMixMatch and
$P_{\text{cutoff}}$ and $\lambda$ for CCSSL. In the experiment with
ReMixMatch’s $\alpha$ values (Figure 7(a)), we discover that the best results
for both datasets are achieved with a value of $0.75$, which is also the
default in the original ReMixMatch paper. We also find that the optimal value
for $\lambda$ was $1.0$ for both datasets. In the experiments on CCSSL’s
$P_{\text{cutoff}}$ value (Figure 7(c)), we observe that lower values of this
parameter yields better results. Specifically, RAF-DB produces the best result
with a value of 0.5, while FER13 achieves the best result with a value of
0.75. Finally, for $\lambda$ values (Fig 7(d)), we obtain the best accuracy
for different values for RAF-DB and FER13 datasets, with 1.0 and 0.25
respectively.

####  4.4.4 Discussion

Table IV summarizes the results of semi-supervised FER using OOD unlabelled
data and compares them to both fully-supervised and semi-supervised methods
with ID unlabelled data. All results are shown for 250 labelled samples per
class. We observe a drop in performance of 1.98% and 3.72% for RAF-DB and
FER13 datasets, respectively, compared to ID semi-supervised learning.
However, the performance of semi-supervised learning with OOD unlabelled
samples is still better than fully-supervised learning by 9.0% and 4.9% for
the two datasets, respectively. Therefore, we can conclude that learning with
the presence of OOD data is still a better choice than relying on a fully
supervised setting alone when presented with limited labelled data.

###  4.5 Semi-supervised FER with unconstrained unlabelled data

####  4.5.1 Setup

In this section, we present the results of semi-supervised FER using
unconstrained unlabelled data, which is considered the most challenging
setting for semi-supervised learning. Here, the unlabelled data are obtained
from a different source than the labelled data and contain images of
expression classes that could be different from the labelled set classes. To
conduct this experiment, we use CelebA as the unlabelled set and FER13, RAF-
DB, and AffectNet as the labelled sets. The training is performed using 10,
25, 100, and 250 labelled samples per class, and all available images from
CelebA are used as the unlabelled set, similar to our previous experiments.

####  4.5.2 Performance

Table V shows the results of different semi-supervised methods with
unconstrained unlabelled data. We make the following two observations from
this table. Firstly, The best-performing method on unconstrained unlabelled
data is similar to that of the OOD setting. Again, ReMixMatch achieves the
best average results compared to the other methods. However, the second-best
method is different from the OOD setting (CCSSL), and is now MixMatch. The
performance of the rest of the methods is significantly lower than these two
methods. Secondly, the performance of semi-supervised methods is much lower
than training with ID data but not too far away from OOD data. For example,
the performance of ReMixMatch on FER-13 with just 10 labelled samples on
unconstrained unlabelled data is 8.58% lower than training with ID labelled
data, but only 0.82% lower than on OOD data. This indicates that for FER,
unconstrained unlabelled data can result in competitive semi-supervised
performance to OOD samples. This is an important finding since it is more
convenient, practical, and economical to collect large unconstrained
unlabelled data than ID data or even OOD data of expressive faces.

####  4.5.3 Sensitivity Study

Next, we present a sensitivity study on the main hyper-parameters of the two
best methods for unconstrained semi-supervised learning (ReMixMatch and
MixMatch). The results are presented in Figure 8.

The first experiment on ReMixMatch’s $\alpha$ values shows that different
datasets have different optimal values. Specifically, the optimal values for
RAF-DB, FER13, and AffectNet are 0.5, 0.75, and 0.1, respectively (Figure
8(a)). Similarly, optimal values for $\lambda$ experiments are different for
each dataset (Figure 8(b)). In this case, the optimal values for RAF-DB,
FER13, and AffectNet are 2.0, 1.0, and 2.0, respectively. MixMatch experiments
follow a similar pattern of having different optimal values. For $\alpha$, the
best values are 0.5, 0.75, and 0.1 (Figure 8(c)), and for $\lambda$ (Figure
8(d)), the optimal values are 0.5, 2.0, and 0.5 for RAF-DB, FER13, and
AffectNet, respectively. In conclusion, hyper-parameters are generally
specific to each dataset when learning from unconstrained unlabelled data.

####  4.5.4 Discussion

Table VI summarizes the results of learning from unconstrained unlabelled
data. The key takeaway from this table is that although semi-supervised
learning with unconstrained data achieves lower performance compared to ID
semi-supervised learning, it still outperforms fully supervised learning when
an equal number of labelled samples are used. For instance, FER13 achieves an
accuracy of 53.58% with fully supervised learning, while the same dataset
reaches 58.04% accuracy through the utilization of semi-supervised learning
with unconstrained unlabelled data.

###  4.6 Semi-supervised FER on small datasets

####  4.6.1 Setup

In addition to the experiments on large in-the-wild datasets, we also conduct
experiments on small datasets collected in lab environments for completeness.
The KDEF [23] and DDCF [24] datasets are used for this purpose, and the
results are presented here for a similar number of labelled samples as in
previous experiments, with the remaining samples as the unlabelled set, which
is very small in size.

####  4.6.2 Performance

Table VII shows the main results of different semi-supervised methods in this
setting. ReMixMatch again performs the best across all methods. With ID data
in this setting, FixMatch again performs well and shows the second-best
average accuracy. The average accuracy for ReMixMatch and FixMatch are 73.09%
and 71.76%, respectively.

####  4.6.3 Sensitivity study

We also analyze the sensitivity of the two best-performing methods, ReMixMatch
and FixMatch, in this setting. Figure 9 illustrates the results of this study.
The experiment on the $\alpha$ parameter of ReMixMatch (Figure 9(a)) shows
that higher values lead to better accuracy. The optimal values are found to be
0.9 and 0.75 for DDCF and KDEF datasets, respectively. Similarly, for the
$\lambda$ parameter (Figure 9(b)), the best results are obtained for 0.9 and
2.0 for DDCF and KDEF datasets. On the other hand, the experiments on FixMatch
reveal that the default parameters of the original FixMatch yield the best
accuracy for both datasets. Specifically, a $P_{\text{cutoff}}$ value of 0.95
(Figure 9(c)) and a $\lambda$ value of 1.0 (Figure 9(d)) achieve the best
performances.

##  5 Conclusion

This research offers a comprehensive analysis of 11 semi-supervised methods
for FER. The study evaluates the performance of these methods in various
unlabelled data scenarios, including ID, OOD, unconstrained, and very small
unlabelled sets. Our primary finding is that FixMatch is the most effective
semi-supervised method for learning from ID unlabelled data. However, for all
other real-world scenarios (OOD, unconstrained, and small set), ReMixMatch
consistently outperforms other semi-supervised methods. Another noteworthy
finding is that semi-supervised learning from any data scenario produces
better results in comparison to fully-supervised learning from the same number
of labelled samples. When learning from an ID unlabelled set, semi-supervised
methods can produce a performance improvement of up to 11% over the fully-
supervised method. Although compared to ID, performance is generally lower for
both OOD and unconstrained unlabelled data, these methods still outperforms
fully-supervised learning. Interestingly, the unconstrained setting, despite
being significantly more challenging than ID or OOD, underperforms OOD by only
a small margin. This is a significant observation since collecting a large
amounts of unconstrained unlabelled data is considerably easier and more
practical than collecting ID or even OOD (but constrained) data. Overall, we
anticipate that this research will serve as a useful guide for further
investigation into semi-supervised learning in the context of FER, as well as
other domains.

## Acknowledgements

We would like to thank BMO Bank of Montreal and Mitacs for funding this
research. We are also thankful to SciNet HPC Consortium for helping with the
computation resources.

## References

  * [1] M. Kolahdouzi, A. Sepas-Moghaddam, and A. Etemad, “Facetoponet: Facial expression recognition using face topology learning,” _IEEE Transactions on Artificial Intelligence_ , 2022.
  * [2] ——, “Face trees for expression recognition,” in _IEEE International Conference on Automatic Face and Gesture Recognition_ , 2021, pp. 1–5.
  * [3] S. Tokuno, G. Tsumatori, S. Shono, E. Takei, T. Yamamoto, G. Suzuki, S. Mituyoshi, and M. Shimura, “Usage of emotion recognition in military health care,” in _Defense Science Research Conference and Expo_ , 2011, pp. 1–5.
  * [4] M. Thrasher, M. D. Van der Zwaag, N. Bianchi-Berthouze, and J. H. Westerink, “Mood recognition based on upper body posture and movement features,” in _International Conference on Affective Computing and Intelligent Interaction_ , 2011, pp. 377–386.
  * [5] D. Sanchez-Cortes, J.-I. Biel, S. Kumano, J. Yamato, K. Otsuka, and D. Gatica-Perez, “Inferring mood in ubiquitous conversational video,” in _12th International Conference on Mobile and Ubiquitous Multimedia_ , 2013, pp. 1–9.
  * [6] H. Leng, Y. Lin, and L. Zanzi, “An experimental study on physiological parameters toward driver emotion recognition,” in _International Conference on Ergonomics and Health Aspects of Work with Computers_ , 2007, pp. 237–246.
  * [7] C. Florea, M. Badea, L. Florea, A. Racoviteanu, and C. Vertan, “Margin-mix: Semi-supervised learning for face expression recognition,” in _European Conference on Computer Vision_ , 2020, pp. 1–17.
  * [8] A. R. Kurup, M. Ajith, and M. M. Ramón, “Semi-supervised facial expression recognition using reduced spatial features and deep belief networks,” _Neurocomputing_ , vol. 367, pp. 188–197, 2019.
  * [9] S. Roy and A. Etemad, “Analysis of semi-supervised methods for facial expression recognition,” in _IEEE International Conference on Affective Computing and Intelligent Interaction_ , 2022, pp. 1–8.
  * [10] M. Sajjadi, M. Javanmardi, and T. Tasdizen, “Regularization with stochastic transformations and perturbations for deep semi-supervised learning,” _Advances in Neural Information Processing Systems_ , vol. 29, 2016.
  * [11] D.-H. Lee _et al._ , “Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks,” in _Workshop on Challenges in Representation Learning, ICML_ , vol. 3, no. 2, 2013, p. 896.
  * [12] A. Tarvainen and H. Valpola, “Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results,” _Advances in Neural Information Processing Systems_ , vol. 30, 2017\.
  * [13] T. Miyato, S.-i. Maeda, M. Koyama, and S. Ishii, “Virtual adversarial training: a regularization method for supervised and semi-supervised learning,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 41, no. 8, pp. 1979–1993, 2018.
  * [14] Q. Xie, Z. Dai, E. Hovy, T. Luong, and Q. Le, “Unsupervised data augmentation for consistency training,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 6256–6268, 2020.
  * [15] D. Berthelot, N. Carlini, I. Goodfellow, N. Papernot, A. Oliver, and C. A. Raffel, “Mixmatch: A holistic approach to semi-supervised learning,” _Advances in Neural Information Processing Systems_ , vol. 32, 2019.
  * [16] D. Berthelot, N. Carlini, E. D. Cubuk, A. Kurakin, K. Sohn, H. Zhang, and C. Raffel, “Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring,” _arXiv preprint arXiv:1911.09785_ , 2019.
  * [17] B. Zhang, Y. Wang, W. Hou, H. Wu, J. Wang, M. Okumura, and T. Shinozaki, “Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 18 408–18 419, 2021.
  * [18] J. Li, C. Xiong, and S. C. Hoi, “Comatch: Semi-supervised learning with contrastive graph regularization,” in _IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 9475–9484.
  * [19] F. Yang, K. Wu, S. Zhang, G. Jiang, Y. Liu, F. Zheng, W. Zhang, C. Wang, and L. Zeng, “Class-aware contrastive semi-supervised learning,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 14 421–14 430.
  * [20] I. J. Goodfellow, D. Erhan, P. L. Carrier, A. Courville, M. Mirza, B. Hamner, W. Cukierski, Y. Tang, D. Thaler, D.-H. Lee _et al._ , “Challenges in representation learning: A report on three machine learning contests,” in _International Conference on Neural Information Processing_ , 2013, pp. 117–124.
  * [21] S. Li, W. Deng, and J. Du, “Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 2852–2861.
  * [22] A. Mollahosseini, B. Hasani, and M. H. Mahoor, “Affectnet: A database for facial expression, valence, and arousal computing in the wild,” _IEEE Transactions on Affective Computing_ , vol. 10, no. 1, pp. 18–31, 2017.
  * [23] D. Lundqvist, A. Flykt, and A. Öhman, “The karolinska directed emotional faces (kdef),” _CD ROM from Department of Clinical Neuroscience, Psychology Section, Karolinska Institutet_ , vol. 91, no. 630, pp. 2–2, 1998.
  * [24] K. A. Dalrymple, J. Gomez, and B. Duchaine, “The dartmouth database of children’s faces: Acquisition and validation of a new face stimulus set,” _PloS One_ , vol. 8, no. 11, p. e79131, 2013.
  * [25] Z. Liu, P. Luo, X. Wang, and X. Tang, “Deep learning face attributes in the wild,” in _Proceedings of International Conference on Computer Vision (ICCV)_ , December 2015.
  * [26] Q. Xie, M.-T. Luong, E. Hovy, and Q. V. Le, “Self-training with noisy student improves imagenet classification,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 10 687–10 698.
  * [27] H. Pham, Z. Dai, Q. Xie, and Q. V. Le, “Meta pseudo labels,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 11 557–11 568.
  * [28] E. D. Cubuk, B. Zoph, J. Shlens, and Q. V. Le, “Randaugment: Practical automated data augmentation with a reduced search space,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , 2020, pp. 702–703.
  * [29] D. Hendrycks, N. Mu, E. D. Cubuk, B. Zoph, J. Gilmer, and B. Lakshminarayanan, “Augmix: A simple data processing method to improve robustness and uncertainty,” _arXiv preprint arXiv:1912.02781_ , 2019.
  * [30] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,” _arXiv preprint arXiv:1710.09412_ , 2017.
  * [31] K. Sohn, D. Berthelot, N. Carlini, Z. Zhang, H. Zhang, C. A. Raffel, E. D. Cubuk, A. Kurakin, and C.-L. Li, “Fixmatch: Simplifying semi-supervised learning with consistency and confidence,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 596–608, 2020.
  * [32] A. Oliver, A. Odena, C. A. Raffel, E. D. Cubuk, and I. Goodfellow, “Realistic evaluation of deep semi-supervised learning algorithms,” _Advances in Neural Information Processing Systems_ , vol. 31, 2018.
  * [33] J.-C. Su, Z. Cheng, and S. Maji, “A realistic evaluation of semi-supervised learning for fine-grained classification,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 12 966–12 975.
  * [34] R. Yoshihashi, W. Shao, R. Kawakami, S. You, M. Iida, and T. Naemura, “Classification-reconstruction learning for open-set recognition,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 4016–4025.
  * [35] L.-Z. Guo, Z.-Y. Zhang, Y. Jiang, Y.-F. Li, and Z.-H. Zhou, “Safe deep semi-supervised learning for unseen-class unlabeled data,” in _International Conference on Machine Learning_ , 2020, pp. 3897–3906.
  * [36] I. Cohen, N. Sebe, F. G. Cozman, and T. S. Huang, “Semi-supervised learning for facial expression recognition,” in _5th ACM SIGMM international workshop on Multimedia information retrieval_ , 2003, pp. 17–22.
  * [37] H. Li, N. Wang, X. Yang, X. Wang, and X. Gao, “Towards semi-supervised deep facial expression recognition with an adaptive confidence margin,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 4166–4175.
  * [38] E. D. Cubuk, B. Zoph, D. Mane, V. Vasudevan, and Q. V. Le, “Autoaugment: Learning augmentation strategies from data,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 113–123.
  * [39] S. Li, W. Deng, and J. Du, “Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 2852–2861.
  * [40] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 770–778.

|  Shuvendu Roy is a Ph.D. student at Department of Electrical and Computer
Engineering, and Ingenuity Labs Research Institute, at Queen’s University in
Canada. He received his B.Sc. in Computer Science and Engineering from Khulna
University of Engineering & Technology, Bangladesh. His current research is
focused on computer vision with deep learning, as well as self-supervised and
semi-supervised learning.
---|---
|  Ali Etemad is an Associate Professor in the Department of Electrical and
Computer Engineering at Queen’s University in Canada, where he also holds the
title of Mitchell Professor in AI for Human Sensing and Understanding. He is a
member of the Ingenuity Labs Research Institute and leads the Ambient
Intelligence and Interactive Machines (Aiim) lab. His primary research
interests lie in machine learning and deep learning, with a focus on human-
centered applications using wearables, smart devices, and smart environments.
Prior to joining Queen’s, he worked in the industry for several years as a
data scientist. His work has been published in top-tier venues such as CVPR,
AAAI, ICCV, ECCV, ACM CHI, ICASSP, Interspeech, ICPR, T-PAMI, T-AFFC, and
more. He has co-invented nine patents and delivered numerous invited talks at
various events. Dr. Etemad serves as an Associate Editor for IEEE Transactions
on Artificial Intelligence and has participated as a PC member/reviewer for
several renowned conferences and journals, including NeurIPS, ICML, CVPR,
ICLR, ICCV, ECCV, AAAI, ACII (senior PC), ICASSP, ISWC, and ICMI, among
others. He has held leadership roles in multiple events, such as General Chair
for the AAAI Workshop on Human-Centric Self-Supervised Learning (2022),
Publicity Co-Chair for the European Workshop on Visual Information Processing
(2022), and Industry Relations Chair for the Canadian Conference on AI (2019).
Dr. Etemad’s lab and research program have received funding from various
organizations, including the Natural Sciences and Engineering Research Council
(NSERC) of Canada, Ontario Centre of Innovation (OCI), Canadian Foundation for
Innovation (CFI), and private sector partners.
---|---