# Spatiotemporal Contrastive Learning of
Facial Expressions in Videos

Shuvendu Roy, Ali Etemad  Dept. ECE and Ingenuity Labs Research Institute
Queen’s University, Kingston, Canada
{shuvendu.roy<EMAIL_ADDRESS>
###### Abstract

We propose a self-supervised contrastive learning approach for facial
expression recognition (FER) in videos. We propose a novel temporal sampling-
based augmentation scheme to be utilized in addition to standard spatial
augmentations used for contrastive learning. Our proposed temporal
augmentation scheme randomly picks from one of three temporal sampling
techniques: (1) pure random sampling, (2) uniform sampling, and (3) sequential
sampling. This is followed by a combination of up to three standard spatial
augmentations. We then use a deep R(2+1)D network for FER, which we train in a
self-supervised fashion based on the augmentations and subsequently fine-tune.
Experiments are performed on the Oulu-CASIA dataset and the performance is
compared to other works in FER. The results indicate that our method achieves
an accuracy of 89.4%, setting a new state-of-the-art by outperforming other
works. Additional experiments and analysis confirm the considerable
contribution of the proposed temporal augmentation versus the existing spatial
ones.

###### Index Terms:

Self-Supervised Learning, Contrastive Learning, Facial Expressions, Affective
Computing

##  I Introduction

The objective of facial expression recognition (FER) is to predict basic
expressions such as happy, disgust, neutral, sad, Surprised, and others, from
facial images or videos [1, 2]. Applications of such systems include emotion-
aware multimedia and smart devices [3], personal mood management [4, 5], and
others. FER is challenging due to a number of reasons including variations in
how different people express different emotions, different lighting or
background conditions, and more. Moreover, the intensities with which emotions
are expressed throughout the face can vary among subjects or even for the same
subject in different contexts. For these reason, there has been an increased
effort toward facial expression recognition in recent years from both images
[1, 2] and videos [6, 2, 7].

A variety of different machine learning and computer vision solutions have
been proposed for FER [8, 9]. More recently, however, due to the immense
success of deep learning algorithms, deep neural networks (DNN) such as
convolutional neural networks (CNN) have been successfully used to perform
automatic feature extraction from training data. Nonetheless, deep networks
generally need considerable amounts of ‘labeled’ data to learn effective and
robust features.

To reduce the reliance on output class labels during training, and also to
learn better representations, self-supervised learning [10, 11, 12] has
emerged as an effective method to pre-train deep neural networks. Different
approaches for self-supervised learning such as contrastive learning
techniques like SimCLR [13] make use of the input data and create pseudo-
labels using augmentations to train the network. Experiments have shown that
such approaches can achieve competitive results even without the full
datasets.

Figure 1: Contrastive self-supervised learning using various augmentations.
Here augmented samples from the same video are considered positive examples,
while other examples are considered negative ones.

In this paper, to take advantage of the desired properties of self-supervised
learning, we propose a new video-based FER solution using contrastive
learning. Our proposed method, Spatiotemporal Contrastive Learning of
Representations (ST-CLR), first uses a novel temporal sampling scheme
(temporal augmentation) to pick individual frames from the unlabeled input
video to construct a sub-video. This will allow for various sub-videos to be
constructed from any given input, resulting in better generalization. It then
performs standard spatial augmentations, namely random cropping, color
distortion, and random flipping to learn by maximizing the agreement between
positive examples (augmentations on the same video) and minimizing the
agreement for the negative pairs. A visual representation of our approach is
shown in Figure 1. The figure demonstrates how the sub-video samples are
projected onto a normalized embedding space, followed by minimizing of the
distance between the positive pairs while maximizing the distance between the
negative samples vs. the positives. We integrate this technique into a deep
neural network that uses a R(2+1)dimensional framework instead of 3D
convolutions to learn spatiotemporal representations. We perform extensive
experiments and show the strong performance of our method on the Oulu-CASIA
dataset [14], outperforming all the previous methods.

In summary, our contributions are as follows.

  * •

We use contrastive self-supervised learning for FER in videos. To the best of
our knowledge, this is the first time contrastive learning is used in this
context.

  * •

We introduce ST-CLR, an extension of the original SimCLR that performs
temporal augmentations (sampling) followed by spatial augmentations for
contrastive learning in videos.

  * •

Using the proposed technique, we develop an end-to-end R(2+1)D network for
video-based FER and perform rigorous experiments on the Oulu-CASIA dataset.
The experiments show that our method outperforms existing works and sets a new
state-of-the-art on this dataset.

The rest of this paper is organized as follows. In the following section we
review the related literature on self-supervised learning and video-based FER.
This is followed by a detailed description of our proposed solution. Next, the
experiment setup and implementation details are described, followed by the
experimental results. Lastly in the final section, concluding remarks,
limitations, and future work are discussed.

##  II Related Works

###  II-A Self-Supervised Learning

Self-supervised learning has been proposed to reduce the reliance of learning
techniques on output labels such as human annotations, by applying specific
augmentations [13] or transformations [15, 16] to input data to generate
pseudo-labels with which the model is trained. Training of the model using the
generated pseudo-labels can be seen as an auxiliary pre-training task, also
sometimes referred to as a ’pretext’ task. Successive to the pre-training step
by means of the pretext tasks, main components of the model, for instance the
convolutional blocks in a CNN, are generally frozen and transfer learning is
used to develop another model for ’down-stream’ tasks, i.e. the main aim
classification problem [16]. In this phase, the fully connected layers are
often trained from scratch with the frozen convolutional blocks using the
original labels in the dataset [13]. In some cases, fine-tuning of the frozen
components can also help achieve a better solution [17]. This approach has
been used on a variety of different problem domains including image analysis
[17, 10], wearable-based activity recognition [18, 12], and affective
computing with bio-signals [19, 20, 21].

In the context of image analysis, when trained with pretext tasks, the
shallower layers of a CNN learn to extract low-level general purpose features
such as corners, textures, and edges, whereas the deeper layers of the model
learn high-level features such as detecting objects and larger parts of scenes
[22]. Recently, a number of different techniques have been proposed for self-
supervised learning in images [23, 17, 24, 25, 26]. The novelties in most of
these techniques lie in the way with which the pretext tasks are defined. For
instance, in one approach, input images were divided into a number of blocks
and shuffled, with the pretext task being to find the right order of blocks
[23, 17]. In [25] images were distorted with different degrees of rotation,
where the pretext task was to predict the amount of rotation that was applied
to a distorted image. In [24] the pretext task was to colorize an input gray-
scale image while in [26] the task was to fill missing parts of an image that
was cropped out.

Recently contrastive learning has emerged as a powerful self-supervised
learning technique and has shown great promise and achieved state-of-the-art
results for a variety of different tasks [27, 28, 29, 30]. In particular, a
more recently proposed contrastive method called SimCLR [13] provides a
generalized learning framework that does not requires any specialized
architecture [30, 31] nor any memory bank [32, 33, 11]. SimCLR uses data
augmentation to obtain two representations of the same input which they
consider as positive pairs. SimCLR learns the visual representation by
maximizing the agreement of positive pairs via a contrastive loss in the
latent space.

###  II-B Self-Supervised Learning in Video

Self-supervised learning with video data often uses the temporal information
in the video sequence, for instance tracking the temporal order of the frames
of a video [34, 35]. The intuition here is that the movement of objects in
frames can be tracked in a sequence of consecutive video frames, thus the
representation of frames that are temporally close in the video must be
similar. In [34], a self-supervised method was proposed that takes a set of
consecutive frames of a video to use as the main input. This was followed by
selecting two additional sets from the same video, where the one temporally
closest to the main set being designated as the positive set, while the other
being designated as the negative set. This approach learned the video
representations by enforcing the embeddings obtained from of the two close
sub-videos to be similar in the latent space, while forcing the embedding of
the negative pair to be far apart in the latent space. In [35], the
performance of [34] was improved by picking the negative sub-video from a
completely different video. In [36], a contrastive learning method was
proposed for videos that, similar to [34], uses consecutive frames to form the
positive and negative pairs.

As another approach to performing temporal self-supervised learning, a pretext
task was designed to validate whether or not the frames of a video are in the
right order successive to performing temporal shuffling [37]. Using this
approach, the performance of video classification on large datasets, for
instance for action recognition, was improved. Later, the Odd-One-Out Network
[38] was proposed with a similar objective. In this case, given a few videos,
the objective was to identity which video does not have the correct temporal
order.

In order to not explicitly learn sequence information, video colorization [39]
was proposed for self-supervised learning of videos. In this approach, the
objective was to colorize input gray scale videos using a pre-colored
reference frame. Unlike colorizing a single image, the temporal coherence in a
video needed to be preserved, hence an loss function was designed to keep
track of correlated pixels in consecutive frames.

###  II-C Expression Recognition from Video

For sequence-based [40, 41] or video-based [1, 2] FER, RNNs such as LSTM
networks have been used to learn temporal information along with CNNs that
exploit spatial information from the individual video frames [42, 43].
Alternatively, 3D CNNs have also been explored for video-based FER [44, 42].
For example, a 3D CNN was used with deformable action parts constraints for
dynamic motion encoding in FER called CNN-DAP [44]. In [42], a 3D CNN was used
in an ensemble with two 2D CNN models. Unlike 3D CNNs that share weights in
the time dimension, in [45], a method was proposed that used a sequence of
images without weight sharing in the time dimension.

Similar to the nature of self-supervised training, fully supervised pre-
training with additional image or video datasets has been widely explored in
the literature. In [46], a frame-to-sequence method was proposed that used a
pre-trained CNN as a feature extractor followed by an RNN to capture the
temporal features. In [1], a multi-modal framework was proposed that used both
image and video -level information with an ensemble of 2D and 3D CNNs. Here
the 2D model acts as a pre-trained feature extractor which is followed by a
NetVLAD layer [47] which aggregates the temporal representation of videos. In
[2], a generative approach was proposed to pre-train the network with a
conditional generative adversarial network [48].

Finally, another approach for enhancing performance in video-based FER has
been to focus on particular parts of the video sequence where expressions are
most prominent [49]. The difficulty with this method is the lack of knowledge
as to when the prominent expressions occur. To tackle this, attention
mechanisms can be used to focus on salient parts of sequences, as performed in
[8, 40, 50].

Figure 2: An example of the temporal and spatial augmentations used in ST-CLR
are depicted.

##  III Method

The proposed method is a 2 step training process. First, the model is pre-
trained by contrastive self-supervised learning. Then the model is fine-tuned
for the task of video-based FER. Following we present our method in detail.

###  III-A Self-Supervised Training

The self-supervised step in our method is inspired by the recent developments
in contrastive learning, which aim to maximize the agreement between augmented
versions of the same input in latent space using contrastive loss. The
proposed method consists of three main components, namely stochastic sampling
and augmentation, an encoder, and a projection head.

####  III-A1 Sampling and Augmentation Module

We propose a temporal sampling step prior to spatial augmentation, which can
also be viewed as a ‘temporal augmentation’ operation. This is followed by the
standard image-based augmentations used in [13]. Accordingly, given input
video $x$, we sample two sub-videos $\tilde{x}_{1}$ and $\tilde{x}_{2}$. Next,
augmentations are performed on each sub-video to yield $\tilde{x}^{Aug}_{1}$
and $\tilde{x}^{Aug}_{2}$. The augmented sub-videos that are sampled from the
same video are considered positive examples, while any augmented sub-video
sampled from a different video is considered a negative example. Following we
describe the two augmentation steps (temporal and spatial).

Temporal Augmentation. To sample a given input video, we defined 3 sampling
strategies: (1) Pure random sampling where $n$ frames are sampled at random
from the full length video, then sorted according to the order in the original
video; (2) Uniform sampling where frames are sampled uniformly starting and
ending with the first and last frames of the input while maintaining uniform
distance in between consecutive sampled frames; (3) Sequential sampling where
an initial frame is randomly picked, followed by the following sequence of
$n-1$ frames. To perform the temporal augmentation step in our method, we
randomly choose between the above strategies every time. Therefore, as we
sample the sub-videos in a different way at each iteration, the positive pairs
not only see different spatial augmentations of the same input video, but also
sees a completely different sub-video. This results in better generalization
and robustness to temporal variations. Figure 2 (top) depicts the temporal
augmentation step where two sub-videos are sampled using the first sampling
strategy.

Spatial Augmentation. Successive to temporal sampling, we apply the following
augmentations to the entire sub-video sequences to obtain
$\tilde{x}^{Aug}_{i}$. (1) Random resized crop. First, random cropping is
applied to a given sub-video. Here, a frame size and position is picked at
random, then all the frames of that sub-video are cropped at the same
position. Finally, the video sequence is resized to the desired input shape of
the CNN model. (2) Random flip. Finally, we randomly choose whether to
horizontally flip all the frames in a sub-video or not. (3) Random color
distortion. This augmentation includes modifying brightness, contrast,
saturation, and hue of the sub-videos. Here, the distortion parameters are
picked randomly and applied to all the frames of the sub-video. It should be
noted that unlike the temporal augmentation step where only one of the
sampling strategies is picked for each sub-video, for spatial augmentation,
all three operations are performed together. Nonetheless, as the parameters
are chosen at random, zeros could be chosen for the parameters of a certain
spatial augmentation, effectively not perform it. Figure 2 (bottom) shows
examples of spatial augmentation operations performed on a sample frame.

Figure 3: Factorization of 3D convolutional block into (2+1)D.

Figure 4: The R(2+1)D encoder network of the proposed method.

####  III-A2 Encoder Module $f(x)$

We use an encoder block (a simple CNN) to transform the input video into a
latent representation. A very common way for modeling 4-dimensional data such
as videos is using 3D convolutional neural networks [51, 44, 42]. This
strategy allows for both spatial and temporal dimensions to be directly
learned. However, 3D CNNs have a large number of parameters, requiring
significantly more data and processing power to train. To address this
drawback, an approach was proposed to factorize the 3D CNNs [52], breaking
down the 3D convolutions into a 2D operation followed by a 1D operation. The
resulting model with residual connections [53] was called R(2+1)D [52]. There
are a couple of advantages to this model. First, the factorized operations
contain a non-linear function in between the layers, resulting in more non-
linearity in the network, which is a desired property. Second, the
factorization helps the optimizer during training by lowering the training
loss and training time. It was also shown in [52] that factorizing the 3D
convolution into two distinct spatial and temporal operations improves the
performance of learning in large scale video classification tasks. In our
paper, we use this concept in the encoder block of the proposed network. This
notion is depicted in Figure 3.

Table I: Architecture of Encoder module Layer Name | Output size |  | Kernel size
---|---|---|---
Conv Block1 | [45, 16, 112, 112] |  | 45, [1, 7, 7]
[64, 16, 112, 112] |  | 64, [3, 1, 1]
[144, 16, 112, 112] | } $\times 4$ | 144, [1, 3, 3]
[64, 16, 112, 112] | 144, [3, 1, 1]
Conv Block2 | [230, 16, 56, 56] |  | 230, [1, 3, 3]
[128, 8, 56, 56] |  | 128, [3, 1, 1]
[230, 8, 56, 56] |  | 230, [1, 3, 3]
[128, 8, 56, 56] |  | 128, [3, 1, 1]
Residual layer1 | [128, 8, 56, 56] |  | 128, [1, 1, 1]
Conv Block3 | [288, 8, 56, 56] | } $\times 2$ | 288, [1, 3, 3]
[128, 8, 56, 56] | 128, [3, 1, 1]
Conv Block4 | [460, 8, 28, 28] |  | 460, [1, 3, 3]
[256, 4, 28, 28] |  | 256, [3, 1, 1]
[460, 4, 28, 28] |  | 460, [1, 3, 3]
[256, 4, 28, 28] |  | 256, [3, 1, 1]
Residual layer2 | [256, 4, 28, 28] |  | 256, [1, 1, 1]
Conv Block5 | [576, 4, 28, 28] | } $\times 2$ | 576, [1, 3, 3]
[256, 4, 28, 28] | 256, [3, 1, 1]
Conv Block6 | [921, 4, 14, 14] |  | 921, [1, 3, 3]
[512, 2, 14, 14] |  | 512, [3, 1, 1]
[921, 2, 14, 14] |  | 921, [1, 3, 3]
[512, 2, 14, 14] |  | 512, [3, 1, 1]
Residual layer3 | [512, 2, 14, 14] |  | 512, [1, 1, 1]
Conv Block7 | [1152, 2, 14, 14] | } $\times 2$ | 1152, [1, 3, 3]
[512, 2, 14, 14] | 512, [3, 1, 1]
Ada. Ave. Pool | [512, 1, 1, 1] |  |

Figure 5: Visual illustration of our proposed method, where the agreement
between different augmented sub-videos is maximized in the latent space.

To form a (2+1)D block from a 3D block of shape $M_{i-1}\times t\times d\times
d$, where $M_{i-1}$ is the number of 3D kernels with each kernel size being
$t\times d\times d$, we replace the $M_{i}$ 3D Convolutional kernels with
(2+1)D blocks which contain $N_{i}$ 2D convolutional kernels of shape
$M_{i-1}\times 1\times d\times d$ and $M_{i}$ temporal convolutional kernels
of shape $N_{i}\times t\times 1\times 1$. Here, $N_{i}$ is the dimension of
the intermediate representations as shown in Figure 3. The encoder with all
the CNNs and residual connections are shown in Figure 4 and the full
architecture of the encoder model using the (2+1)D blocks is laid out in Table
I. Lastly, we represent the output embedding of the encoder block as
$h_{i}=f(\tilde{x}_{i})$, where $h_{i}\in\mathbb{R}^{d}$.

####  III-A3 Projection head $g(x)$

Following the intuition of [13], we use a projection head $g(x)$ consisting of
2 dense layers, with a ReLU non-linearity in between, which takes the output
of $f(x)$ as input and outputs the final latent embedding. Experiments in [13]
showed improvements in the learnt representation by including such a non-
linear learnable transformation. The final embedding is represented as
$z_{i}=g(h_{i})$ on which the contrastive loss is applied.

####  III-A4 Contrastive loss

We use a contrastive loss to train the proposed self-supervised network. The
loss is designed to learn visual representation from seeing positive and
negative pairs. Let’s define the similarity between the embedding $z_{i}$ and
$z_{j}$ (the outputs from the encoder) as
$cosine(z_{i},z_{j})=z_{i}^{T}z_{j}/||z_{i}||.||z_{j}||$. If $i$ and $j$ are
positive examples, then the contrastive loss function is defined as:

$\displaystyle
Loss_{i,j}=-log\frac{exp(cosine(z_{i},z_{j})/\tau)}{\sum_{k=1}^{2N}\mathbf{1}_{[k\neq
i]}exp(cosine(z_{i},z_{k})/\tau)},$ (1)

where $\mathbf{1}_{[k\neq i]}\in\\{0,1\\}$ is a function of $i$, $k$ which
outputs 1 when $i\neq k$. Here $\tau$ represents a temperature parameter which
scales the cosine similarity. In some previous works such as [13], loss
function defined above is termed normalized temperature-scaled cross entropy
loss. Figure 5 depicts a visual illustration of the proposed ST-CLR method.

###  III-B Model Fine-tuning

After pre-training the model with self-supervised learning, we no longer use
the projection head at the end of the encoder block. During the supervised
fine-tuning phase, we drop the projection head and add a single-layer linear
transformation for generating the output probability. This fine-tuning is
performed with categorical cross-entropy loss.

##  IV Experiments and Results

###  IV-A Experiments

In this section we present the experiments and the results for the proposed
method on video-based FER. We conduct the experiments on a popular video FER
dataset called Oulu-CASIA [14].

Dataset. The Oulu-CASIA [14] is a facial expression recognition dataset that
contains 6 expressions: Happy, Sad, Surprised, Angry, Fear, and Disgust. This
dataset was collected from 80 people aging from 23 to 58. Among them 73.8%
were male and 26.2% were female. The dataset was collected in a lab
environment where the subjects were asked to sit about 60 cm away from the
camera. The videos were collected in 25 frames per second at a spatial
resolution of $320\times 240$.

Implementation Details As mentioned in the method section, we use the R(2+1)D
architecture for the encoder of our method. At the end of the encoder we have
a projection head, which is a 2-layer network. The output of the projection
head is a 128 dimensional vector, on which the contrastive loss is calculated.
The model is trained with an SGD optimizer for 1000 epochs with a momentum of
0.9, a learning rate of 0.001, and weight decay of 1e-4. Oue solution was
implemented in PyTorch and trained using 4 NVIDIA V100 GPUs. The model takes
sub-videos with 16 frames, and spatial resolution of $224\times 224$ as its
inputs.

In the fine-tuning step, we train the final model for 100 epochs. First, we
train the newly initialized linear layer for 30 epoch and then fine-tune the
entire model for 70 epochs. The model is trained using Adam optimizer with a
learning rate of 1e-4 and plateau learning rate decay with a patience of 3.

###  IV-B Results

We train the proposed method on Oulu-CASIA dataset. To compare our method with
previous works on this dataset, we have conducted all the experiments with
10-fold cross-validation. Table II compares the result of our solution with
previous methods on the Oulu-CASIA dataset. Our ST-CLR method performs the
best among all the methods in this comparison. The confusion matrix depicted
in Figure 6 shows that our method performs the best on classifying ‘disgust’
and also performs well on expressions ‘fear’ and ‘happy’. We observe the least
effective performance on ‘angry’ and ‘sad’. The figure shows that ‘Angry’ is
often confused with ‘Disgust’ while ‘Sad’ is generally confused with ‘Fear’
and ‘Surprised’. Lastly, ‘Surprised’ is mostly confused with ‘Fear’ and
‘Angry’. This analysis points to relatively reasonable mistakes by the model.

Table II: Comparison with previous methods on Oulu-CASIA dataset Authors | Method | Accuracy
---|---|---
Zhao et al. [54] | LBP-TOP | 68.13%
Liu et al. [55] | STM-ExpLet | 74.59%
Guo et al. [56] | Atlases | 75.52%
Jung et al. [45] | DTAGN-Joint | 81.46%
Zhao et al. [49] | PPDN | 84.59%
Yu et al. [57] | DCPN | 86.23%
Zhangh et al. [58] | PHRNN-MSCNN | 86.25%
Ding et al. [7] | FN2EN | 87.71%
Yang et al. [2] | DeRL | 88.00%
Zhang et al. [6] | WMCNN-LSTM | 88.00%
Ours | ST-CLR |  89.38%

Figure 6: Confusion matrix of the prediction of the model on Oulu-CASIA
dataset. Model predictions are presented in the $x$ axis while the ground-
truths are in the $y$ axis.

Next, we conduct ablation studies on the main elements of the proposed method.
As discussed earlier the proposed method requires a CNN backbone as the
encoder suitable for 4 dimensional input video data, for which we used
R(2+1)D. We conduct experiments by replacing this backbone with two popular
alternative encoder architectures, namely a CNN with 3D convolutions and a CNN
with mixed 2D and 3D convolutions. For the 3D convolutional network, we
replace all the R(2+1)D blocks mentioned in Table I with 3D blocks. As for the
mixed 3D-2D alternative, 3D blocks are used for all the convolutions in the
$ConvBlock_{1}$ and 2D blocks are used for rest of the network. Table III
shows the accuracy of our model versus its variants when the alternative
networks are used as the encoder block. As we can see, R(2+1)D performs better
than other networks, confirming our choice of using the R(2+1)D architecture.

Table III: Comparison of different encoder blocks on Oulu-Casia dataset Encoder Network | Accuracy
---|---
3D Convolution | 87.50%
Mixed 2D and 3D Convolution | 85.21%
R(2+1)D Convolution | 89.38%
Table IV: Effect of different augmentations on Oulu-Casia dataset. FS: Frame Sampling, RRC: Random Resized Crop, RCD: Random Color Distortion, RF: Random Flip. Augmentations | FS | RRC | RCD | RF | Accuracy
---|---|---|---|---|---
All (temp. + sp.) | ✓ | ✓ | ✓ | ✓ | 89.38%
w/o temp. | ✗ | ✓ | ✓ | ✓ | 85.41%
temp. + partial sp. 1 | ✓ | ✗ | ✓ | ✓ | 87.71%
temp. + partial sp. 2 | ✓ | ✓ | ✗ | ✓ | 86.67%
temp. + partial sp. 3 | ✓ | ✓ | ✓ | ✗ | 89.17%
Table V: Comparison of performance for different amount of ssl epochs on Oulu-Casia dataset Epochs | Accuracy
---|---
100 | 87.71%
500 | 89.17%
1000 | 89.38%
Table VI: Comparison of performance of the method using different amount of supervised data on Oulu-CASIA dataset Labeled Data (%) | Accuracy
---|---
75% | 87.50%
50% | 83.33%
25% | 77.08%
10% | 54.17%

Another important component of contrastive self-supervised leaning is the use
of augmentations. One of the novelties in our proposed method is the use of
frame sampling -based temporal augmentations along with popular spatial
augmentations. Table IV shows the ablation study on spatial and temporal
augmentations used in our method, where the impact of each augmentation is
investigated by systematic exclusion. We see that the temporal augmentation
clearly has the highest impact on the final performance as removing this
augmentation module reduces the accuracy of the model by about 4%. Among the
spatial augmentations, ‘random color distortion’ shows the highest impact,
which is in conformity with other works on contrastive learning [13].

It was shown in previous works such as [13] that self-supervised learning
usually benefits from longer training compared to fully supervised learning.
In Table V, we show the impact of different number of epochs for pre-training
our model. We find that the accuracy improves significantly when the number of
epochs increases from 100 to 500. However, only slight improvements are
achieved when training it for another 500 epochs (for a total of 1000 epochs).
So, for our final model, we pre-train the model for 1000 epochs.

Finally, to show the robustness of our model, we conduct experiments with
partially labeled data. Table VI shows the accuracy of the model with
different portions of the dataset being labeled and used for fine-tuning. We
find that, using 75% and 50% data drops the accuracy by only 2% and 6%
respectively. Even with only 25% of the data being labeled, the accuracy is
better than few of the previous works shown in Table II. This indicates that
our model is robust to having fewer labeled data, which can be attributed to
the self-supervised training strategy used.

##  V Conclusion and Future Work

This paper presents a self-supervised approach for learning facial expressions
in video. Our method uses a R(2+1)D style architecture to avoid using 3D
convolutions for faster and better performance. To perform self-supervised
training for our model, we propose a novel temporal augmentation scheme along
with standard spatial (frame-based) augmentations. We test our solution on the
Oulu-CASIA dataset and perform rigorous experiments. Our experiments show that
the proposed method outperforms existing work on this dataset to set a new
state-of-the-art. Extensive experiments were conducted to show the importance
of each of the utilized augmentations toward the final performance of the
model, highlighting the importance of our proposed temporal augmentation.
Additional experiments were performed which validated our choice of R(2+1)D
convolutions versus 3D and hybrid approached. Lastly, our analysis shows the
advantage and robustness of our self-supervised method when significant
portions of output labels were not used for training.

For future work, the proposed temporal augmentation scheme can be used for
contrastive self-supervised learning in other video-based representation
learning tasks such as action recognition, pose estimation, object detection
and tracking, and others. Accordingly, additional modifications may need to be
made to the network architecture and hyper-parameters in order to obtain
optimum performance in other domains and with other datasets.

## Acknowledgements

We would like to thank BMO Bank of Montreal and Mitacs for funding this
research.

## References

  * [1] S. Pini, O. B. Ahmed, M. Cornia, L. Baraldi, R. Cucchiara, and B. Huet, “Modeling multimodal cues in a deep learning-based framework for emotion recognition in the wild,” in _Proceedings of the 19th ACM International Conference on Multimodal Interaction_ , 2017, pp. 536–543.
  * [2] H. Yang, U. Ciftci, and L. Yin, “Facial expression recognition by de-expression residue learning,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 2168–2177.
  * [3] Y. Cho, S. J. Julier, and N. Bianchi-Berthouze, “Instant stress: detection of perceived mental stress through smartphone photoplethysmography and thermal imaging,” _JMIR Mental Health_ , vol. 6, no. 4, 2019.
  * [4] M. Thrasher, M. D. Van der Zwaag, N. Bianchi-Berthouze, and J. H. Westerink, “Mood recognition based on upper body posture and movement features,” in _International Conference on Affective Computing and Intelligent Interaction_ , 2011, pp. 377–386.
  * [5] D. Sanchez-Cortes, J.-I. Biel, S. Kumano, J. Yamato, K. Otsuka, and D. Gatica-Perez, “Inferring mood in ubiquitous conversational video,” in _Proceedings of the 12th International Conference on Mobile and Ubiquitous Multimedia_ , 2013, pp. 1–9.
  * [6] H. Zhang, B. Huang, and G. Tian, “Facial expression recognition based on deep convolution long short-term memory networks of double-channel weighted mixture,” _Pattern Recognition Letters_ , vol. 131, pp. 128–134, 2020.
  * [7] H. Ding, S. K. Zhou, and R. Chellappa, “Facenet2expnet: Regularizing a deep face recognition net for expression recognition,” in _12th IEEE International Conference on Automatic Face & Gesture Recognition_, 2017, pp. 118–126.
  * [8] J. Li, K. Jin, D. Zhou, N. Kubota, and Z. Ju, “Attention mechanism-based cnn for facial expression recognition,” _Neurocomputing_ , vol. 411, pp. 340–350, 2020.
  * [9] X. Liang, L. Xu, J. Liu, Z. Liu, G. Cheng, J. Xu, and L. Liu, “Patch attention layer of embedding handcrafted features in cnn for facial expression recognition,” _Sensors_ , vol. 21, no. 3, p. 833, 2021.
  * [10] L. Chen, P. Bentley, K. Mori, K. Misawa, M. Fujiwara, and D. Rueckert, “Self-supervised learning for medical image analysis using image context restoration,” _Medical Image Analysis_ , vol. 58, p. 101539, 2019.
  * [11] I. Misra and L. v. d. Maaten, “Self-supervised learning of pretext-invariant representations,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 6707–6717.
  * [12] S. Rahimi Taghanaki and A. Etemad, “Self-supervised wearable-based activity recognition by learning to forecast motion,” _arXiv e-prints_ , pp. arXiv–2010, 2020.
  * [13] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in _International Conference on Machine Learning_ , 2020, pp. 1597–1607.
  * [14] G. Zhao, X. Huang, M. Taini, S. Z. Li, and M. PietikäInen, “Facial expression recognition from near-infrared videos,” _Image and Vision Computing_ , vol. 29, no. 9, pp. 607–619, 2011.
  * [15] M. Noroozi, H. Pirsiavash, and P. Favaro, “Representation learning by learning to count,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 5898–5906.
  * [16] A. Dosovitskiy, P. Fischer, J. T. Springenberg, M. Riedmiller, and T. Brox, “Discriminative unsupervised feature learning with exemplar convolutional neural networks,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 38, no. 9, pp. 1734–1747, 2015.
  * [17] M. Noroozi and P. Favaro, “Unsupervised learning of visual representations by solving jigsaw puzzles,” in _European Conference on Computer Vision_ , 2016, pp. 69–84.
  * [18] A. Saeed, T. Özçelebi, and J. J. Lukkien, “Multi-task self-supervised learning for human activity detection,” _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ , vol. 3, no. 2, p. 61, 2019.
  * [19] P. Sarkar and A. Etemad, “Self-supervised learning for ecg-based emotion recognition,” in _IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2020, pp. 3217–3221.
  * [20] P. Sarkar, S. Lobmaier, B. Fabre, G. Berg, A. Mueller, M. G. Frasch, M. C. Antonelli, and A. Etemad, “Detection of maternal and fetal stress from ecg with self-supervised representation learning,” _arXiv preprint arXiv:2011.02000_ , 2020.
  * [21] P. Sarkar and A. Etemad, “Self-supervised ecg representation learning for emotion recognition,” _IEEE Transactions on Affective Computing_ , 2020.
  * [22] D. Bau, B. Zhou, A. Khosla, A. Oliva, and A. Torralba, “Network dissection: Quantifying interpretability of deep visual representations,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 6541–6549.
  * [23] C. Doersch, A. Gupta, and A. A. Efros, “Unsupervised visual representation learning by context prediction,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2015, pp. 1422–1430.
  * [24] R. Zhang, P. Isola, and A. A. Efros, “Colorful image colorization,” in _European Conference on Computer Vision_ , 2016, pp. 649–666.
  * [25] S. Gidaris, P. Singh, and N. Komodakis, “Unsupervised representation learning by predicting image rotations,” _arXiv preprint arXiv:1803.07728_ , 2018\.
  * [26] D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros, “Context encoders: Feature learning by inpainting,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 2536–2544.
  * [27] R. Hadsell, S. Chopra, and Y. LeCun, “Dimensionality reduction by learning an invariant mapping,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , vol. 2, 2006, pp. 1735–1742.
  * [28] A. Dosovitskiy, J. T. Springenberg, M. Riedmiller, and T. Brox, “Discriminative unsupervised feature learning with convolutional neural networks,” in _Proceedings of the 27th International Conference on Neural Information Processing Systems_ , 2014, pp. 766–774.
  * [29] A. v. d. Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” _arXiv preprint arXiv:1807.03748_ , 2018\.
  * [30] P. Bachman, R. D. Hjelm, and W. Buchwalter, “Learning representations by maximizing mutual information across views,” _arXiv preprint arXiv:1906.00910_ , 2019.
  * [31] O. Henaff, “Data-efficient image recognition with contrastive predictive coding,” in _International Conference on Machine Learning_ , 2020, pp. 4182–4192.
  * [32] Z. Wu, Y. Xiong, S. X. Yu, and D. Lin, “Unsupervised feature learning via non-parametric instance discrimination,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 3733–3742.
  * [33] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 9729–9738.
  * [34] X. Wang and A. Gupta, “Unsupervised learning of visual representations using videos,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2015, pp. 2794–2802.
  * [35] F. Schroff, D. Kalenichenko, and J. Philbin, “Facenet: A unified embedding for face recognition and clustering,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2015, pp. 815–823.
  * [36] R. Qian, T. Meng, B. Gong, M.-H. Yang, H. Wang, S. Belongie, and Y. Cui, “Spatiotemporal contrastive video representation learning,” _arXiv preprint arXiv:2008.03800_ , 2020.
  * [37] I. Misra, C. L. Zitnick, and M. Hebert, “Shuffle and learn: unsupervised learning using temporal order verification,” in _European Conference on Computer Vision_ , 2016, pp. 527–544.
  * [38] B. Fernando, H. Bilen, E. Gavves, and S. Gould, “Self-supervised video representation learning with odd-one-out networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 3636–3645.
  * [39] C. Vondrick, A. Shrivastava, A. Fathi, S. Guadarrama, and K. Murphy, “Tracking emerges by colorizing videos,” in _Proceedings of the European Conference on Computer Vision_ , 2018, pp. 391–408.
  * [40] A. Sepas-Moghaddam, A. Etemad, F. Pereira, and P. L. Correia, “Facial emotion recognition using light field images with deep attention-based bidirectional lstm,” in _IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2020, pp. 3367–3371.
  * [41] A. Sepas-Moghaddam, A. Etemad, F. Pereira, and P. Lobato Correia, “Capsfield: Light field-based face and expression recognition in the wild using capsule routing,” _arXiv e-prints_ , pp. arXiv–2101, 2021.
  * [42] C. Lu, W. Zheng, C. Li, C. Tang, S. Liu, S. Yan, and Y. Zong, “Multiple spatio-temporal feature learning for video-based emotion recognition in the wild,” in _Proceedings of the 20th ACM International Conference on Multimodal Interaction_ , 2018, pp. 646–652.
  * [43] B. Sun, Q. Wei, L. Li, Q. Xu, J. He, and L. Yu, “Lstm for dynamic emotion and group emotion recognition in the wild,” in _Proceedings of the 18th ACM International Conference on Multimodal Interaction_ , 2016, pp. 451–457.
  * [44] M. Liu, S. Li, S. Shan, R. Wang, and X. Chen, “Deeply learning deformable facial action parts model for dynamic expression analysis,” in _Asian Conference on Computer Vision_ , 2014, pp. 143–157.
  * [45] H. Jung, S. Lee, J. Yim, S. Park, and J. Kim, “Joint fine-tuning in deep neural networks for facial expression recognition,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2015, pp. 2983–2991.
  * [46] C.-M. Kuo, S.-H. Lai, and M. Sarkis, “A compact deep learning model for robust facial expression recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2018, pp. 2121–2129.
  * [47] R. Arandjelovic, P. Gronat, A. Torii, T. Pajdla, and J. Sivic, “Netvlad: Cnn architecture for weakly supervised place recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 5297–5307.
  * [48] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial networks,” _arXiv preprint arXiv:1406.2661_ , 2014.
  * [49] X. Zhao, X. Liang, L. Liu, T. Li, Y. Han, N. Vasconcelos, and S. Yan, “Peak-piloted deep network for facial expression recognition,” in _European Conference on Computer Vision_ , 2016, pp. 425–442.
  * [50] G. Zhang, V. Davoodnia, A. Sepas-Moghaddam, Y. Zhang, and A. Etemad, “Classification of hand movements from eeg using a deep attention-based lstm network,” _IEEE Sensors Journal_ , vol. 20, no. 6, pp. 3113–3122, 2019.
  * [51] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, “Learning spatiotemporal features with 3d convolutional networks,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2015, pp. 4489–4497.
  * [52] D. Tran, H. Wang, L. Torresani, J. Ray, Y. LeCun, and M. Paluri, “A closer look at spatiotemporal convolutions for action recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 6450–6459.
  * [53] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 770–778.
  * [54] G. Zhao and M. Pietikainen, “Dynamic texture recognition using local binary patterns with an application to facial expressions,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 29, no. 6, pp. 915–928, 2007\.
  * [55] M. Liu, S. Shan, R. Wang, and X. Chen, “Learning expressionlets on spatio-temporal manifold for dynamic facial expression recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2014, pp. 1749–1756.
  * [56] Y. Guo, G. Zhao, and M. Pietikäinen, “Dynamic facial expression recognition using longitudinal facial expression atlases,” in _European Conference on Computer Vision_ , 2012, pp. 631–644.
  * [57] Z. Yu, Q. Liu, and G. Liu, “Deeper cascaded peak-piloted network for weak expression recognition,” _The Visual Computer_ , vol. 34, no. 12, pp. 1691–1699, 2018.
  * [58] K. Zhang, Y. Huang, Y. Du, and L. Wang, “Facial expression recognition based on deep evolutional spatial-temporal networks,” _IEEE Transactions on Image Processing_ , vol. 26, no. 9, pp. 4193–4203, 2017.