Zengqun<EMAIL_ADDRESS>Ioannis<EMAIL_ADDRESS>School
of Electronic Engineering and Computer Science,
Queen Mary University of London,
London, E1 4NS, UK Prompting Visual-Language Models for DFER

# Prompting Visual-Language Models for Dynamic Facial Expression Recognition

###### Abstract

This paper presents a novel visual-language model called DFER-CLIP, which is
based on the CLIP model and designed for in-the-wild Dynamic Facial Expression
Recognition (DFER). Specifically, the proposed DFER-CLIP consists of a visual
part and a textual part. For the visual part, based on the CLIP image encoder,
a temporal model consisting of several Transformer encoders is introduced for
extracting temporal facial expression features, and the final feature
embedding is obtained as a learnable "class" token. For the textual part, we
use as inputs textual descriptions of the facial behaviour that is related to
the classes (facial expressions) that we are interested in recognising – those
descriptions are generated using large language models, like ChatGPT. This, in
contrast to works that use only the class names and more accurately captures
the relationship between them. Alongside the textual description, we introduce
a learnable token which helps the model learn relevant context information for
each expression during training. Extensive experiments demonstrate the
effectiveness of the proposed method and show that our DFER-CLIP also achieves
state-of-the-art results compared with the current supervised DFER methods on
the DFEW, FERV39k, and MAFW benchmarks. Code is publicly available at
https://github.com/zengqunzhao/DFER-CLIP.

##  1 Introduction

Facial expression is an important aspect of daily conversation and
communication [Darwin and Prodger(1998)]. Because of its application in
various fields, such as human-computer interaction (HCI) [Duric et
al.(2002)Duric, Gray, Heishman, Li, Rosenfeld, Schoelles, Schunn, and
Wechsler], driving assistance [Wilhelm(2019)], and mental health [Bishay et
al.(2019)Bishay, Palasek, Priebe, and Patras, Foteinopoulou et
al.(2021)Foteinopoulou, Tzelepis, and Patras], facial expression recognition
(FER) attracts increasing attention from researchers in psychology, computer
science, linguistics, neuroscience, and related disciplines [Corneanu et
al.(2016)Corneanu, Simón, Cohn, and Guerrero]. When using a discrete emotion
model [Ekman(1971)], FER aims to classify an image or video sequence into one
of several basic emotions, i.e., neutral, happiness, sadness, surprise, fear,
disgust, and anger. Traditional facial expression recognition approaches have
mainly focused on static images or video frames, which do not capture the
temporal dynamics of facial expressions [Li and Deng(2020), Zhao et
al.(2021a)Zhao, Liu, and Wang]. However, the recognition of dynamic facial
expressions involves capturing the changes in facial movements over time,
which can provide more information for accurate emotion recognition.
Therefore, the recognition of dynamic facial expressions has become an
increasingly important research area within the field of computer vision and
affective computing [Li and Deng(2020), Wang et al.(2022a)Wang, Song, Tao,
Liotta, Yang, Li, Gao, Sun, Ge, Zhang, et al.].

Early studies for dynamic facial expression recognition (DFER) are mainly
focused on lab-controlled conditions [Li and Deng(2020)], in which the faces
are all frontal and have no occlusions. However, in the real world, facial
expressions always occur accompanied by variations such as illumination, non-
frontal pose, and occlusion. Therefore, works on in-the-wild DFER are getting
popular in recent years [Jiang et al.(2020)Jiang, Zong, Zheng, Tang, Xia, Lu,
and Liu, Zhao and Liu(2021), Wang et al.(2022b)Wang, Sun, Huang, Liu, Gao,
Zhang, Ge, and Zhang, Liu et al.(2022)Liu, Dai, Feng, Wang, Yin, Zeng, and
Shan, Liu et al.(2023)Liu, Wang, Feng, Zhang, Chen, and Zhan]. The main focus
is trying to learn discriminative and robust spatial-temporal feature
representations for the DFER task. These methods can be roughly categorized
into 3DCNN-based [Jiang et al.(2020)Jiang, Zong, Zheng, Tang, Xia, Lu, and
Liu, Lee et al.(2019)Lee, Kim, Kim, Park, and Sohn], CNN-RNN-based [Baddar and
Ro(2019), Liu et al.(2021)Liu, Zhang, and Zhou, Wang et al.(2022c)Wang, Sun,
Song, Gao, Huang, Chen, Ge, and Zhang], and Transformer-based [Zhao and
Liu(2021), Foteinopoulou and Patras(2022), Li et al.(2023b)Li, Niu, Zhu, and
Zhao, Liu et al.(2023)Liu, Wang, Feng, Zhang, Chen, and Zhan], in which
Transformer-based methods have achieved state-of-the-art performance. In
addition, many in-the-wild datasets were proposed in recent years for DFER
[Jiang et al.(2020)Jiang, Zong, Zheng, Tang, Xia, Lu, and Liu, Liu et
al.(2022)Liu, Dai, Feng, Wang, Yin, Zeng, and Shan, Wang et al.(2022b)Wang,
Sun, Huang, Liu, Gao, Zhang, Ge, and Zhang]. However, the high cost of
labelling facial expression data limits the amount of training data available
for DFER models. This can have an impact on the model’s performance,
especially if the training data does not fully capture the diversity of facial
expressions that may occur in the real world.

Figure 1: Illustration of the (a) previous approach for DFER, which relies on
a classifier to predict the facial expression label. (b) Standard vision-
language model CLIP. (c) Based on the CLIP, we propose a DFER-CLIP, which
further models temporal facial features and incorporates fine-grained
descriptors.

Recently, vision-language pre-training (V-LP) models such as CLIP [Radford et
al.(2021)Radford, Kim, Hallacy, Ramesh, Goh, Agarwal, Sastry, Askell, Mishkin,
Clark, et al.] have emerged as promising alternatives for visual
representation learning. The main idea of the V-LP models is to align large
connections of images and raw texts using two separate encoders, so as to
learn semantic information between visual and textual data. The success of
V-LP models has led to a growing interest in using them for various downstream
computer vision tasks, such as video understanding [Ju et al.(2022)Ju, Han,
Zheng, Zhang, and Xie, Rasheed et al.(2023)Rasheed, Khattak, Maaz, Khan, and
Khan], image synthesis [Patashnik et al.(2021)Patashnik, Wu, Shechtman, Cohen-
Or, and Lischinski], and semantic segmentation [Ma et al.(2022)Ma, Yang, Wang,
Zhang, and Xie]. Given the powerful representation learning capabilities of
V-LP models, a relevant question is how to best exploit their potential for
the DFER task. One proposed approach is to fine-tune the image encoder of a
V-LP model on DFER datasets. However, this approach faces two challenges that
must be addressed. Firstly, while the original CLIP model can recognize
objects and scenes in images, it may not be as effective in recognizing subtle
facial expressions, which require more fine-grained descriptors and modelling
of the similarities between expressions. Secondly, learning robust temporal
facial features to understand emotions is crucial for DFER. Unfortunately, the
standard CLIP visual encoder encodes static images and therefore lacks ways of
capturing temporal information.

To address these challenges, based on the CLIP model, we propose a novel
architecture, namely the DFER-CLIP model. An overview of the differences
between DFER-CLIP with previous approaches and CLIP is shown in Fig. 1 and the
detailed architecture is shown in Fig. 2. The proposed method mainly consists
of a visual part and a textual part. Regarding the visual part, based on the
CLIP image encoder, we introduce a temporal model, which consists of several
Transformer encoders [Dosovitskiy et al.(2021)Dosovitskiy, Beyer, Kolesnikov,
Weissenborn, Zhai, Unterthiner, Dehghani, Minderer, Heigold, Gelly, et al.],
for modelling the temporal facial features. The final video-level facial
features are obtained by a learnable class token. Regarding the textual part,
considering that different facial expressions have both common properties and
unique or special properties at the level of local behaviour [Zhong et
al.(2012)Zhong, Liu, Yang, Liu, Huang, and Metaxas, Zhao et al.(2021b)Zhao,
Liu, and Zhou], we utilize descriptions related to facial behaviour instead of
class names for the text encoder. As a result, the text embedding can provide
more detailed and precise information about the specific movements or
positions of muscles involved in each expression. Furthermore, inspired by
CoOp [Zhou et al.(2022b)Zhou, Yang, Loy, and Liu], we adopt the learnable
prompt as a context for descriptors of each class – this does not require
experts to design context words and allows the model to learn relevant context
information for each expression during training. To evaluate our DFER-CLIP
model, we conducted experiments on three datasets. The results show that the
temporal model can clearly enhance performance, and adopting the fine-grained
expression descriptions with learnable context is superior to class-level
prompts. Furthermore, compared with the current supervised DFER methods, the
proposed DFER-CLIP achieves state-of-the-art results on DFEW, FERV39k, and
MAFW benchmarks.

Figure 2: The structure of the proposed DFER-CLIP model. cos() denotes cosine
similarity. M denotes the learnable context number. C denotes expression
classes.

##  2 Related Work

###  2.1 DFER in the Wild

For in-the-wild DFER, deep learning-based methods have achieved superior
performance to methods using hand-crafted features in recent years [Li and
Deng(2020)]. Those methods are mainly focusing on learning robust spatial-
temporal facial expression features, and can be roughly categorized into
3DCNN-based [Jiang et al.(2020)Jiang, Zong, Zheng, Tang, Xia, Lu, and Liu, Lee
et al.(2019)Lee, Kim, Kim, Park, and Sohn], CNN-RNN-based [Baddar and
Ro(2019), Liu et al.(2021)Liu, Zhang, and Zhou, Wang et al.(2022c)Wang, Sun,
Song, Gao, Huang, Chen, Ge, and Zhang], and Transformer-based [Zhao and
Liu(2021), Foteinopoulou and Patras(2022), Li et al.(2023b)Li, Niu, Zhu, and
Zhao, Liu et al.(2023)Liu, Wang, Feng, Zhang, Chen, and Zhan] methods. To
incorporate more modalities for emotion understanding, multi-modal learning is
also prevalent in DFER in the wild [Chen et al.(2016)Chen, Chen, Chi, and Fu,
Mittal et al.(2020)Mittal, Bhattacharya, Chandra, Bera, and Manocha]. Zhao et
al. [Zhao and Liu(2021)] first introduced the Transformer into DFER and
achieved state-of-the-art results. After that, many transformer-based methods
[Ma et al.(2023)Ma, Sun, and Li, Li et al.(2022a)Li, Sui, Zhu, and Zhao, Li et
al.(2023b)Li, Niu, Zhu, and Zhao] were proposed to further enhance the
performance. In the latest work, Li et al. [Li et al.(2023a)Li, Niu, Zhu, and
Zhao] proposed a model called CLIPER that uses visual language models for the
FER task, achieving state-of-the-art performance. However, the model is
limited in its ability to capture temporal information for dynamic FER and
uses a two-stage training paradigm instead of end-to-end training. By
contrast, our DFER-CLIP model is capable of learning spatial-temporal features
and is trained in an end-to-end manner.

###  2.2 Visual-Language Models

Vision-language models have made significant progress in zero-shot performance
through contrastive learning on pairs of images and text [Radford et
al.(2021)Radford, Kim, Hallacy, Ramesh, Goh, Agarwal, Sastry, Askell, Mishkin,
Clark, et al., Zhou et al.(2022b)Zhou, Yang, Loy, and Liu, Zhou et
al.(2022a)Zhou, Yang, Loy, and Liu, Menon and Vondrick(2023), Ju et
al.(2022)Ju, Han, Zheng, Zhang, and Xie, Rasheed et al.(2023)Rasheed, Khattak,
Maaz, Khan, and Khan, Oldfield et al.(2023)Oldfield, Tzelepis, Panagakis,
Nicolaou, and Patras]. This approach leverages a vast and virtually limitless
supply of image-text pairs collected from the internet, making the process
both highly efficient and cost-effective. The most recent vision-language
learning frameworks CLIP [Radford et al.(2021)Radford, Kim, Hallacy, Ramesh,
Goh, Agarwal, Sastry, Askell, Mishkin, Clark, et al.] and ALIGN [Jia et
al.(2021)Jia, Yang, Xia, Chen, Parekh, Pham, Le, Sung, Li, and Duerig] benefit
from 400 million and 1.8 billion image-text pairs collected from the Internet.
These large-scale vision-language models are then applied to various
downstream computer vision tasks and achieve stage-of-the-art performance
[Patashnik et al.(2021)Patashnik, Wu, Shechtman, Cohen-Or, and Lischinski, Ni
et al.(2022)Ni, Peng, Chen, Zhang, Meng, Fu, Xiang, and Ling, Nag et
al.(2022)Nag, Zhu, Song, and Xiang, Ju et al.(2022)Ju, Han, Zheng, Zhang, and
Xie, Rasheed et al.(2023)Rasheed, Khattak, Maaz, Khan, and Khan]. For the
visual-language modes, the researchers usually utilize a prompt (e.g., “a
photo of a”) along with the name of the class to generate a text descriptor
for each class. Zhou et al. [Zhou et al.(2022b)Zhou, Yang, Loy, and Liu]
introduced a learnable continuous prompt method named context optimization
(CoOp), which can adaptively adjust according to different datasets.
Furthermore, Zhou et el. [Zhou et al.(2022a)Zhou, Yang, Loy, and Liu] proposed
conditional context optimization (CoCoOp) based on CoOp, which can generate a
prompt conditioned on each image and make better inferences on unseen classes.
Li et al. [Li et al.(2022b)Li, Li, Xiong, and Hoi] proposed a new V-LP model
named BLIP which achieves state-of-the-art performance on both understanding-
based and generation-based tasks. BLIP trains a multimodal mixture of encoder-
decoder models on large-scale noisy image-text pairs. In the latest work, Li
et al. [Li et al.(2023c)Li, Fan, Hu, Feichtenhofer, and He] proposed a simple
and more efficient method for training CLIP, named Fast Language-Image Pre-
training (FLIP). They randomly mask out and remove a large portion of image
patches during training, allowing more image-text pairs can be used for
training under the same overheads. Our method is designed based on the CLIP
model.

##  3 Methods

###  3.1 Overview

The structure of the DFER-CLIP is shown in Fig. 2. As we can see, the proposed
DFER-CLIP consists of a visual part and a textual part. Regarding the visual
part, based on the CLIP image encoder, we introduce a temporal model, which
consists of several Transformer Encoders, for modelling the temporal facial
features. The final video-level facial features are obtained by a learnable
class token. Regarding the textual part, we utilize descriptions related to
facial behaviour instead of class names for the text encoder. Furthermore, we
adopt the learnable prompt as a context for descriptors of each class, which
does not require experts to design context words and allows the model to learn
relevant context information for each expression during training. We will
first introduce the detail of the DFER-CLIP model and then show how we build
textural descriptions for facial expressions.

###  3.2 DFER-CLIP

Regarding the visual part, on top of the CLIP image encoder, the temporal
model consisting of several ViT encoder layers is adopted for modelling the
temporal relationship. Each encoder also consists of a multi-headed self-
attention and feed-forward network, which are all trained from scratch. The
frame-level features are first learnt by the shared CLIP visual encoder. Then
all of the frame-level features along with an additional learnable class token
will feed into the temporal model, in which the learnable position embedding
is added to encode the temporal position.

Concretely, given a facial video, we sample $T$ frames of the size $H\times W$
so as to form an input $\textit{x}\in\mathbb{R}^{T\times 3\times H\times W}$.
For each frame $\textit{x}_{i}$, we first utilize a shared CLIP image encoder
$f(\cdot)$ to extract feature vectors $\textbf{{f}}^{v}_{i}\in\mathbb{R}^{L}$,
where $i\in\\{1,2,\cdots,T\\}$, $L$ is the length of the feature vectors. Then
$T$ feature vectors will feed into a temporal model $g(\cdot)$ for learning
temporal features, and the final visual representation
$\textbf{{f}}^{V}\in\mathbb{R}^{L}$ can be obtained:

$\displaystyle\textbf{{f}}^{v}_{i}=f(x_{i})$ (1)
$\displaystyle\textbf{{f}}^{V}=g(\textbf{{f}}^{v}_{0}+\textbf{{e}}_{0},\textbf{{f}}^{v}_{1}+\textbf{{e}}_{1},...,\textbf{{f}}^{v}_{T}+\textbf{{e}}_{T})$
(2)

where $\textbf{{f}}^{v}_{0}$ is a special learnable vector for the class
token, e represents a learnable position embedding added to encode the
temporal position.

Regarding the textual part, we utilize descriptions related to facial
behaviour instead of class names for the text encoder. Furthermore, we adopt
the learnable prompt as a context for descriptors of each class. The form of
the prompt is as follows:

$\displaystyle\textbf{{P}}_{k}=[p]_{k}^{1}[p]_{k}^{2}\cdots[p]_{k}^{M}[Tokenizer(description)]_{k}$
(3)

where $M$ is a hyperparameter specifying the number of context tokens,
$k\in\\{1,2,\cdots,C\\}$, $C$ is the number of the facial expression classes,
and each $[p]_{k}^{m}$, $m\in\\{1,2,\cdots,M\\}$, is a vector with the same
dimension as word embeddings. Here, we adopt class-specific context where
context vectors are independent of each description. By forwarding a prompt
$\textbf{{P}}_{k}$ to the text encoder $h(\cdot)$, we can obtain $C$
classification weight vector $\textbf{{f}}^{T}_{k}\in\mathbb{R}^{L}$
representing a visual concept:

$\displaystyle\textbf{{f}}^{T}_{k}=h(\textbf{{P}}_{k})$ (4)

Then the prediction probability can be computed as:

$\displaystyle
p(y=k|x)=\dfrac{exp(cos(\textbf{{f}}^{V},\textbf{{f}}^{T}_{k})/\tau)}{\sum_{k^{\prime}=1}^{C}exp(cos(\textbf{{f}}^{V},\textbf{{f}}^{T}_{k^{\prime}})/\tau)}$
(5)

where $\tau$ is a temperature parameter learned by CLIP and $cos(\cdot,\cdot)$
denotes cosine similarity.

During the training phase, the CLIP text encoder is fixed and we fine-tune the
CLIP image encoder. The temporal model, learnable class token, and learnable
context are all learned from scratch. DFER-CLIP is trained end to end and the
cross-entropy loss is adopted for measuring the distance between the
prediction and the ground-truth labels.

###  3.3 Building Descriptions

Previous research has indicated that human facial expressions share some
common properties while also possessing unique or distinctive features [Zhong
et al.(2012)Zhong, Liu, Yang, Liu, Huang, and Metaxas]. This suggests that
there are similarities in the movements of local facial muscles across
different expressions. For example, the expressions of happiness and surprise
both involve the raising of the eyebrows, and the expressions of sadness and
anger both involve the lowering of the eyebrows and the furrowing of the
forehead. Considering the fact that the CLIP text encoder learned semantic
information from natural language text [Radford et al.(2021)Radford, Kim,
Hallacy, Ramesh, Goh, Agarwal, Sastry, Askell, Mishkin, Clark, et al.], we
propose taking the facial expression action description as the input for the
text encoder. Instead of manually designing the facial expression description,
we prompt a large language model such as ChatGPT to automatically generate
descriptions based on contextual information. We prompt the language model
with the input:
Q: What are useful visual features for the facial expression of {class name}?
A: Some useful visual features for facial expressions of {class name} include:
…
All the generated descriptors of each facial expression class will be combined
to form a comprehensive description. The descriptions for the classification
of the seven basic emotions and of the classes in the datasets we used can be
found in the supplementary material.

##  4 Experiments

###  4.1 Datasets

To evaluate our method, we conduct experiments on three popular DFER
benchmarks.

DFEW [Jiang et al.(2020)Jiang, Zong, Zheng, Tang, Xia, Lu, and Liu] dataset
contains 11,697 in-the-wild video clips, and all the samples have been split
into five same-size parts without overlap. Each video is individually
annotated by ten annotators under professional guidance and assigned to one of
the seven basic expressions (i.e., happiness, sadness, neutral, anger,
surprise, disgust, and fear). The video clips are collected from over 1,500
movies worldwide, covering various challenging interferences, such as extreme
illuminations, occlusions, and variant head pose. The 5-fold cross-validation
is adopted as an evaluation protocol.

FERV39k [Wang et al.(2022b)Wang, Sun, Huang, Liu, Gao, Zhang, Ge, and Zhang]
dataset contains 38,935 in-the-wild video clips, which is currently the
largest in-the-wild DFER dataset. All the video clips were collected from 4
scenarios, which can be further divided into 22 fine-grained scenarios, such
as crime, daily life, speech, and war. Each clip is annotated by 30 individual
annotators and assigned to one of the seven basic expressions as DFEW. Video
clips of all scenes are randomly shuffled and split into training (80%), and
testing (20%) without overlapping.

MAFW [Liu et al.(2022)Liu, Dai, Feng, Wang, Yin, Zeng, and Shan] dataset
contains 10,045 in-the-wild video clips, which is the first large-scale,
multi-modal, multi-label affective database with 11 single-expression
categories (anger, disgust, fear, happiness, sadness, surprise, contempt,
anxiety, helplessness, disappointment, and neutral), 32 multiple-expression
categories, and emotional descriptive texts. The 5-fold cross-validation is
adopted as an evaluation protocol.

###  4.2 Implementation Details

Training Setting: For the visual input, all the fixed 16 frames of sequence in
our experiments follow the sampling strategy in [Jiang et al.(2020)Jiang,
Zong, Zheng, Tang, Xia, Lu, and Liu, Zhao and Liu(2021), Ma et al.(2023)Ma,
Sun, and Li, Li et al.(2023b)Li, Niu, Zhu, and Zhao, Li et al.(2022a)Li, Sui,
Zhu, and Zhao, Li et al.(2023a)Li, Niu, Zhu, and Zhao], and then resized to
$224\times 224$. The random resized crop, horizontal flipping, random
rotation, and color jittering are used to avoid overfitting. Considering the
high computational cost of video data, ViT-B/32-based CLIP [Radford et
al.(2021)Radford, Kim, Hallacy, Ramesh, Goh, Agarwal, Sastry, Askell, Mishkin,
Clark, et al.] is employed in our DFER-CLIP. The maximum number of textual
tokens is 77 (following the official CLIP design), and the temperature hyper-
parameter $\tau$ is set to 0.01. Our models are trained on the Tesla A100 GPU
based on the open-source PyTorch platform. The parameters are optimized via
the SGD optimizer with a mini-batch size of 48. The initial learning rate for
the CLIP image encoder, temporal model, and learnable prompt is set as
$1\times 10^{-5}$, $1\times 10^{-2}$, and $1\times 10^{-3}$, respectively. The
MultiStepLR with milestones of {30, 40} and gamma of 0.1 is employed as a
scheduler to adjust the learning rate. The DFER-CLIP model is trained for 50
epochs in an end-to-end manner. To obtain more stable and reliable results, we
train models three times with different random seeds and then use the average
as the final result.

Evaluation Metrics: Consistent with most previous methods [Jiang et
al.(2020)Jiang, Zong, Zheng, Tang, Xia, Lu, and Liu, Zhao and Liu(2021), Ma et
al.(2023)Ma, Sun, and Li, Li et al.(2023b)Li, Niu, Zhu, and Zhao, Li et
al.(2022a)Li, Sui, Zhu, and Zhao, Li et al.(2023a)Li, Niu, Zhu, and Zhao], the
weighted average recall (WAR, i.e., accuracy) and the unweighted average
recall (UAR, i.e., the accuracy per class divided by the number of classes
without considering the number of instances per class) are utilized for
evaluating the performance of methods.

Number of the Context Prompts |  Depth of the Temporal Model | DFEW | FERV39k | MAFW
---|---|---|---|---
UAR | WAR | UAR | WAR | UAR | WAR
4 | ✗ | 56.91 | 69.01 | 40.26 | 50.96 | 38.03 | 50.62
8 | ✗ | 57.39 | 69.00 | 40.64 | 50.92 | 38.51 | 50.91
16 | ✗ | 57.32 | 68.96 | 40.22 | 50.64 | 37.98 | 50.40
8 | 1 | 59.61 | 71.25 | 41.27 | 51.65 | 39.89 | 52.55
8 | 2 | 58.87 | 70.92 | 40.41 | 51.08 | 39.13 | 52.10
8 | 3 | 58.64 | 70.80 | 40.35 | 50.98 | 38.90 | 51.86
Table 1: Evaluation of the learnable context prompt numbers & the temporal
model depths.

###  4.3 Ablation Analysis

To evaluate the effectiveness of each component in our DFER-CLIP, we conduct
exhaustive ablation analysis on all three benchmarks. Specifically, we will
first demonstrate the effectiveness of the temporal model, the effect of the
different number of context prompts, and the depth of the temporal model. We
will then compare different training strategies, including classifier-based
and text-based (classifier-free). Finally, we will compare different prompts
for the DFER task.

Evaluation of Temporal Model & Context Prompts: As aforementioned, learning
temporal facial features is crucial for video-based FER tasks. From Tab. 1, we
can see that by adopting the temporal model, the UAR performance can be
improved by 2.22%, 0.63%, and 1.38%, and WAR performance can be improved by
2.25%, 0.73%, and 1.64% on DFER, FERV39k, and MAFW datasets, respectively. In
general, deeper models can get better performance, but in our DFER-CLIP, the
best performance is obtained under the one-layer temporal model. This is
because the temporal model is trained from scratch and may overfit if it is
complex. This is also a consideration in the learnable context, in which more
learnable vectors do not improve the results. We believe that increasing the
learnable context number or temporal model depth will cause overfitting on the
training data, resulting in a worse generalization performance on test data.

Strategies | DFEW | FERV39k | MAFW
---|---|---|---
UAR | WAR | UAR | WAR | UAR | WAR
Classifier-based | Linear Probe | 45.46 | 57.40 | 32.47 | 43.72 | 30.74 | 42.95
Fully Fine-Tuning (w/o TM) | 55.70 | 68.41 | 39.64 | 50.77 | 37.53 | 50.48
Fully Fine-Tuning (w/ TM) | 58.28 | 70.27 | 40.55 | 51.22 | 38.39 | 50.92
Text-based (Classifier-free) |  Zero-shot CLIP [Radford et al.(2021)Radford, Kim, Hallacy, Ramesh, Goh, Agarwal, Sastry, Askell, Mishkin, Clark, et al.] | 23.34 | 20.07 | 20.99 | 17.09 | 18.42 | 19.16
Zero-shot FaRL [Zheng et al.(2022)Zheng, Yang, Zhang, Bao, Chen, Huang, Yuan, Chen, Zeng, and Wen] | 23.14 | 31.54 | 21.67 | 25.65 | 14.18 | 11.78
CoOp [Zhou et al.(2022b)Zhou, Yang, Loy, and Liu] | 44.98 | 56.68 | 31.72 | 42.55 | 30.79 | 42.77
Co-CoOp [Zhou et al.(2022a)Zhou, Yang, Loy, and Liu] | 46.80 | 57.52 | 32.91 | 44.25 | 30.81 | 43.23
DFER-CLIP (w/o TM) (Ours) | 57.39 | 69.00 | 40.64 | 50.92 | 38.51 | 50.91
| DFER-CLIP (w TM) (Ours) | 59.61 | 71.25 | 41.27 | 51.65 | 39.89 | 52.55
Table 2: Evaluation of different training strategies. Linear Probe: Adding a linear classifier to the CLIP image encoder and optimizing the parameters of the linear layer with the image encoder fixed. Fully fine-tuning: fine-tuning both the linear layer and the CLIP image encoder. Zero-shot CLIP/FaRL: Using the prompt “an expression of [class]”. CoOp & Co-CoOp: Using the prompt “[learnable prompt] [class]”. The image encoder in FaRL is ViT-B/16-based and the rest are all ViT-B/32-based. TM stands for the temporal model. Prompts | DFEW | FERV39k | MAFW
---|---|---|---
UAR | WAR | UAR | WAR | UAR | WAR
w/o TM | a photo of [Class] | 56.21 | 68.44 | 39.44 | 49.94 | 37.91 | 50.87
an expression of [Class] | 56.16 | 68.73 | 39.28 | 50.41 | 37.71 | 51.08
[Learnable Prompt] [Class] | 57.37 | 68.86 | 40.42 | 50.50 | 38.01 | 50.81
[Learnable Prompt] [Descriptors] | 57.39 | 69.00 | 40.64 | 50.92 | 38.51 | 50.91
w/ TM | [Learnable Prompt] [Class] | 58.28 | 70.29 | 40.60 | 51.18 | 39.64 | 51.21
[Learnable Prompt] [Descriptors] | 59.61 | 71.25 | 41.27 | 51.65 | 39.89 | 52.55
Table 3: Evaluation of different prompts. We set the number of the learnable context prompts as 8 and with or without the temporal model for comparison. Description Position |  Class Specific | DFEW | FERV39k | MAFW
---|---|---|---|---
UAR | WAR | UAR | WAR | UAR | WAR
Middle | ✗ | 58.91 | 71.19 | 40.36 | 51.22 | 39.55 | 52.54
Middle | ✓ | 59.24 | 71.33 | 40.71 | 51.32 | 39.26 | 52.19
End | ✗ | 58.97 | 71.12 | 40.50 | 51.21 | 39.46 | 52.45
End | ✓ | 59.61 | 71.25 | 41.27 | 51.65 | 39.89 | 52.55
Table 4: Evaluation of different settings for learnable context. All the
methods adopt 8 learnable context tokens, with the temporal model.

Evaluation of Different Training Strategies: The final recognition result in
our DFER-CLIP is obtained by computing the similarities between visual
embedding and all the textual class-level embeddings, which is different from
the conventional classifier-based training strategies. Therefore, we compare
DFER-CLIP with Linear Probe and Fully Fine-tuning methods. The results in Tab.
2 show that our method outperforms Fully Fine-tuning in UAR by 3.91%, 1.63%,
and 2.36%, and in WAR by 2.84%, 0.88%, and 2.07% on DFER, FERV39k, and MAFW
datasets, respectively. Even without the temporal model, our method is better
than all the classifier-based methods. We also add a temporal model for the
Fully Fine-tuning strategy, the results demonstrate our method is still
superior to it. We also compare our method with zero-shot CLIP [Radford et
al.(2021)Radford, Kim, Hallacy, Ramesh, Goh, Agarwal, Sastry, Askell, Mishkin,
Clark, et al.] and zero-shot FaRL [Zheng et al.(2022)Zheng, Yang, Zhang, Bao,
Chen, Huang, Yuan, Chen, Zeng, and Wen], in which FaRL is pre-trained on the
large-scale visual-language face data. The results in Tab. 2 show that fine-
tuning the image encoder can improve the performance remarkably. CoOp [Zhou et
al.(2022b)Zhou, Yang, Loy, and Liu] and Co-CoOp [Zhou et al.(2022a)Zhou, Yang,
Loy, and Liu] are all using the learnable context, in which the Co-CoOp also
add projected image features into the context prompt. The comparison between
our method without a temporal model and these two methods in Tab. 2 show the
effectiveness of our strategy.

Methods | DFEW | FERV39k | MAFW
---|---|---|---
UAR | WAR | UAR | WAR | UAR | WAR
C3D [Tran et al.(2015)Tran, Bourdev, Fergus, Torresani, and Paluri] | 42.74 | 53.54 | 22.68 | 31.69 | 31.17 | 42.25
P3D [Qiu et al.(2017)Qiu, Yao, and Mei] | 43.97 | 54.47 | 23.20 | 33.39 | - | -
I3D-RGB [Carreira and Zisserman(2017)] | 43.40 | 54.27 | 30.17 | 38.78 | - | -
3D ResNet18 [Hara et al.(2018)Hara, Kataoka, and Satoh] | 46.52 | 58.27 | 26.67 | 37.57 | - | -
R(2+1)D18 [Tran et al.(2018)Tran, Wang, Torresani, Ray, LeCun, and Paluri] | 42.79 | 53.22 | 31.55 | 41.28 | - | -
ResNet18-LSTM [He et al.(2016)He, Zhang, Ren, and Sun, Hochreiter and Schmidhuber(1997)] | 51.32 | 63.85 | 30.92 | 42.95 | 28.08 | 39.38
ResNet18-ViT [He et al.(2016)He, Zhang, Ren, and Sun, Dosovitskiy et al.(2021)Dosovitskiy, Beyer, Kolesnikov, Weissenborn, Zhai, Unterthiner, Dehghani, Minderer, Heigold, Gelly, et al.] | 55.76 | 67.56 | 38.35 | 48.43 |  35.80 | 47.72
EC-STFL [Jiang et al.(2020)Jiang, Zong, Zheng, Tang, Xia, Lu, and Liu] [MM’20] | 45.35 | 56.51 | - | - | - | -
Former-DFER [Zhao and Liu(2021)] [MM’21] | 53.69 | 65.70 | 37.20 | 46.85 | 31.16 | 43.27
NR-DFERNet [Li et al.(2022a)Li, Sui, Zhu, and Zhao] [arXiv’22] | 54.21 | 68.19 | 33.99 | 45.97 | - |
DPCNet [Wang et al.(2022c)Wang, Sun, Song, Gao, Huang, Chen, Ge, and Zhang] [MM’22] | 57.11 | 66.32 | - | - | - | -
T-ESFL [Liu et al.(2022)Liu, Dai, Feng, Wang, Yin, Zeng, and Shan] [MM’22] | - | - | - | - | 33.28 |  48.18
EST [Liu et al.(2023)Liu, Wang, Feng, Zhang, Chen, and Zhan] [PR’23] | 53.94 | 65.85 | - | - | - | -
LOGO-Former [Ma et al.(2023)Ma, Sun, and Li] [ICASSP’23] | 54.21 | 66.98 | 38.22 | 48.13 | - | -
IAL [Li et al.(2023b)Li, Niu, Zhu, and Zhao] [AAAI’23] | 55.71 | 69.24 | 35.82 | 48.54 | - | -
CLIPER [Li et al.(2023a)Li, Niu, Zhu, and Zhao] [arXiv’23] |  57.56 |  70.84 |  41.23 |  51.34 | - | -
M3DFEL [Wang et al.(2023)Wang, Li, Wu, Shen, Liu, Ding, and Zhou] [CVPR’23] | 56.10 | 69.25 | 35.94 | 47.67 | - | -
AEN [Lee et al.(2023)Lee, Shin, Ku, and Ko] [CVPRW’23] | 56.66 | 69.37 | 38.18 | 47.88 | - | -
DFER-CLIP (Ours) | 59.61 | 71.25 | 41.27 | 51.65 | 39.89 | 52.55
Table 5: Comparison with the state-of-the-art methods.

Evaluation of Different Prompts: In a supervised learning setting, V-LP models
differ from conventional classification models in that they can be prompted to
design classifier-free models for prediction. This makes prompt engineering a
critical aspect of adapting V-LP models for downstream tasks. We compare our
method with two kinds of manually designed prompts: “a photo of [class]” and
“an expression of [class]”. Results in Tab. 3 show that our method outperforms
manually designed prompts on both DFEW and FERV39k datasets, but has a
slightly lower WAR than the prompt of “an expression of [class]” on the MAFW
dataset. We note that the MAFW dataset comprises eleven facial expression
classes and that the video samples are imbalanced, with the proportions of the
“contempt”, “helplessness”, and “disappointment” expressions accounting for
only 2.57%, 2.86% and 1.98%, respectively. The results in Tab. 3 demonstrate
that the learning-based context consistently achieves the best WAR results.
Furthermore, our method outperforms using the prompt of the class name with
the learnable context approach, which indicates the effectiveness of using
descriptions.

In our DFER-CLIP, the description is at the end of the prompt and the
learnable context is class-specific, namely, each description will have its
own learned context. We also conduct experiments with different settings: a)
put the description in the middle of the learnable context, and b) all the
descriptions share the same learned context. The results on three datasets are
shown in Tab. 4. The results indicate that positioning the description tokens
to the end gains more improvement. Moreover, for the end position, adopting
the class-specific is always better than a shared context prompt, achieving
the best results.

###  4.4 Comparison with State-of-the-Art Methods

In this section, we compare our results with several state-of-the-art methods
on the DFEW, FERV39k, and MAFW benchmarks. Consistent with previous methods
[Jiang et al.(2020)Jiang, Zong, Zheng, Tang, Xia, Lu, and Liu, Zhao and
Liu(2021), Ma et al.(2023)Ma, Sun, and Li, Li et al.(2023b)Li, Niu, Zhu, and
Zhao, Li et al.(2023a)Li, Niu, Zhu, and Zhao, Liu et al.(2022)Liu, Dai, Feng,
Wang, Yin, Zeng, and Shan], the experiments on DFEW and MAFW are conducted
under 5-fold cross-validation and use training and test set on FERV39k.
Furthermore, we train models three times with different random seeds and then
use the average for more stable and reliable results. The comparative
performance in Tab. 5 demonstrates that the proposed DFER-CLIP outperforms the
compared methods both in UAR and WAR. Specifically, compared with the previous
best results, our method shows a UAR improvement of 2.05%, 0.04%, and 4.09%
and a WAR improvement of 0.41%, 0.31%, and 4.37% on DFEW, FERV39k, and MAFW,
respectively. It should be pointed out that FERV39k is the current largest
DFER benchmark with 38,935 videos. Given this substantial scale, making
significant enhancements becomes a formidable task.

##  5 Conclusion

This paper presents a novel visual-language model called DFER-CLIP for in-the-
wild dynamic facial expression recognition. In the visual part, based on the
CLIP image encoder, a temporal model consisting of several Transformer
Encoders was introduced for modelling the temporal facial expression features.
In the textual part, the expression descriptors related to facial behaviour
were adopted as the textual input to capture the relationship between facial
expressions and their underlying facial behaviour, in which the expression
descriptors were generated by large language models like ChatGPT. The
learnable contexts for these descriptors were also designed to help the model
learn relevant context information for each expression during training.
Extensive experiments demonstrate the effectiveness of each component in DFER-
CLIP. Moreover, the proposed method achieves state-of-the-art results on three
benchmarks.

## Acknowledgements

Zengqun Zhao is funded by Queen Mary Principal’s PhD Studentships, and this
work is supported by the EU H2020 AI4Media No. 951911 project. We thank Niki
Foteinopoulou and James Oldfield for the valuable discussion.

## References

  * [Baddar and Ro(2019)] Wissam J Baddar and Yong Man Ro.  Mode variational lstm robust to unseen modes of variation: Application to facial expression recognition.  In _AAAI_ , pages 3215–3223, 2019.
  * [Bishay et al.(2019)Bishay, Palasek, Priebe, and Patras] Mina Bishay, Petar Palasek, Stefan Priebe, and Ioannis Patras.  Schinet: Automatic estimation of symptoms of schizophrenia from facial behaviour analysis.  _IEEE Transactions on Affective Computing_ , 12(4):949–961, 2019.
  * [Carreira and Zisserman(2017)] Joao Carreira and Andrew Zisserman.  Quo vadis, action recognition? a new model and the kinetics dataset.  In _CVPR_ , pages 6299–6308, 2017.
  * [Chen et al.(2016)Chen, Chen, Chi, and Fu] Junkai Chen, Zenghai Chen, Zheru Chi, and Hong Fu.  Facial expression recognition in video with multiple feature fusion.  _IEEE Transactions on Affective Computing_ , 9(1):38–50, 2016.
  * [Corneanu et al.(2016)Corneanu, Simón, Cohn, and Guerrero] Ciprian Adrian Corneanu, Marc Oliu Simón, Jeffrey F Cohn, and Sergio Escalera Guerrero.  Survey on rgb, 3d, thermal, and multimodal approaches for facial expression recognition: History, trends, and affect-related applications.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 38(8):1548–1568, 2016.
  * [Darwin and Prodger(1998)] Charles Darwin and Phillip Prodger.  _The expression of the emotions in man and animals_.  Oxford University Press, 1998.
  * [Dosovitskiy et al.(2021)Dosovitskiy, Beyer, Kolesnikov, Weissenborn, Zhai, Unterthiner, Dehghani, Minderer, Heigold, Gelly, et al.] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _ICLR_ , 2021.
  * [Duric et al.(2002)Duric, Gray, Heishman, Li, Rosenfeld, Schoelles, Schunn, and Wechsler] Zoran Duric, Wayne D Gray, Ric Heishman, Fayin Li, Azriel Rosenfeld, Michael J Schoelles, Christian Schunn, and Harry Wechsler.  Integrating perceptual and cognitive modeling for adaptive and intelligent human-computer interaction.  _Proceeding of the IEEE_ , 90(7):1272–1289, 2002.
  * [Ekman(1971)] Paul Ekman.  Universals and cultural differences in facial expressions of emotion.  In _Nebraska symposium on motivation_. University of Nebraska Press, 1971.
  * [Foteinopoulou and Patras(2022)] Niki Maria Foteinopoulou and Ioannis Patras.  Learning from label relationships in human affect.  In _MM_ , pages 80–89, 2022.
  * [Foteinopoulou et al.(2021)Foteinopoulou, Tzelepis, and Patras] Niki Maria Foteinopoulou, Christos Tzelepis, and Ioannis Patras.  Estimating continuous affect with label uncertainty.  In _ACII_ , pages 1–8, 2021.
  * [Hara et al.(2018)Hara, Kataoka, and Satoh] Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh.  Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet?  In _CVPR_ , pages 6546–6555, 2018.
  * [He et al.(2016)He, Zhang, Ren, and Sun] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _CVPR_ , pages 770–778, 2016.
  * [Hochreiter and Schmidhuber(1997)] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  _Neural Computation_ , 9(8):1735–1780, 1997.
  * [Jia et al.(2021)Jia, Yang, Xia, Chen, Parekh, Pham, Le, Sung, Li, and Duerig] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _ICML_ , pages 4904–4916, 2021.
  * [Jiang et al.(2020)Jiang, Zong, Zheng, Tang, Xia, Lu, and Liu] Xingxun Jiang, Yuan Zong, Wenming Zheng, Chuangao Tang, Wanchuang Xia, Cheng Lu, and Jiateng Liu.  Dfew: A large-scale database for recognizing dynamic facial expressions in the wild.  In _MM_ , pages 2881–2889, 2020.
  * [Ju et al.(2022)Ju, Han, Zheng, Zhang, and Xie] Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie.  Prompting visual-language models for efficient video understanding.  In _ECCV_ , pages 105–124, 2022.
  * [Lee et al.(2023)Lee, Shin, Ku, and Ko] Bokyeung Lee, Hyunuk Shin, Bonhwa Ku, and Hanseok Ko.  Frame level emotion guided dynamic facial expression recognition with emotion grouping.  In _CVPRW_ , pages 5680–5690, 2023.
  * [Lee et al.(2019)Lee, Kim, Kim, Park, and Sohn] Jiyoung Lee, Seungryong Kim, Sunok Kim, Jungin Park, and Kwanghoon Sohn.  Context-aware emotion recognition networks.  In _ICCV_ , pages 10143–10152, 2019.
  * [Li et al.(2022a)Li, Sui, Zhu, and Zhao] Hanting Li, Mingzhe Sui, Zhaoqing Zhu, and Feng Zhao.  Nr-dfernet: Noise-robust network for dynamic facial expression recognition.  _arXiv preprint arXiv:2206.04975_ , 2022a.
  * [Li et al.(2023a)Li, Niu, Zhu, and Zhao] Hanting Li, Hongjing Niu, Zhaoqing Zhu, and Feng Zhao.  Cliper: A unified vision-language framework for in-the-wild facial expression recognition.  _arXiv preprint arXiv:2303.00193_ , 2023a.
  * [Li et al.(2023b)Li, Niu, Zhu, and Zhao] Hanting Li, Hongjing Niu, Zhaoqing Zhu, and Feng Zhao.  Intensity-aware loss for dynamic facial expression recognition in the wild.  In _AAAI_ , pages 67–75, 2023b.
  * [Li et al.(2022b)Li, Li, Xiong, and Hoi] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In _ICML_ , pages 12888–12900, 2022b.
  * [Li and Deng(2020)] Shan Li and Weihong Deng.  Deep facial expression recognition: A survey.  _IEEE Transactions on Affective Computing_ , 13(3):1195–1215, 2020.
  * [Li et al.(2023c)Li, Fan, Hu, Feichtenhofer, and He] Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He.  Scaling language-image pre-training via masking.  In _CVPR_ , 2023c.
  * [Liu et al.(2021)Liu, Zhang, and Zhou] Daizong Liu, Hongting Zhang, and Pan Zhou.  Video-based facial expression recognition using graph convolutional networks.  In _ICPR_ , pages 607–614, 2021.
  * [Liu et al.(2022)Liu, Dai, Feng, Wang, Yin, Zeng, and Shan] Yuanyuan Liu, Wei Dai, Chuanxu Feng, Wenbin Wang, Guanghao Yin, Jiabei Zeng, and Shiguang Shan.  Mafw: A large-scale, multi-modal, compound affective database for dynamic facial expression recognition in the wild.  In _MM_ , 2022.
  * [Liu et al.(2023)Liu, Wang, Feng, Zhang, Chen, and Zhan] Yuanyuan Liu, Wenbin Wang, Chuanxu Feng, Haoyu Zhang, Zhe Chen, and Yibing Zhan.  Expression snippet transformer for robust video-based facial expression recognition.  _Pattern Recognition_ , 138:109368, 2023.
  * [Ma et al.(2022)Ma, Yang, Wang, Zhang, and Xie] Chaofan Ma, Yuhuan Yang, Yanfeng Wang, Ya Zhang, and Weidi Xie.  Open-vocabulary semantic segmentation with frozen vision-language models.  In _BMVC_ , pages 1–15, 2022.
  * [Ma et al.(2023)Ma, Sun, and Li] Fuyan Ma, Bin Sun, and Shutao Li.  Logo-former: Local-global spatio-temporal transformer for dynamic facial expression recognition.  In _ICASSP_ , pages 1–5, 2023.
  * [Menon and Vondrick(2023)] Sachit Menon and Carl Vondrick.  Visual classification via description from large language models.  In _ICLR_ , 2023.
  * [Mittal et al.(2020)Mittal, Bhattacharya, Chandra, Bera, and Manocha] Trisha Mittal, Uttaran Bhattacharya, Rohan Chandra, Aniket Bera, and Dinesh Manocha.  M3er: Multiplicative multimodal emotion recognition using facial, textual, and speech cues.  In _AAAI_ , pages 1359–1367, 2020.
  * [Nag et al.(2022)Nag, Zhu, Song, and Xiang] Sauradip Nag, Xiatian Zhu, Yi-Zhe Song, and Tao Xiang.  Zero-shot temporal action detection via vision-language prompting.  In _ECCV_ , pages 681–697, 2022.
  * [Ni et al.(2022)Ni, Peng, Chen, Zhang, Meng, Fu, Xiang, and Ling] Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xiang, and Haibin Ling.  Expanding language-image pretrained models for general video recognition.  In _ECCV_ , pages 1–18, 2022.
  * [Oldfield et al.(2023)Oldfield, Tzelepis, Panagakis, Nicolaou, and Patras] James Oldfield, Christos Tzelepis, Yannis Panagakis, Mihalis A Nicolaou, and Ioannis Patras.  Parts of speech-grounded subspaces in vision-language models.  _arXiv preprint arXiv:2305.14053_ , 2023.
  * [Patashnik et al.(2021)Patashnik, Wu, Shechtman, Cohen-Or, and Lischinski] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski.  Styleclip: Text-driven manipulation of stylegan imagery.  In _CVPR_ , pages 2085–2094, 2021.
  * [Qiu et al.(2017)Qiu, Yao, and Mei] Zhaofan Qiu, Ting Yao, and Tao Mei.  Learning spatio-temporal representation with pseudo-3d residual networks.  In _ICCV_ , pages 5533–5541, 2017.
  * [Radford et al.(2021)Radford, Kim, Hallacy, Ramesh, Goh, Agarwal, Sastry, Askell, Mishkin, Clark, et al.] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _ICML_ , pages 8748–8763, 2021.
  * [Rasheed et al.(2023)Rasheed, Khattak, Maaz, Khan, and Khan] Hanoona Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan.  Fine-tuned clip models are efficient video learners.  In _CVPR_ , pages 6545–6554, 2023.
  * [Tran et al.(2015)Tran, Bourdev, Fergus, Torresani, and Paluri] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri.  Learning spatiotemporal features with 3d convolutional networks.  In _CVPR_ , pages 4489–4497, 2015.
  * [Tran et al.(2018)Tran, Wang, Torresani, Ray, LeCun, and Paluri] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri.  A closer look at spatiotemporal convolutions for action recognition.  In _CVPR_ , pages 6450–6459, 2018.
  * [Wang et al.(2023)Wang, Li, Wu, Shen, Liu, Ding, and Zhou] Hanyang Wang, Bo Li, Shuang Wu, Siyuan Shen, Feng Liu, Shouhong Ding, and Aimin Zhou.  Rethinking the learning paradigm for dynamic facial expression recognition.  In _CVPR_ , pages 17958–17968, 2023.
  * [Wang et al.(2022a)Wang, Song, Tao, Liotta, Yang, Li, Gao, Sun, Ge, Zhang, et al.] Yan Wang, Wei Song, Wei Tao, Antonio Liotta, Dawei Yang, Xinlei Li, Shuyong Gao, Yixuan Sun, Weifeng Ge, Wei Zhang, et al.  A systematic review on affective computing: Emotion models, databases, and recent advances.  _Information Fusion_ , 2022a.
  * [Wang et al.(2022b)Wang, Sun, Huang, Liu, Gao, Zhang, Ge, and Zhang] Yan Wang, Yixuan Sun, Yiwen Huang, Zhongying Liu, Shuyong Gao, Wei Zhang, Weifeng Ge, and Wenqiang Zhang.  Ferv39k: a large-scale multi-scene dataset for facial expression recognition in videos.  In _CVPR_ , pages 20922–20931, 2022b.
  * [Wang et al.(2022c)Wang, Sun, Song, Gao, Huang, Chen, Ge, and Zhang] Yan Wang, Yixuan Sun, Wei Song, Shuyong Gao, Yiwen Huang, Zhaoyu Chen, Weifeng Ge, and Wenqiang Zhang.  Dpcnet: Dual path multi-excitation collaborative network for facial expression representation learning in videos.  In _MM_ , pages 101–110, 2022c.
  * [Wilhelm(2019)] Torsten Wilhelm.  Towards facial expression analysis in a driver assistance system.  In _FG_ , pages 1–4, 2019.
  * [Zhao and Liu(2021)] Zengqun Zhao and Qingshan Liu.  Former-dfer: Dynamic facial expression recognition transformer.  In _MM_ , pages 1553–1561, 2021.
  * [Zhao et al.(2021a)Zhao, Liu, and Wang] Zengqun Zhao, Qingshan Liu, and Shanmin Wang.  Learning deep global multi-scale and local attention features for facial expression recognition in the wild.  _IEEE Transactions on Image Processing_ , 30:6544–6556, 2021a.
  * [Zhao et al.(2021b)Zhao, Liu, and Zhou] Zengqun Zhao, Qingshan Liu, and Feng Zhou.  Robust lightweight facial expression recognition network with label distribution training.  In _AAAI_ , pages 3510–3519, 2021b.
  * [Zheng et al.(2022)Zheng, Yang, Zhang, Bao, Chen, Huang, Yuan, Chen, Zeng, and Wen] Yinglin Zheng, Hao Yang, Ting Zhang, Jianmin Bao, Dongdong Chen, Yangyu Huang, Lu Yuan, Dong Chen, Ming Zeng, and Fang Wen.  General facial representation learning in a visual-linguistic manner.  In _CVPR_ , pages 18697–18709, 2022.
  * [Zhong et al.(2012)Zhong, Liu, Yang, Liu, Huang, and Metaxas] Lin Zhong, Qingshan Liu, Peng Yang, Bo Liu, Junzhou Huang, and Dimitris N Metaxas.  Learning active facial patches for expression analysis.  In _CVPR_ , pages 2562–2569, 2012.
  * [Zhou et al.(2022a)Zhou, Yang, Loy, and Liu] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Conditional prompt learning for vision-language models.  In _CVPR_ , pages 16816–16825, 2022a.
  * [Zhou et al.(2022b)Zhou, Yang, Loy, and Liu] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Learning to prompt for vision-language models.  _IJCV_ , 130(9):2337–2348, 2022b.

## Supplementary Material

###  A Facial Expression Descriptions

Expressions | Descriptors
---|---
happiness |  | a smiling mouth, raised cheeks, wrinkled eyes, and arched eyebrows.
---
sadness |  | tears, a downward turned mouth,
---
drooping upper eyelids,and a wrinkled forehead.
neutral |  | relaxed facial muscles, a straight mouth,
---
a smooth forehead, and unremarkable eyebrows.
anger |  | furrowed eyebrows, narrow eyes,
---
tightened lips, and flared nostrils.
surprise |  | widened eyes, an open mouth, raised eyebrows,
---
and a frozen expression.
disgust |  | a wrinkled nose, lowered eyebrows,
---
a tightened mouth, and narrow eyes.
fear |  | raised eyebrows, parted lips,
---
a furrowed brow, and a retracted chin.
contempt |  | one side of its mouth raised,
---
one eyebrow lower and one raised,
narrowed eyes, and a raised chin.
anxiety |  | a tensed forehead, tightly pressed lips,
---
pupil dilation, and tensed facial muscles.
helplessness |  | drooping eyebrows, a downward gaze,
---
a downturned mouth, and lacking expression.
disappointment |  | a downturned mouth, lowered eyebrows,
---
narrowed eyes, and a sighing face.

###  B Additional Ablation Studies

The descriptions used in our method are class-level instead of sample-level,
and our point is to create discriminative class-level text embedding for
supervising the visual part. However, we should admit that the unique
description has limitations. To this end, we also conducted experiments with
prompt ensembles; the results are shown in Tab. A and indicate that employing
more descriptions is beneficial.

In our method, we prompt a large language model such as ChatGPT to
automatically generate useful visual descriptors for each facial expression
with the process described in Section 3.3. Furthermore, we also investigated
the effect of the different descriptions. Specifically, we study the effect of
the different numbers of the facial-action-unit-level descriptors. We select
the top-2, top-4, and top-6 descriptors generated from the LLMs – the results
are shown in Tab. B. We believe fewer descriptors cause the lack of
correlation among different expressions and more descriptors result in
diminishing the discriminative features.

Table A: Evaluation of the prompt ensembling. descriptions number | DFEW | FERV39k | MAFW
---|---|---|---
UAR | WAR | UAR | WAR | UAR | WAR
1 | 59.61 | 71.25 | 41.27 | 51.65 | 39.89 | 52.55
2 | 60.42 | 72.01 | 40.84 | 51.60 | 40.42 | 52.92
Table B: Evaluation of the different number of descriptors. descriptiors number | DFEW | FERV39k | MAFW
---|---|---|---
UAR | WAR | UAR | WAR | UAR | WAR
2 | 59.65 | 71.91 | 40.79 | 51.57 | 39.65 | 52.26
4 | 59.61 | 71.25 | 41.27 | 51.65 | 39.89 | 52.55
6 | 59.59 | 71.87 | 40.54 | 51.46 | 39.25 | 52.37

###  C Confusion Matrix on Three Benchmarks

Figure A: Confusion matrix on 5-fold DFEW, FERV39k test set and 5-fold MAFW.