# Improving Personalisation in Valence and Arousal Prediction using Data
Augmentation

Munachiso Nwadike1 Jialin Li1, Hanan Salam1
1 Department of Computer Science, New York University Abu Dhabi, UAE This work
was supported by the Center for Artificial Intelligence and Robotics at NYUAD

###### Abstract

In the field of emotion recognition and Human-Machine Interaction (HMI),
personalised approaches have exhibited their efficacy in capturing individual-
specific characteristics and enhancing affective prediction accuracy. However,
personalisation techniques often face the challenge of limited data for target
individuals. This paper presents our work on an enhanced personalisation
strategy, that leverages data augmentation to develop tailored models for
continuous valence and arousal prediction. Our proposed approach, Distance
Weighting Augmentation (DWA), employs a weighting-based augmentation method
that expands a target individual’s dataset, leveraging distance metrics to
identify similar samples at the segment-level. Experimental results on the
MuSe-Personalisation 2023 Challenge dataset demonstrate that our method
significantly improves the performance of features sets which have low
baseline performance, on the test set. This improvement in poor-performing
features comes without sacrificing performance on high-performing features. In
particular, our method achieves a maximum combined testing CCC of 0.78,
compared to the reported baseline score of 0.76 (reproduced at 0.72). It also
achieved a peak arousal and valence scores of 0.81 and 0.76, compared to
reproduced baseline scores of 0.76 and 0.67 respectively. Through this work,
we make significant contributions to the advancement of personalised affective
computing models, enhancing the practicality and adaptability of data-level
personalisation in real world contexts.

##  I INTRODUCTION

Automatic emotion recognition (ER) has been an active area of research in the
field of Human-Machine Interaction (HMI) during the past decade. ER utility
spans a wide range of applications in areas like healthcare [22, 7, 45], and
robotics [51, 54]. Research approaches in automatic ER has focused on either
the categorical model [59, 23] or the dimensional model [66, 67]. The
categorical model focuses on the recognition of discrete emotion classes
(anger, disgust, fear, happiness, sadness, surprise, contempt). This model of
ER has been widely explored in affective computing for its simplicity, due to
the ease with which humans can readily interpret distinct emotional classes.
The dimensional model, on the other hand, focuses on the prediction of
continuous emotional attributes representing them in a continuous dimensional
space such as arousal-valence space. The dimensional model is more suitable
for modeling the complex emotions, which cannot easily be fit into distinct
categories, but rather vary along a spectrum. Both models of emotions have
been investigated and contrasted in the affective computing literature [44,
10, 69].

ER presents various complex challenges due to the diverse nature of emotional
expressions across individuals and cultures. Each individual expresses emotion
slightly differently [26, 63], in a range of modalities such as voice audio,
video facial expression, and physiological stress signals such as heart rate
(BPM) [70, 71]. The question naturally arises, as to whether personalisation
strategies can be used with multimodal supervised learning techniques to
improve state-of-the-art performance on dimensional emotion recognition
problems. Efforts such as the 4th Multimodal Sentiment Analysis Challenge and
Workshop: Mimicked Emotions, Humour and Personalisation (MuSe 2023) [14] have
sought to address this problem. Specifically, the MuSe-personalisation sub-
challenge focuses on investigating customised approaches to enhance the
accuracy of machine learning for customised predictions of emotional valence
and arousal. A multimodal dataset, Ulm-Trier Social Stress Test (Ulm-TSST)
[70], has been provided, together with seven pre-extracted feature sets.

Early research in affective machine learning has yielded significant insights
into the potential of personalisation for predicting emotional states [29,
75]. Techniques such as transfer learning [28, 47, 78, 9, 34, 46, 49], user-
specific training [80, 60, 62, 31, 1, 61], group-specific training [75, 29,
63, 27] and multitask learning [37, 25, 72, 50, 13] have demonstrated notable
improvements in estimation performance [33]. The MuSe-Personalisation
challenge baseline paper [14], for instance, adopted the transfer learning
approach. This approach involves training a generic model on global data and
fine-tuning it using individualised data specific to each user. However,
achieving effective personalisation in real-world settings remains a
considerable challenge due to data constraints [33]. The limited data
availability and quality for target users poses significant challenges to the
performance of the personalised models. To address these constraints, previous
research has explored various techniques, including transfer learning [48],
weighting-based approaches [12, 15], feature augmentation [53, 81, 52], and
generative-based models [39, 5, 35, 8, 79, 77]. Among them, weighting-based
methods offer a unique advantage by allowing the fine-tuning of weights as
hyperparameters during augmentation, providing added flexibility for model
training. However, traditional approaches in weighting-based methods have not
fully tapped into their potential. Most methods overlook the opportunity to
utilise a two-stage procedure involving training a generic model on global
non-personalised data. Consequently, they do not fully exploit the abundant
information contained within this global data.

Our work aims to contribute to and advance the field of multimodal
personalised affective computing, by leveraging the insights of weighting-
based methods. Building upon the foundations laid by previous studies,
particularly the curriculum learning and grouping methods introduced in [57],
we propose a distance weighting augmentation (DWA) approach. DWA may be
interpreted as existing at the intersection of weighting-based methods and
transfer learning. By experimenting on the Ulm-TSST dataset using this
approach, we seek to shed light on its potential and explore its effectiveness
in enhancing affective understanding. The main contributions of this paper can
be summarised as follows:

  * •

We highlight the importance of personalisation across 8 different features in
4 different modalities, by showing the difference in model performance with
and without personalisation.

  * •

We introduce the DWA method, as a data augmentation strategy to enhance model
performance in valence and/or arousal prediction, showing how DWA aids
performance of features which are poorly performing.

  * •

Through experimentation on competing distance metrics, we illustrate how
different feature modalities respond differently to DWA, and how valence and
arousal signals may be improved independently of one another.

##  II RELATED WORK

This work addresses multi-modal emotion recognition and personalisation, and
it is based on sample re-weighting and individual model fine-tuning
approaches. In this section, we give a brief review of these techniques.

###  II-A Generic Multimodal Emotion Recognition

Emotion recognition refers to the field developing technical skills such as
facial recognition, speech recognition, voice recognition, deep learning, and
pattern recognition for recognizing human emotions. Among all the emotion
recognition tasks, arousal-valence estimation has gained a lot of attention
due to its ability to capture a wide range of emotional states in a nuanced
manner. As essential dimensions in affect modeling, they represent emotional
polarity and intensity, respectively. The estimation of these natural emotions
can be derived from biosignals collected via wearable sensors [64], text [40],
speech [16], video [56] and so on.

Emotion recognition from multi-modal signals has gained considerable scholarly
attention in recent years [2]. Generic multi-modal approaches combine more
than two different modalities like speech [42], visual [41], audio [17], text
[4], and physiological signals [68] to recognize emotions. They extract
informative features from each modality and fuse them, either through feature-
level fusion or decision-level fusion after separate modeling per modality.
The recent advancements in deep learning and multi-modal emotion recognition
have significantly improved the accuracy of fine-grained valence-arousal
estimation [30]. These include but are not limited to the studies on audio-
visual scenarios [43, 36], and audio-textual scenarios [17]. For example, the
multi-modal fusion framework proposed by [43] has enhanced valence-arousal
estimation by leveraging complementary audio-visual modalities. The framework
employs a joint cross-attentional architecture that calculates attention
weights based on the correlation between combined audio-visual features and
individual audio and video features, thus effectively extracts salient
features across modalities while preserving intra-modal relationships. Another
audio-textual framework explored multiple temporal models to incorporate
sequential information in the visual signals with various ensemble strategies
at decision level [36]. On an audio-textual level, the method proposed in [17]
developed a distilled encoder using audio-text features, significantly
improving valence prediction while addressing the size constraints of large
self-supervised learning (SSL) models. However, these generic models make
general assumptions and cannot capture individual differences in emotional
expression well. Consequently, personalised models that leverage on individual
characteristics seem promising.

###  II-B Personalised Multi-modal Emotion Recognition

Early works in personalising affective computing models present from two
angles, data-level techniques and model-level techniques [33]. Model-level
techniques focus on directly incorporating personalisation in model
architecture or training [9, 47, 25], while data-level techniques operate on
data before feeding it into the model [57, 46, 80, 75]. Prior works have
notably advances by accounting individual differences in social behavior and
emotional states through tailoring the generic affective models to individual-
specific and group-specific levels. In [82], for example, a multi-task
learning approach was proposed for personalised emotion recognition from
physiological signals. By modeling relationships between signals and
personality in a hypergraph framework, emotion recognition was formulated as
related tasks for multiple features, improving personalised emotion prediction
accuracy.

####  II-B1 Individual Model Transfer Learning

Individual transfer learning is a two-step process that is widely adapted in
personalised deep learning algorithms. It begins with the training of a deep
learning model on a generic dataset, followed by fine-tuning the model on a
dataset that is specific to an individual. This approach strikes a balance
between generalisability and specialisation, ensuring that the datasets are
utilised to their fullest extent. The research by [57] introduces the concept
of “late shaping,” which aims to enhance the model’s exposure to individual-
specific data during the training phase using transfer learning. It fine-tunes
a pre-trained model using the data of the individual of interest. Similarly,
[28] proposes to adapt the pre-trained general model for individual-level
humor recognition by manually fine-tuning the generic model on an individual
humor dataset. In the context of facial expression recognition, [9] also
applies individual model transfer learning to facilitate the recognition of
non-universal expressions. It builds on the feature extraction capabilities of
a pre-trained convolutional encoder, and augments the existing encoder with an
additional convolutional layer. This layer is specifically designed to learn
and capture the unique representations inherent to the target dataset, thereby
enhancing the model’s performance in specialised facial expression recognition
tasks.

####  II-B2 Sample Re-weighting

Re-weighting methods primarily attach weights to training samples, or to
personalised model predictions that come from similar user datasets, based on
their similarity to the target user’s data. By effectively utilising the
available dataset, the model becomes better at capturing the unique
characteristics of the target user, leading to more accurate predictions. In
[57], “data grouping” was proposed to augment the individual’s small MNIST
dataset with similar data points from the global MNIST dataset, with its
similarity identified using an autoencoder.

Motivated by such existing investigations into personalisation strategies, we
propose to investigate the potential of the personalised multi-modal emotion
recognition using a deep network trained under a combined re-weighting and
transfer learning scheme.

Algorithm 1 Distance Weighting Augmentation

Input: Global Data $D_{G}$, Data of Individual $D_{I}$
Output: Augmented data of individual $I$

1:All segment lengths == winlen

2:

3:Generate AugPool $AP$$\triangleright$ Build Augmentation Pool from $D_{G}$

4:$AP:=\\{\\}$

5:for each individual $D_{G_{i}}\in D_{G}$ do

6: for each sample $S\in D_{G_{i}}$ do $\triangleright$ each sample is a
segment

7: $AP:=AP\cup S$

8: end for

9:end for

10:

11:Perform Augmentation

12:for each individual $D_{I_{i}}\in D_{I}$ do

13: $D_{I_{i}aug}:=D_{I_{i}}$

14: for each sample $D_{I_{i}W^{T}}\in D_{I_{i}}$ do

15: $W^{T}:=D_{I_{i}W^{T}}$

16: for each sample $S\in AP$ do

17: if $\text{distance\\_metric}==\text{``cosine''}$ then

18: $distance:=\text{cosine\\_distance}(S,W^{T})$

19: $\ \ \ \ \ \ \ \ \ \ \ \ \
=\displaystyle\sum_{j=1}^{winlen}[1-\frac{\overrightarrow{S_{j}}\cdot\overrightarrow{W^{T}_{j}}}{\parallel\overrightarrow{S_{j}}\parallel^{2}\parallel\overrightarrow{W_{j}^{T}}\parallel^{2}}]$

20: else if $\text{distance\\_metric}==\text{``centroid DP''}$ then

21: $\overrightarrow{W}:=\overline{W^{T}}\ ;\ \overrightarrow{S}$:=
$\overline{S}$ $\triangleright$ centroid of both $\triangleright$ segments

22: $distance:=-\overrightarrow{W}\cdot\overrightarrow{S}$

23: else if $\text{distance\\_metric}==\text{``centroid L2''}$ then

24: $\overrightarrow{W}:=\overline{W^{T}}\ ;\
\overrightarrow{S}:=\overline{S}$

25: $distance:=\parallel\overrightarrow{W}-\overrightarrow{S}\parallel^{2}$

26: end if

27: end for

28: $D_{aug}:=n\ samples\ S\in AP$

29: $with\ \min distance$

30: $D_{I_{i}aug}:=D_{I_{i}aug}\cup D_{aug}$

31: end for

32:end for

##  III PROPOSED METHODOLOGY

In this section, we present a data-level augmentation approach, aimed at
enhancing personalised models’ accuracy in predicting continuous valence and
arousal under data label constraints. We provide a detailed explanation of our
approach, organised into two sections. First, in section III-A, we introduce
the distance weighting framework. Then, in section III-B, we discuss the
criteria employed for the distance metrics used to calculate the weights
during augmentation.

###  III-A Distance Weighting Augmentation

Our objective is to enhance emotion recognition performance, on each unique
individual, with respect to our evaluation metric. Overall, we adopt a
transfer learning technique as illustrated in Figure 1. Recognizing the
importance of the size and quality of the individual dataset $D_{I_{i}}$, we
propose the Distance Weighting Augmentation (DWA) method. DWA seeks to augment
the limited $D_{I_{i}}$, by incorporating similar samples from the global
dataset $D_{G}$. Psuedo-code is provided in Algorithm 1 for a comprehensive
breakdown of DWA.

As illustrated in Figure 2, DWA begins by constructing an augmentation pool
$AP$ from $D_{G}$. $AP$ comprises all samples (i.e., segments) $S$, from all
individuals in the training and development sets. “Training and development
sets” here refer to $D_{G}$, and not to the training and development sets
Train_I and Devel_I in $D_{I_{i}}$ (see table II). DWA then iterates through
all segments in the target $D_{I_{i}}$. For each target segment $W^{T}$, DWA
computes the distance to $W^{T}$ from all segments in $AP$. The nearest $n$
segments, measured via a defined distance metric (cf. section ), are selected,
i.e given a weight of 1, and appended to $D_{I_{i}}$. All other segments in
the augmentation pool are momentarily given a weight of 0, till the next
iteration. The result is $D_{I_{i}{aug}}$, an augmented dataset, for each
individual in the test set. DWA then returns $D_{I_{i}aug}$, which consists of
the union of $D_{I_{i}}$ with these $n$ samples for all segments in
$D_{I_{i}}$. $D_{I_{i}aug}$ is then utilised for fine-tuning a generic model
$M_{G}$, to obtain a personalised model $M_{I_{i}}$ for the target individual
in question. All individuals in the testing set make use of the same
augmentation pool, from which we draw samples with replacement.

The choice of the distance metric influences the calculated distance values
and rankings of the $n$ nearest segments. The subsequent section provides a
detailed description of these metrics.

Figure 1: Our personalisation framework leverages transfer learning together
with distance weighting augmentation.

###  III-B Distance Metrics

We employed three distance metrics during augmentation, to compute the
similarity between two time segments of equal length: (1) centroid L2
distance, (2) centroid dot product, and (3) cosine distance. These were chosen
as well studied distance measures suitable to our problem statement. For a
time segment $W^{T}$, we have a set of features
$\\{W^{T}_{j}\\}_{j=1}^{winlen}$, where the number of features depends on the
segment length. For instance, ${winlen}=10$ implies that 10 timestamps are
grouped together within each segment, with 1 feature vector for each
timestamp. By $\overline{W^{T}}$, we denote the average of all ${winlen}$
feature vectors belonging to segment $W^{T}$. This is equivalent to computing
the centroid feature vector, and it is utilised in Algorithm 1. For two time
segments $W^{1}$ and $W^{2}$, our three distance metrics can be defined as
follows:

####  III-B1 Centroid L2 Distance

Centroid L2 Distance calculates segment similarity by measuring the L2
distance between the mean vectors of two given segments. The mean vector
within a given segment represents the centroid, or average feature over that
time frame. L2 distance, also denoted as euclidean distance, has been proven
effective in the literature for measuring similarity between feature vectors
in various domains, such as face recognition [38], speaker voice similarity
[65, 55], and text similarity [74, 24] systems. A smaller Euclidean distance
between two vectors suggests a higher degree of similarity, with a value of
zero indicating that the vectors are identical.

$\lVert\overline{W^{1}}-\overline{W^{2}}\rVert_{2}$ (1)

####  III-B2 Centroid Dot Product (centroid DP) Distance

Similar to the Centroid L2 calculation, we utilise Centroid Dot Product
distance to compute the distance between the centroids of two respective
segments. However, as dot product is inherently a measure of similarity, we
take the negative of the dot product as a measure of distance. The
effectiveness of dot product as a measure of similarity is time-tested, not
the least by virtue of its widespread usage in transformer models [73].

$-\overline{W^{1}}\cdot\overline{W^{2}}$ (2)

####  III-B3 Cosine Distance

Cosine distance is a robust and efficient metric that excels in high-
dimensional applications such as text mining, information retrieval, and
computer vision [74]. Originating from the widely-used cosine distance measure
in vector space [32, 3], it quantifies the distance between pair-wise vectors
or segments. Cosine distance is scale-invariant, focusing on the orientation
of vectors rather than their magnitude, ensuring its effectiveness for
computing distance for sparse and high-dimensional feature vectors in Ulm-TSST
dataset. Cosine distance has a bounded range from 0 to 1, with 0 indicating
identical vectors.

$\displaystyle\sum_{j}\left(1-\frac{W^{1}_{j}\cdot W^{2}_{j}}{\lVert
W^{1}_{j}\rVert_{2}-\lVert W^{2}_{j}\rVert_{2}}\right)$ (3)

Figure 2:  DWA generates an dataset $D_{I_{i}{aug}}$ from the original dataset
$D_{I_{i}}$, by generating an augmentation pool, and selecting most similar
samples from that pool, for target segment $W_{T}$.

##  IV EXPERIMENTAL SETUP

In this section, we describe the experimental setup including the baseline we
compare our approach to, dataset, loss function, evaluation metric, and
experimental details.

###  IV-A Baseline Models

The baseline work from [14], much like our work, uses a two-stage transfer
learning approach. In the first stage, it trains a generic model $M_{G}$,
using the generic dataset $D_{G}$. It then copies the generic model multiple
times for each individual in the testing set, and trains separate personalised
models using each $D_{I_{i}}$. Similarly to our approach, the training and
development parts, Train_I and Devel_I, of the Test individuals, are used to
personalise the respective generic models. Our approach seeks to build upon
this baseline using DWA, whereby each Train_${I_{i}}$ is augmented with
similar samples from $D_{G}$.

###  IV-B Dataset

We utilise the Ulm-TSST dataset from the MuSe 2023 Personalisation sub-
challenge [14]. This dataset comprises video, audio, written transcripts, and
physiological signals obtained from approximately 5 minutes of solo
conversations with 69 different individuals. In each solo conversation, a
single individual stands in front of a camera and speaks continuously in the
German language. The dataset labels consist of continuous valence and arousal
labels for small incremental timestamps through the length of each
conversation. It is vital to note that the dataset is divided into training,
development, and testing sets based on the individuals themselves, rather than
time segments. That is, 41 individuals are assigned to the training set, 14
individuals to the development set, and 14 individuals to the testing set. In
this paper we refer to the 55 training and development individuals as $D_{G}$,
while the 14 testing individuals are represented as
$D_{I}=\\{D_{I_{i}}\\}_{i=1}^{14}$.

Our methodological objective is to accurately predict the arousal and valence
values on the 14 individuals for the testing set. Notably, while labels are
available for the entire 5 minutes of conversation for both the training and
development individuals, we are only provided labels for 2 minutes of
conversation of each testing individual. That is, for each testing individual,
the first labeled minute is for personalised training and the second minute is
for personalised development/validation. This allows us to conduct
personalised fine-tuning for each testing individual $D_{I_{i}}$ after
training a generic model on $D_{G}$. A summary of the available labels for the
train-test split is provided in table II. Notably, the testing portion of the
testing set lacks labels, and our objective in the MuSe-Personalisation
challenge is to accurately predict the valence and arousal labels for this
unlabeled part of the testing set.

Features.  For the different modalities, pre-extracted feature sets were made
available by [14]. These are summarised in table I. Among the video features,
Fau most directly encodes facial action units [83], while FaceNet512 features
are associated with deep facial features [58]. While ViT [20] features encode
information about visual semantics, these semantics may not be facial
information. This is because the ViT model is pre-trained on Imagenet [18]
using DINO [11] self-supervision. Among the audio features, eGeMAPS [21] are
based on a set of interpretable speech parameters, which can be computed in
closed form from the data. Each eGeMAPS dimension corresponds to a parameter
such as pitch, shimmer, loudness, or alpha-ratio. DeepSpectrum and Wav2Vec
features are extracted via neural network. However, DeepSpectrum [6] features
are extracted from spectrogram images using a convolutional neural network
(CNN), while Wav2Vec [76] features are extracted from audio waveforms using a
combination of CNNs and transformers. Bert-4 [19] features are extracted using
a BERT transformer, which has been pretrained on the German language. The
Biosignals features consist of machine-recorded readings of electrocardiograms
(ECG), Respiration (RESP) and heart rate beats per minute (BPM).

TABLE I: Understanding the features in our dataset.

Modality | Feature | Dim. | Information
---|---|---|---
Video |  FaceNet512 [58] | 512 | Facial structure
Fau111https://py-feat.org/pages/au_reference.html [83] | 20 | Facial actions
ViT [11] | 384 | Visual semantics
(not necessarily facial)
Audio | eGeMAPS[21] | 88 | Speech features
(frequency, amplitude, etc.)
DeepSpectrum [6] | 1024 | Spectrogram image features
Wav2Vec [76] | 1024 | Audio wave features
Text |  Bert-4222https://huggingface.co/dbmdz/bert-base-german-cased [19] | 768 | Transcription semantics
Physiological | Biosignals | 3 | Heart rate/electrical
activity, breathing

TABLE II: Understanding the structure of the dataset labels provided. The
number indicate how many individuals belong to the respective tasks.

$D_{G}$ | $\\{D_{I_{i}}\\}$
---|---
Train_G (41) | Devel_G (14) | Test (14)
Train_I | Devel_I | Test
✓ | ✓ | ✓ | ✓ | ✗

###  IV-C Loss Function and Evaluation Metric

In line with [14], we adopt the Concordance Correlation Coefficient (CCC) as
the loss function to train our model. It is typically used to assess how well
two sets of observations agree with each other in terms of both their mean
values and their dispersion, i.e. accuracy and precision. As a product of the
Pearson Correlation Coefficient (PCC) and the Bias Correction Factor (BCF),
the resulting value of CCC ranges between $-1$ and $1$, where a value of $1$
indicates perfect agreement, $0$ indicates no agreement, and $-1$ indicates
perfect disagreement. The CCC loss is formulated as follows:

$\centering Loss=1-CCC\@add@centering$ (4) $\centering
CCC=\frac{2\rho\sigma_{x}\sigma_{y}}{\sigma_{x}^{2}+\sigma_{y}^{2}+(\mu_{x}-\mu_{y})^{2}}\@add@centering$
(5)

where $\mu_{\hat{y}}$ and $\mu_{y}$ represent the mean of the prediction
$\hat{y}$ and the label $y$, respectively. Similarly, $\sigma_{\hat{y}}$ and
$\sigma_{y}$ correspond to their standard deviations. $\rho$ denotes the PCC
between $\hat{y}$ and $y$. It measures the strength and direction of the
linear relationship between $X$ and $Y$. A value of $\rho$ close to $1$
indicates a strong positive linear relationship, while a value close to $-1$
indicates a strong negative linear relationship. A value of $0$ indicates no
linear relationship. In addition,
$\frac{2\sigma_{x}\sigma_{y}}{\sigma_{x}^{2}+\sigma_{y}^{2}+(\mu_{x}-\mu_{y})^{2}}$
denotes the BCF. It corrects for any deviation of the means of $X$ and $Y$
from a perfect linear relationship. It ensures that the CCC accounts for the
accuracy component as well. BCF ranges between $0$ and $1$, where a value of
$1$ indicates perfect accuracy.

TABLE III: Performance of Generic models on $D_{I}$, as a benchmark prior to
personalisation. Yellow: maximum and minimum scores. Bold: Best performing
feature in the specific modality. Bold: Best performance overall the features.

Feature | Arousal | Valence | Combined
---|---|---|---
Devel_I | Test | Devel_I | Test | Test
FaceNet512 | 0.43 | 0.49 | 0.76 | 0.56 | 0.52
Fau | 0.21 | -0.00 | 0.52 | 0.47 | 0.23
ViT | 0.61 | 0.51 | 0.40 | 0.40 | 0.45
eGeMAPS | 0.58 | 0.56 | 0.58 | 0.49 | 0.52
DeepSpectrum | 0.54 | 0.42 | 0.57 | 0.45 | 0.44
Wav2Vec | 0.52 | 0.54 | 0.46 | 0.42 | 0.48
Bert-4 | 0.51 | 0.49 | 0.34 | 0.25 | 0.37
Biosignals | -0.14 | -0.14 | 0.23 | 0.19 | 0.03

###  IV-D Experimental Details

Baseline Models. The baseline work of [14], while insightful, only reports
results for 6 audio and video features, namely, Fau, FaceNet512, ViT, eGeMAPS,
DeepSpectrum, and Wav2Vec. We conduct all our experiments on these features,
but also include results for the Bert-4, and Biosignals features. We also
employ the same generic model from the baseline methodology for consistent
comparison.

Hyperparameters. DWA was used to augment each individual dataset and obtain
$D_{I_{\text{aug}}}$. For fair comparison with the baseline, we set consistent
ranges for any tuned hyerparameters such as learning rates, number of RNN
layers, and model dimensions, during hyperparameter search. All other
hyperparameters, including early stopping patience, segment length, and hop
length, are kept unchanged from the baseline method in [14]. However, we
perform the hyperparameter search for all 8 features in the dataset, expanding
upon the 6 reported features of the [14].

Our experiments sought to understand the impact of different choices of
distance metrics and numbers of augmentation samples per segment. We trained
models using data augmented with one of the three distance metrics: centroid
DP, cosine, and centroid L2. We also explored varying values for number of
augmentation samples per segment $n$, specifically $n=1$ up to $3$.

TABLE IV:  Performance of baseline personalised models on $D_{I}$, to
highlight the effect of personalisation. Yellow: greatest increases due to
personalisation w.r.t generic models. Bold: Best performing feature in the
specific modality. Bold: Best performance overall the features.

Feature | Arousal | Valence | Combined
---|---|---|---
Devel_I | Test | Devel_I | Test | Test
FaceNet512 | 0.88 | 0.68 | 0.89 | 0.69 | 0.69
Fau | 0.88 | 0.79 | 0.90 | 0.78 | 0.79
ViT | 0.85 | 0.64 | 0.67 | 0.55 | 0.60
eGeMAPS | 0.70 | 0.53 | 0.68 | 0.59 | 0.56
DeepSpectrum | 0.90 | 0.55 | 0.72 | 0.56 | 0.55
Wav2Vec | 0.89 | 0.53 | 0.90 | 0.80 | 0.66
Bert-4 | 0.90 | 0.76 | 0.87 | 0.60 | 0.68
Biosignals | 0.78 | 0.61 | 0.45 | 0.35 | 0.48

TABLE V: Performance of DWA with exploration of $n$ and distance metrics. Green: DWA outperforms the baseline on Devel_I. Blue: DWA outperforms baseline on Test set. Yellow: used in place of blue if improvement is $>0.1$ Feature | Distance Metric | Arousal | Valence | Combined
---|---|---|---|---
n=1 | n=2 | n=3 | Test | n=1 | n=2 | n=3 | Test | Test
FaceNet512 | Centroid_DP | 0.75 | 0.81 | 0.71 | 0.68 | 0.92 | 0.92 | 0.91 | 0.78 | 0.72
Centroid_L2 | 0.83 | 0.68 | 0.82 | 0.69 | 0.94 | 0.95 | 0.94 | 0.79 | 0.74
Cosine | 0.83 | 0.83 | 0.72 | 0.65 | 0.90 | 0.92 | 0.93 | 0.83 | 0.74
Fau | Centroid_DP | 0.72 | 0.76 | 0.81 | 0.60 | 0.76 | 0.77 | 0.80 | 0.64 | 0.67
Centroid_L2 | 0.79 | 0.73 | 0.71 | 0.55 | 0.90 | 0.74 | 0.66 | 0.83 | 0.69
Cosine | 0.74 | 0.83 | 0.82 | 0.71 | 0.77 | 0.79 | 0.72 | 0.65 | 0.74
ViT | Centroid_DP | 0.84 | 0.84 | 0.92 | 0.71 | 0.78 | 0.54 | 0.54 | 0.65 | 0.67
Centroid_L2 | 0.89 | 0.78 | 0.85 | 0.64 | 0.66 | 0.69 | 0.52 | 0.54 | 0.59
Cosine | 0.85 | 0.87 | 0.93 | 0.70 | 0.51 | 0.59 | 0.54 | 0.52 | 0.61
eGeMAPS | Centroid_DP | 0.75 | 0.69 | 0.75 | 0.71 | 0.68 | 0.81 | 0.81 | 0.68 | 0.70
Centroid_L2 | 0.74 | 0.66 | 0.74 | 0.09 | 0.73 | 0.59 | 0.69 | 0.46 | 0.27
Cosine | 0.82 | 0.82 | 0.62 | 0.09 | 0.65 | 0.60 | 0.64 | 0.59 | 0.34
DeepSpectrum | Centroid_DP | 0.90 | 0.68 | 09.73 | 0.59 | 0.78 | 0.76 | 0.67 | 0.38 | 0.49
Centroid_L2 | 0.85 | 0.69 | 0.66 | 0.58 | 0.84 | 0.78 | 0.76 | 0.63 | 0.60
Cosine | 0.79 | 0.79 | 0.60 | 0.28 | 0.78 | 0.80 | 0.75 | 0.72 | 0.50
Wav2Vec | Centroid_DP | 0.80 | 0.81 | 0.70 | 0.52 | 0.72 | 0.58 | 0.59 | 0.70 | 0.61
Centroid_L2 | 0.74 | 0.76 | 0.60 | 0.54 | 0.50 | 0.72 | 0.60 | 0.65 | 0.59
Cosine | 0.73 | 0.80 | 0.83 | 0.60 | 0.84 | 0.70 | 0/64 | 0.68 | 0.64
Bert-4 | Centroid_DP | 0.82 | 0.79 | 0.88 | 0.67 | 0.82 | 0.59 | 0.78 | 0.60 | 0.63
Centroid_L2 | 0.85 | 0.89 | 0.77 | 0.57 | 0.78 | 0.53 | 0.76 | 0.46 | 0.51
Cosine | 0.90 | 0.82 | 0.91 | 0.73 | 0.80 | 0.82 | 0.75 | 0.47 | 0.60
Biosignals | Centroid_DP | 0.28 | 0.19 | 0.50 | 0.50 | 0.42 | 0.26 | 0.29 | 0.14 | 0.32
Centroid_L2 | 0.35 | 0.12 | 0.09 | 0.20 | 0.51 | 0.77 | 0.64 | 0.69 | 0.44
Cosine | 0.67 | 0.48 | 0.53 | 0.56 | 0.74 | 0.67 | 0.61 | 0.45 | 0.50
TABLE VI: Results from late fusion, for features where DWA showed improvements
in ablation. Yellow: improvements over the baseline in Blue.

Feature | Arousal | Valence | Combined
---|---|---|---
Devel_I | Test | Devel_I | Test | Test
Baseline(s)
FaceNet512 \+ ViT | 0.90 | 0.74 | 0.82 | 0.65 | 0.70
Reproduced (FaceNet512 \+ eGeMAPS) | 0.85 | 0.67 | 0.90 | 0.76 | 0.72
Reported (FaceNet512 \+ eGeMAPS) | 0.92 | 0.75 | 0.86 | 0.78 | 0.76
Late Fusion, Proportional Weighting
ViT\+ eGeMAPS \+ DWA | 0.92 | 0.44 | 0.86 | 0.77 | 0.61
FaceNet512 \+ eGeMAPS+DWA | 0.89 | 0.61 | 0.95 | 0.81 | 0.71
ViT+Bert-4+DWA | 0.93 | 0.76 | 0.88 | 0.68 | 0.72
ViT+DeepSpectrum+DWA | 0.94 | 0.71 | 0.88 | 0.74 | 0.73
Facenet+ViT+DWA | 0.92 | 0.76 | 0.94 | 0.80 | 0.78

##  V RESULTS AND DISCUSSION

We organise our discussion into 2 sections. Firstly, we seek to highlight the
experimental effects of personalisation. To do this, we reproduce the generic
model and personalised model results of the baseline paper on our dataset, and
compare their respective performances on the testing set (section V-A). This
is particularly important in the context of personalisation, as the generic
model results on the testing set were not provided in the baseline work [14].
Secondly, we perform a series of experiments to evaluate the performance of
the DWA personalisation approach (section V-B). Our objective is to show that
augmentation with individual-specific samples from our augmentation pool can
enhance the performance of personalised model training on valence and arousal
prediction for each testing individual.

###  V-A The Effect of Personalisation

Generic Model Performance Evaluation. The findings presented in table III
serve to demonstrate the need for personalisation, by demonstrating scores of
the generic model $M_{G}$, obtained without the use of personalisation, in
order to have a reference for any effects that may result from a personalised
training step. We observe that nearly all models have combined CCC scores
below 0.53. The highest performing features are FaceNet512 and eGeMAPS, with
combined testing scores of 0.52. In particular, eGeMAPS scored highest on
arousal with a testing score of 0.56. While FaceNet512 had highest such score
of 0.56 on valence. This suggests that without personalisation, a model
trained on a general dataset will be less effective on a given individual. The
lowest testing CCC scores are obtained by Fau, and Biosignals features, which
score 0.23 and 0.03 respectively. The Biosignals feature scores lowest in both
cases of arousal and valence, obtaining -0.14, and 0.19 respectively. For each
timestamp, the Biosignals feature vector comprises 3 measurements: 1 for ECG,
1 for resps, and 1 for BPM. We hypothesise that the low dimensionality of the
Biosignals feature may be one additional reason for its extraordinarily low
performance. However, all in all, the generic model results serve as a
backdrop to compare personalised model scores with and without DWA.

Baseline Personalisation Performance Evaluation. The development and testing
set CCC scores obtained from personalised training using the baseline approach
[14] across all features, are presented in table IV. The Fau feature, which
previously scored 0.23 (combined CCC) in our reproduced results, without
personalisation, now obtains the highest CCC across all features, of 0.79,
owed to respective arousal and valence scores of 0.79 and 0.78 respectively.
The Biosignals feature still has the lowest score, at 0.48, but dramatically
outperforms its corresponding generic model score of 0.03. Biosignals has the
5th highest score in arousal of 0.61, but scores only 0.35 in valence
performance. In our experiments, while FaceNet512 and eGeMAPS improved from a
testing CCC of 0.52 to 0.69, and 0.56 respectively, they are by no means our
best performing respective audio and video features. These were instead Fau
and Wav2Vec, which improved from 0.23 to 0.79, and from 0.48 to 0.66
respectively, thanks to personalisation. In the case of Wav2Vec, which was the
best performing audio feature, it scores 0.80 in valence, making it the
highest performing feature in valence. However, it scored 0.53 on arousal.

###  V-B Performance Evaluation of DWA

The 8 different feature types utilised in our experiments, each capture
different characteristics of a target individual. For example, FaceNet512
features capture facial characteristics, while Fau features capture facial
actions, and eGeMAPS capture variations in prosody. Yet fundamentally, DWA
augments a given segment with other segments from an augmentation pool, based
on measurements of distance. Therefore, the efficacy of DWA for a given
feature depends on the meaning of the distance metric in that feature space.
For example, FaceNet512 features encode a compact representation of facial
features in the Euclidean space by which distances correspond to a measure of
face similarity, thus DWA may result in augmentation with similar faces. Thus,
in order to better understand the performance of DWA, we present our
experiments across all features in table V, recognising that DWA may be more
suitable for some features than others. Table V also captures the effects of
alternative distance metrics, the Centroid DP, Centroid L2, and Cosine
metrics. We present varying values of number of augmentation samples per
segment between $n=1$ and $n=3$ as well.

Arousal vs. Valence. A key observation from table V, is that DWA may result in
improvement in arousal or valence tasks independently. Our experiments suggest
that an improvement in valence prediction due to DWA, may not guarantee an
improvement in arousal prediction, and vice versa. However, for feature-task
combinations where DWA does result in improvement, such improvements are
generally seen among all distance metrics, and for most values of $n$, as can
be observed by the neat patterns of green highlighting seen in the table. For
instance, while the DeepSpectrum feature improves on valence to a score of
0.72, it fails to improve on arousal, peaking at 0.59. On the other hand, the
eGemMAPS feature is able to improve on both arousal and valence, reaching
scores of 0.71 and 0.68 respectively.

Video & Physiological Features. Among the video features, only FaceNet512 and
ViT features seem to benefit from DWA, with the latter seeing the most
improvement compared to the baseline model. For FaceNet512, the combined
testing CCC peaks at 0.74 with DWA, compared to 0.69 without DWA (baseline
model). Much of this improvement is due to the valence prediction, which
increases from a testing CCC of 0.69 to 0.83. Arousal prediction does not see
similar improvement for FaceNet512. With ViT features, the final combined
testing CCC increases from 0.60 to 0.67. This is due to significant
improvements in both arousal prediction from 0.64 to 0.71, and in valence
prediction from 0.55 to 0.65 with respect to the baseline. The Fau features,
which had the highest performance without DWA, saw a drop in performance with
DWA. This suggests that DWA can only be used as a tool to improve the
performance of features which are lagging behind on specific tasks. Indeed,
this hypothesis is further supported by the improvement in Biosignals features
testing CCC. With the use of DWA a compelling performance increase can be seen
from 0.35 to 0.69 for the arousal prediction task.

Audio Features. Results with audio features are also presented in table V. The
performance increase due to DWA is most noticeable using the eGeMAPS feature.
For the eGeMAPS, DWA brings about an increase in performance from 0.56 to
0.70. This is due to corresponding improvements in testing arousal and valence
CCC, which increased from 0.50 and 0.59 respectively, to 0.71 and 0.68
respectively. This combined increase in eGeMAPS performance with to DWA is not
only the highest among the audio features, but among all features which were
tested. DWA also results in an improved performance in the DeepSpectrum
feature from 0.56 to 0.63. However, the improvement is largely limited to the
valence prediction task. As a result, since the arousal prediction of
DeepSpectrum does not appear to benefit from DWA, the combined CCC score only
improves from 0.55 to 0.60.

Number of Augmentations per segment. We ran experiments on the value of $n$ to
determine the effect of the number of augmentation samples on model
performance. What we find, as per table V, is that the ideal number of
augmentation samples varies depending on the feature and distance metric. For
example, consider the case of valence prediction using DeepSpectrum features
and cosine distance. The performance on Devel_I increases as n is increasing
from 1 to 2, and decreases when n is further increased to 3. This suggests a
diminishing return pattern Such is also the case for Biosignals feature. On
the other hand, performance on FaceNet512, with cosine feature, seems to be
steadily increasing with n. This suggests that the value of n must be tuned
uniquely as a hyperparamter for each feature-distance-metric combination.

Late Fusion. Finally, we adopt the same late fusion strategy deployed in our
baseline, for comparison purposes. For late fusion, we utilise proportional
weighting, whereby two sets of predictions are weighted based on their
performance on the development set. For fair comparison, we include the
reported results for late fusion from the baseline, of FaceNet512 and eGeMAPS
features. However, in reproducing these results, we utilise consistent
hyperparameter search between all methods, causing the reproduced late fusion
score to be different from the reported baseline. The results are presented in
table VI.

In the baseline work, the best performing feature fusion is of FaceNet512 and
eGeMAPS, reaching a testing CCC of 0.76. Our reproduced score is slightly
lower, at 0.72. However, our DWA method results in a score of 0.78 when fusing
ViT and eGeMAPS features. This improvement is significant, as without DWA, the
late fusion of ViT and eGeMAPS would only result in a CCC testing score of
0.70. As expected, the fusion of eGeMAPS with ViT scores higher than with
FaceNet512, as the latter did not see signficant benefit in arousal prediction
from DWA, as earlier discussed. However, this but reinforces our hypothesis,
that DWA can be quite useful as a boost to personalisation performance for
features which are not performing well.

##  VI CONCLUSIONS AND FUTURE WORKS

In this paper, we introduced the Distance Weighting Augmentation (DWA) method
for enhancing predictions of valence and arousal, at the data level. In simple
terms, DWA expands a personalised dataset with samples from a global dataset
that are the most similar. We implement DWA on all 8 features provided in our
target dataset, with the best performance observed from a fusion of ViT and
eGeMAPS predictions. Among the 3 distance metrics employed for similarity
measurement, all proved to be effective for either valence or arousal, or
both, in 4 out of 8 features. Our findings indicate that DWA is most useful in
improving personalisation performance when utilising features which have poor
performance for a particular task. Late fusion of the DWA-trained ViT and
eGeMAPS models, yields a combined CCC score of 0.78, indicating the
effectiveness of DWA, and opening the door for further research into data-
level personalisation.

Future work could expand upon DWA in 3 key ways. Firstly, we can explore
further distance metrics to see if they could improve over the 3 used thus
far. Secondly, it may be beneficial to explore fractional weighting of similar
samples. This could be implemented via sample weighting, whereby we would
alternate between normal samples and augmentation samples within a training
loop. In effect, this could allow us to weight samples in the interval [0,1]
rather than relying solely on binary weights. Thirdly, it is important to
investigate the reasons for which DWA brings limited performance to features
which already performed well. In effect, modifications to the algorithm
itself, may help in expanding upon its applicability to different feature
sets. This last point will be easier to explore as we seek to apply DWA to
additional datasets for dimensional emotion prediction.

## References

  * [1] A. R. Abbasi, N. V. Afzulpurkar, and T. Uno.  Towards emotionally-personalized computing: Dynamic prediction of student mental states from self-manipulatory body movements.  In 2009 International Conference on Emerging Technologies, pages 235–240. IEEE, 2009.
  * [2] N. Ahmed, Z. Al Aghbari, and S. Girija.  A systematic survey on multimodal emotion recognition using learning algorithms.  Intelligent Systems with Applications, 17:200171, 2023.
  * [3] F. S. Al-Anzi and D. AbuZeina.  Toward an enhanced arabic text classification using cosine similarity and latent semantic indexing.  Journal of King Saud University-Computer and Information Sciences, 29(2):189–195, 2017.
  * [4] N. Alswaidan and M. E. B. Menai.  A survey of state-of-the-art approaches for emotion recognition in text.  Knowledge and Information Systems, 62:2937–2987, 2020.
  * [5] N. Alyuz, E. Okur, E. Oktay, U. Genc, S. Aslan, S. E. Mete, B. Arnrich, and A. A. Esme.  Semi-supervised model personalization for improved detection of learner’s emotional engagement.  In Proceedings of the 18th ACM International Conference on Multimodal Interaction, pages 100–107, 2016.
  * [6] S. Amiriparian, M. Gerczuk, S. Ottl, N. Cummins, M. Freitag, S. Pugachevskiy, A. Baird, and B. Schuller.  Snore sound classification using image-based deep spectrum features.  2017\.
  * [7] D. Ayata, Y. Yaslan, and M. E. Kamasak.  Emotion recognition from multimodal physiological signals for emotion aware healthcare systems.  Journal of Medical and Biological Engineering, 40:149–157, 2020.
  * [8] P. Barros, G. Parisi, and S. Wermter.  A personalized affective memory model for improving emotion recognition.  In International Conference on Machine Learning, pages 485–494. PMLR, 2019.
  * [9] P. Barros and A. Sciutti.  Ciao! a contrastive adaptation mechanism for non-universal facial expression recognition.  In 2022 10th International Conference on Affective Computing and Intelligent Interaction (ACII), pages 1–8. IEEE, 2022.
  * [10] O. Bruna, H. Avetisyan, and J. Holub.  Emotion models for textual emotion classification.  In Journal of physics: conference series, volume 772, page 012063. IOP Publishing, 2016.
  * [11] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin.  Emerging properties in self-supervised vision transformers.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650–9660, 2021.
  * [12] R. Chattopadhyay, Q. Sun, W. Fan, I. Davidson, S. Panchanathan, and J. Ye.  Multisource domain adaptation and its application to early detection of fatigue.  ACM Transactions on Knowledge Discovery from Data (TKDD), 6(4):1–26, 2012.
  * [13] V. V. Chithrra Raghuram, H. Salam, J. Nasir, B. Bruno, and O. Celiktutan.  Personalized productive engagement recognition in robot-mediated collaborative learning.  In Proceedings of the 2022 International Conference on Multimodal Interaction, pages 632–641, 2022.
  * [14] L. Christ, S. Amiriparian, A. Baird, A. Kathan, N. Müller, S. Klug, C. Gagne, P. Tzirakis, E.-M. Meßner, A. König, et al.  The muse 2023 multimodal sentiment analysis challenge: Mimicked emotions, cross-cultural humour, and personalisation.  arXiv preprint arXiv:2305.03369, 2023.
  * [15] W.-S. Chu, F. De la Torre, and J. F. Cohn.  Selective transfer machine for personalized facial expression analysis.  IEEE transactions on pattern analysis and machine intelligence, 39(3):529–545, 2016.
  * [16] S. Cunningham, H. Ridley, J. Weinel, and R. Picking.  Audio emotion recognition using machine learning to support sound design.  In Proceedings of the 14th International Audio Mostly Conference: A Journey in Sound, pages 116–123, 2019.
  * [17] D. de Oliveira, N. R. Prabhu, and T. Gerkmann.  Leveraging semantic information for efficient self-supervised emotion recognition with audio-textual distilled models.  arXiv preprint arXiv:2305.19184, 2023.
  * [18] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [19] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [20] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [21] F. Eyben, K. R. Scherer, B. W. Schuller, J. Sundberg, E. André, C. Busso, L. Y. Devillers, J. Epps, P. Laukka, S. S. Narayanan, et al.  The geneva minimalistic acoustic parameter set (gemaps) for voice research and affective computing.  IEEE transactions on affective computing, 7(2):190–202, 2015.
  * [22] M. A. Hasnul, N. A. A. Aziz, S. Alelyani, M. Mohana, and A. A. Aziz.  Electrocardiogram-based emotion recognition systems and their applications in healthcare—a review.  Sensors, 21(15):5015, 2021.
  * [23] A. R. Hazourli, A. Djeghri, H. Salam, and A. Othmani.  Multi-facial patches aggregation network for facial expression recognition and facial regions contributions to emotion display.  Multimedia Tools and Applications, 80:13639–13662, 2021.
  * [24] A. Huang et al.  Similarity measures for text document clustering.  In Proceedings of the sixth new zealand computer science research student conference (NZCSRSC2008), Christchurch, New Zealand, volume 4, pages 9–56, 2008.
  * [25] N. Jaques, S. Taylor, E. Nosakhare, A. Sano, and R. Picard.  Multi-task learning for predicting health, stress, and happiness.  In NIPS Workshop on Machine Learning for Healthcare, 2016.
  * [26] M. Jensen.  Personality traits and nonverbal communication patterns.  Int’l J. Soc. Sci. Stud., 4:57, 2016.
  * [27] A. Kasparova, O. Celiktutan, and M. Cukurova.  Inferring student engagement in collaborative problem solving from visual cues.  In Companion Publication of the 2020 International Conference on Multimodal Interaction, pages 177–181, 2020.
  * [28] A. Kathan, S. Amiriparian, L. Christ, A. Triantafyllopoulos, N. Müller, A. König, and B. W. Schuller.  A personalised approach to audiovisual humour recognition and its individual-level fairness.  In Proceedings of the 3rd International on Multimodal Sentiment Analysis Workshop and Challenge, pages 29–36, 2022.
  * [29] J. Kim, E. André, and T. Vogt.  Towards user-independent classification of multimodal emotional signals.  In 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops, pages 1–7. IEEE, 2009.
  * [30] D. Kollias, P. Tzirakis, A. Baird, A. Cowen, and S. Zafeiriou.  Abaw: Valence-arousal estimation, expression recognition, action unit detection & emotional reaction intensity estimation challenges.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5888–5897, 2023.
  * [31] D. Leyzberg, S. Spaulding, and B. Scassellati.  Personalizing robot tutors to individuals’ learning differences.  In Proceedings of the 2014 ACM/IEEE international conference on Human-robot interaction, pages 423–430, 2014.
  * [32] B. Li and L. Han.  Distance weighted cosine similarity measure for text classification.  In Intelligent Data Engineering and Automated Learning–IDEAL 2013: 14th International Conference, IDEAL 2013, Hefei, China, October 20-23, 2013. Proceedings 14, pages 611–618. Springer, 2013.
  * [33] J. Li, A. Waleed, and H. Salam.  A survey on personalized affective computing in human-machine interaction.  arXiv preprint arXiv:2304.00377, 2023.
  * [34] L. Li, H. Zhu, S. Zhao, G. Ding, and W. Lin.  Personality-assisted multi-task learning for generic and personalized image aesthetics assessment.  IEEE Transactions on Image Processing, 29:3898–3910, 2020.
  * [35] G. Liang, S. Wang, and C. Wang.  Pose-aware adversarial domain adaptation for personalized facial expression recognition.  arXiv preprint arXiv:2007.05932, 2020.
  * [36] X. Liu, L. Sun, W. Jiang, F. Zhang, Y. Deng, Z. Huang, L. Meng, Y. Liu, and C. Liu.  Evaef: Ensemble valence-arousal estimation framework in the wild.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5862–5870, 2023.
  * [37] D. Lopez-Martinez, K. Peng, S. C. Steele, A. J. Lee, D. Borsook, and R. Picard.  Multi-task multiple kernel machines for personalized pain recognition from functional near-infrared spectroscopy brain signals.  In 2018 24th International Conference on Pattern Recognition (ICPR), pages 2320–2325. IEEE, 2018.
  * [38] M. Malkauthekar.  Analysis of euclidean distance and manhattan distance measure in face recognition.  In Third International Conference on Computational Intelligence and Information Technology (CIIT 2013), pages 503–507. IET, 2013.
  * [39] K. Niinuma, I. Önal Ertuğrul, J. F. Cohn, L. A. Jeni, et al.  Facial expression manipulation for personalized facial action estimation.  Frontiers in Signal Processing, 2:1–16, 2022.
  * [40] Y. Odaka and K. Kaneiwa.  Block-segmentation vectors for arousal prediction using semi-supervised learning.  Applied Soft Computing, 142:110327, 2023.
  * [41] S. Poria, I. Chaturvedi, E. Cambria, and A. Hussain.  Convolutional mkl based multimodal emotion recognition and sentiment analysis.  In 2016 IEEE 16th international conference on data mining (ICDM), pages 439–448. IEEE, 2016.
  * [42] F. Povolny, P. Matejka, M. Hradis, A. Popková, L. Otrusina, P. Smrz, I. Wood, C. Robin, and L. Lamel.  Multimodal emotion recognition for avec 2016 challenge.  In Proceedings of the 6th International Workshop on Audio/Visual Emotion Challenge, pages 75–82, 2016.
  * [43] R. G. Praveen, P. Cardinal, and E. Granger.  Audio-visual fusion for emotion recognition in the valence-arousal space using joint cross-attention.  IEEE Transactions on Biometrics, Behavior, and Identity Science, 2023.
  * [44] S. PS and G. Mahalakshmi.  Emotion models: a review.  International Journal of Control Theory and Applications, 10(8):651–657, 2017.
  * [45] F. A. Pujol, H. Mora, and A. Martínez.  Emotion recognition to improve e-healthcare systems in smart cities.  In Research & Innovation Forum 2019: Technology, Innovation, Education, and their Social Impact 1, pages 245–254. Springer, 2019.
  * [46] B. Ren, E. G. Balkind, B. Pastro, E. S. Israel, D. A. Pizzagalli, H. Rahimi-Eichi, J. T. Baker, and C. A. Webb.  Predicting states of elevated negative affect in adolescents from smartphone sensors: A novel personalized machine learning approach.  Psychological Medicine, pages 1–9, 2022.
  * [47] M. Rescigno, M. Spezialetti, and S. Rossi.  Personalized models for facial emotion recognition through transfer learning.  Multimedia Tools and Applications, 79(47-48):35811 – 35828, 2020.  Cited by: 12; All Open Access, Hybrid Gold Open Access.
  * [48] M. Rescigno, M. Spezialetti, and S. Rossi.  Personalized models for facial emotion recognition through transfer learning.  Multimedia Tools and Applications, 79:35811–35828, 2020.
  * [49] O. Rudovic, J. Lee, M. Dai, B. Schuller, and R. W. Picard.  Personalized machine learning for robot perception of affect and engagement in autism therapy.  Science Robotics, 3(19), 2018.
  * [50] A. Saeed and S. Trajanovski.  Personalized driver stress detection with multi-task neural networks using physiological signals.  arXiv preprint arXiv:1711.06116, 2017.
  * [51] H. Salam, O. Celiktutan, H. Gunes, and M. Chetouani.  Automatic context-aware inference of engagement in hmi: A survey.  IEEE Transactions on Affective Computing, 2023.
  * [52] H. Salam, O. Celiktutan, I. Hupont, H. Gunes, and M. Chetouani.  Fully automatic analysis of engagement and its relationship to personality in human-robot interactions.  IEEE Access, 5:705–721, 2016.
  * [53] H. Salam, O. Celiktutan, V. Manoranjan, I. Ismail, and H. Mukherjee.  Learning personalised models for automatic self-reported personality recognition.  In ICCV 2021 Understanding Social Behavior in Dyadic and Small Group Interactions Challenge Fact sheet: Automatic self-reported personality recognition Track, 2021.
  * [54] H. Salam and M. Chetouani.  Engagement detection based on mutli-party cues for human robot interaction.  In 2015 International Conference on Affective Computing and Intelligent Interaction (ACII), pages 341–347. IEEE, 2015.
  * [55] E. San Segundo, A. Tsanas, and P. Gómez-Vilda.  Euclidean distances as measures of speaker similarity including identical twin pairs: a forensic investigation using source and filter voice characteristics.  Forensic Science International, 270:25–38, 2017.
  * [56] A. V. Savchenko.  Emotieffnets for facial processing in video-based valence-arousal prediction, expression classification and action unit detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5715–5723, 2023.
  * [57] J. Schneider and M. Vlachos.  Personalization of deep learning.  In Data Science–Analytics and Applications: Proceedings of the 3rd International Data Science Conference–iDSC2020, pages 89–96. Springer, 2021.
  * [58] F. Schroff, D. Kalenichenko, and J. Philbin.  Facenet: A unified embedding for face recognition and clustering.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815–823, 2015.
  * [59] T. Senechal, V. Rapp, H. Salam, R. Seguier, K. Bailly, and L. Prevost.  Facial action recognition combining heterogeneous features via multikernel learning.  IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 42(4):993–1005, 2012.
  * [60] R. V. Shah, G. Grennan, M. Zafar-Khan, F. Alim, S. Dey, D. Ramanathan, and J. Mishra.  Personalized machine learning of depressed mood using wearables.  Translational psychiatry, 11(1):1–18, 2021.
  * [61] M. Shahabinejad, Y. Wang, Y. Yu, J. Tang, and J. Li.  Toward personalized emotion recognition: A face recognition based attention method for facial emotion recognition.  In Proceedings of IEEE International Conference on Face & Gesture, 2021.
  * [62] Z. Shao, S. Song, S. Jaiswal, L. Shen, M. Valstar, and H. Gunes.  Personality recognition by modelling person-specific cognitive processes using graph representation.  In proceedings of the 29th ACM international conference on multimedia, pages 357–366, 2021.
  * [63] F. A. Shaqra, R. Duwairi, and M. Al-Ayyoub.  Recognizing emotion from speech based on age and gender using hierarchical models.  Procedia Computer Science, 151:37–44, 2019.
  * [64] P. Siirtola, S. Tamminen, G. Chandra, A. Ihalapathirana, and J. Röning.  Predicting emotion with biosignals: A comparison of classification and regression models for estimating valence and arousal level using wearable sensors.  Sensors, 23(3):1598, 2023.
  * [65] M. K. Singh, N. Singh, and A. Singh.  Speaker’s voice characteristics and similarity measurement using euclidean distances.  In 2019 International Conference on Signal Processing and Communication (ICSC), pages 317–322. IEEE, 2019.
  * [66] C. Soladié, H. Salam, C. Pelachaud, N. Stoiber, and R. Séguier.  A multimodal fuzzy inference system using a continuous facial expression representation for emotion detection.  In Proceedings of the 14th ACM international conference on Multimodal interaction, pages 493–500, 2012.
  * [67] C. Soladié, H. Salam, N. Stoiber, and R. Séguier.  Continuous facial expression representation for multimodal emotion detection.  International Journal of Advanced Computer Science (IJACSci), 3(5), 2013.
  * [68] B. C. Song and D. H. Kim.  Hidden emotion detection using multi-modal signals.  In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1–7, 2021.
  * [69] Y. Song, S. Dixon, M. T. Pearce, and A. R. Halpern.  Perceived and induced emotion responses to popular music: Categorical and dimensional models.  Music Perception: An Interdisciplinary Journal, 33(4):472–492, 2016.
  * [70] L. Stappen, A. Baird, L. Christ, L. Schumann, B. Sertolli, E.-M. Messner, E. Cambria, G. Zhao, and B. W. Schuller.  The muse 2021 multimodal sentiment analysis challenge: sentiment, emotion, physiological-emotion, and stress.  In Proceedings of the 2nd on Multimodal Sentiment Analysis Challenge, pages 5–14. 2021.
  * [71] L. Stappen, E.-M. Meßner, E. Cambria, G. Zhao, and B. W. Schuller.  Muse 2021 challenge: Multimodal emotion, sentiment, physiological-emotion, and stress detection.  In Proceedings of the 29th ACM International Conference on Multimedia, pages 5706–5707, 2021.
  * [72] S. Taylor, N. Jaques, E. Nosakhare, A. Sano, and R. Picard.  Personalized multitask learning for predicting tomorrow’s mood, stress, and health.  IEEE Transactions on Affective Computing, 11(2):200–213, 2017.
  * [73] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [74] M. Vijaymeena and K. Kavitha.  A survey on similarity measures in text mining.  Machine Learning and Applications: An International Journal, 3(2):19–28, 2016.
  * [75] T. Vogt and E. André.  Improving automatic emotion recognition from speech via gender differentiaion.  In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06), Genoa, Italy, May 2006. European Language Resources Association (ELRA).
  * [76] J. Wagner, A. Triantafyllopoulos, H. Wierstorf, M. Schmitt, F. Burkhardt, F. Eyben, and B. W. Schuller.  Dawn of the transformer era in speech emotion recognition: closing the valence gap.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
  * [77] C. Wang and S. Wang.  Personalized multiple facial action unit recognition through generative adversarial recognition network.  In Proceedings of the 26th ACM international conference on Multimedia, pages 302–310, 2018.
  * [78] K. Woodward, E. Kanjo, D. J. Brown, and T. McGinnity.  Towards personalised mental wellbeing recognition on-device using transfer learning “in the wild”.  In 2021 IEEE International Smart Cities Conference (ISC2), pages 1–7. IEEE, 2021.
  * [79] H. Yang, Z. Zhang, and L. Yin.  Identity-adaptive facial expression recognition through expression regeneration using conditional generative adversarial networks.  In 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pages 294–301. IEEE, 2018.
  * [80] R. Yusuf, D. G. Sharma, I. Tanev, and K. Shimohara.  Individuality and user-specific approach in adaptive emotion recognition model.  In 2017 International Conference on Biometrics and Kansei Engineering (ICBAKE), pages 1–6. IEEE, 2017.
  * [81] H. Zhao, N. Ye, and R. Wang.  Transferring age and gender attributes for dimensional emotion prediction from big speech data using hierarchical deep learning.  In 2018 IEEE 4th International Conference on Big Data Security on Cloud (BigDataSecurity), IEEE International Conference on High Performance and Smart Computing,(HPSC) and IEEE International Conference on Intelligent Data and Security (IDS), pages 20–24. IEEE, 2018.
  * [82] S. Zhao, A. Gholaminejad, G. Ding, Y. Gao, J. Han, and K. Keutzer.  Personalized emotion recognition by personality-aware high-order learning of physiological signals.  ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 15(1s):1–18, 2019.
  * [83] R. Zhi, M. Liu, and D. Zhang.  A comprehensive survey on automatic facial action unit analysis.  The Visual Computer, 36:1067–1093, 2020.