# An Open-source Benchmark of Deep Learning Models for Audio-visual Apparent
and Self-reported Personality Recognition

Rongfan Liao, Siyang Song∗, and Hatice Gunes Rongfan Liao is with SONY China
Software Center. E-mail<EMAIL_ADDRESS>Siyang Song and Hatice Gunes
are with the AFAR Lab, Department of Computer Science and Technology,
University of Cambridge, Cambridge, CB3 0FT, United Kingdom. E-mail:
<EMAIL_ADDRESS><EMAIL_ADDRESS>(∗ Corresponding Author: Siyang
Song, E-mail<EMAIL_ADDRESS>Manuscript received September 18, 2022.

###### Abstract

Personality determines a wide variety of human daily and working behaviours,
and is crucial for understanding human internal and external states. In recent
years, a large number of automatic personality computing approaches have been
developed to predict either the apparent personality or self-reported
personality of the subject based on non-verbal audio-visual behaviours.
However, the majority of them suffer from complex and dataset-specific pre-
processing steps and model training tricks. In the absence of a standardized
benchmark with consistent experimental settings, it is not only impossible to
fairly compare the real performances of these personality computing models but
also makes them difficult to be reproduced. In this paper, we present the
first reproducible audio-visual benchmarking framework to provide a fair and
consistent evaluation of eight existing personality computing models (e.g.,
audio, visual and audio-visual) and seven standard deep learning models on
both self-reported and apparent personality recognition tasks. Building upon a
set of benchmarked models, we also investigate the impact of two previously-
used long-term modelling strategies for summarising short-term/frame-level
predictions on personality computing results. We conduct a comprehensive
investigation into all the benchmarked models to demonstrate their
capabilities in modelling personality traits on two publicly available
datasets, audio-visual apparent personality (ChaLearn First Impression) and
self-reported personality (UDIVA) datasets. The experimental results conclude:
(i) apparent personality traits, inferred from facial behaviours by most
benchmarked deep learning models, show more reliability than self-reported
ones; (ii) visual models frequently achieved superior performances than audio
models on personality recognition; (iii) non-verbal behaviours contribute
differently in predicting different personality traits; and (iv) our
reproduced personality computing models generally achieved worse performances
than their original reported results. We make all the code and settings of
this personality computing benchmark publicly available at
https://github.com/liaorongfan/DeepPersonality.

###### Index Terms:

Self-reported (true) personality recognition, Apparent personality
(impression) recognition, Audio-visual personality computing benchmark,
Spatio-temporal modelling, Deep Learning

##  1 Introduction

Personality defines a characteristic sets of human cognitive processes and
emotional patterns that evolve from various biological and environmental
factors [1], which are well associated with a wide range of human behaviours
and status such as purchasing behaviours [2], health conditions [3, 4], social
relationships [5], and even criminal activities [6]. Consequently, automatic
personality computing systems have drawn a lot of attention in recent years,
and have been frequently developed for real-world human behaviour
understanding applications, including computer assisted tutoring systems [7],
human resource management [8], job interviews [9], and recommendation systems
[10]. In these systems, the complex personality is usually described by trait-
based models [11, 12, 13, 14] which focus on modelling personality aspects
that are stable over time for the target person but differ in others [15].

Existing automatic personality computing approaches can be roughly categorized
into two types: (i) automatic personality recognition that recognises the
target subject’s self-reported personality traits (also called true
personality traits); and (ii) automatic personality perception recognition
that predicts external human observers’ impression on the target subject (also
called first impression or apparent personality). The majority of these
approaches were focused on automatic apparent personality recognition and were
evaluated on the ChaLearn First Impression dataset [16]. They recognise
apparent personality using either human visual behaviours (e.g., facial
behaviours) [17, 18, 19, 20, 21, 22, 23, 24, 25, 24] or audio-visual
behaviours [26, 27, 28, 29, 30]. More specifically, a large part of these [31,
32, 17, 26, 21, 22] infer personality traits from every single frame or short
segment [19], and then make the clip-level personality prediction by combining
all frame/segment-level predictions, where video-level personality labels were
used as the frame/short segment-level labels to train models. While subjects
with different personalities can behave similarly in a frame or a short video
segment, models trained by such strategies would be problematic as they pair
similar behaviour samples with different personality labels [33].
Alternatively, recent studies frequently emphasized that long-term behaviours
enable more reliable personality inference [34, 29, 25], as personality traits
are stable over time [15]. These approaches propose to encode a clip-level
personality representation from long-term behaviours of the subject, and
generally achieve superior recognition results than most frame/segment-level
approaches.

However existing approaches usually employ different pre-processing, post-
processing and training strategies. Such differences lead to their reported
results not being fairly reflected by their models’ capabilities in
recognizing personality traits. Meanwhile, very few number of deep learning-
based approaches [18, 35, 36] have been proposed to recognize self-reported
personality traits, and consequently there is lack of a comprehensive
understanding for the capability of existing widely-developed apparent
personality recognition approaches and standard deep learning models in
recognizing self-reported personality traits. In summary, there is no
comprehensive study that fairly demonstrates and compares the performances of
existing personality computing models and widely-used deep learning models on
both apparent personality and self-reported personality recognition (Research
gap 1). In addition, only a small number of these studies made their code
publicly available, which are even built on different platforms, e.g., Keras
[19, 20], TensorFlow [37], caffe [24], Torch [27] and Chainer [38] (listed in
Table I), where some of these codes are incomplete and unimplementable. As a
result, it is difficult for other researchers to reproduce or extend most of
them (Research gap 2).

In this paper, we bridge the aforementioned research gaps by introducing two
main contributions: (i) we benchmark eight existing audio-visual or visual
only automatic apparent personality recognition approaches as well as seven
widely-used static or spatio-temporal deep learning models with standardized
pre-processing, training and post-processing strategies, on both widely-used
self-reported personality (i.e., UDIVA [39]) and apparent personality datasets
(ChaLearn First Impression [40]). The reported results provide a fair
evaluation and comparison of all these models’ capabilities for both apparent
personality and self-reported personality recognition tasks (addressing
research gap 1); and (ii) we make the code of all benchmarked models as well
as their standardized pre-processing, training and post-processing scripts
publicly available to further the science of _automatic personality
recognition_. This provides new researchers entering this field a set of
strong personality computing baselines, which will facilitate their
exploration of new models and will enable them to apply these to new datasets
in personality computing. To the best of our knowledge, this is the first
study that benchmarks and fairly evaluates the existing personality computing
approaches and standard deep learning models on both self-reported and
apparent personality recognition tasks, with consistent and reproducible
experimental settings.

TABLE I: The list of publicly available code of previous automatic apparent personality recognition approaches Method | Platform | Link | Details
---|---|---|---
Moreno-Armendazi et al.[19] | Keras | https://github.com/miguelmore/personality | Jupyter notebook-based script
Helm et al.[20] | Keras | https://github.com/dahe-cvl/apa_paper | Python2.7-based code
Hayat et al.[37] | TensorFlow | https://github.com/HassanHayat08/ | Unimplementable (Bugs exist)
Zhang et al.[24] | Caffe | https://github.com/ZhangLeUestc/PersEmoN | C++ based code
Subraman et al.[27] | Torch (Lua) | https://github.com/InnovArul/first-impressions | Torch (Lua) is out of date
Güclütürk et al.[38] | Chainer | https://github.com/yagguc/deep_impression | Chainer is out of date

##  2 Related Work

In this section, we systematically review and summarize recent audio-visual
automatic apparent personality recognition (Sec. 2.1) and self-reported
personality recognition (Sec. 2.2) approaches.

###  2.1 Automatic apparent personality Recognition

Existing approaches frequently attempt to recognize apparent personality from
non-verbal facial behaviours, where a large part of them infer apparent
personality only from a single facial display [17, 19, 20]. Joo et al. [41]
conduct studies on face images of $650$ American politicians of white
ethnicity, which extracts Histogram of oriented gradients (HOG) features as
the low-level static facial representation to predict personality traits.
Dhall et al. [42] utilize both hand-crafted and deep-learned features to
describe Twitter profile facial images and infer personality traits from them,
where background information was also considered. Moreno-Armendáriz et al.
[19] first extract one portrait picture as the representation for each video,
and then train a CNN to predict video-level perception. In addition, although
facial videos are provided, some studies also directly infer apparent
personality from static facial appearance without considering temporal
dynamics. For example, Ventura et al.[17] use Classification Activation Map
(CAM) [43] to investigate the relationship between facial expressions and
apparent personality, which feeds each frame to a CNN to infer video-level
perception traits.

While very few studies [37] infer personality traits from only audio
behaviours, the majority of the studies focus on audio-visual methods that
frequently show enhanced performances than the corresponding single modality
(e.g., visual-based or audio-based) systems. Besides some early studies [44,
45] investigated the relationship between frame-level facial cues and apparent
personality traits in the context of human-robot interactions, most existing
approaches attempts to predict video-level personality traits. Zhang et al.
[26] first choose 6 images from each video, and extend the Descriptor
Aggregation Network (DAN) to deep learn facial perception features at the
frame-level. Meanwhile, a linear regressor is used to process log filter bank-
based clip-level audio representation. The late fusion scheme is employed to
get the final prediction by averaging audio and visual predictions. Guccluturk
et al. [32] propose a two-stream ResNet to deep learn both audio and visual
personality cues at the frame-level, which are combined at fully connected
(FC) layer to predict perceptions. Subramaniam et al.[27] divide each audio-
visual clip into several short segments, where one facial frame is selected
from each segment to produce a segment-level visual feature. Then, the
segment-level audio-visual features are combined via a FC layer to make the
apparent personality prediction. To recognize apparent personality from clip-
level (long-term) audio-visual behaviours, Gürpinar et al.[28] employ a pre-
trained network to extract frame-level facial emotion features. The video-
level visual representation is then obtained by computing statistics of all
frame-level features while clip-level audio feature is extracted via OpenSMILE
[46]. The final prediction is made by the weighted average of audio and visual
predictions. As discussed in Sec. 1, these static or short-term behaviour-
based approaches usually re-use video-level personality labels as the
frame/segment-level labels during the training, which may result in
problematic models.

To avoid the above problem, many recent approaches are proposed to recognize
apparent personality from long-term behaviours (i.e., applying video-level
behaviour representation to recognize video-level personality traits). For
example, Beyan et al. [25] summarize a video into a set of dynamic images, and
then select a small part of key dynamic images based on their spatio-temporal
saliency. Then, the produced key dynamic image sequence is used as the video-
level representation for the perception prediction. Helm et al.[20] extract a
face image sequence by down-sampling the target video, whose features is used
as the video-level facial behaviour representation. Then, a 3D-CNN is employed
to infer apparent personality from this video-level representation. Gürpınar
et al.[23] jointly extract facial expression-related features from all aligned
face images and the ambient information from the first frame of a down-sampled
video, both of which are combined as the video-level visual representation for
apparent personality prediction. Li et al. [29] propose to down-sample each
video into $32$ frames, and extract video-level visual features from the
corresponding facial region sequence as well as the whole image sequence. In
addition, the long-term audio features and text features are also combined
with the visual features for apparent personality recognition. Since the above
long-term behaviour modelling approach depend on down-sample videos, Song et
al. [34] propose a self-supervised learning strategy to use all frames for
video-level representation construction. This method trains a person-specific
layer for each individual using facial behaviours of the whole video, whose
weights are then used as the video-level personality representation.

###  2.2 Automatic Self-reported Personality Recognition

Compared to apparent personality, only a small number of studies attempt to
infer self-reported personality traits. Qin et al. [47] extract five types of
hand-crafted features (i.e., HOG, Local Binary Pattern (LBP), Gabor, Scale-
invariant feature transform (SIFT) and Generalized Search Trees (Gist)) from
each face image, which are fed to standard regressors (e.g., decision tree) to
estimate self-reported personality (16PF) and intelligence. Besides such
frame-level self-reported personality recognition approaches, recently
proposed audio-visual self-reported personality analysis datasets [39, 48]
lead some studies to develop video-level self-reported personality recognition
approaches. Similar to apparent personality recognition solutions, Curto et
al. [35] propose a Transformer-based model to extract individual and
interpersonal behaviour features from each short dyadic interaction segment
using variable time windows, which can jointly recognize self-reported
personality traits for both individuals. In addtion, Celiktutan et al. [49]
specifically investigated the self-reported personality recognition under the
human-robot interaction scenarios. Since personality is well associated with
human cognitive process, Song and Shao et al. [18, 36] propose the first
audio-visual approach that employs Neural Architecture Search (NAS) to explore
a person-specific network for each subject using all available frames, whose
architecture and parameters are encoded as the person-specific cognitive
process graph representation for self-reported personality recognition. This
NAS-based idea was then followed by [50], which also search a personalized
deep learning architectures for each subject to recognize Big Five personality
traits.

##  3 The proposed benchmarking framework

In this paper, we propose a benchmarking framework that aims to provide a
rigorous and reproducible evaluation of the existing personality computing
models and the widely-used deep learning models for both automatic self-
reported and apparent personality recognition. This framework will equip the
future researchers with a set of strong audio-visual personality computing
baselines, with standardized data pre-processing, training and post-processing
strategies. Specifically, this section introduces our benchmarking framework
by presenting its coding infrastructure (Sec. 3.1), benchmarked models and
their settings (Sec. 3.2), and datasets used for evaluation (Sec. 3.3). The
pipeline of the proposed personality benchmarking framework is illustrated in
Fig. 1.

Figure 1: Illustration of our personality computing benchmark framework.

###  3.1 Coding infrastructure

A unified experimental setting is much needed given the heterogeneity in data
processing and training strategies of existing personality computing
approaches. Our benchmarking framework is built on PyTorch [51] library and it
unifies the components of the data input, data pre-processing, data post-
processing, training, optimization and evaluation, for all deep learning
model-based experiments. The only difference in experiments resides in
different models’ architectures and their training hyper-parameter settings as
the optimal hyper-parameter settings are model-dependent. This way, the
personality computing results achieved for all benchmarked models can be
compared fairly.

####  3.1.1 Data input and pre-processing

For all experiments, we consistently employ the same training, validation and
test sets that are predefined in the corresponding self-reported personality
and apparent personality datasets for training the models and reporting the
results. Instead of using complex and dataset-specific data pre-processing
pipelines, we employ a widely-used and unified data pre-processing pipeline
for each compared model (e.g., static models and spatio-temporal models) as
follows:

  * •

Static image-based models: We first evenly divide each video into $K$ short
segments as suggested in [31, 27] and only select the one frame of each
segment (To make experiments to be reproducible, we consistently use the first
frame of each segment), where we follow the winner [31] of ChaLearn 2016
impression challenge to set $K$ as $100$ for experiments conducted on ChaLearn
First Impression dataset, and empirically set $K$ as $2000$ for UDVIA dataset.
We then apply MTCNN [52] to obtain an cropped and aligned face image from each
selected frame. During training, validation or testing, we again follow [31]
to resize the resolution of these images to $456\times 256$, where only a
$224\times 224$ sub-region in the center of the image is used as the final
full frame. Meanwhile, the face region is contained in a cropped image with
the resolution of $112\times 112$ using the Dlib library [53]. Specifically,
standard data augmentation including center crop, random horizontal flip and
pixel normalization are employed to process all training images. We follow
previous studies [31, 17, 32] that compute personality traits prediction of
each test video by averaging all frame-level predictions.

  * •

Spatio-temporal models: For each spatio-temporal model, we benchmark two types
of systems: (i) short segment-level system, where we divide each video into
several segments, where each segment that contains $n$ frames (i.e., we
empirically employed $n=32$ or $n=64$ according to (1) the setting employed in
the original publication; and (2) the experimental results) is used as the
input. The personality trait prediction of each test video is computed by
averaging all segment-level predictions; and (ii) video-level system, where we
again follow [29] that evenly divides each video into $32$ segments, and
select the first frame of each segment to construct a sequence of $32$ frames
to represent the entire video. The sequence is then used as the input for
spatio-temporal models to produce the video-level personality prediction. For
both systems, MTCNN and Dlib library are again employed to obtain each full
frame and its corresponding face image based on the same setting as the static
image-based models.

  * •

Audio models: As most audio-visual studies did not provide the details of
their raw audio signal extraction, we follow previous studies [31, 54] to use
the FFmpeg tool 111https://github.com/FFmpeg/FFmpeg to extract the raw audio
signal from original audio-visual clips for all audio-based models. The
processing details for each audio model are introduced in Sec. 3.2.2.

####  3.1.2 Training, validation and testing protocol

For all benchmarked static models and short segment-based spatio-temporal
models, we follow the previous studies [31, 17, 32] to re-use the clip-level
label as the corresponding frame-level or short segment-level labels to train
them. We also use the clip-level label as the label to train models that take
clip-level sequence as the input. For all experiments, we consistently employ
the same training, validation and testing protocol. Each static image-based
model or short segment-based spatio-temporal model is trained on the training
dataset with the shuffled data samples, and is evaluated on the validation
set. We choose the model that obtains the best performance on the validation
set as the final model for each experiment, which generates the reported
testing results.

####  3.1.3 Evaluation metrics

To measure the performance of the trained models, we first employ two standard
evaluation metrics (i.e., ACC and Mean Squared Error (MSE)) proposed in [16]
and [39]. The ACC is defined as:

$\mbox{ACC}=1-\frac{1}{N}\sum_{n=1}^{N}|y_{p_{n}}-y_{g_{n}}|$ (1)

and the MSE can be computed as:

$\mbox{MSE}=\frac{1}{N}\sum_{n=1}^{N}\left(y_{p_{n}}-y_{g_{n}}\right)^{2}$ (2)

where $N$ indicates the number of test videos; $y_{p_{i}}$ and $y_{g_{i}}$
denote the prediction and ground-truth of the $n_{th}$ test video,
respectively. We also follow recent studies [18, 34, 22] that employ
Concordance Correlation Coefficient (CCC, Eq. 3) to measure the correlation
between predictions and ground-truths, which is defined as:

$\mbox{CCC}=\frac{2\sigma_{Y_{p}}\sigma_{Y_{g}}\mbox{PCC}_{Y_{p},Y_{g}}}{\sigma_{Y_{p}}^{2}+\sigma_{Y_{g}}^{2}+\left(\mu_{Y_{p}}-\mu_{Y_{g}}\right)^{2}},$
(3)

where $\mu_{Y_{p}}$ and $\mu_{Y_{g}}$ denote the mean values of predictions
and ground-truths of all test videos, respectively; $\sigma_{Y_{p}}$ and
$\sigma_{Y_{g}}$ are the corresponding standard deviations; and
$\mbox{PCC}_{Y_{p},Y_{g}}$ denotes the Pearson Correlation Coefficient (PCC)
between $Y_{p}$ and $Y_{g}$. In this paper, we follow previous studies that
individually employ MSE and CCC to evaluate the self-reported personality
recognition systems while we use ACC and CCC to evaluate the apparent
personality recognition systems.

###  3.2 Benchmarked personality computing models

In this section, we provide the details of all the benchmarked models
including (i) seven visual models, six audio models and five audio-visual
models that have been employed by existing personality computing approaches;
(ii) seven typical spatial/spatio-temporal deep learning CNN/Transformer
models that have been widely-used in image or video processing tasks but have
not yet been employed for personality recognition; and (iii) two previously
used clip-level encoding models that allow all frame/short segment-level
personality predictions of a clip to be combined for clip-level personality
recognition.

####  3.2.1 Personality computing model inclusion and exclusion criteria

To the best of our knowledge, there are more than one hundred visual or audio-
visual personality computing models that have been proposed in the literature.
In this paper, we propose the following inclusion and exclusion criteria to
benchmark the most representative personality computing models:

  * •

Inclusion criteria: The main criteria of this benchmark is to choose
representative approaches that have been evaluated and compared in the well-
known and widely-used publicly available audio-visual personality computing
dataset: ChaLearn 2016 First Impression. As a result, we first choose the
Top-3 ranked models [26, 27, 32] from the ChaLearn 2016 First Impression
Recognition Challenge. Then, we choose the best end-to-end audio-visual deep
learning model that have been published in recent three years, respectively,
which ([24] (2019), [29] (2020), and [54] 2022) claimed that they have
achieved the state-of-the-art performances on ChaLearn 2016 impression dataset
and provided all details for model settings and training (the best model of
2021 [34] is not an end-to-end approach). In addition, we also reproduce two
models [17, 37] that can visualize the relationship between human audio/visual
non-verbal behaviours and personality traits.

  * •

Exclusion criteria: In this paper, we exclude methods that are not based on:
(i) audio-visual deep learning models; (ii) an end-to-end framework, such as
[22, 34, 36, 18] which require relatively complex pre-processing or pre-
training; (iii) evaluated on the ChaLearn 2016 dataset [35, 55, 36, 18, 35,
50]; (iv) using the Big-five personality traits as the target recognition
task; and (v) treating personality recognition as a regression task.

####  3.2.2 Existing personality computing models

We first present seven deep learning visual models, six audio models and five
audio-visual models that have been proposed to recognize apparent personality
traits, which are evaluated on the ChaLearn 2016 First Impression dataset
[16]. For each deep learning-based model, the last FC layer has five neurons
to jointly predict five personality traits.

Visual models: The details of the seven existing personality computing visual
models reproduced in this paper are explained as follows:

  * •

(i) Wei et al.[31]: DAN is a CNN model that infers personality from each
static image (including both face and background). It consists of several
convolution-ReLU blocks and an additional block that is equipped with both
average-pooling and max-pooling, which is added between the last convolutional
layer and the final FC layer. Before being fed to the model, each image is
resized to $456\times 256$ resolution. The training process is conducted using
the SGD optimizer [56] with initial learning rate of 0.05, weight decay of
0.0005 and momentum of 0.8, where MSE is employed as the loss function.

  * •

(ii) Ventura et al.[17]: CAM-DAN+ has a similar architecture to the DAN
network, which also infers personality from each static image. This model
applies a max pooling and an average pooling as two parallel branches in
addition to convolution layers, and is equipped with a Class Activation Map
(CAM) module in order to visualize the most salient facial regions for
personality recognition. The training process is conducted using the SGD
optimizer [56] with initial learning rate of 0.05, weight decay of 0.0005 and
momentum of 0.8, where MSE is employed as the loss function.

  * •

(iii) Subramaniam et al. [27]: The visual part of this approach consists of
three 3D convolutional layers and two FC layers. It randomly takes three face
images from each video as the video-level representation, and concatenates
them as the clip-level input. The model is trained using the SGD optimizer
with initial learning rate of 0.05, weight decay of 0.0005 and momentum of
0.9, where MSE is employed as the loss function.

  * •

(iv) Güclütürk et al. [32]: The visual part of this model is a $17$-layer deep
residual network, where an FC layer is attached at the top of the model. It
takes a static image (both face and background) as the input. The training
process is conducted using the SGD optimizer with initial learning rate of
0.05, weight decay of 0.0005 and momentum of 0.9, where MSE is employed as the
loss function.

  * •

(v) Zhang et al. [24]: The model first employs four convolution layers and an
FC layer to extract features from each face image of the video. Then, three
parallel FC layers are used to individually predict valence, arousal and
personality traits at the same time. Besides, a coherence module (an FC layer)
is introduced to process both images from the personality dataset and emotion
dataset, allowing the final learned representation to be dataset invariant,
while a RAM module that consists of two FC layers is employed to produce the
apparent personality attributes from arousal and valence inputs, aiming to
enhance the learned personality representations. The training process of this
model is achieved by using the SGD optimizer with initial learning rate of
0.001, weight decay of 0.0005 and momentum of 0.9, while MSE is used as the
loss function.

  * •

(vi) Li et al.[29]: The visual module of the CR-Net consists of two streams,
where each uses a $34$-layer deep residual network as the backbone to learn
personality features from either down-sampled video-level face sequences and
full image sequences. Then, a CR-block is attached to first obtain personality
classification features (processed by softmax) from the latent feature
produced by each backbone, based on which the regression feature is extracted.
The Extra Trees Regressor (ETR) is finally employed to predict personality
trait intensities from the extracted regression features. To obtain the video-
level face sequence and full image sequence, each video is divided into $32$
segments, where a single frame is randomly selected from each segment (i.e.,
each video is down-sampled to 32 frames). The SGD optimizer with initial
learning rate of 0.002, weight decay of 0.005 and momentum of 0.9 is employed
for training, where the Bell Loss function [29] is also applied.

  * •

(vii) Suman et al. [54]: The visual part of this system is a two-stream
ResNet-101 model that extracts two different visual features: ambient features
(extracted from full frames) and facial features. The system consists of three
steps: (i) Pre-processing: the system selects six equally spaced images from
each video clip, and applies MTCNN to obtain the face area for extracting
facial features; (ii) ambient and facial feature extraction: ResNet-101 pre-
trained on ImageNet is re-trained to extract ambient features and facial
features from each selected frame; and (iii) the ambient features and facial
features of each selected frame individually are fed to a MLP (containing FC
layers and a sigmoid layer) to make frame-level personality prediction. The
video-level personality trait prediction is obtained by averaging all frame-
level predictions. The training process is conducted using the SGD optimizer
with initial learning rate of 0.01, weight decay of 0.0005 and momentum of
0.9, where MSE is employed as the loss function.

Audio models: The details of the six existing personality computing audio
models reproduced in this paper are explained as follows:

  * •

(i) Hayat et al. [57]: The clip-level raw audio signal is obtained from each
audio-visual clip at 4000 HZ, and is represented by a $61208$ dimensional
vector. The obtained audio signal is normalized to have zero mean and unit
variance, and converted to the frequency domain using FFT. The produced
spectral signal is then fed to a network that consists of three temporal
convolution layers followed by max pooling and dropout. Here, the training is
conducted using SGD optimizer with initial learning rate of 0.00083, weight
decay of 0.065 and a momentum of 0.9, and the MSE is applied as the loss
function.

  * •

(ii) Wei et al. [31]: The clip-level raw audio signal is obtained from each
audio-visual clip at 44,100 Hz, which are divided into 3059 audio frames. A
13-D MFCC feature and a 26-D logfbank feature are extracted from each audio
frame. Then, the clip-level 39767-D MFCC feature and 79534-D logfbank feature
are produced by concatenating all frame-level MFCC and logfbank features,
respectively. During the model training, SGD optimizer with initial learning
rate of 0.05, weight decay of 0.0005 and momentum of 0.8 is applied, and the
MSE is employed as the loss function.

  * •

(iii) Subramaniam et al. [27]: The audio signal of each clip is first divided
into six non-overlapping segments, where the mean and standard deviation of
several properties of each audio segment are extracted, and combined as a 68
dimension vector using pyAudioAnalysis [58] (please check [27] for details).
We then concatenate six vectors from six segments as a matrix with size of
$6\times 68$, which is fed to a FC-LSTM-FC block to jointly predict five
personality traits. During the model training, SGD optimizer with initial
learning rate of 0.05, weight decay of 0.0005 and momentum of 0.9 is applied,
and the MSE is employed as the loss function.

  * •

(iv) Güclütürk et al.[32]: The entire audio signal of each clip is sampled at
16,000 HZ using the librosa library[59], resulting in a 244,832-D vector to
represent each clip-level audio signal. For each iteration, we randomly select
a continuous 50,176-D sub-vector from the clip-level vector as the clip-level
audio representation. Then, a $17$-layer ResNet is employed, where a FC layer
is attached at the top of it. The model training is achieved using the SGD
optimizer with initial learning rate of 0.0002, weight decay of 0.0005 and
momentum of 0.9 while the MSE is employed as the loss function.

  * •

(v) Li et al. [29]: The entire audio signal of each clip is converted to a
244,832-D vector using the librosa library [59], which is then used as the
clip-level audio representation. The ResNet-34 model is employed to process
this audio representation to extract the clip-level audio personality
features, from which the CR-block and ETR are used to jointly predict five
personality traits. The entire audio model is trained using the SGD optimizer
with initial learning rate of 0.0002, weight decay of 0.0005 and momentum of
0.9 and the Bell loss is employed as the loss function.

  * •

(vi) Suman et al. [54]: The raw audio clip is re-sampled to 16 kHz. The
spectrogram of the audio is then created, and these features are framed into
non-overlapping examples of 0.96s. A pre-trained VGGish CNN is then introduced
to extract audio personality features from each spectrogram frame. The
features of all frames are finally concatenated and fed to a MLP consisting of
FC layers and a sigmoid layer to make a clip-level personality prediction. The
entire audio model is trained using the SGD optimizer with initial learning
rate of 0.001, weight decay of 0.0005 and a momentum of 0.9, and the Bell loss
is employed as the loss function.

Audio-visual models: Based on the reproduced audio and visual systems, we also
reproduce four audio-visual models that have been used for apparent
personality recognition on the ChaLearn 2016 First Impression dataset as
follows:

  * •

(i) Wei et al. [31]: Following the same settings introduced in [31], we build
an audio-visual model that is made up of the corresponding visual and the
audio model (the visual model (i) and the audio model (ii)) described above.
The final clip-level prediction is obtained by averaging clip-level audio
prediction and clip-level video prediction.

  * •

(ii) Subramaniam et al. [27]: The audio and visual clip of each subject is
firstly divided into 6 segments, where each visual segment is processed by the
visual model (iii) described above and each audio segment is processed by the
audio model (iii) described above. Then, each pair of segment-level audio and
visual latent features are concatenated as a 160-D vector. Finally, the clip-
level personality prediction is obtained by feeding the audio-visual vectors
of six segments to an LSTM model with 128 hidden units. Here, the LSTM
contains a single hidden layer with 128 neurons. The entire model is trained
using the SGD optimizer with initial learning rate of 0.05, weight decay of
0.0005 and momentum of 0.9, and MSE is employed as the loss function.

  * •

(iii) Güclütürk et al.[32]: A two-steam ResNet model is employed, which
consists of the visual model (iv) and the audio model (iv) described above.
The audio feature (256-D) and the latent frame-level visual feature (256-D)
are concatenated as a 512-D feature to a fully-connected layer to jointly
predict the five personality traits. Specifically, the model randomly takes an
audio and visual sample at each training iteration. The entire model is
trained using the SGD optimizer with initial learning rate of 0.0002, weight
decay of 0.0005 and momentum of 0.9, and MSE is employed as the loss function.

  * •

(iv) Li et al.[29]: A three-stream ResNet-34 model is employed, which consists
of two visual streams (the visual model (vi) described above), and an audio
stream (the visual model (v) described above). Specifically, the two-stream
visual model takes both the face image sequence and full frame sequence as the
visual inputs. The latent features produced by audio and visual streams are
combined as a single 512-D vector by element-wise sum. The entire model is
trained using the SGD optimizer with initial learning rate of 0.002, weight
decay of 0.005 and momentum of 0.9, where the Bell loss is employed as the
loss function.

  * •

(v) Suman et al. [54]: A multi-modal system that contains a visual stream (the
visual model (vii) described above) which takes both face and full frames as
the input, and an VGGish-based audio stream (the audio model (vii) described
above) which learns personality features from the audio spectrogram. The
entire model is trained using the SGD optimizer with initial learning rate of
0.001, weight decay of 0.005 and momentum of 0.9, where the Bell loss is
employed as the loss function.

####  3.2.3 Widely-used static/spatio-temporal visual deep learning models

Name | Learning rate | Weight decay | Momentum
---|---|---|---
SENet[60] | 0.01 | 0.0005 | 0.8
HRNet[61] | 0.01 | 0.0005 | 0.9
Swin[62] | 0.01 | 0.0005 | 0.9
3DResNet[63] | 0.01 | 0.005 | 0.9
Slow-Fast[64] | 0.002 | 0.0005 | 0.9
TPN[65] | 0.001 | 0.0005 | 0.9
VAT[66] | 0.01 | 0.0005 | 0.9

TABLE II: The settings of the widely-used deep learning models. The SGD [56]
and Mean Square Error (MSE) are employed for training all models. Static full
frames and face frames are fed to SENet, HRNet and Swin (Swin-Transformer)
while spatio-temporal full frame/face frame sequences are fed to 3DResNet,
Slow-Fast, TPN and VAT.

Since visual information are much more informative for personality recognition
(validated in Sec. 4.1), we additional benchmark seven standard visual deep
learning models that previously have not been applied to personality computing
but have been widely used for static image or video analysis. Specifically,
these include three models that take the static face/full image as the input
and four models that infer personality from spatio-temporal visual data. We
list all these models in Table II.

####  3.2.4 Clip-level representation generation models

To combine frame/segment-level personality predictions of an audio-visual clip
for a clip-level personality trait prediction, this section also introduces
two standard methods that have been employed in existing personality computing
publications. In particular, besides the widely-used strategy which simply
averages all frame/segment-level predictions at the clip-level prediction
(used by [17, 31, 32, 24]), we also benchmark the spectral representation [33,
67] to summarize frame/segment-level predictions at the clip-level, which has
been used by [34]. This is because the spectral representation can not only
effectively encodes the temporal dependencies among all frame-level
predictions of a clip, but also summarises time-series signal of an arbitrary
length into a representation of a fixed size without any distortion.

  * •

Averaging frame/segment-level predictions (AFP): The clip-level personality
prediction is obtained by averaging all frame/segment-level predictions.

  * •

Spectral representation of frame/segment-level predictions (SFP): We first
encode a pair of spectral heatmaps (please see [33] for details) from the
five-channel (corresponding to five traits) frame/segment-level personality
trait prediction time-series. Then, we train a 1D-CNN to predict clip-level
personality traits from the produced spectral heatmaps.

###  3.3 Evaluation datasets

There are several publicly available audio-visual personality computing
datasets for apparent personality recognition [16, 68, 69], and self-reported
personality recognition [70, 39, 71] (please see [40] for a survey of existing
personality databases). Among them, the two most widely used publicly
available audio-visual self-reported personality and apparent personality
datasets: the UDIVA Dataset [39] and ChaLearn First Impression dataset [16],
are employed to evaluate all models described in Sec. 3.2. Both datasets
provide the Big-Five personality traits (i.e., Extraversion (Ext),
Agreeableness (Agr), Openness (Ope), Conscientiousness (Con), and Neuroticism
(Neu)) as the label for each audio-visual clip.

UDIVA dataset was released in 2021. It records 188 dyadic interaction clips
between $147$ voluntary participants, with total $90.5$h of recordings. Each
clip contains two audio-visual files, where each records a single
participant’s behaviours. For each dyadic interaction session, participants
were matched based on their availability, language and three variables:
gender, age group and the relationship among interlocutors. In particular, all
participants were matched to enforce a close-to-uniform distribution among all
possible combinations between these variables. During the recordings,
participants were asked to sit at 90 degrees to the conversational partner
around a table, and under the dyadic interactions based on five tasks: Talk,
’Animal games’, Lego building, “Ghost blitz” card game, and Gaze events, where
each pair of participants were interacting using one of the three languages
(i.e., English, Spanish or Catalan). The self-reported personality labels are
obtained for each participant based on the following rules: (i) parents of
children up to 8 years old completed the Children Behavior Questionnaire (CBQ)
[72]; participants from 9 to 15 years old completed the Early Adolescent
Temperament Questionnaire (EATQ-R) [73]; and the rest of the participants
completed both the Big Five Inventory [74] and the Honesty-Humility axis of
the HEXACO personality inventory [75].

ChaLearn First Impression dataset was released in 2016. It contains talking-
to-the-camera $10,000$ audio-visual clips that come from $2,764$ YouTube
users, where each video lasts for about $15$ seconds with $30$ fps. Each video
is labelled with the Big-Five personality traits that are annotated by
external human annotators using the Amazon Mechanical Turk. The intensity of
each trait was normalized to the range of $[0,1]$. The dataset provides
official splits for training ($6,000$ videos), validation ($2,000$ videos) and
test ($2,000$ videos). It is the largest audio-visual dataset openly available
for research purposes, which has been annotated with Big-Five apparent
personality traits.

##  4 Experimental Results

In this section, we first present the results of all benchmarked models for
both self-reported personality and apparent personality traits recognition in
Sec. 4.1. Then, in Sec. 4.2 we discuss the influences of different settings on
personality recognition performances. Finally, we provide a systematic
discussion of the benchmarking results in Sec. 4.3

###  4.1 Benchmarking personality computing models

This section reports the self-reported and apparent personality recognition
results achieved by all benchmarked models. According to Table III and Table
IV, all benchmarked models failed to accurately infer the self-reported
personality traits from human audio/visual/audio-visual behaviours, with only
three visual models (e.g., Interpret-img, SENet, and HRNet) achieving more
than $0.15$ CCC performances. Standard audio feature extraction methods can
barely extract self-reported personality-related information from non-verbal
audio signals, as most of the benchmarked models achieved near-zero CCC
values, while only the predictions of the VGGish achieved CCC $>0.1$ with the
ground-truth. Some audio models even degraded the corresponding visual systems
under the audio-visual setting. The low CCC and high MSE results of almost all
models on self-reported personality recognition indicate that it may difficult
and unreliable for standard deep learning models to be used to directly infer
human self-reported personality traits from external non-verbal behaviours.

In contrast, the relatively higher CCC and ACC results shown in Table V and
Table VI suggest that apparent personality traits can be better predicted from
human non-verbal behaviours when using deep learning models, i.e., five visual
models and two audio-visual models achieved CCC $>0.5$. In particular, the
predictions produced by HRNet and VAT from only facial behaviours have more
than $0.6$ correlation (measured by CCC) with the ground-truth. In addition,
we found that the CR-Net and VGGish models can also deep learn valuable
apparent personality-related cues from non-verbal audio signals. The
aforementioned results suggest that compared to self-reported personality,
apparent personality is generally more feasible and easier to be directly
predicted from human non-verbal behaviours. Since apparent personality is
defined as the external human obverses’ perception of the target subject,
these results can be explained by the fact that it is straight-forward for
deep learning models to be used as an external observer and make judgments
based on behaviours that can be directly observed. In contrast, self-reported
personality is the internal state/attribute of the target person, which is not
straight-forward to be directly inferred from external behaviours, and thus
can be more difficult for deep learning models to simulate such processes. The
detailed discussions on applying ML models for inferring self-reported
personality traits can be found in [18, 36].

| Traits | Open | Consc | Extrav | Agree | Neuro | Avg.
---|---|---|---|---|---|---|---
Audio | FFT [57] | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000
MFCC/Logfbank [31] | 0.0002 | -0.0002 | 0.0002 | 0.0001 | 0.0001 | 0.0001
Bi-modal CNN-LSTM [27] | 0.0000 | -0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.0000
ResNet [32] | -0.0459 | 0.1045 | -0.0416 | 0.0429 | 0.0015 | 0.0123
CRNet [29] | 0.0005 | 0.0290 | 0.0201 | 0.0335 | 0.0267 | 0.0220
VGGish [54] | 0.0688 | 0.1882 | 0.1310 | 0.1069 | 0.0412 | 0.1072
Video | DAN [31] | 0.0009 | 0.0087 | 0.0061 | 0.0035 | -0.0016 | 0.0036
Bi-modal CNN-LSTM[27] | -0.0001 | 0.0000 | 0.0000 | 0.0001 | 0.0000 | 0.0000
ResNet [32] | 0.0648 | 0.2424 | 0.0349 | 0.0305 | -0.0009 | 0.0743
CRNet [29] | -0.0206 | 0.0879 | 0.012 | 0.1454 | 0.0645 | 0.0578
CAM-DAN+ [17] | 0.0336 | 0.3359 | 0.0270 | 0.2014 | 0.2709 | 0.1738
PersEmoN [24] | -0.0095 | 0.0133 | 0.0045 | 0.0085 | 0.0058 | 0.0045
Amb-Fac [54] | 0.0532 | 0.1941 | 0.0442 | 0.0453 | 0.0204 | 0.0714
SENet [60] | 0.1678 | 0.2776 | 0.0538 | 0.0299 | 0.3093 | 0.1510
HRNet [61] | 0.2175 | 0.2998 | -0.0039 | 0.1680 | 0.1945 | 0.1752
Swin-Transformer [62] | -0.0273 | 0.0470 | 0.0361 | -0.0142 | 0.0860 | 0.0256
3D-Resnet [63] | -0.0478 | 0.0102 | 0.0478 | 0.0499 | -0.0240 | 0.0072
Slow-Fast [64] | -0.0102 | 0.0076 | 0.0010 | -0.0063 | 0.0161 | 0.0016
TPN [65] | 0.0448 | 0.0348 | 0.0287 | -0.0177 | -0.0281 | 0.0125
| VAT [66] | -0.0139 | -0.0016 | 0.0016 | -0.0013 | -0.0006 | -0.0031
Aud-vis | Wei et al. [31] | 0.0008 | 0.1154 | -0.0132 | -0.0222 | 0.1219 | 0.0405
Subramaniam et al. [27] | 0.0001 | -0.0001 | 0.0001 | -0.0001 | 0.0005 | 0.0001
Güclütürk et al. [32] | -0.0530 | 0.1290 | 0.0230 | 0.0310 | 0.0002 | 0.0260
Li et al. [29] | 0.0998 | 0.1780 | 0.1158 | 0.2168 | 0.0449 | 0.1311
Suman et al. [54] | -0.0348 | 0.0468 | 0.0302 | 0.0397 | 0.0041 | 0.0171

TABLE III: The CCC results achieved for the self-reported personality
recognition on the UDIVA dataset. The reported results are obtained by
averaging the results achieved for four sessions. For different modality
groups, audio, video, audio-visual, the values in bold denote the highest ones
in the columns and the values with underline denote the second highest ones.

| Traits | Open | Consc | Extrav | Agree | Neuro | Avg.
---|---|---|---|---|---|---|---
Audio | FFT [57] | 1.0122 | 0.7114 | 1.5056 | 0.9342 | 1.2851 | 1.0897
MFCC/logfbank [31] | 0.9489 | 1.0256 | 2.0083 | 1.2377 | 1.3526 | 1.3144
Bi-modal CNN-LSTM [27] | 0.9544 | 0.7717 | 1.2788 | 0.8676 | 1.1965 | 1.0138
ResNet [32] | 0.9540 | 0.6985 | 1.2325 | 0.8611 | 1.0744 | 0.9641
CRNet [29] | 0.9145 | 0.6978 | 1.4821 | 0.9623 | 1.3858 | 1.0885
VGGish [54] | 0.9312 | 0.6576 | 1.3345 | 0.9002 | 1.2227 | 1.0092
Video | DAN [31] | 1.0841 | 1.1174 | 2.0979 | 1.4846 | 1.2400 | 1.4048
Bi-modal CNN-LSTM[27] | 0.8947 | 0.6690 | 1.3459 | 0.9269 | 1.1518 | 0.9977
ResNet [32] | 0.9637 | 0.6709 | 1.5770 | 1.0364 | 1.1526 | 1.0801
CRNet [29] | 1.0985 | 0.6877 | 1.3079 | 0.909 | 1.4140 | 1.0834
CAM-DAN+ [17] | 1.0223 | 0.5471 | 1.4800 | 0.8446 | 1.1493 | 1.0087
PersEmoN [24] | 0.9034 | 0.6985 | 1.4097 | 0.9446 | 1.1469 | 1.0206
Amb-Fac [54] | 0.8529 | 0.6394 | 1.4212 | 0.9334 | 1.1410 | 0.9976
SENet [60] | 1.1398 | 0.6571 | 1.5526 | 1.1313 | 1.2733 | 1.1508
HRNet [61] | 1.1224 | 0.8163 | 1.8893 | 1.0362 | 1.5447 | 1.2818
Swin-transformer [62] | 1.0398 | 0.7119 | 1.3394 | 1.0186 | 1.2767 | 1.0773
3D-Resnet [63] | 1.3016 | 0.7051 | 1.2360 | 0.8741 | 1.3855 | 1.1004
Slow-Fast [64] | 1.0652 | 0.9862 | 2.0394 | 1.2631 | 1.2956 | 1.3299
TPN [65] | 0.9259 | 0.7596 | 1.2524 | 0.9458 | 1.4147 | 1.0597
| VAT [66] | 0.8554 | 0.7076 | 1.1398 | 0.8520 | 1.2286 | 0.9566
Aud-vis | Wei et al. [31] | 1.1657 | 0.7358 | 2.0523 | 1.1006 | 1.0745 | 1.2258
Subramaniam et al. [27] | 0.9281 | 0.7315 | 1.2944 | 0.8625 | 1.3152 | 1.0263
Güclütürk et al. [32] | 1.1417 | 0.7787 | 1.6838 | 1.0732 | 1.1512 | 1.1657
Li et al. [29] | 1.2834 | 0.8644 | 1.2876 | 1.0442 | 1.6987 | 1.2357
Suman et al. [54] | 0.9391 | 0.6675 | 1.4956 | 0.9588 | 1.1590 | 1.0440

TABLE IV: The MSE results achieved for the self-reported personality
recognition on the UDIVA dataset. The reported results are obtained by
averaging the results achieved for four sessions. For different modality
groups, audio, video, audio-visual, the values in bold denote the smallest
ones in the columns and the values with underline denote the second smallest
ones.

| Traits | Open | Consc | Extrav | Agree | Neuro | Avg.
---|---|---|---|---|---|---|---
Audio | FFT [57] | 0.0002 | -0.0001 | -0.0002 | -0.0002 | -0.0003 | -0.0002
MFCC/logfbank [31] | 0.1968 | 0.1497 | 0.1738 | 0.1295 | 0.1780 | 0.1655
Bi-modal CNN-LSTM [27] | -0.0004 | -0.0005 | 0.0004 | -0.0005 | -0.0008 | 0.0004
ResNet [32] | 0.1293 | 0.0830 | 0.0458 | 0.1101 | 0.1548 | 0.1046
CRNet [29] | 0.4122 | 0.3406 | 0.3846 | 0.2857 | 0.4306 | 0.3707
VGGish [54] | 0.4516 | 0.4493 | 0.4429 | 0.3127 | 0.4500 | 0.4213
Visual | DAN [31] | 0.5693 | 0.6254 | 0.6070 | 0.4855 | 0.6025 | 0.5779
Bi-modal CNN-LSTM[27] | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000
ResNet [32] | 0.1561 | 0.1902 | 0.1355 | 0.0838 | 0.1373 | 0.1406
CRNet [29] | 0.3748 | 0.3646 | 0.3987 | 0.2390 | 0.3226 | 0.3399
CAM-DAN+ [17] | 0.5882 | 0.6550 | 0.6326 | 0.5003 | 0.6199 | 0.5992
PersEmoN [24] | 0.2067 | 0.2441 | 0.2675 | 0.1369 | 0.1768 | 0.2064
Amb-Fac [54] | 0.5858 | 0.6750 | 0.5997 | 0.4971 | 0.5765 | 0.5868
SENet [60] | 0.5300 | 0.5580 | 0.5815 | 0.4493 | 0.5708 | 0.5379
HRNet [61] | 0.5923 | 0.6912 | 0.6436 | 0.5195 | 0.6273 | 0.6148
Swin-transformer [62] | 0.2223 | 0.2426 | 0.2531 | 0.1224 | 0.1942 | 0.2069
3D-Resnet [63] | 0.3248 | 0.3601 | 0.3601 | 0.2120 | 0.3352 | 0.3185
Slow-Fast [64] | 0.0256 | 0.0320 | 0.0185 | 0.0105 | 0.0184 | 0.0210
TPN [65] | 0.4427 | 0.4767 | 0.4998 | 0.3230 | 0.4675 | 0.4420
| VAT [66] | 0.6216 | 0.6753 | 0.6836 | 0.5228 | 0.6456 | 0.6298
Aud-vis | Wei et al. [31] | 0.4341 | 0.4645 | 0.4553 | 0.3519 | 0.4588 | 0.4329
Subramaniam et al. [27] | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000
Güclütürk et al. [32] | 0.4150 | 0.3671 | 0.3889 | 0.2679 | 0.4181 | 0.3714
Li et al. [29] | 0.5193 | 0.5106 | 0.5024 | 0.4026 | 0.5119 | 0.4894
Suman et al. [54] | 0.5618 | 0.6421 | 0.5921 | 0.4620 | 0.5734 | 0.5663

TABLE V: The CCC results achieved for the apparent personality recognition on
the ChaLearn 2016 First impression dataset.For different modality groups,
audio, video, audio-visual, the values in bold denote the highest ones in the
columns and the values with underline denote the second highest ones.

| Traits | Open | Consc | Extrav | Agree | Neuro | Avg.
---|---|---|---|---|---|---|---
Audio | FFT[57] | 0.8303 | 0.8174 | 0.8502 | 0.8309 | 0.8331 | 0.8324
MFCC/logfbank [31] | 0.8891 | 0.8790 | 0.8835 | 0.8967 | 0.8802 | 0.8857
Bi-modal CNN-LST [27] | 0.8835 | 0.8747 | 0.8785 | 0.8937 | 0.8774 | 0.8816
ResNet [32] | 0.8822 | 0.8780 | 0.8782 | 0.8958 | 0.8820 | 0.8832
CRNet [29] | 0.9001 | 0.8895 | 0.8951 | 0.9022 | 0.8964 | 0.8967
Vggish-feat [54] | 0.9010 | 0.8962 | 0.8959 | 0.9028 | 0.8953 | 0.8982
Visual | DAN [31] | 0.9098 | 0.9106 | 0.9096 | 0.9102 | 0.9061 | 0.9093
Bi-modal CNN-LSTM[27] | 0.8832 | 0.8742 | 0.8778 | 0.8933 | 0.8770 | 0.8811
ResNet [32] | 0.8896 | 0.8835 | 0.8837 | 0.8968 | 0.8830 | 0.8873
CRNet [29] | 0.8987 | 0.8932 | 0.8952 | 0.9018 | 0.8908 | 0.8960
CAM-DAN+ [17] | 0.9115 | 0.9139 | 0.9126 | 0.9118 | 0.9089 | 0.9118
PersEmoN [24] | 0.8934 | 0.8893 | 0.8913 | 0.8994 | 0.8866 | 0.8920
Amb-Fac [54] | 0.9101 | 0.9141 | 0.9082 | 0.9095 | 0.9038 | 0.9091
SENet [60] | 0.9076 | 0.906 | 0.908 | 0.9097 | 0.9061 | 0.9075
HRNet [61] | 0.9101 | 0.9154 | 0.9111 | 0.9113 | 0.9084 | 0.9113
Swin-transformer [62] | 0.8937 | 0.8870 | 0.8893 | 0.8983 | 0.8860 | 0.8909
3D-Resnet [63] | 0.8964 | 0.8921 | 0.8933 | 0.9008 | 0.8915 | 0.8948
Slow-Fast [64] | 0.8780 | 0.8604 | 0.8443 | 0.8809 | 0.8613 | 0.8650
TPN [65] | 0.9025 | 0.8963 | 0.9019 | 0.9013 | 0.8992 | 0.9003
| VAT [66] | 0.9115 | 0.9123 | 0.9153 | 0.9099 | 0.9098 | 0.9118
Aud-vis | Wei et al. [31] | 0.9049 | 0.9020 | 0.9024 | 0.9081 | 0.9015 | 0.9038
Subramaniam et al. [27] | 0.8833 | 0.8744 | 0.8779 | 0.8935 | 0.8773 | 0.8813
Güclütürk et al. [32] | 0.8996 | 0.8918 | 0.8945 | 0.9015 | 0.8948 | 0.8964
Li et al. [29] | 0.9075 | 0.9019 | 0.9017 | 0.9055 | 0.9034 | 0.9040
Suman et al. [54] | 0.9127 | 0.9169 | 0.9117 | 0.9133 | 0.9088 | 0.9127

TABLE VI: The ACC results achieved for the apparent personality recognition on
the ChaLearn 2016 First impression dataset.For different modality groups,
audio, video, audio-visual, the values in bold denote the highest ones in the
columns and the values with underline denote the second highest ones.

###  4.2 Ablation studies

In this section, we evaluate the influences of different pre-processing, post-
processing and model settings on personality recognition performances.

####  4.2.1 Full frames VS. Face regions

We compare the results achieved by feeding aligned face regions and full
frames (containing both the face and backgrounds) to standard visual deep
learning models in Fig. 2 and Fig. 3, where the standard visual deep learning
models generally provide more reliable self-reported and apparent personality
predictions by using face regions, despite the fact that neither full frames
nor face regions provide reliable self-reported personality predictions. In
contrast, apparent personality predictions generated by all visual models
using both face regions and full frames have CCC $>0.37$ on average with the
ground-truth, where face region-based predictions still have $1.62\%$ CCC
advantage over full frame-based predictions. However, it can be seen that
using face regions and full frames does not cause a large difference for most
models, where only HRNet is sensitive to this variable on both tasks. These
results suggest that although some previous studies claimed that backgrounds
and spatial contextual information can provide informative cues for
personality recognition, they may also contain personality-unrelated noise
that can negatively impact the personality recognition. In short, the results
achieved by the benchmarked standard visual deep learning models show that the
personality-related cues contained in background and spatial contextual
information can not always compensate their negative impact for personality
recognition.

(a) CCC results achieved for the UDIVA dataset.

(b) MSE results achieved for the UDIVA dataset.

Figure 2: Comparison between the self-reported personality recognition results
achieved by different visual models using either aligned faces or original
full frames.

(a) CCC results achieved for the ChaLearn First Impression dataset.

(b) MSE results achieved for the ChaLearn First Impression dataset.

Figure 3: Comparison between the apparent personality recognition CCC results
achieved by different visual models using either aligned faces or original
full frames.

####  4.2.2 Spatial VS. Spatio-temporal visual models

We compare the average results achieved by spatial visual models (i.e., frame-
level systems) and spatio-temporal visual models (i.e., short segment-level
systems and video-level systems) on both datasets in Fig. 4. It can be
observed that the average CCC performance achieved by the spatial models
outperforms the spatio-temporal settings for both self-reported and apparent
personality recognition tasks, i.e., while no model is reliable for inferring
self-reported personality from facial behaviours, spatial models generally
provide better self-reported personality predictions for all five traits than
short segment-level spatio-temporal models as well as four of the five traits
than video-level spatio-temporal models. Nevertheless, we found that a spatio-
temporal model VAT achieved the best CCC performance in predicting apparent
personality traits, with CCC $>0.6$ for predicting four of the five traits,
which suggest that the VAT’s temporal modelling strategy is suitable to
extract apparent personality-related facial behaviour cues. Overall, spatial
visual models consistently achieve better average performance than spatio-
temporal models in predicting all apparent personality traits. These results
indicate that the most standard spatio-temporal deep learning models (except
VAT) failed to extract personality-related cues from human non-verbal spatio-
temporal facial behaviours. In contrast, spatial visual models encode more
reliable and discriminative personality cues from static facial
expressions/identities.

(a) CCC results achieved for the UDIVA dataset.

(b) CCC results achieved for the ChaLearn First Impression dataset.

Figure 4: Comparison between average results achieved by frame-level models
and spatio-temporal models, where both short segment-level system and video-
level system are evaluated for spatio-temporal models.

####  4.2.3 Short segment-level VS. video-level personality modelling

Fig. 4 also compares short segment-level modelling and video-level modelling
systems, where four widely-used spatio-temporal models introduced in Sec.
3.2.3 are benchmarked. Although both short segment-level modelling and video-
level modelling systems generally failed to provide reliable self-reported
personality predictions, the short segment-level systems still achieved
slightly better average performances than the video-level systems for four
traits (i.e., Ope, Con, Agr, and Neu traits). Moreover, the short segment-
level systems have clear advantages over the video-level systems for
predicting all apparent personality traits. These results suggest that the
short-term non-verbal behaviours contain crucial apparent personality-related
cues while the standard video-level models which ignored short-term behaviours
can not extract reliable apparent personality cues from the down-sampled long-
term behaviours (i.e., down-sampling a long clip that contains around 400
frames to a segment containing 32 frames results in the lose of short-term
behavioural cues).

####  4.2.4 Audio VS. Visual VS. Audio-visual models

As shown in Fig. 5, visual models and audio-visual models on average generated
superior personality recognition performances than audio models for all traits
on both datasets. We found that CAM-DAN+, HRNet and SENet visual models
achieved relatively promising results for both self-reported and apparent
personality recognition tasks. However, the low CCC performance on self-
reported personality recognition still shows that standard audio/visual
machine learning models can hardly extract self-reported personality-related
cues from audio-visual behaviours. As a result, the five audio-visual systems
displayed in Table V do not show clear and consistent advantages over their
corresponding visual counterparts. In contrast, the audio and facial
behaviours sometimes are more reliable in reflecting apparent personality
traits. According to Table V, adding the audio modality allows three out of
the five audio-visual systems [29, 32, 54] achieved better performances over
their corresponding visual methods in predicting apparent personality traits.
However, when the extracted audio features are not well associated with
apparent personality traits, they may even negatively impact the personality
recognition ability of the visual system (e.g., he MFCC/logbank audio features
degrade the DAN-based visual system [31]). In summary, these results suggest
that (i) non-verbal facial behaviours contain more discriminative personality-
related cues than non-verbal audio signals for both self-reported and apparent
personality recognition tasks; and (ii) if the audio modality can individually
provide reliable personality predictions, it is expected to further enhance
the predictions of the visual system.

(a) CCC results achieved for the UDIVA dataset.

(b) CCC results achieved for the ChaLearn First Impression dataset.

Figure 5: Comparison between the average results achieved by the audio models,
the visual models and the audio-visual models.

####  4.2.5 Summarising frame/segment-level personality predictions

We then compare the results achieved by the different strategies for
summarising frame/segment-level personality predictions for obtaining clip-
level personality prediction by: (i) simply averaging frame-level personality
predictions as the clip-level personality prediction; and (ii) encoding a
spectral representation of frame/segment-level predictions. Specifically, we
first encode a pair of spectral heatmaps from the five-channel frame/segment-
level personality trait predictions. Then, we train a regressor that contains
seven 1D convolution layers followed by two fully connected layers to predict
clip-level personality traits from the produced spectral heatmaps. As shown in
Table VII, applying spectral representation to encode frame/segment-level
personality predictions has consistent advantages over the averaging strategy
for summarising clip-level apparent personality predictions, as the
performances of all models improved using spectral representation. This also
indicates that the spectral representation can capture reliable personality-
related spatio-temporal cues from frame/segment-level personality predictions.
The improvements of applying spectral representation for self-reported
personality recognition are not as clear as recognising apparent personality
traits. We assume this is because frame-level predictions of self-reported
personality traits are not reliable, and thus it is difficult to capture
reliable personality-related spatio-temporal cues from them.

|  Clip-level representation |  DAN[31] |  ResNet[32] |  CRNet[61] |  PersEmoN [24] |  CAM-DAN+ [17] |  SENet[60] |  HRNet [61] |  Swin[62]
---|---|---|---|---|---|---|---|---|---
(i) |  AFP |  0.5779 |  0.3714 |  0.4894 |  0.2064 |  0.5992 |  0.5122 |  0.5270 |  0.2333
SFP |  0.5829(+0.0050) |  0.3980(+0.0266) |  0.5003(0.0109) |  0.2881(+0.0817) |  0.6209(+0.0217) |  0.5154(0.0032) |  0.5314(+0.0044) |  0.2366(+0.0033)
(ii) |  AFP |  0.0736 |  0.0898 |  0.1311 |  0.0068 |  0.1059 |  0.1191 |  0.1007 |  0.0644
SFP |  0.0355(-0.0381) |  0.1021(+0.0123) |  0.0146(-0.1165) |  0.0020(-0.0048) |  0.0106(-0.0953) |  0.0239(-0.0952) |  0.0200(-0.0807) |  0.0011(-0.0633)
TABLE VII: Long-term modelling results of the benchmarked frame/short segment-
level personality computing visual models, where (i) denotes the results
achieved on ChaLearn 2016 First Impression dataset, and (ii) denotes the
results achieved on UDIVA dataset. AFP and SFP represent the averaging frame-
level predictions and spectral representation of frame-level predictions,
respectively.

###  4.3 Discussion

In addition to reporting the results achieved by the benchmarked models and
discussing the results achieved by the different settings, this section
discusses other relevant aspects of the benchmarked models, including: (i) the
reproduced models generally achieving lower results than the originally
reported results; (ii) personality traits having different relationships with
the non-verbal behaviours; and (iii) challenges and research gaps in the
current audio-visual personality computing models.

####  4.3.1 Lower performance than originally reported

As compared in Table VIII, one important finding in this work is that all
reproduced models failed to achieve the same or superior results as their
originally reported results on the ChaLearn 2016 First Impression dataset,
even when we followed the exact same pre-processing and training settings they
have reported. This could be caused by various reasons, including the original
papers not reporting all training/evaluation/pre-processing tricks and model
details they used to obtain the reported models, or the random factors
involved in training processes (e.g., the shuffled training samples).
Moreover, the results achieved by most reproduced models are not as good as
the corresponding systems that utilized the proposed standardised pre-
processing and training steps. This validated the effectiveness of the
proposed standardised pre-processing pipeline, which can be adopted by future
researchers to report on their proposed models for fair comparisons.

Model | Reproduced | Standardised | Reported
---|---|---|---
DAN[31] | 0.9069 | 0.9093 | 0.9130
Bi-modal CNN-LSTM[27] | 0.8812 | 0.8813 | 0.9121
ResNet[32] | 0.8939 | 0.8964 | 0.9109
CRNet[29] | 0.9019 | 0.9040 | 0.9188
PersEmoN[24] | 0.8912 | 0.8920 | 0.9170
CAM-DAN+[17] | 0.9070 | 0.9118 | 0.9120
FFT[57] | 0.8598 | 0.8324 | 0.9009
Amb-Fac-VGGish[54] | 0.9127 | 0.9102 | 0.9146

TABLE VIII: Mean ACC results for the benchmarked/reproduced models on the
ChaLearn 2016 First Impression dataset. Reproduced denotes the data were
processed based on the method described in their papers; Standardised
indicates the results obtained from our uniformed data processing pipeline;
and Reported reports the performances provided by their original papers.

####  4.3.2 Relationships between the personality traits and non-verbal
behaviours

Fig. 4 show that self-reported Conscientiousness and Neuroticism traits are
relatively reliable to be inferred by static facial displays than other
traits. Self-reported Extraversion trait is less related to static facial
behaviours but more associated with long-term spatio-temporal facial
behaviours. In terms of apparent personality traits, it can be seen that they
have similar relationship patterns with facial behaviours of three temporal
scales (frame-level, short segment-level and video-level), i.e., the
behaviours of all three temporal scales are more reliable to infer
Conscientiousness, Extraversion, Neuroticism, and Openness traits, while all
of their associations with the Agreeableness trait are clearly weaker.

While facial behaviours are clearly more informative than audio behaviours in
predicting all personality traits, Fig. 5 illustrates that the self-reported
Conscientiousness and Neuroticism traits can be better reflected by facial
behaviours while Extraversion and Openness traits are less associated with
facial behaviours. Again, both audio and facial behaviours are more reliable
in inferring apparent Conscientiousness, Extraversion, Neuroticism, and
Openness traits but are less correlated with the Agreeableness trait.

Based on these results, we can conclude that each true or apparent personality
trait has a unique relationship with human non-verbal behaviours, including
their temporal scales and modalities. In particular, we found that: (i) non-
verbal audio-visual behaviours are more reliable in inferring apparent
personalities; (ii) facial behaviours are more informative than audio
behaviours in inferring personality traits; (iii) self-reported
Conscientiousness and Neuroticism traits can be easier to be inferred from
non-verbal facial behaviours than other self-reported traits; and (iv)
apparent Agreeableness trait is more difficult to be predicted from facial
behaviours than other apparent personality traits.

####  4.3.3 Challenges and research gaps

Although an increasing number of automatic personality computing approaches
have been proposed in recent years, both their reported results and our
benchmarking results show that existing approaches are not very reliable for
inferring personality traits, especially for self-reported personality. We can
summarise the research gaps of existing personality computing approaches as
follows:

  * •

(i) Most existing studies [31, 32, 17, 26, 21, 76, 22] attempt to infer
personality traits directly from static frames or short audio-visual segments
without considering contextual information. Since subjects that have different
personalities can show similar behaviours in a single frame or a short audio-
visual segment (e.g., different subjects can show a happy expression in a
single frame), a static frame or a short audio-visual segment is not reliable
for inferring personality. Also pairing static frame or a short audio-visual
segment with clip-level personality labels to train models results in an ill-
posed machine learning problem, making the trained model theoretically not
have good generalization capability. Our results show that the SFP-based long-
term modelling of frame/short segment-level predictions produced by reliable
frame/short segment-based systems (e.g., most frame/short segment-level
apparent personality recognition models) leads to the enhanced personality
recognition results (Table VII).

  * •

(ii) Existing clip-level personality models can partially address the
aforementioned ill-posed machine learning problem during model training.
However, our benchmark shows that their results are worse as compared to the
static frames or short audio-visual segments-based approaches. This can be
explained by the fact that most clip-level spatio-temporal personality
computing models [29] (including the benchmarked standard spatio-temporal
models) require down-sampling the original video, i.e., a large number of
frames are discarded, i.e., crucial short-term personality-related cues are
ignored during training and inference stages. While some clip-level
personality computing studies [34, 36, 18] retain almost all frames in a
video, these methods are time-consuming to implement and train as they need to
individually train a person-specific network for each subject. In short, there
lacks an efficient clip-level audio-visual personality computing framework
that can extract personality-related cues from all frames of the input data.

  * •

(iii) Most existing deep learning-based personality computing approaches
simply pair the given audio-visual data and labels to train models, i.e., they
do not take the personality-related domain knowledge into consideration when
designing and training their models. As we can see from the benchmarked
results, such strategies can not produce models that are able to provide
reliable personality predictions. In other words, it is necessary for future
studies to systematically review the physiological and psychological findings
to concretely model the relationship between true/apparent personality traits
and human non-verbal behaviours (e.g., the behaviours/temporal patterns that
are more frequently displayed by subjects that have a certain personality
traits pattern), and integrate such knowledge to design more advanced
personality computing models and training strategies.

Besides the research gaps, we also discuss the challenges for both current and
future researchers in developing new and more advanced personality models as
follows:

  * •

(i) There is a need of open-source and easily implemented code that allow
researcher to improve the existing approaches or develop more advanced models
based on it.

  * •

(ii) Existing approaches usually have different and complex pre-processing
strategies. A large number of the published approaches do not provide full
details of their pre-processing steps, model settings and training strategies
due to the limited page numbers. Subsequently, it is difficult for researchers
to fully reproduce their methods and results, i.e., our benchmarking results
are not as the same as the results reported in their original papers.

  * •

(iii) Due to the limited number of publicly available datasets, most existing
personality computing models are only evaluated on a single dataset (e.g.,
ChaLearn 2016 First Impression dataset). Based on the results achieved by our
benchmark, we found the same model can have completely different performances
in predicting self-reported or apparent personality traits on different a
dataset. Therefore, we assume that personality computing models should be
evaluated on more than one dataset (i.e., multiple apparent personality
datasets or self-reported personality datasets) which have different data
recording settings and scenarios. Otherwise, if a model achieves excellent
performance on a single dataset, we can only conclude that this model is well
adapted to the dataset at hand, but can not be made about its reliability and
capability in predicting personality in general.

##  5 Conclusion

In this paper, we propose the first audio-visual personality computing
benchmark for both self-reported and apparent personality recognition tasks,
which are evaluated on the two widely-used and publicly available personality
computing datasets: the ChaLearn First Impression dataset and the ChaLearn
UDIVA self-reported personality dataset. We first benchmarked seven visual
models, six audio models and five audio-visual models that have been published
and evaluated on the ChaLearn First Impression datasets. We also benchmarked
seven visual deep learning models that are widely used for visual problems,
which have not been applied to video-based personality computing before. Our
open-source, easy-to-use and standardised framework can be utilized to not
only develop new personality computing models but also conduct quick yet
robust evaluation of new models on both self-reported and apparent personality
traits recognition tasks.

Building on our benchmarked models, we systematically evaluated various
factors that impact personality computing. This led us to make the following
conclusions: (i) using cropped and aligned face images as input generally led
models to produce slightly better personality predictions than using images
with background; (ii) the static models frequently achieved better
performances than most spatio-temporal models for personality recognition;
(iii) visual models frequently achieved superior results than audio models on
personality recognition, indicating that subjects’ facial behaviours contain
more discriminative apparent personality-related cues than non-verbal audio
behaviours; (iv) most models achieved better apparent personality recognition
results than self-reported personality recognition even when different
datasets were used, which suggests that subjects do not show their self-
reported personality through non-verbal audio-visual behaviours; and (v) each
personality trait has a unique relationship with human non-verbal behaviours
(i.e., modalities and temporal scale), and thus even the same data contribute
differently for personality recognition.

We also discussed the current research gaps and challenges in audio-visual
personality computing. Our work provides a standardized data loading, pre-
processing and model training framework as well as a set of reproduced
personality computing models, which partially address the discussed issues.
However, creating an extensive benchmark must be a common effort, and thus our
benchmark does not include all previous personality computing models. Also,
our benchmark does not include other modalities (e.g., language) as current
version only focuses on inferring personality traits from non-verbal audio-
visual behavioural signals. Our future work will focus on updating the
benchmark every year by adding new published personality computing models,
evaluating the benchmarked models on new personality datasets as well as
adding more modalities (e.g., language) with our own extensions and with the
help of the current and the future researchers’ contributions. We hope that
this paper serves as an open call to all interested researchers to join forces
with us to integrate their personality computing models into our publicly
available framework.

## Acknowledgments

Funding: The work of Siyang Song was funded partially by the EPSRC/UKRI under
grant ref. EP/R030782/1, and partially by the European Union’s Horizon 2020
research and innovation programme project WorkingAge, under grant agreement
No. 82623. Hatice Gunes is supported by the EPSRC/UKRI under grant ref.
EP/R030782/1.

Open Access Statement: For the purpose of open access, the authors have
applied a Creative Commons Attribution (CC BY) license to any Accepted
Manuscript version arising.

Data Access Statement: This study involves secondary analyses of existing
datasets, that are described and cited in the text. Licensing restrictions
prevent sharing of the datasets.

Code Access: https://github.com/liaorongfan/DeepPersonality.

## References

  * [1] P. J. Corr and G. Matthews, _The Cambridge handbook of personality psychology_. Cambridge University Press, 2020.
  * [2] S. Youn and R. J. Faber, “Impulse buying: its relation to personality traits and cues,” _ACR North American Advances_ , 2000.
  * [3] A. Sano, A. J. Phillips, Z. Y. Amy, A. W. McHill, S. Taylor, N. Jaques, C. A. Czeisler, E. B. Klerman, and R. W. Picard, “Recognizing academic performance, sleep quality, stress level, and mental health using personality traits, wearable sensors and mobile phones,” in _2015 IEEE 12th International Conference on Wearable and Implantable Body Sensor Networks (BSN)_. IEEE, 2015, pp. 1–6.
  * [4] S. Jaiswal, S. Song, and M. Valstar, “Automatic prediction of depression and anxiety from behaviour and personality attributes,” in _2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII)_. IEEE, 2019, pp. 1–7.
  * [5] J. B. Asendorpf and S. Wilpers, “Personality effects on social relationships.” _Journal of personality and social psychology_ , vol. 74, no. 6, p. 1531, 1998.
  * [6] D. A. Briley and E. M. Tucker-Drob, “Genetic and environmental continuity in personality development: a meta-analysis.” _Psychological bulletin_ , vol. 140, no. 5, p. 1303, 2014.
  * [7] S. Afzal, B. Dempsey, C. D’Helon, N. Mukhi, M. Pribic, A. Sickler, P. Strong, M. Vanchiswar, and L. Wilde, “The personality of ai systems in education: experiences with the watson tutor, a one-on-one virtual tutoring system,” _Childhood Education_ , vol. 95, no. 1, pp. 44–52, 2019.
  * [8] D. Mansour, A. B. Bhardwaj, and A. Chopra, “Relating ocean (big five) to job satisfaction in aviation,” in _2021 International Conference on Computational Intelligence and Knowledge Economy (ICCIKE)_. IEEE, 2021, pp. 285–289.
  * [9] H. Kaya, F. Gurpinar, and A. Ali Salah, “Multi-modal score fusion and decision trees for explainable automatic job candidate screening from video cvs,” in _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , 2017, pp. 1–9.
  * [10] F. Xia, N. Y. Asabere, H. Liu, Z. Chen, and W. Wang, “Socially aware conference participant recommendation with personality traits,” _IEEE Systems Journal_ , vol. 11, no. 4, pp. 2255–2266, 2014.
  * [11] H. Eysenck, “Dimensions of personality: 16, 5 or 3?—criteria for a taxonomic paradigm,” _Personality and Individual Differences_ , vol. 12, no. 8, pp. 773–790, 1991. [Online]. Available: https://www.sciencedirect.com/science/article/pii/019188699190144Z
  * [12] J. T. Mitchell, N. A. Kimbrel, N. E. Hundt, A. R. Cobb, R. O. Nelson-Gray, and C. M. Lootens, “An analysis of reinforcement sensitivity theory and the five-factor model,” _European Journal of Personality: Published for the European Association of Personality Psychology_ , vol. 21, no. 7, pp. 869–887, 2007.
  * [13] F. De Fruyt, L. Van de Wiele, and C. Van Heeringen, “Cloninger’s psychobiological model of temperament and character and the five-factor model of personality,” _Personality and individual differences_ , vol. 29, no. 3, pp. 441–452, 2000.
  * [14] R. R. McCrae and P. T. Costa Jr, “The five-factor theory of personality.” 2008\.
  * [15] G. Matthews, I. J. Deary, and M. C. Whiteman, _Personality traits_. Cambridge University Press, 2003.
  * [16] V. Ponce-López, B. Chen, M. Oliu, C. Corneanu, A. Clapés, I. Guyon, X. Baró, H. J. Escalante, and S. Escalera, “Chalearn lap 2016: First round challenge on first impressions-dataset and results,” in _European conference on computer vision_. Springer, 2016, pp. 400–418.
  * [17] C. Ventura, D. Masip, and A. Lapedriza, “Interpreting cnn models for apparent personality trait regression,” in _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , 2017, pp. 55–63.
  * [18] S. Song, Z. Shao, S. Jaiswal, L. Shen, M. Valstar, and H. Gunes, “Learning graph representation of person-specific cognitive processes from audio-visual behaviours for automatic personality recognition,” _arXiv preprint arXiv:2110.13570_ , 2021.
  * [19] M. A. Moreno-Armendáriz, C. A. D. Martínez, H. Calvo, and M. Moreno-Sotelo, “Estimation of personality traits from portrait pictures using the five-factor model,” _IEEE Access_ , vol. 8, pp. 201 649–201 665, 2020.
  * [20] D. Helm and M. Kampel, “Single-modal video analysis of personality traits using low-level visual features,” in _2020 Tenth International Conference on Image Processing Theory, Tools and Applications (IPTA)_. IEEE, 2020, pp. 1–6.
  * [21] S. Eddine Bekhouche, F. Dornaika, A. Ouafi, and A. Taleb-Ahmed, “Personality traits and job candidate screening via analyzing facial videos,” in _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , 2017, pp. 10–13.
  * [22] M. K. Tellamekala, T. Giesbrecht, and M. Valstar, “Dimensional affect uncertainty modelling for apparent personality recognition,” _IEEE Transactions on Affective Computing_ , 2022.
  * [23] F. Gürpınar, H. Kaya, and A. A. Salah, “Combining deep facial and ambient features for first impression estimation,” in _European conference on computer vision_. Springer, 2016, pp. 372–385.
  * [24] L. Zhang, S. Peng, and S. Winkler, “Persemon: a deep network for joint analysis of apparent personality, emotion and their relationship,” _IEEE Transactions on Affective Computing_ , 2019.
  * [25] C. Beyan, A. Zunino, M. Shahid, and V. Murino, “Personality traits classification using deep visual activity-based nonverbal features of key-dynamic images,” _IEEE Transactions on Affective Computing_ , 2019.
  * [26] C.-L. Zhang, H. Zhang, X.-S. Wei, and J. Wu, “Deep bimodal regression for apparent personality analysis,” in _European conference on computer vision_. Springer, 2016, pp. 311–324.
  * [27] A. Subramaniam, V. Patel, A. Mishra, P. Balasubramanian, and A. Mittal, “Bi-modal first impressions recognition using temporally ordered deep audio and stochastic visual features,” in _European conference on computer vision_. Springer, 2016, pp. 337–348.
  * [28] F. Gürpinar, H. Kaya, and A. A. Salah, “Multimodal fusion of audio, scene, and face features for first impression estimation,” in _2016 23rd International conference on pattern recognition (ICPR)_. IEEE, 2016, pp. 43–48.
  * [29] Y. Li, J. Wan, Q. Miao, S. Escalera, H. Fang, H. Chen, X. Qi, and G. Guo, “Cr-net: A deep classification-regression network for multimodal apparent personality analysis.” _International Journal of Computer Vision_ , vol. 128, no. 12, 2020.
  * [30] S. Aslan and U. Güdükbay, “Multimodal video-based apparent personality recognition using long short-term memory and convolutional neural networks,” _arXiv preprint arXiv:1911.00381_ , 2019.
  * [31] X.-S. Wei, C.-L. Zhang, H. Zhang, and J. Wu, “Deep bimodal regression of apparent personality traits from short video sequences,” _IEEE Transactions on Affective Computing_ , vol. 9, no. 3, pp. 303–315, 2018.
  * [32] Y. Güçlütürk, U. Güçlü, M. A. van Gerven, and R. van Lier, “Deep impression: Audiovisual deep residual networks for multimodal apparent personality trait recognition,” in _European conference on computer vision_. Springer, 2016, pp. 349–358.
  * [33] S. Song, S. Jaiswal, L. Shen, and M. Valstar, “Spectral representation of behaviour primitives for depression analysis,” _IEEE Transactions on Affective Computing_ , 2020.
  * [34] S. Song, S. Jaiswal, E. Sanchez, G. Tzimiropoulos, L. Shen, and M. Valstar, “Self-supervised learning of person-specific facial dynamics for automatic personality recognition,” _IEEE Transactions on Affective Computing_ , 2021\.
  * [35] D. Curto, A. Clapés, J. Selva, S. Smeureanu, J. Junior, C. Jacques, D. Gallardo-Pujol, G. Guilera, D. Leiva, T. B. Moeslund _et al._ , “Dyadformer: A multi-modal transformer for long-range modeling of dyadic interactions,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 2177–2188.
  * [36] Z. Shao, S. Song, S. Jaiswal, L. Shen, M. Valstar, and H. Gunes, “Personality recognition by modelling person-specific cognitive processes using graph representation,” in _Proceedings of the 29th ACM International Conference on Multimedia_ , 2021, pp. 357–366.
  * [37] H. Hayat, C. Ventura, and A. Lapedriza, “On the use of interpretable cnn for personality trait recognition from audio.” in _CCIA_ , 2019, pp. 135–144.
  * [38] Y. Güçlütürk, U. Güçlü, M. A. J. van Gerven, and R. van Lier, “Deep impression: Audiovisual deep residual networks for multimodal apparent personality trait recognition,” _Computer Vision – ECCV 2016 Workshops_ , p. 349–358, 2016. [Online]. Available: http://dx.doi.org/10.1007/978-3-319-49409-8_28
  * [39] C. Palmero, J. Selva, S. Smeureanu, J. C. J. Junior, A. Clapés, A. Moseguí, Z. Zhang, D. Gallardo, G. Guilera, D. Leiva _et al._ , “Context-aware personality inference in dyadic scenarios: Introducing the udiva dataset.” in _WACV (Workshops)_ , 2021, pp. 1–12.
  * [40] J. C. S. J. Junior, Y. Güçlütürk, M. Pérez, U. Güçlü, C. Andujar, X. Baró, H. J. Escalante, I. Guyon, M. A. Van Gerven, R. Van Lier _et al._ , “First impressions: A survey on vision-based apparent personality trait analysis,” _IEEE Transactions on Affective Computing_ , 2019.
  * [41] J. Joo, F. F. Steen, and S.-C. Zhu, “Automated facial trait judgment and election outcome prediction: Social dimensions of face,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 3712–3720.
  * [42] A. Dhall and J. Hoey, “First impressions-predicting user personality from twitter profile images,” in _International Workshop on Human Behavior Understanding_. Springer, 2016, pp. 148–158.
  * [43] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba, “Learning deep features for discriminative localization,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 2921–2929.
  * [44] O. Celiktutan and H. Gunes, “Continuous prediction of perceived traits and social dimensions in space and time,” in _Image Processing (ICIP), 2014 IEEE International Conference on_. IEEE, 2014, pp. 4196–4200.
  * [45] O. Celiktutan, F. Eyben, E. Sariyanidi, H. Gunes, and B. Schuller, “Maptraits 2014-the first audio/visual mapping personality traits challenge-an introduction: Perceived personality and social dimensions,” in _Proceedings of the 16th International Conference on Multimodal Interaction_. ACM, 2014, pp. 529–530.
  * [46] F. Eyben, M. Wöllmer, and B. Schuller, “Opensmile: the munich versatile and fast open-source audio feature extractor,” in _Proceedings of the 18th ACM international conference on Multimedia_ , 2010, pp. 1459–1462.
  * [47] R. Qin, W. Gao, H. Xu, and Z. Hu, “Modern physiognomy: an investigation on predicting personality traits and intelligence from the human face,” _Science China Information Sciences_ , vol. 61, no. 5, p. 058105, 2018.
  * [48] A. Cafaro, J. Wagner, T. Baur, S. Dermouche, M. Torres Torres, C. Pelachaud, E. André, and M. Valstar, “The noxi database: multimodal recordings of mediated novice-expert interactions,” in _Proceedings of the 19th ACM International Conference on Multimodal Interaction_ , 2017, pp. 350–359.
  * [49] O. Celiktutan, E. Skordos, and H. Gunes, “Multimodal human-human-robot interactions (mhhri) dataset for studying personality and engagement,” _IEEE Transactions on Affective Computing_ , vol. 10, no. 4, pp. 484–497, 2017.
  * [50] H. Salam, V. Manoranjan, J. Jiang, and O. Celiktutan, “Learning personalised models for automatic self-reported personality recognition,” in _Understanding Social Behavior in Dyadic and Small Group Interactions_. PMLR, 2022, pp. 1–21.
  * [51] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga _et al._ , “Pytorch: An imperative style, high-performance deep learning library,” _Advances in neural information processing systems_ , vol. 32, pp. 8026–8037, 2019.
  * [52] K. Zhang, Z. Zhang, Z. Li, and Y. Qiao, “Joint face detection and alignment using multitask cascaded convolutional networks,” _IEEE Signal Processing Letters_ , vol. 23, no. 10, pp. 1499–1503, 2016.
  * [53] N. Boyko, O. Basystiuk, and N. Shakhovska, “Performance evaluation and comparison of software for face recognition, based on dlib and opencv library,” in _2018 IEEE Second International Conference on Data Stream Mining & Processing (DSMP)_. IEEE, 2018, pp. 478–482.
  * [54] C. Suman, S. Saha, A. Gupta, S. K. Pandey, and P. Bhattacharyya, “A multi-modal personality prediction system,” _Knowledge-Based Systems_ , vol. 236, p. 107715, 2022.
  * [55] H.-Y. Suen, K.-E. Hung, and C.-L. Lin, “Tensorflow-based automatic personality recognition used in asynchronous video interviews,” _IEEE Access_ , vol. 7, pp. 61 018–61 023, 2019.
  * [56] S. Ruder, “An overview of gradient descent optimization algorithms,” _arXiv preprint arXiv:1609.04747_ , 2016.
  * [57] H. Hayat, C. Ventura, and À. Lapedriza, “On the use of interpretable cnn for personality trait recognition from audio,” in _CCIA_ , 2019.
  * [58] T. Giannakopoulos, “pyaudioanalysis: An open-source python library for audio signal analysis,” _PloS one_ , vol. 10, no. 12, p. e0144610, 2015.
  * [59] B. McFee, C. Raffel, D. Liang, D. P. Ellis, M. McVicar, E. Battenberg, and O. Nieto, “librosa: Audio and music signal analysis in python,” in _Proceedings of the 14th python in science conference_ , vol. 8. Citeseer, 2015, pp. 18–25.
  * [60] J. Hu, L. Shen, S. Albanie, G. Sun, and E. Wu, “Squeeze-and-excitation networks,” 2019.
  * [61] J. Wang, K. Sun, T. Cheng, B. Jiang, C. Deng, Y. Zhao, D. Liu, Y. Mu, M. Tan, X. Wang, W. Liu, and B. Xiao, “Deep high-resolution representation learning for visual recognition,” 2020.
  * [62] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” 2021.
  * [63] K. Hara, H. Kataoka, and Y. Satoh, “Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet?” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018, pp. 6546–6555.
  * [64] C. Feichtenhofer, H. Fan, J. Malik, and K. He, “Slowfast networks for video recognition,” 2019.
  * [65] C. Yang, Y. Xu, J. Shi, B. Dai, and B. Zhou, “Temporal pyramid network for action recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * [66] R. Girdhar, J. Carreira, C. Doersch, and A. Zisserman, “Video action transformer network,” 2019.
  * [67] S. Song, L. Shen, and M. Valstar, “Human behaviour-based automatic depression analysis using hand-crafted statistics and deep learned spectral features,” in _2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018)_. IEEE, 2018, pp. 158–165.
  * [68] J.-I. Biel and D. Gatica-Perez, “Voices of vlogging,” in _Fourth International AAAI Conference on Weblogs and Social Media_ , 2010.
  * [69] D. Sanchez-Cortes, O. Aran, M. S. Mast, and D. Gatica-Perez, “A nonverbal behavior approach to identify emergent leaders in small groups,” _IEEE Transactions on Multimedia_ , vol. 14, no. 3, pp. 816–832, 2011.
  * [70] J. A. Miranda-Correa, M. K. Abadi, N. Sebe, and I. Patras, “Amigos: A dataset for affect, personality and mood research on individuals and groups,” _IEEE Transactions on Affective Computing_ , vol. 12, no. 2, pp. 479–493, 2018.
  * [71] J. A. M. Correa, M. K. Abadi, N. Sebe, and I. Patras, “Amigos: A dataset for affect, personality and mood research on individuals and groups,” _IEEE Transactions on Affective Computing_ , 2018.
  * [72] M. K. Rothbart, S. A. Ahadi, K. L. Hershey, and P. Fisher, “Investigations of temperament at three to seven years: The children’s behavior questionnaire,” _Child development_ , vol. 72, no. 5, pp. 1394–1408, 2001.
  * [73] L. K. Ellis and M. K. Rothbart, “Revision of the early adolescent temperament questionnaire,” in _Poster presented at the 2001 biennial meeting of the society for research in child development, Minneapolis, Minnesota_. Citeseer, 2001.
  * [74] C. J. Soto and O. P. John, “The next big five inventory (bfi-2): Developing and assessing a hierarchical model with 15 facets to enhance bandwidth, fidelity, and predictive power.” _Journal of personality and social psychology_ , vol. 113, no. 1, p. 117, 2017.
  * [75] M. C. Ashton and K. Lee, “The hexaco–60: A short measure of the major dimensions of personality,” _Journal of personality assessment_ , vol. 91, no. 4, pp. 340–345, 2009.
  * [76] M. K. Tellamekala, T. Giesbrecht, and M. Valstar, “Apparent personality recognition from uncertainty-aware facial emotion predictions using conditional latent variable models,” in _2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021)_. IEEE, 2021, pp. 1–8.

|  Rongfan Liao is an Algorithm Engineer with Sony China Software Center,
Shanghai, China. He received his master degree in Shanghai Institute of
Technology. His research interests include object detection by various weakly
supervised or self-supervised learning, brain inspired intelligence, knowledge
distillation for light weight neural models and automatic personality
analysis.
---|---
|  Siyang Song is a Research Associate with the Department of Computer Science
and Technology, University of Cambridge, Cambridge, U.K.. He received his PhD
in the Computer Vision Lab and Horizon Center for Doctoral Training at the
University of Nottingham, UK. His research interests include automatic
emotion, personality and depression analysis by developing various self-
supervised learning, Neural Architecture Search and graph modelling
techniques.
---|---
|  Hatice Gunes (Senior Member, IEEE) is a Professor of Affective Intelligence
and Robotics (AFAR) at the Department of Computer Science and Technology,
University of Cambridge, United Kingdom, leading the AFAR Lab. Her expertise
is in the areas of affective computing and social signal processing cross-
fertilizing research in human behaviour understanding, computer vision, signal
processing, machine learning, and human–robot interaction. She has published
over 130 papers in the above areas, and her research highlights include
RSJ/KROS Distinguished Interdisciplinary Research Award Finalist at IEEE RO-
MAN’21, Distinguished PC Award at IJCAI’21, Best Paper Award Finalist at IEEE
RO-MAN’20, Finalist for the 2018 Frontiers Spotlight Award, Outstanding Paper
Award at IEEE FG’11, and Best Demo Award at IEEE ACII’09. Prof Gunes is the
former President (2017-2019) of the Association for the Advancement of
Affective Computing, was the General Co-Chair of ACII 2019, and Program Co-
Chair of ACM/IEEE HRI 2020 and IEEE FG 2017. Her research has been supported
by various competitive grants, with funding from the Engineering and Physical
Sciences Research Council, UK (EPSRC), Innovate UK, British Council, Alan
Turing Institute and EU Horizon 2020. She is currently a Fellow of the EPSRC
and was previously a Faculty Fellow of the Alan Turing Institute.
---|---