#  _Head Matters_ : Explainable Human-centered Trait Prediction from Head
Motion Dynamics

Surbhi Madan  Indian Institute of Technology (IIT), Ropar
<EMAIL_ADDRESS>,  Monika Gahalawat  Indian Institute of
Technology (IIT), Ropar<EMAIL_ADDRESS>,  Tanaya Guha
University of Warwick<EMAIL_ADDRESS>and  Ramanathan Subramanian
IIT, Ropar and University of Canberra<EMAIL_ADDRESS>
(2021)

###### Abstract.

We demonstrate the utility of elementary head-motion units termed _kinemes_
for behavioral analytics to predict _personality_ and _interview_ traits.
Transforming head-motion patterns into a sequence of kinemes facilitates
discovery of latent temporal signatures characterizing the targeted traits,
thereby enabling both _efficient_ and _explainable_ trait prediction.
Utilizing Kinemes and Facial Action Coding System (FACS) features to predict
(a) OCEAN personality traits on the First Impressions Candidate Screening
videos, and (b) Interview traits on the MIT dataset, we note that: (1) A Long-
Short Term Memory (LSTM) network trained with kineme sequences performs better
than or similar to a Convolutional Neural Network (CNN) trained with facial
images; (2) Accurate predictions and explanations are achieved on combining
FACS action units (AUs) with kinemes, and (3) Prediction performance is
affected by the time-length over which head and facial movements are observed.

Kinemes, Head-motion Units, Action Units, Behavioral Analytics, Explainable
Prediction, Personality and Interview Traits

††journalyear: 2021††copyright: acmcopyright††conference: Proceedings of the
2021 International Conference on Multimodal Interaction; October 18–22, 2021;
Montréal, QC, Canada††booktitle: Proceedings of the 2021 International
Conference on Multimodal Interaction (ICMI ’21), October 18–22, 2021,
Montréal, QC, Canada††price: 15.00††doi: 10.1145/3462244.3479901††isbn:
978-1-4503-8481-0/21/10††copyright: acmcopyright††ccs: Human-centered
computing Human computer interaction (HCI)††ccs: Computing methodologies
Computer vision representations

##  1\. Introduction

The importance of non-verbal behavioral cues towards human-centric trait
estimation has been acknowledged for long. Case in point, multifarious cues
like proxemics or the use of physical space (Takayama and Pantofaru, 2009),
speaking time and speech energy (Jayagopi et al., 2009), head orientation
patterns denoting social attention distribution (Subramanian et al., 2013,
2010), head and facial movements (Escalante et al., 2020; Gucluturk et al.,
2018) and physiological responses to emotional stimuli (Subramanian et al.,
2018) have been studied for predicting personality traits driving human
behavior. While a majority of these cues are relevant and effectively captured
in social settings, there has been increasing interest in predicting human-
centered traits from self-presentation videos (alternatively termed
_multimedia CVs_) (Batrinca et al., 2011) recently.

Figure 1. Framework overview. Kinemes and action units are depicted via a 3D
model. View in color and under zoom.

Multimodal non-verbal cues especially play a critical role in demonstrating an
individual’s personality and inter-personal skills in the context of
multimedia CVs (Raza and Carpenter, 1987; Batrinca et al., 2011). Subjective
impressions of interviewee personality traits influence hiring decisions (Van
Dam, 2003), and even one behavioral information channel such as _visual_ can
explain personality attributions (DeGroot and Gooty, 2009). _E.g._ , among the
big-five or OCEAN personality traits (Chmielewski and Morgan, 2013),
_Conscientiousness_ characterizing responsible and attentive behavior is
reflected via an upright posture and minimal head movements in self-
presentation videos. Likewise, _Neuroticism_ indicating anxiety and stress is
revealed via hand movement and posture dynamics such as fidgeting and camera
aversion (Batrinca et al., 2011).

The above examples convey that head-motion plays a critical role in conveying
personality impressions. Recently, _kinemes_ denoting elementary head-motion
units akin to phonemes in human speech (Birdwhistell, 1970), are shown to be
effective emotional cues (Samanta and Guha, 2021). Apart from enabling human-
comparable recognition performance, kinemes also allow for _explanations_
relating to predictions; e.g., the sad emotion is associated with slow head
movements (Gross and Crane, 2010).

This work explores the utility of kinemes for predicting _personality_ and
_interview_ traits from multimedia CVs. As in Figure 1, we extract head-motion
from an input video via the 3D _yaw_ , _pitch_ and _roll_ Euler rotation
angles. The head motion time-series is then unsupervisedly decomposed into a
kineme sequence comprising elementary head motion units. This decomposition
enables discovery of kinemes characteristic of a target trait, e.g., head
nodding is typical of courteous or sympathetic behavior (Ishii et al., 2020;
Osugi and Kawahara, 2018). In addition to head motion, we also utilize action
units (AUs) which encode facial motion and expressions, for trait prediction.
Our experiments confirm that encoding head and facial movements in terms of
kinemes and AUs enables efficient and explanative trait predictions. Overall,
we make the following research contributions:

  * (1)

We novelly employ kinemes for personality and interview trait prediction.
While medium-grained head motion behaviors like social gaze have been shown to
predict traits like Extraversion (Lepri et al., 2012), we show that kinemes
denoting fine-grained and elementary head motion are effective non-verbal
behavioral cues; Kinemes are also interpretable, can be learned without
supervision from input videos, and can enable discovery of latent temporal
signatures of the target trait.

  * (2)

Our experiments reveal that kinemes are highly predictive of personality and
interview traits. A kineme-based long short term memory (LSTM) network
performs comparable to a 2D-convolutional neural network (CNN) for personality
trait prediction on the First Impressions Candidate Screening (FICS) dataset
constituting 10K videos. On the 138-videos MIT dataset annotated for
interview-specific traits, the kineme LSTM considerably outperforms 2D-CNN.
Fusing kinemes with AUs also improves prediction performance owing to the
complementary information they encode.

  * (3)

Apart from being predictive, kinemes and AUs enable behavioral explanations
for the target traits. As examples, frequent head nodding and smiling convey
high Agreeableness, while head shaking and frowning are indicative of low
Agreeableness; likewise, upward head-tilt indicating upright demeanor conveys
high Conscientiousness; conversely, looking down to avoid eye-contact is
indicative of low Conscientiousness.

  * (4)

We present empirical results and ablative studies on two diverse datasets,
namely the FICS and MIT interview datasets. With both kinemes and AUs, we
examine personality and interview trait prediction from very short behavioral
episodes known as _thin-slices_. While reasonable prediction performance is
achieved even with 5s-long slices, more precise predictions are achieved with
longer behavioral slices. Also, we assess the suitability of video labels to
thin-slices by comparing slice and video-level predictions; video labels are
consistent with longer-slice behaviors for both datasets.

##  2\. Related Work

This section reviews literature on (a) personality and interview trait
prediction, and (b) employing head motion features for behavioral analytics,
and highlighting the research gaps thereof.

###  2.1. Personality and Interview Trait Prediction

It is well known that personality drives human behavior, and the big-five or
OCEAN personality trait model (Costa and McCrae, 1992) describes human
personality in terms of the Openness (curious vs cautious), Conscientiousness
(diligent vs insincere), Extraversion (outgoing vs reserved), Agreeableness
(sympathetic vs dispassionate) and Neuroticism (nervous vs emotionally stable)
dimensions. Numerous studies have attempted personality prediction from
behavioral, and specifically non-verbal cues such as proxemics (Takayama and
Pantofaru, 2009), speaking and head movement behavior (Jayagopi et al., 2009;
Subramanian et al., 2013), facial characteristics (Escalante et al., 2020;
Gucluturk et al., 2018) and physiological responses to emotional scenes
(Subramanian et al., 2018).

Many studies have also examined the relationship between personality traits
and job/job-interview performance. _E.g._ , Mount et al. (Mount et al., 1998)
observe that Conscientiousness, Agreeableness and Emotional stability
positively impact job performance involving interpersonal interactions.
Authors of (Moy and Lam, 2003) observe that the OCEAN traits are among the
major attributes influencing hiring decisions. Two recent studies that examine
the relation between personality traits and human factors in job interviews
are (Gucluturk et al., 2018; Escalante et al., 2020). Naim et al. (Naim et
al., 2018) perform multimodal analyses of interview videos and conclude that
prosodic features (speaking style) critically influence impressions of
interview-specific traits.

Explainable personality and interview trait prediction: While deep learning
architectures such as CNNs and LSTMs have achieved excellent performance on
multiple pattern recognition problems, their predictions are often not
interpretable (Ventura et al., 2017). A deep residual network (ResNet) is
proposed in (Gucluturk et al., 2018) to predict personality trait impressions,
and linear regression is employed to predict interview scores from personality
trait annotations; regression coefficients are employed to assess the
influence of the OCEAN traits on interview scores. Face visualizations are
also presented to demonstrate similarities among individuals achieving high
and low trait scores. CNN-based trait prediction is explored in (Ventura et
al., 2017), and analyses show that CNNs primarily analyze key facial regions
such as eyes and mouth for prediction. The FICS dataset developed for a
candidate screening challenge is presented in (Escalante et al., 2020).
Decision trees and visual-plus-verbal explanations are presented to convey
relationships among personality and interview scores. Facial action units,
which are typically used to describe emotions are shown to effectively predict
personality traits in (Gavrilescu, 2015), and the correlations among AUs and
learned CNN features is demonstrated in (Ventura et al., 2017).

###  2.2. Head Motion for Behavioral Analytics

A majority of existing works on head-motion rely on extracting low-level
features from head motion data. For example, Ding et al. (Ding et al., 2018)
use amplitude of representative Fourier components. Samanta and Guha (Samanta
and Guha, 2017) propose to extract energy of displacement, velocity and
acceleration of the Euler rotation angles, while Gunes and Pantic (Gunes and
Pantic, 2010) employ magnitude and direction of the 2D head motion. The use of
semantically meaningful head gestures has been limited to extracting nods and
shakes (Gunes and Pantic, 2010). However, head motion generated during dyadic
interactions and self-presentation videos is complex; therefore high-level
head gestures may not be limited to only nods and shakes. To this end, Yang
and Narayanan (Yang and Narayanan, 2017) propose to extract arbitrary head
gesture segments. These head gesture segments are abstract, and do not have
physical interpretation.

Head motion in human-centered traits: In the context of affect analysis, head
motion patterns have been used to study coordination between mothers and
infants (Hammal et al., 2015a, b), emotion recognition (Samanta and Guha,
2021), measuring engagement levels of dementia patients (Parekh et al., 2018)
and for analyzing interpersonal coordination in couple therapy (Hammal et al.,
2014; Xiao et al., 2015).

###  2.3. Inference Summary

Analysis of the literature reveals the following shortcomings: (1) While many
works examine prediction of personality and interview traits, and explain
predictions via statistical analysis or visualizations, these explanations are
limited to discovering salient facial features or examining connections
between interview performance and personality traits. (2) While head motion
patterns have been identified as critical non-verbal behavioral cues in
interactive scenarios, they have nevertheless not been employed for
personality or interview trait prediction.

Differently, we novelly attempt explanations for personality and interview
traits from kinemes which are inherently explanatory; apart from generic
explanations such as head-nodding evoking positive and head-shaking eliciting
negative impressions, trait-specific characteristics such as Openness and
emotional stability associating with persisting head movements, and
Conscientiousness being impacted by head tilting to maintain/avoid eye-contact
are also evident. More intuitive explanations are achieved on combining
kinemes and facial action units such as nodding and expressive facial behavior
achieving high interview scores, and head shaking and frowning being seen as
less friendly and sociable. In addition to their explanatory characteristics,
kinemes and AUs are also found to effectively predict the targeted traits.

##  3\. Kineme Formulation

Given a self-presentation video, we extract the 3D head pose in terms of Euler
rotation angles about $x,y,z$, namely, _pitch_ ($\theta_{p}$), _yaw_
($\theta_{y}$) and _roll_ ($\theta_{r}$), for each frame using the computer
vision tool OpenFace (Baltrušaitis et al., 2016). Thus, head motion is denoted
as a multivariate time-series of head orientations:
$\boldsymbol{\theta}=\\{\theta_{p}^{1:T},\theta_{y}^{1:T},\theta_{r}^{1:T}\\}$
of length $T$. We propose to model head motion as a sequence of fundamental
and _interpretable_ motion units termed kinemes, which consequently enable
_explanative_ trait predictions. Past work on motion modeling has focused on
extracting head-motion patterns such as nods and shakes (Gunes and Pantic,
2010), or learning arbitrary head gestures (Yang and Narayanan, 2017) with no
physical meaning. Differently, we unsupervisedly learn meaningful motion
patterns following (Samanta and Guha, 2021) to translate head motion into a
sequence of kinemes.

###  3.1. Head motion as kineme sequence

Given the head-motion time-series extracted from a set of videos, we divide
each time-series $\boldsymbol{\theta}$ into short overlapping segments of
length $\ell$ (overlapping segments enable shift-invariance, are empirically
found to generate better representations in (Samanta and Guha, 2021)). The
$i^{th}$ segment is denoted by a vector
$\mathbf{h}^{(i)}=[\theta_{p}^{i:i+\ell}\,\theta_{y}^{i:i+\ell}\,\theta_{r}^{i:i+\ell}]^{\intercal}$.
The characterization matrix $\mathbf{H}_{\boldsymbol{\theta}}$ is defined as
$\mathbf{H}_{\boldsymbol{\theta}}=[\mathbf{h}^{(1)},\mathbf{h}^{(2)},\cdots,\mathbf{h}^{(s)}]$,
where $s$ is the total number of segments in $\boldsymbol{\theta}$. Given $N$
training samples, a head motion matrix
$\mathbf{H}\in\mathbb{R}_{+}^{3\ell\times Ns}$ is created as
$\mathbf{H}=[\mathbf{H}_{\boldsymbol{\theta}_{1}}|\mathbf{H}_{\boldsymbol{\theta}_{2}}|\cdots|\mathbf{H}_{\boldsymbol{\theta}_{N}}]$.
$\mathbf{H}$ is then subject to a Non-negative Matrix Factorization (NMF),
yielding a basis matrix $\mathbf{B}$ and a coefficient matrix $\mathbf{C}$. We
cluster head motion segments in the transformed space by grouping the
coefficient vectors (columns in $\mathbf{C}$) via a Gaussian Mixture Model
(GMM) into $k<<Ns$ clusters. This produces a $k$ column matrix
${\mathbf{C}^{*}}$. Kinemes are transformed back to the original _yaw_
-_pitch_ -_roll_ space via $\mathbf{H}^{*}=\mathbf{B}\mathbf{C}^{*}$, whose
columns yield the set of $k$ kinemes $\mathcal{K}$.

Upon learning kinemes for the input video set, we can represent any head
motion time-series as a sequence of kinemes by associating each $\ell$-long
segment to one of the $K$ kinemes. Consider the $i^{th}$ segment in
$\boldsymbol{\theta}$ characterized by $\mathbf{h}^{(i)}$. We project
$\mathbf{h}^{(i)}$ onto the learned subspace to obtain $\mathbf{c}^{(i)}$:

$\hat{\mathbf{c}}=\underset{\mathbf{c}^{(i)}\geq 0}{\text{arg
min}}\lVert{\mathbf{h}^{(i)}-\mathbf{B}\mathbf{c}^{(i)}}\rVert_{F}^{2}$

The corresponding kineme $K^{(i)}$ for the $i^{th}$ segment is given by
maximizing the posterior probability $P({K}|\hat{\mathbf{c}})$, over all
$K\in\mathcal{K}$. On mapping each head motion time-series segment to a
kineme, we get the corresponding kineme sequence
$\boldsymbol{\theta}:\\{K^{(1)}\cdots K^{(s)}\\},K^{(j)}\in\mathcal{K}$.

###  3.2. Trait prediction from kinemes

Prior studies have shown that human-centered traits are characterized by
specific non-verbal (and specifically, head-motion) behaviors; e.g., an
upright posture maintaining eye-contact conveys high Conscientiousness, while
gaze avoidance is indicative of low Conscientiousness (Hoppe et al., 2018;
Jarrett, 2014). Likewise, frequent head nodding is seen as courteous and
agreeable behavior, while head shaking and frowning indicates a cold demeanor
(Ishii et al., 2020; Osugi and Kawahara, 2018). Kinemes inherently enable
discovery of temporal head-motion patterns characteristic of a given trait,
and sequence learning methods such as Hidden Markov Models (HMMs) and Long-
short term memory (LSTM) networks can be employed to learn these latent
temporal signatures for continuous or categorical trait prediction.

##  4\. Explainable Trait Prediction

In this work, we examined the First Impression Candidate Screening (FICS)
(Escalante et al., 2020) and the MIT interview (Naim et al., 2018) datasets
for trait prediction. A detailed description of these datasets is presented in
Sec. 5.1. The FICS dataset is curated with the objective of training
algorithms to predict apparent personality traits from multimedia CVs, and has
annotations for the OCEAN traits on a $[0,1]$ scale. The MIT dataset comprises
recordings of mock interviews with prospective interns, and observer ratings
for 16 interviewee-specific traits. Among them, we examined the following
traits, hypothesizing that they could be adequately explained by head and
facial behaviors: level of friendliness (Fr), excitement (Ex) and eye-contact
(EC), and recommended hiring score (RH) conveying the likelihood of the
candidate being invited for further interviews. Figure 2 presents exemplar
frames from the FICS and MIT datasets. While the FICS self-presentation videos
are recorded under varied conditions such as differing scene background,
camera perspective and camera distance from subject, the MIT videos are
captured with a relatively stable background via two wall-mounted cameras.

Figure 2. FICS (top) and MIT (bottom) video examples.

Upon extracting the _yaw_ , _pitch_ and _roll_ angles and 17 facial action
unit (AU) intensities per frame with the _Openface_ (Baltrušaitis et al.,
2016) toolkit, we computed kinemes for the FICS and MIT datasets with $K=16$
as per the procedure outlined in Sec. 3.1. We employed 2s-long segments with
50% overlap for kineme extraction. Figure 4 presents the 16 kinemes extracted
for the FICS dataset, while Fig. 4 shows selected kinemes for the MIT data. A
head nod corresponds to a sudden change (spike) in pitch or head-tilt, while a
head shake translates to a spike in yaw. To discover dominant AUs, we again
examined 2s time-windows with 1s overlap and regarded an AU as dominant if
within the window its maximum value exceeded the mean intensity over all AUs.
Fig. 5 presents commonly seen AUs for both datasets.

Figure 3. Plots of 16 kinemes extracted for the FICS dataset following raster
ordering (left to right, top to bottom.

Figure 4. Selected kineme plots for the MIT dataset. Figure 5. Common AUs in the FICS and MIT datasets. Table 1. Explaining OCEAN and interview traits via kinemes and AUs. MIT Kinemes in bold font are visualized in Figure 4. Dataset | Trait | Dominant Kin | Dominant AUs | Inferences
---|---|---|---|---
FICS | O (H) | 2, 8, 10, 16 | 7, 12, 14, 25, 26 |  Persistent head movements (as noted in (Koppensteiner, 2013)) with nodding and smiling.
C (H) | 1, 8, 10, 16 | 7, 12, 17, 25, 26 | Upward head-tilt indicative of upright demeanor and head nodding.
E (H) | 2, 10, 14, 16 | 10, 12, 17, 25, 26 | Head tilt-down with nodding, and facial gestures related to speaking.
A (H) | 3, 8, 10, 16 | 7, 12, 14, 25, 26 |  Frequent head nodding and smiling (associated with courteous behavior (Ishii et al., 2020; Osugi and Kawahara, 2018)).
N (H) | 2, 8, 10, 16 | 7, 12, 17, 25, 26 | Frequent head movements with nodding and smiling.
O (L) | 1, 6, 11, 16 | 4, 10, 14, 17, 26 | Relatively fewer head movements and frowning.
C (L) | 2, 4, 8, 16 | 4, 7, 10, 14, 25 | Head tilt-down avoiding eye-contact, head shaking and frowning.
E (L) | 1, 4, 10, 16 | 4, 7, 10, 14, 17 | Tilt-up, head shaking and frowning.
A (L) | 1, 8, 9, 16 | 4, 14, 17, 25, 26 | Frequent head movements and frowning.
N (L) | 1, 5, 12, 16 | 4, 7, 10, 14, 25 | Few head movements, head shaking and frowning.
MIT | RH (H) |  16, 14, 3, 4 | 5, 10, 12, 14, 25 | Head nodding and smiling, and being expressive.
Ex (H) |  14, 3, 4, 9 | 5, 10, 12, 14, 25 | Head nodding and exhibiting persistent head motion. Smiling and being expressive.
EC (H) |  14, 12, 4, 5 | 6, 7, 10, 14, 25 | Head up, nodding and showing limited facial emotions.
Fr (H) |  16, 3, 11, 14 | 5, 10, 12, 14, 25 | Frequent head movements and smiling.
RH (L) |  11, 1, 2, 5 | 6, 7, 12, 14, 25 | Head shaking and exhibiting minimal facial expressions.
Ex (L) |  11, 16, 2, 3 | 4, 6, 7, 14, 25 | Head shaking and nodding. Frowning and showing minimal facial expressions.
EC (L) |  13, 7, 16, 11 | 6, 7, 10, 12, 25 | Frequent nodding is perceived as avoiding eye-contact.
Fr (L) |  3, 11, 4, 9 | 1, 4, 6, 7, 25 | Head shaking, frowning and otherwise being minimally expressive.

To examine kineme and AU-based explanations, we considered the top and bottom
10 percentile videos for each trait and computed the most frequent kinemes and
AUs in those videos. Table 1 lists the frequently occurring kinemes and AUs
for the high (H) and low (L)-rated OCEAN and Interview trait videos (top 4
kinemes based on frequency and top 5 dominant AUs are shown), and inferences
thereof. We make the following observations from Table 1:

  * •

Kineme 16 is common for all OCEAN traits, while kineme 10 denoting a head-nod
is seen in high OCEAN trait videos. Therefore, head-nodding in general evokes
positive personality trait impressions. Likewise, AUs 25 and 26 typical of
talking behavior are noted for all videos.

  * •

Focusing on other kinemes, high Openness is characterized by kinemes 2 and 8,
which signify persistent head movements. This finding is echoed in
(Koppensteiner, 2013), where large motion variations are found to associate
with high O impressions. Presence of AUs 12 and 14 indicates that a smiling
demeanor characterizes high O. Conversely, kineme 6 denoting minimal head
motion and AUs 4 and 17 typical of frowning and diffident behavior are
commonly noted for low O videos.

  * •

Kineme 1 denoting an upward head tilt is associated with high C, while kinemes
2 and 4 depicting tilt-down and head-shaking are associated with low C. This
indicates that attempting to maintain eye-contact conveys diligence and
honesty, while avoiding eye-contact conveys insincerity.

  * •

Extraversion appears to be conveyed better by AUs than kinemes; Dominant AUs
for high E include 10, 12 and 17 indicating a friendly and talkative nature,
while dominant kinemes 2 and 14 convey significant head movements. Conversely,
low E is associated with kineme 4 denoting head-shaking and AUs 4, 7 and 17
indicating frowning, overall conveying a socially distant nature.

  * •

High Agreeableness is characterized by kineme 3 (head-nod), and AUs 12 and 14
which constitute a smile. Conversely, kinemes 1, 8 and 9 dominate low A, and
they collectively convey persistent head motion. Also, AUs dominant for low A
are 4, 14 and 17, cumulatively describing a frown; overall, nodding and
smiling is viewed as courteous, while frequent head movements and frowning
convey hostility.

  * •

Emotional stability (high N) is associated with kinemes 2 and 8, and AUs 7, 12
and 17, indicating persistent head motion and facial expressiveness. On the
other hand, a neurotic trait is conveyed via limited head motion and head-
shaking (kinemes 1, 5, 12) and frowning (described by AUs 4, 7, 10).

  * •

While kinemes for the MIT videos are less discernible, due to smaller face
size (Fig. 2) and the fact that they capture an interactional setting, some
patterns are nevertheless evident as seen in Fig. 4; these kinemes are
highlighted in Table 1. As with FICS, Kineme 14 denoting a head-nod is
commonly observed for all high trait videos, while kineme 11 depicting a head-
shake is common for all low-trait videos.

  * •

High RH scores are elicited with expressive facial behavior involving head-
nodding and smiling. Conversely, low RH scores are associated with head-
shaking and exhibiting limited facial expressions. Highly excited behavior is
associated with identical AUs as high RH, and persistent head motion.
Inversely, low excitement scores are connected with head shaking, and limited
facial emotions.

  * •

Identical AUs are observed for both high and low eye-contact, implying that
head movements primarily impact eye-contact impressions. Head nodding (kineme
14) is associated with high EC, while kinemes 11 and 16 depicting head shaking
and frequent head-nodding elicit low EC scores. Therefore interestingly, while
head nodding is beneficial, frequent nodding is perceived as avoiding eye-
contact.

  * •

High friendliness is characterized by kinemes 11, 14 and 16, signifying
persistent head motion along with expressive and smiling facial movements (AUs
5, 12 and 14). Conversely, low friendliness is associated with head-shaking
(kineme 11) and frowning (AUs 4, 6, 7).

Overall, kineme and AU patterns offer intuitive explanations for the
considered personality and interview traits. In addition, they enable
efficient trait prediction as described in the next section.

##  5\. Experiments and Results

###  5.1. Datasets

FICS: The FICS dataset (Escalante et al., 2020) comprises 10K ($\approx$15s
long) _YouTube_ self-presentation videos, and is labeled via crowdworkers for
the OCEAN personality traits. As Neuroticism is a negative trait, N scores
encode inverse of Neuroticism scores in this dataset. FICS contains 6K
_training_ , 2K _validation_ and 2K _test_ videos.

MIT: The MIT interview (Naim et al., 2018) dataset comprises 138 mock
internship interview videos, with an average length of 4.7 minutes. These
videos were rated for 16 traits by crowdworkers; along with the four traits
considered in Sec. 4, we also evaluate prediction performance on the _overall_
interview score (Ov) in this section.

###  5.2. Experimental settings

Prediction Type : For our experiments, we modeled the personality and
interview trait scores as either continuous or discrete variables (upon
thresholding trait scores at their median value), and present regression
(Tables 3, 4) and classification (Tables 5, 6) results. As the FICS dataset is
already partitioned into the train, validation and test sets, we employed the
validation set for fine-tuning model parameters, or for early stopping of
model training for LSTM. On the other hand, as the MIT dataset only comprises
138 videos, we report performance in the form of $\mu\pm\sigma$ over five
repetitions of 10-fold video cross validation (total of 50 runs) in Tables 4
and 6. Validation sets were obtained by randomly holding out 10% of the
training data during each run.

Chunk vs video-level predictions: In line with the thin-slice approach for
behavioral trait prediction, we segmented the original videos into smaller
chunks, 3, 5 and 7s for the FICS data and 5–60s chunks for the MIT data (see
Fig. 9) and repeated the video label for all chunks. We then computed metrics
over a) all chunks (chunk-level performance), and b) over all videos by
assigning the majority label over all chunks to each video (video-level
performance).

###  5.3. Performance Metrics

For regression, we considered the accuracy (Acc) and Pearson Correlation
Coefficient (PCC) metrics. As in (Gucluturk et al., 2018), Acc is measured as
1-MAE, the mean absolute error in the predictions with respect to groud-truth
scores; the PCC measures how well the predictions correlate with the ground-
truth scores (PCC=1 for precise predictions). For classification, we employed
the accuracy (Acc) and F1-score metrics. Given the imbalanced class
distributions for some traits (see Table 2), the F1-score denoting the
harmonic mean of precision and recall is more suited for performance
evaluation.

###  5.4. Models

Linear regression with principal components (PCA-Lin-Reg): We collated the
yaw, pitch and roll values over each chunk, and then performed principal
component analysis (PCA) over the training data. As many principal components
explaining 90% data variance were preserved, and linear regression was
performed on the same. This method ignores the temporal head and facial motion
dynamics.

Hidden Markov Model for classification (HMM Kin): To learn trait-wise temporal
head motion patterns, we employed a HMM to iteratively deduce model parameters
and class labels from kineme sequences via the Baum-Welch algorithm.

Long short-term memory for regression and classification: LSTMs denote another
efficient methodology to learn latent temporal head and facial motion
dynamics. We trained LSTMs with the kineme sequences (LSTM Kin), AU sequences
(LSTM AU) and their combination (LSTM Kin+AU FF). The kineme sequences were
input in the one-hot encoding form, where the kineme corresponding to a given
time-window is coded to 1 and the other 15 coded to 0. AUs on the other hand
were input in a coded vector form, where all AUs deemed dominant upon
thresholding within a time-window were set to 1 and others set to 0. To fuse
the kineme and AU-based representations (denoted as feature fusion or FF), we
employed the architecture shown in Fig. 6. Kineme features form a 3D matrix of
16 (one-hot-kineme-vector) $\times$ data-points $\times$ chunk-length, while
the AU matrix is of size 17 (vector of active AUs coded to 1) $\times$ data-
points $\times$ chunk-length. A single hidden LSTM layer with 32 neurons was
employed for kinemes and AUs, while these outputs were merged to obtain 64
neurons for FF. The LSTM layer is followed by a dense layer involving two
neurons with sigmoidal activation for classification, and one neuron with
linear activation for regression. In both cases, a dropout value of 0.2 was
employed for the LSTM layer to prevent overfitting, and an Adam optimizer with
learning rate of 0.01 was utilized for training. Binary-cross entropy was
defined as the loss function for classification, and mean absolute error for
regression.

2D CNN for regression and classification: A 19 layered VGG model, which
processes images (video frames) was used. Upon removing the output layer, two
hidden dense layers with 512 and 64 neurons respectively were added along with
output layer involving 5 neurons (one neuron each for the OCEAN/Interview
traits). Mean squared error (MSE) for regression, and binary cross-entropy
(BCE) loss for classification were used to train on a random frame from each
video, with learning rate of 1e-4 and a batch size of 64.

Decision fusion for regression and classification (LSTM Kin+AU (DF)): Apart
from LSTM-based kineme AU fusion, we attempted fusion of the unimodal LSTM
predictions. We adopted the fusion weight estimation approach proposed in
(Koelstra and Patras, 2013). Assuming that the test sample score is $\alpha
p_{Kin}+(1-\alpha)p_{AU},\alpha\in[0,1]$, where $p_{Kin}$ and $p_{AU}$ are the
individual classifier (regressor) scores, we performed grid-search
incrementing $\alpha$ in steps of 0.01 to estimate the optimal $\alpha^{*}$
maximizing PCC for regression and F1-score for classification; $\alpha^{*}$
was then applied to compute test sample scores.

Figure 6. Kineme$+$AU LSTM classification architecture. The dense layer output involves a single neuron with linear activation for regression. Best viewed in color and zoom. Table 2. Trait-wise train (Tr) and test (Te) class distributions for the FICS and MIT datasets obtained for classification experiments. MIT class distributions correspond to 1-minute video samples employed for analysis. | FICS | MIT
---|---|---
| O | C | E | A | N | Ov | RH | Ex | EC | Fr
Label | Tr | Te | Tr | Te | Tr | Te | Tr | Te | Tr | Te | Tr | Te | Tr | Te | Tr | Te | Tr | Te | Tr | Te
-ve | 0.53 | 0.52 | 0.51 | 0.51 | 0.51 | 0.51 | 0.53 | 0.53 | 0.52 | 0.52 | 0.50 | 0.50 | 0.59 | 0.59 | 0.50 | 0.50 | 0.43 | 0.43 | 0.39 | 0.39
+ve | 0.47 | 0.48 | 0.49 | 0.49 | 0.49 | 0.49 | 0.47 | 0.47 | 0.48 | 0.48 | 0.50 | 0.50 | 0.41 | 0.41 | 0.50 | 0.50 | 0.57 | 0.57 | 0.61 | 0.61
Table 3. FICS Regression results: Accuracy and PCC values for different methods are tabulated. Trait | PCA Lin-Reg | LSTM Kin | LSTM AU | LSTM Kin+AU (FF) | LSTM Kin+AU (DF) | 2D-CNN
---|---|---|---|---|---|---
| Acc | PCC | Acc | PCC | Acc | PCC | Acc | PCC | Acc | PCC | Acc | PCC
Open | 0.884 | 0.085 | 0.872 | 0.060 | 0.889 | 0.370 | 0.892 | 0.368 | 0.893 | 0.382 | 0.906 | 0.392
Con | 0.875 | 0.086 | 0.864 | 0.027 | 0.882 | 0.317 | 0.880 | 0.304 | 0.882 | 0.282 | 0.908 | 0.295
Extra | 0.877 | 0.060 | 0.869 | 0.048 | 0.891 | 0.491 | 0.893 | 0.474 | 0.891 | 0.485 | 0.907 | 0.492
Agree | 0.892 | 0.035 | 0.885 | 0.046 | 0.897 | 0.251 | 0.892 | 0.253 | 0.896 | 0.275 | 0.906 | 0.283
Neuro | 0.877 | 0.071 | 0.867 | 0.051 | 0.885 | 0.370 | 0.884 | 0.365 | 0.887 | 0.387 | 0.903 | 0.395
Table 4. MIT Regression results: Accuracy and PCC values are tabulated. Trait | PCA Lin-Reg | LSTM Kin | LSTM AU | LSTM Kin+AU (FF) | LSTM Kin+AU (DF) | 2D-CNN
---|---|---|---|---|---|---
| Acc | PCC | Acc | PCC | Acc | PCC | Acc | PCC | Acc | PCC | Acc | PCC
Ov | 0.86$\pm$0.00 | 0.15$\pm$0.26 | 0.93$\pm$0.04 | 0.84$\pm$0.26 | 0.93$\pm$0.04 | 0.84$\pm$0.26 | 0.97$\pm$0.03 | 0.92$\pm$0.17 | 0.95$\pm$0.04 | 0.89$\pm$0.21 | 0.86$\pm$0.03 | 0.47±0.25
RH | 0.83$\pm$0.03 | 0.13$\pm$0.26 | 0.95$\pm$0.03 | 0.93$\pm$0.10 | 0.95$\pm$0.03 | 0.93$\pm$0.10 | 0.96$\pm$0.04 | 0.91$\pm$0.21 | 0.94$\pm$0.04 | 0.90$\pm$0.19 | 0.85$\pm$0.03 | 0.45$\pm$0.24
Ex | 0.82$\pm$0.04 | 0.05$\pm$0.20 | 0.94$\pm$0.04 | 0.89$\pm$0.20 | 0.94$\pm$0.04 | 0.89$\pm$0.20 | 0.96$\pm$0.05 | 0.91$\pm$0.20 | 0.94$\pm$0.04 | 0.91$\pm$0.17 | 0.85$\pm$0.03 | 0.57$\pm$0.18
EC | 0.83$\pm$0.03 | 0.10$\pm$0.21 | 0.94$\pm$0.04 | 0.89$\pm$0.13 | 0.94$\pm$0.04 | 0.89$\pm$0.22 | 0.96$\pm$0.04 | 0.94$\pm$0.13 | 0.95$\pm$0.04 | 0.91$\pm$0.16 | 0.84$\pm$0.03 | 0.44$\pm$0.22
Fr | 0.82$\pm$0.04 | 0.02$\pm$0.28 | 0.95$\pm$0.03 | 0.93$\pm$0.10 | 0.95$\pm$0.03 | 0.93$\pm$0.10 | 0.97$\pm$0.03 | 0.96$\pm$0.08 | 0.95$\pm$0.03 | 0.94$\pm$0.08 | 0.86$\pm$0.03 | 0.58 $\pm$0.20
Table 5. FICS Classification results: Accuracy and F1 values are tabulated. Trait | HMM Kin | LSTM Kin | LSTM AU | LSTM Kin+AU (FF) | LSTM Kin+AU (DF) | 2D-CNN
---|---|---|---|---|---|---
| Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1
Open | 0.529 | 0.505 | 0.519 | 0.516 | 0.635 | 0.634 | 0.629 | 0.628 | 0.632 | 0.632 | 0.697 | 0.697
Con | 0.524 | 0.515 | 0.513 | 0.513 | 0.618 | 0.618 | 0.604 | 0.604 | 0.599 | 0.599 | 0.705 | 0.705
Extra | 0.514 | 0.509 | 0.505 | 0.505 | 0.651 | 0.651 | 0.648 | 0.648 | 0.657 | 0.653 | 0.712 | 0.712
Agree | 0.528 | 0.497 | 0.481 | 0.479 | 0.580 | 0.580 | 0.593 | 0.586 | 0.584 | 0.583 | 0.656 | 0.656
Neuro | 0.524 | 0.506 | 0.523 | 0.518 | 0.627 | 0.624 | 0.626 | 0.623 | 0.620 | 0.616 | 0.717 | 0.717
Table 6. MIT Classification results: Accuracy and F1 values are tabulated. Trait | HMM | LSTM Kin | LSTM AU | LSTM Kin+AU (FF) | LSTM Kin+AU (DF) | 2D-CNN
---|---|---|---|---|---|---
| Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1
Ov | 0.56$\pm$0.17 | 0.48$\pm$0.18 | 0.83$\pm$0.11 | 0.82$\pm$0.13 | 0.82$\pm$0.14 | 0.81$\pm$0.15 | 0.80$\pm$0.14 | 0.80$\pm$0.14 | 0.85$\pm$0.13 | 0.85$\pm$0.14 | 0.65$\pm$0.09 | 0.64$\pm$0.08
RH | 0.55$\pm$0.14 | 0.34$\pm$0.27 | 0.79$\pm$0.12 | 0.79$\pm$0.12 | 0.83$\pm$0.13 | 0.83$\pm$0.14 | 0.81$\pm$0.12 | 0.80$\pm$0.12 | 0.84$\pm$0.11 | 0.83$\pm$0.12 | 0.59$\pm$0.07 | 0.58$\pm$0.07
Ex | 0.57$\pm$0.14 | 0.51$\pm$0.10 | 0.82$\pm$0.13 | 0.82$\pm$0.13 | 0.82$\pm$0.12 | 0.82$\pm$0.10 | 0.79$\pm$0.13 | 0.79$\pm$0.13 | 0.83$\pm$0.11 | 0.82$\pm$0.12 | 0.72$\pm$0.08 | 0.71$\pm$0.08
EC | 0.53$\pm$0.17 | 0.44$\pm$0.24 | 0.79$\pm$0.13 | 0.79$\pm$0.13 | 0.81$\pm$0.12 | 0.80$\pm$0.13 | 0.78$\pm$0.12 | 0.76$\pm$0.13 | 0.84$\pm$0.13 | 0.83$\pm$0.14 | 0.57$\pm$0.07 | 0.55$\pm$0.09
Fr | 0.57$\pm$0.18 | 0.54$\pm$0.12 | 0.80$\pm$0.15 | 0.80$\pm$0.16 | 0.86$\pm$0.09 | 0.85$\pm$0.09 | 0.84$\pm$0.10 | 0.84$\pm$0.11 | 0.87$\pm$0.11 | 0.86$\pm$0.12 | 0.61$\pm$0.08 | 0.61$\pm$0.08
Figure 7. Exemplar Gradcam (Selvaraju et al., 2019) maps for CNN predictions.

Figure 8. Chunk vs video-level predictions with kinemes (left) and AUs (right)
for the FICS dataset.

Figure 9. Chunk vs video-level predictions with kinemes (left) and AUs (right)
for the MIT dataset.

###  5.5. Results and Discussion

Tables 3 and 4 present regression results, while Tables 5, 6 tabulate
classification results. We make the following remarks therefrom:

  * •

One can note from Tables 3 and 4 that very low PCC values are noted even for
relatively high accuracies for the FICS dataset; these results convey that
larger variability in prediction scores is observed for the FICS videos, and
that PCC is more suited for MIT performance evaluation. Also, comparing Tables
3, 4 with Tables 5, 6, much higher performance is achieved for regression than
classification. This can be attributed to the Gaussian-distributed FICS OCEAN
trait scores around 0.5 (see Figs 12 and 13 in (Escalante et al., 2020)), and
the MIT Interview trait scores about the median.

  * •

Focusing on Table 3, PCA Lin-Reg which ignores temporal head-motion dynamics,
performs better than LSTM Kin; this implies that the extracted kinemes cannot
adequately describe the FICS OCEAN scores. Very poor PCC scores are obtained
with both PCA Lin-Reg and LSTM Kin.

  * •

AUs achieve superior LSTM-based prediction than kinemes. However,
complementarity of the kineme and AU encodings in general enables slightly
superior performance with both feature and decision fusion, with both fusion
schemes performing comparably. 2D-CNN achieves optimal predictions on the FICS
videos; the explanatory power of CNNs is however limited. Grad-cam
visualizations (Fig. 7) which highlight regions deemed critical for model
prediction focus on the eye, mouth and nasal regions as in (Ventura et al.,
2017). Unlike kinemes however, trait-characteristic temporal dynamics of these
regions are not conveyed by these saliency maps.

  * •

Kinemes however demonstrate superior predictive power for interview traits
(Table 4), considerably outperforming Lin-Reg and performing identical to AUs.
Fusing unimodal features/decisions is beneficial for the MIT dataset, with
feature fusion outperforming decision fusion.

  * •

Comparing against the Resnet-based trait prediction model proposed in
(Gucluturk et al., 2018), we note that the 2D-CNN achieves similar or superior
performance, while the Kineme/AU-based LSTM performs inferiorly.

  * •

On the smaller MIT dataset however, LSTM Kin considerably outperforms 2D-CNN,
revealing the need for large training data to effectively tune VGG-type
networks.

  * •

In Table 3, 2D-CNN achieves highest accuracy for Conscientiousness, while
Agreeableness is predicted best via all other methods. Highest PCC values are
achieved for Friendliness with all methods excepting PCA Lin-Reg in Table 4.

  * •

Focusing on Table 5, HMM Kin and LSTM Kin perform worst for OCEAN trait
prediction. LSTM AU performs considerably better than LSTM Kin and similar to
the feature and decision fusion schemes, implying that multimodal fusion is
not very beneficial for the FICS dataset. The 2D-CNN achieves optimal trait
classification, substantially outperforming other methods. With all schemes,
least classification performance is observed for Agreeableness, while
Extraversion is best classified by all schemes excepting HMM Kin and LSTM Kin.

  * •

With respect to Table 6, the LSTM network employing kineme features achieves
considerably higher classification performance than HMM Kin. This trend again
confirms that kinemes are effective predictors of interview traits. LSTM AU
performs slightly better than LSTM Kin, while the decision fusion framework
achieves optimal classification, marginally outperforming LSTM AU.

  * •

As with regression, LSTM Kin achieves substantially better F1-scores than
2D-CNN, confirming the efficacy of the kineme representation with fewer
training data. The Friendliness trait is best isolated by four of the six
classifiers.

The above results are achieved by considering 15s chunks for the FICS videos
and 1-minute chunks for the MIT videos, and assigning the video label as the
majority label over all chunk predictions. As mentioned in Sec. 5.2, we
attempted trait prediction from thin behavioral slices (chunks) of varying
lengths and evaluated prediction performance at the chunk and video levels.

Figures 9 and 9 present corresponding results. In both figures, higher PCC
values are achieved with video-level labels than chunk-level labels, implying
that while episodic behaviors may be inconsistent with one another, trait-
specific behaviors are indeed homogeneous over a long time-span. From Fig. 9,
we note that kineme-based trait prediction performance generally decreases
with larger time-slices, while AU-based predictions become more accurate with
larger time-slices. Conversely, one can note a general increase in video and
chunk-level PCC values with both Kineme and AU features for the MIT dataset.
Interestingly, even with 5s slices, we obtain a PCC ¿ 0.3 at the chunk-level
and PCC ¿ 0.65 at the video level with kineme features, and still higher
values are obtained with AUs. These trends suggest that AUs describing facial
behavior, encode more trait-specific information than kinemes characterizing
head motion, consistent with one’s expectation.

##  6\. Conclusion

This work demonstrates that kinemes, denoting fundamental and interpretable
head-motion units, achieve effective personality and interview trait
prediction and also provide behavioral explanations for different traits
consistent with prior findings. Our empirical results confirm that
characteristic head gesture behaviors indeed exist for different traits, and
this work contributes to the understanding of the role of head gestures in
behavioral trait prediction.

Combining facial action unit information with kinemes enables more efficient
trait prediction and intuitive explanations. This work extracts kinemes over
fixed time windows, which represents a limitation as head movement behaviors
can vary across individuals, and also be conditioned on speaking behavior
(relaxed vs animated speech). Existence of trait-specific kineme patterns can
be exploited to synthesize head motion for virtual agents with specified
personality traits. The finding that apparent trait impressions are
explainable in terms of head and facial motion patterns can also be utilized
to developing assistive technologies to help users improve their public
speaking and interactional skills. Nevertheless, the proposed behavioral
analytic tools are intended to support and complement human decision-making,
and the authors do not advocate the exclusive use of such technology for
complex processes such as job recruitment.

## Acknowledgement

Thanks to A. Samanta, IIT Kanpur for sharing the kineme code.

## References

  * (1)
  * Baltrušaitis et al. (2016) Tadas Baltrušaitis, Peter Robinson, and Louis-Philippe Morency. 2016\.  OpenFace: An open source facial behavior analysis toolkit. In _2016 IEEE Winter Conference on Applications of Computer Vision (WACV)_. 1–10.  https://doi.org/10.1109/WACV.2016.7477553
  * Batrinca et al. (2011) Ligia Batrinca, Nadia Mana, Bruno Lepri, Fabio Pianesi, and Nicu Sebe. 2011.  Please, tell me about yourself: Automatic personality assessment using short self-presentations.  _ICMI’11 - Proceedings of the 2011 ACM International Conference on Multimodal Interaction_ , 255–262.  https://doi.org/10.1145/2070481.2070528
  * Birdwhistell (1970) Ray Birdwhistell. 1970\.  Kinesics and Context, Essays on Body-Motion Communication.  _Philadel-phia: University of Pennsylvania Press_ (1970).
  * Chmielewski and Morgan (2013) Michael S. Chmielewski and Theresa A. Morgan. 2013.  _Five-Factor Model of Personality_.  Springer New York, New York, NY, 803–804.
  * Costa and McCrae (1992) Paul T. Jr. Costa and Robert R. McCrae. 1992.  _NEO-PI-R professional manual: Revised NEO personality and NEO Five-Factor Inventory (NEO-FFI)_. Vol. 4.  Psychological Assessment Resources, Odessa, Florida. 5–13 pages.
  * DeGroot and Gooty (2009) Timothy DeGroot and Janaki Gooty. 2009.  Trait Perception in the Employment Interview: A Five–Factor Model Perspective.  _Journal of Business and Psychology_ 24, 2 (2009).
  * Ding et al. (2018) Yu Ding, Lei Shi, and Zhigang Deng. 2018.  Low-level Characterization of Expressive Head Motion through Frequency Domain Analysis.  _IEEE Transactions on Affective Computing_ (2018).
  * Escalante et al. (2020) Jair Hugo Escalante, Meysam Madadi, Stephane Ayache, Evelyne Viegas, Furkan Gurpinar, Sukma Achmadnoer Wicaksana, Cynthia Liem, A. J. Van Marcel Gerven, Van Rob Lier, Heysem Kaya, Ali Albert Salah, Sergio Escalera, Yagmur Gucluturk, Umut Guclu, Xavier Baro, Isabelle Guyon, and C. S. Julio Jacques. 2020.  Modeling, Recognizing, and Explaining Apparent Personality from Videos.  _IEEE Transactions on Affective Computing_ (2020), 1–1.
  * Gavrilescu (2015) Mihai Gavrilescu. 2015\.  Study on determining the Big-Five personality traits of an individual based on facial expressions. In _2015 E-Health and Bioengineering Conference (EHB)_. 1–6.  https://doi.org/10.1109/EHB.2015.7391604
  * Gross and Crane (2010) M. Melissa Gross and Elizabeth A. Crane. 2010.  Methodology for Assessing Bodily Expression of Emotion.  _Journal of Nonverbal Behavior_ 34(4) (2010), 223–248.
  * Gucluturk et al. (2018) Yagmur Gucluturk, Umut Guclu, Xavier Baro, Hugo Jair Escalante, Isabelle Guyon, Sergio Escalera, Marcel A. J. van Gerven, and Rob van Lier. 2018.  Multimodal First Impression Analysis with Deep Residual Networks.  _IEEE Transactions on Affective Computing_ 9, 3 (July 2018), 316–329.  https://doi.org/10.1109/TAFFC.2017.2751469
  * Gunes and Pantic (2010) Hatice Gunes and Maja Pantic. 2010.  Dimensional emotion prediction from spontaneous head gestures for interaction with sensitive artificial listeners. In _International conference on intelligent virtual agents_. Springer, 371–377.
  * Hammal et al. (2014) Zakia Hammal, Jeffrey F Cohn, and David T George. 2014\.  Interpersonal Coordination of Head Motion in Distressed Couples.  _IEEE transactions on affective computing_ 5, 2 (2014), 155–167.
  * Hammal et al. (2015b) Zakia Hammal, Jeffrey F Cohn, Carrie Heike, and Matthew L Speltz. 2015b.  What can head and facial movements convey about positive and negative affect?. In _Affective Computing and Intelligent Interaction (ACII), 2015 International Conference on_. IEEE, 281–287.
  * Hammal et al. (2015a) Zakia Hammal, Jeffrey F Cohn, and Daniel S Messinger. 2015a.  Head movement dynamics during play and perturbed mother-infant interaction.  _IEEE transactions on affective computing_ 6, 4 (2015), 361–370.
  * Hoppe et al. (2018) Sabrina Hoppe, Tobias Loetscher, Stephanie A. Morey, and Andreas Bulling. 2018.  Eye Movements During Everyday Behavior Predict Personality Traits.  _Frontiers in Human Neuroscience_ 12 (2018), 105.  https://doi.org/10.3389/fnhum.2018.00105
  * Ishii et al. (2020) Ryo Ishii, Chaitanya Ahuja, Yukiko I. Nakano, and Louis-Philippe Morency. 2020.  Impact of Personality on Nonverbal Behavior Generation. In _ACM International Conference on Intelligent Virtual Agents_.  https://doi.org/10.1145/3383652.3423908
  * Jarrett (2014) Christian Jarrett. 2014\.  The psychology of first impressions – digested.  _Research Digest_ (2014).  https://digest.bps.org.uk/2014/07/22/the-psychology-of-first-impressions-digested/
  * Jayagopi et al. (2009) D. B. Jayagopi, H. Hung, C. Yeo, and D. Gatica-Perez. 2009.  Modeling Dominance in Group Conversations Using Nonverbal Activity Cues.  _IEEE Transactions on Audio, Speech, and Language Processing_ 17, 3 (2009), 501–513.  https://doi.org/10.1109/TASL.2008.2008238
  * Koelstra and Patras (2013) Sander Koelstra and Ioannis Patras. 2013.  Fusion of facial expressions and EEG for implicit affective tagging.  _Image and Vision Computing_ 31, 2 (2013), 164–174.  https://doi.org/10.1016/j.imavis.2012.10.002 Affect Analysis In Continuous Input.
  * Koppensteiner (2013) Markus Koppensteiner. 2013\.  Motion cues that make an impression: Predicting perceived personality by minimal motion information.  _Journal of Experimental Social Psychology_ 49, 6 (2013), 1137–1143.  https://doi.org/10.1016/j.jesp.2013.08.002
  * Lepri et al. (2012) Bruno Lepri, Ramanathan Subramanian, Kyriaki Kalimeri, Jacopo Staiano, Fabio Pianesi, and Nicu Sebe. 2012.  Connecting Meeting Behavior with Extraversion—A Systematic Study.  _IEEE Transactions on Affective Computing_ 3, 4 (2012), 443–455.  https://doi.org/10.1109/T-AFFC.2012.17
  * Mount et al. (1998) M. K. Mount, M. R. Barrick, and G. L. Stewart. 1998\.  Five-Factor Model of personality and performance in jobs involving interpersonal interactions.  _Human Performance_ 11, 3 (1998), 145–165.
  * Moy and Lam (2003) J.W. Moy and K.F. Lam. 2003\.  Selection criteria and the impact of personality on getting hired.  _Personnel review_ (2003).
  * Naim et al. (2018) Iftekhar Naim, Md. Iftekhar Tanveer, Daniel Gildea, and Mohammed Ehsan Hoque. 2018\.  Automated Analysis and Prediction of Job Interview Performance.  _IEEE Transactions on Affective Computing_ 9, 2 (2018), 191–204.  https://doi.org/10.1109/TAFFC.2016.2614299
  * Osugi and Kawahara (2018) Takayuki Osugi and Jun I Kawahara. 2018.  Effects of Head Nodding and Shaking Motions on Perceptions of Likeability and Approachability.  _Perception_ 47, 1 (2018), 16–29.  https://doi.org/10.1177/0301006617733209
  * Parekh et al. (2018) Viral Parekh, Pin Sym Foong, Shengdong Zhao, and Ramanathan Subramanian. 2018.  AVEID: Automatic Video System for Measuring Engagement In Dementia. In _International Conference on Intelligent User Interfaces_. 409–413.
  * Raza and Carpenter (1987) S. Raza and Bruce Carpenter. 1987.  A model of hiring decisions in real employment interviews.  _Journal of Applied Psychology_ 72 (1987), 596–603.
  * Samanta and Guha (2017) Atanu Samanta and Tanaya Guha. 2017.  On the role of head motion in affective expression. In _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2886–2890.
  * Samanta and Guha (2021) Atanu Samanta and Tanaya Guha. 2021.  Emotion sensing from head motion capture.  _IEEE Sensors Journal_ 21(4) (2021), 5035–5043.
  * Selvaraju et al. (2019) Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2019.  Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization.  _International Journal of Computer Vision_ 128, 2 (Oct 2019), 336–359.  https://doi.org/10.1007/s11263-019-01228-7
  * Subramanian et al. (2010) Ramanathan Subramanian, Jacopo Staiano, Kyriaki Kalimeri, Nicu Sebe, and Fabio Pianesi. 2010\.  Putting the Pieces Together: Multimodal Analysis of Social Attention in Meetings. In _International Conference on Multimedia_. 659–662.
  * Subramanian et al. (2018) R. Subramanian, J. Wache, M. K. Abadi, R. L. Vieriu, S. Winkler, and N. Sebe. 2018\.  ASCERTAIN: Emotion and Personality Recognition Using Commercial Sensors.  _IEEE Transactions on Affective Computing_ 9, 2 (2018), 147–160.  https://doi.org/10.1109/TAFFC.2016.2625250
  * Subramanian et al. (2013) Ramanathan Subramanian, Yan Yan, Jacopo Staiano, Oswald Lanz, and Nicu Sebe. 2013\.  On the Relationship between Head Pose, Social Attention and Personality Prediction for Unstructured and Dynamic Group Interactions. In _International Conference on Multimodal Interaction_. Association for Computing Machinery, New York, NY, USA, 3–10.  https://doi.org/10.1145/2522848.2522862
  * Takayama and Pantofaru (2009) L. Takayama and C. Pantofaru. 2009.  Influences on proxemic behaviors in human-robot interaction. In _2009 IEEE/RSJ International Conference on Intelligent Robots and Systems_. 5495–5502.  https://doi.org/10.1109/IROS.2009.5354145
  * Van Dam (2003) Karen Van Dam. 2003\.  Trait Perception in the Employment Interview: A Five–Factor Model Perspective.  _International Journal of Selection and Assessment_ 11, 1 (2003), 43–55.  https://doi.org/10.1111/1468-2389.00225
  * Ventura et al. (2017) Carles Ventura, David Masip, and Agata Lapedriza. 2017\.  Interpreting CNN Models for Apparent Personality Trait Regression. In _Computer Vision and Pattern Recognition Workshops_. 1705–1713.  https://doi.org/10.1109/CVPRW.2017.217
  * Xiao et al. (2015) Bo Xiao, Panayiotis Georgiou, Brian Baucom, and Shrikanth S Narayanan. 2015.  Head motion modeling for human behavior analysis in dyadic interaction.  _IEEE transactions on multimedia_ 17, 7 (2015), 1107–1119.
  * Yang and Narayanan (2017) Zhaojun Yang and Shrikanth S Narayanan. 2017.  Modeling dynamics of expressive body gestures in dyadic interactions.  _IEEE Transactions on Affective Computing_ 8, 3 (2017), 369–381.