# Improving face generation quality and prompt following with synthetic
captions

Michail Tarasiou Stylianos Moschoglou Jiankang Deng Stefanos Zafeiriou
Imperial College London

###### Abstract

Recent advancements in text-to-image generation using diffusion models have
significantly improved the quality of generated images and expanded the
ability to depict a wide range of objects. However, ensuring that these models
adhere closely to the text prompts remains a considerable challenge. This
issue is particularly pronounced when trying to generate photorealistic images
of humans. Without significant prompt engineering efforts models often produce
unrealistic images and typically fail to incorporate the full extent of the
prompt information. This limitation can be largely attributed to the nature of
captions accompanying the images used in training large scale diffusion
models, which typically prioritize contextual information over details related
to the person’s appearance. In this paper we address this issue by introducing
a training-free pipeline designed to generate accurate appearance descriptions
from images of people. We apply this method to create approximately 250,000
captions for publicly available face datasets. We then use these synthetic
captions to fine-tune a text-to-image diffusion model. Our results demonstrate
that this approach significantly improves the model’s ability to generate
high-quality, realistic human faces and enhances adherence to the given
prompts, compared to the baseline model. We share our synthetic captions,
pretrained checkpoints and training code
`https://github.com/michaeltrs/Text2Face`

##  1 Introduction

Recent years have witnessed a significant resurgence in generative modelling
for a range of modalities such as text [13, 11, 19], vision [5, 18, 9, 17, 15,
4] and audio [10].

Particularly in vision, diffusion-based image generators have dramatically
enhanced the quality and expressiveness of images generated from various input
signals such as text, depth maps, and semantic labels just to name a few.
Text-to-image models, trained on large-scale, internet-based image-caption
pairs have demonstrated remarkable generative capabilities [15, 4]. However,
captions describing images sourced from the internet often lack information
that is useful for training such models. Although there is some signal to
learn the content and style of images from their textual descriptions, most
internet-based captions provide context for the image rather than accurately
describing what is depicted. This discrepancy is particularly pronounced when
working with images of humans, where internet-sourced captions can encompass
multiple aspects such as the setting, profession or even the name of the
person depicted, without necessarily describing their appearance. Some
examples of image-caption pairs that exhibit these characteristics are shown
in Fig.1. As a result, “in-the-wild” data exhibit a low signal-to-noise ratio,
which hinders effective training. Consequently, very large datasets are
required. Furthermore, models trained in this manner often fall short of
generating images that accurately follow detailed prompt descriptions.

A solution to this problem can be found through utilizing curated datasets or
synthetic captioning. Several studies have found that smaller, higher-quality
annotations can lead to improved modeling outcomes [1]. Building on this
insight and utilizing the extensive literature and resources around face
analysis, this paper proposes a pipeline for generating synthetic captions
describing a person’s appearance only from face images.

Our contributions are twofold:

  * •

We propose a training-free captioning pipeline for face images based on
extensive prior work in 2D and 3D face analysis. Specifically, we leverage a
set of publicly available models, pre-trained on specific face analysis tasks
(e.g., face and landmark detectors), to extract detailed appearance
attributes. These attributes are far more detailed than the information
typically contained in large-scale web captions. Then, we transform these
outputs into word descriptions of attributes, producing a bag-of-words
description for each face/image. These descriptions are then processed by a
pretrained, instruction-tuned Large Language Model (LLM) to produce natural
language image captions. We share the synthetic captions for approximately
250k face images from public datasets.

  * •

Using our synthetic captions, we fine-tune a text-to-image diffusion model to
produce high-quality, realistic face images that closely adhere to the
provided text descriptions. We also make this model publicly available.

Figure 1: Comparison with captions from the LAION-Face dataset [21]. For each
image we show the caption provided in LAION-Face (top) and the output of our
synthetic captioning pipeline (bottom).

##  2 Synthetic Captions for Human Faces

In this section, we present our training-free pipeline for obtaining textual
descriptions of humans. Section 2.1 describes the feature extraction process,
specifically the pre-trained models used and their respective outputs. Section
2.2 details how we use the extracted bag-of-words features to generate
coherent textual descriptions.

###  2.1 Feature extraction

We begin by applying a powerful face detection model to all face images using
a ResNet50 backbone from [3]. The output of the face detector provides the
number of faces detected in each image, along with their locations and five
facial landmarks for: the eye pupils, tip of the nose and left/right
commissures of the mouth. We exclude all images containing more than one face
and use the facial landmarks to crop square images around the face region.
These crops are optionally aligned to predefined templates for inference with
subsequent models. At this stage, we ensure that only the facial region is
shown, with the face centered within the crop.

Next, we proceed with a second round of filtering aimed at removing images of
animated humans and cartoons, which the face detector incorrectly identifies
as humans. We have found CLIP [14] to be an effective solution for identifying
these images. We design CLIP text prompts to detect images either not
depicting real humans or containing significant amounts of text overlaid over
the depicted faces, such as album covers or film posters. Following this
filtering, we use CLIP to assess if the person’s teeth or tongue are visible,
as this information allows for better rendering, especially for teeth.

Subsequently, we proceed to feature extraction. First, we extract facial
attributes using [21] to obtain a list of attributes characterizing the person
in the image. The pretrained model detects the presence of each of the
following classes independently: “5 o’clock shadow”, “arched eyebrows”,
“attractive”, “bags under eyes”, “bald”, “bangs”, “big lips”, “big nose”,
“black hair”, “blond hair”, “blurry”, “brown hair”, “bushy eyebrows”,
“chubby”, “double chin”, “eyeglasses”, “goatee”, “gray hair”, “heavy makeup”,
“high cheekbones”, “male”, “mouth slightly open”, “mustache”, “narrow eyes”,
“no beard”, “oval face”, “pale skin”, “pointy nose”, “receding hairline”,
“rosy cheeks”, “sideburns”, “smiling”, “straight hair”, “wavy hair”, “wearing
earrings”, “wearing hat”, “wearing lipstick”, “wearing necklace”, “wearing
necktie”, “young”.

We obtain a description of the emotions exhibited by the person through a
emotion recognition with single class categorical outputs using [12]. Detected
emotions are: “anger”, “disgust”, “fear”, “happiness”, “sadness”, “surprise”
and “neutral”. We retain the most dominant emotion detected. If no single
emotion is clearly dominant, we retain the top two emotions and describe the
person as exhibiting both.

Next, all images are processed for face parsing using [21] resulting in dense
class maps for all facial regions. We use these outputs to get a set of
derivative attributes, such as the length of the hair and descriptions of the
openness of the eyes and mouth.

Finally, we use a model to gather information about the age, gender, and
race/ethnicity of the person [16]. We find that gender is predicted with a
high degree of accuracy. However, age is a subjective attribute that varies
significantly among individuals, as people age at different rates. Therefore,
using a specific age based on predictive models can be misleading. To address
this, we treat age as a range rather than a precise value. This approach
allows for more flexibility and realism in the synthetic labeling of human
images. Specifically, we either sample an age from a distribution centered
around the predicted age or categorize ages into five-year intervals to better
accommodate the variability in aging among different people and the
inaccuracies in model predictions. For ethnicity, the model predicts the
closest output among classes: “black”, “white”, “asian”, “middle eastern”,
“indian”, “hispanic”.

###  2.2 Language Fusion

Following the completion of the feature extraction stage, we can associate
each extracted feature with a descriptive word. Thus, we now have a set of
features describing the person in an image in the form of a bag-of-words. To
obtain a set of descriptions in the form of natural language, we simply use an
instruct-based pre trained LLM, which we condition to output a description
based on the provided bag of words. For this purpose, we utilise the Vicuna
1.5 13B model [20]. Through experimentation, we find that the smallest 7B
parameter model is not good enough for our purposes as it fails to follow
prompt instructions adequately. In particular, we have found that the model
fails to produce short descriptions and often hallucinates either descriptions
that were not provided or goes into an effort to describe further information
about a person that is not part of their appearance description. The 13B
parameter model is able to follow directions more precisely and is adequate
for the task at hand. Interestingly, we have found that because the 7B
parameter model fails to provide short descriptions, its runtime can exceed
that of the 13B model due to needless long generations.

We follow the process below to obtain sentence descriptions using the LLM. For
each sample, we collect the bag-of-words description for all features
discussed in sec. 2.1. For all features, we distinguish between F1 level
features, which are primary descriptors such as age, gender and ethnicity and
F2 features, which are more ephemeral, e.g., wearing a hat or beard. All bag-
of-features are randomly permuted prior to feeding to the LLM to vary the
order they appear in the textual outputs. For age, we either provide the
predicted age by adding some noise (plus minus 1/15 of this value), define an
age bracket (plus minus 5 years) or define some descriptive categories for
each age (“baby”, “toddler”, “preschooler”, “child”, “teenager”, “young
adult”, “adult”, “middle-aged adult”, “senior adult”, “elderly”) with equal
probability. Finally, we have noticed a high degree of correlation among
detected features. For example, in FFHQ, more than $90\%$ of images classified
as attractive using [21] are women, and among these, more than $80\%$ are
classified as wearing makeup. To avoid our data being biased to relate
attractiveness with wearing makeup, we randomly drop the ”attractive” label
when it coincides with makeup with $80\%$ probability.

We use the following prompt to obtain textual descriptions with Vicuna 13B:

“Without elaborating, describe a person with all of the following
characteristics: F1. They have the following attributes: F2. Combine specific
characteristics to produce a coherent description. You are encouraged to use
synonyms for the provided attributes but not to add information other than
what is provided. It is important to use all provided characteristics. Do not
repeat characteristics that are provided more than once. Do not repeat these
instructions.”

Optionally, we append the ”The image is blurry” or ”The image is black and
white” in case we have a blurry or a monochrome image. For each image, we
obtain more than one LLM output, which we randomly sample from during
training, to emphasize the multiplicity of descriptions corresponding to a
facial image.

##  3 Experiments

In this section we train a text-to-image generation model on the images and
captions generated following the process presented in sec 2. We initialize
model parameters from Stable Diffusion 2.1 [15, 2] which we finetune using
LoRA [6]. In particular we train LoRAs attached to the UNet denoiser and the
CLIP text encoder of SD2.1.

###  3.1 Datasets

We apply our caption generation pipeline to three publicly available datasets:
EasyPortrait [7], FFHQ [8] and LAION-Face [21]. For LAION-Face we apply
filtering with the face detector and CLIP as discussed in sec 2 to exclude
non-human images as well as low resolution images less than $250\times 250$
face region. For EasyPortrait and FFHQ we retain all samples. This process
results in 40k, 70k and 156k images (and generated captions) for EasyPortrait,
FFHQ and LAION-Face respectively.

###  3.2 Implementation details

We apply LoRAs on the UNet denoiser and the CLIP text encoder of SD2.1. We
finetune for 30k steps at $768\times 768$ pixels reesolution using a constant
learning rate 1E-5 and $\times 8$ Nvidia V100 cards, each loaded with four
samples per training step (total batch size 32). The finetuning process takes
approximately two days to complete.

Figure 2: Comparison with SD2.1 base model. For each prompt we show the image
generated by the SD2.1 base model (left) as well as our finetuned LoRA model
(right). Figure 3: Generated images varying the person’s age. Images are
generated through the prompt ”A $\\{$age$\\}$ year old white male with black
hair and happy expression.” where age is substituted by the number shown in
each image. We observe that identity characteristics are decoupled from age.
Figure 4: Generated images varying a person’s ethnicity, emotion and gender.
Images are generated through the prompt ”A 40 year old $\\{$ethnicity$\\}$
$\\{$gender$\\}$ with black hair and $\\{$expression$\\}$ expression.” where
ethnicity, gender and expression are modified accordingly. Gender varies
across the left and right figures. Rows represent Black, White and Asian race.
Respective figure columns show generations for happy, neutral and sad
expressions.

###  3.3 Analysis

In Fig.2 we present a comparison with the base SD2.1 text-to-image model.
Based on this figure we can make two observations: 1) the finetuned model
clearly outputs realistic face images in comparison with the baseline which
often includes a cartoonish effect. It is possible to constrain the base model
to produce realistic images with prompt engineering, however, this takes
considrable effort and computing resourses to generate multiple images
varrying the prompt and seed and does not guarantee realistic results. 2) The
finetuned models typically follows the prompt more closely, especially when
the propmt is complicated and contains multiple instructions.

In Fig.3 we use the same prompt and random seed and vary the person’s age from
age 20 to 50, at 5 year intervals. We observe that the identity of the person
is preserved to some degree for different ages. More specifically, we note a
clustering of identity characteristics around lower (20-25) and higher ($>$35)
age values with images in between serving as a transition between the two.

Similarly, in Fig.4 we vary the gender, ethnicity and emotion of a person.
Again, identity characteristics are preserved to some degree between these
attributes, altough not as clearly as for age.

###  3.4 Limitations

Since the pipeline involves the use of pretrained face analysis models, their
outputs will unavoidably contain all biases found in the employed models.
Similarly, the trained model is finetuned from SD2.1 and as such, inherits all
the limitations and biases associated with the base model. Overall, biases may
manifest in skewed representations across different ethnicities, emotions, age
and genders due to the nature of the training data originally used for either
the facial analysis models or Stable Diffusion 2.1. Specific limitations
include:

  * •

Ethnic and gender biases: The model may generate images that do not equally
represent the diversity of human features in different ethnic and gender
groups, potentially reinforcing or exacerbating existing stereotypes.

  * •

Selection bias in finetuning datasets: The datasets used for finetuning this
model were selected with specific criteria in mind, which may not encompass a
wide enough variety of data points to correct for the inherited biases of the
base model.

  * •

Caption generation bias: The synthetic annotations used to finetune this model
were generated by automated face analysis models, which themselves may be
biased. This could lead to inaccuracies in facial feature interpretation and
representation, particularly for less-represented demographics in the training
data.

##  4 Conclusion

We presented a training-free pipeline for the automatic extraction of facial
image descriptions to generate large scale image-caption pairs for facial
images. These data can be used in training vision-language models for humans.
In that direction we used the generated captions to train a text-to-image
diffusion model that generates realistic images of humans following prompt
conditioning more closely than the base model. The generated captions and
finetuned model are made publicly available to encourage further research into
this space.

## References

  * [1] https://cdn.openai.com/papers/dall-e-3.pdf.
  * sd [2] https://huggingface.co/stabilityai/stable-diffusion-2-1.
  * Deng et al. [2020] Jiankang Deng, Jia Guo, Evangelos Ververas, Irene Kotsia, and Stefanos Zafeiriou.  Retinaface: Single-shot multi-level face localisation in the wild.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Esser et al. [2024] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach.  Scaling rectified flow transformers for high-resolution image synthesis, 2024.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _CoRR_ , abs/2006.11239, 2020.
  * Hu et al. [2022] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  LoRA: Low-rank adaptation of large language models.  In _International Conference on Learning Representations_ , 2022.
  * Kapitanov et al. [2023] Alexander Kapitanov, Karina Kvanchiani, and Kirillova Sofia.  Easyportrait - face parsing and portrait segmentation dataset.  _arXiv preprint arXiv:2304.13509_ , 2023.
  * Karras et al. [2018] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  _CoRR_ , abs/1812.04948, 2018.
  * Karras et al. [2022] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine.  Elucidating the design space of diffusion-based generative models, 2022.
  * Kong et al. [2021] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro.  Diffwave: A versatile diffusion model for audio synthesis.  In _International Conference on Learning Representations_ , 2021.
  * Ouyang et al. [2022] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe.  Training language models to follow instructions with human feedback, 2022.
  * Pham et al. [2021] Luan Pham, The Huynh Vu, and Tuan Anh Tran.  Facial expression recognition using residual masking network.  In _2020 25th International Conference on Pattern Recognition (ICPR)_ , pages 4513–4519. IEEE, 2021.
  * Radford et al. [2019] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  2019\.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  _CoRR_ , abs/2103.00020, 2021.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 10684–10695, 2022.
  * Serengil and Ozpinar [2020] Sefik Ilkin Serengil and Alper Ozpinar.  Lightface: A hybrid deep face recognition framework.  In _2020 Innovations in Intelligent Systems and Applications Conference (ASYU)_ , pages 23–27. IEEE, 2020.
  * Song et al. [2020a] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models, 2020a.
  * Song et al. [2020b] Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  _CoRR_ , abs/2011.13456, 2020b.
  * Touvron et al. [2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.  Llama: Open and efficient foundation language models, 2023.
  * Zheng et al. [2023] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica.  Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
  * Zheng et al. [2022] Yinglin Zheng, Hao Yang, Ting Zhang, Jianmin Bao, Dongdong Chen, Yangyu Huang, Lu Yuan, Dong Chen, Ming Zeng, and Fang Wen.  General facial representation learning in a visual-linguistic manner.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18697–18709, 2022.


11institutetext: Massachusetts Institute of Technology, Cambridge MA 02139,
USA 11email<EMAIL_ADDRESS><EMAIL_ADDRESS>22institutetext: Personal Robotics
Group, MIT Media Lab, Cambridge MA 02139, USA 22email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>33institutetext: Japan
Broadcasting Corporation, Tokyo, Japan
33email<EMAIL_ADDRESS>
# Explainable AI for Suicide Risk Assessment
Using Eye Activities and Head Gestures

Siyu Liu 11 Catherine Lu 11 Sharifa Alghowinem 22 0000-0002-9391-0163 Lea
Gotoh 33 Cynthia Breazeal 22 0000-0002-0587-2065 Hae Won Park 22
0000-0001-9638-1722

###### Abstract

The prevalence of suicide has been on the rise since the 20th century, causing
severe emotional damage to individuals, families, and communities alike.
Despite the severity of this suicide epidemic, there is so far no reliable and
systematic way to assess suicide intent of a given individual. Through efforts
to automate and systematize diagnosis of mental illnesses over the past few
years, verbal and acoustic behaviors have received increasing attention as
biomarkers, but little has been done to study eyelids, gaze, and head pose in
evaluating suicide risk. This study explores statistical analysis, feature
selection, and machine learning classification as means of suicide risk
evaluation and nonverbal behavioral interpretation. Applying these methods to
the eye and head signals extracted from our unique dataset, this study finds
that high-risk suicidal individuals experience psycho-motor retardation and
symptoms of anxiety and depression, characterized by eye contact avoidance,
slower blinks and a downward eye gaze. By comparing results from different
methods of classification, we determined that these features are highly
capable of automatically classifying different levels of suicide risk
consistently and with high accuracy, above 98%. Our conclusion corroborates
psychological studies, and shows great potential of a systematic approach in
suicide risk evaluation that is adoptable by both healthcare providers and
naïve observers.

###### Keywords:

Affective Computing Suicide Risk Nonverbal Behaviour Explainable AI.

##  1 Introduction

Among the many suicide attempts each year, 800,000 people die [37]. Often,
they suffered from depression, anxiety, or have histories of self-injury for
years [18]. Those that pass leave tremendous emotional strain to their
families and communities. Japan is at the center of this suicide epidemic.
Historical trauma, war, unemployment, natural disasters, high stress, all
contribute to the alarmingly high rate of 20,000 deaths per year [26]. Yet,
suicide is preventable. Given the association between suicide inclination and
mental disorder, early identification and intervention can mitigate many
tragedies [28].

Current suicide assessment criteria — including Behavioral Health Screening
(BHS), Manchester self harm rule (MSHR), and Södersjukhuset self-harm rule
(SOS-4) — lack specificity and sensitivity, and are difficult to administrate,
and those with suicide inclinations are often blamed for not seeking help [35,
31]. This hinders early detection and intervention.

Recent efforts to automate suicide risk assessment using machine learning on
metadata has shown promising results, yielding high classification accuracy
($>$ 90%) in suicide ideation prediction using electronic medical records and
co-morbidity biomarkers [6]. However, significant challenges exist in
application of these results, as no model has successfully been transferred
into new settings or populations [6, 13]. While recent works have explored
using objective verbal and acoustic markers for suicide risk assessment [20,
9, 15], to our best knowledge, the three studies led by Laksana [23], Eigbe
[11] and Shah [33] are the only works that investigate observable facial
behavior markers.

The purpose of this study is to construct a systematic and interpretable
approach for suicide risk assessment. We investigate whether nonverbal
behaviors from the eyelids, head pose, and eye gaze hold discriminative power
for evaluating suicidal risks, considering a wider array of behavioral
features than previous works using statistical analysis, feature selection,
and classification with machine learning. Given the interpretability approach
of this work, we hope to propose a systematic approach in suicide risk
evaluation that is executable for both healthcare providers and naïve
observers. The main contributions of this research are as follows:

  * •

extract objective and interpretable behaviors that are discernible to naïve
observers

  * •

provide analyses using statistical methods, feature selection, and machine
learning with a focus on explainability

##  2 Background

Nonverbal indicators of suicide ideation have received far less attention than
verbal ones. Laksana et al. found simple facial features including smiling and
head and eyebrow movement effective for suicide risk assessment [23], and that
suicidal individuals produce more non-Duchenne smiles. Eigbe et al. verified
this result and also found that suicidal subjects look down more often than
their non-suicidal counterparts [11].

Suicidal intent have strong correlation with other mental disorders, including
depression and anxiety. Symptoms of anxiety and depression — high stress,
fatigue, and social withdrawal — have been observed among suicidal individuals
as well [35]. Particularly, statistical analysis and machine learning
classification of the movement of facial and ocular landmarks found that
depressed patients have narrower eye openings, longer duration of blinks, and
slower and less frequent head movements that signify fatigue or eye contact
avoidance [2, 36, 3]. Fossi et al. attribute this phenomenon as a form of
psycho-motor retardation [14]. Additionally, by analyzing data collected with
eye trackers, infra-red corneal reflection techniques, or human observation,
many studies have confirmed that patients with anxiety disorder avoid eye
contact and direct gaze at the person’s face in a photo, but scan non-facial
features and the surrounding environment more extensively [27, 19, 22]. As
depression and anxiety disorder are significant risk factors for suicidal
thoughts and behaviors, we may leverage from nonverbal indicators of
depression and anxiety to formulate our research.

To classify such indicators, machine learning has been utilized extensively.
Automatic assessment of mental health disorders such as depression,
schizophrenia, and bipolar disorder has primarily used machine learning
classification to provide objective diagnosis [25]. These efforts have been
successful. For instance, Cohn et al. [10], Abaei and Osman [1], and Shah et
al. [33] classified for depression, bipolar disorder, and suicidal intent by
analyzing visual, acoustic, and verbal features using machine learning
techniques including support vector machines (SVMs), logistic regression, and
convolutional neural-nets (CNNs). Given the correlation between these
disorders and suicide, classification of verbal and nonverbal behaviors may
yield good results for predicting suicidal risk.

Yet, few studies have attempted automatic machine learning classification of
suicide risk. Laksana et al. [23] and Eigbe et al. [11] have achieved 40% and
69% accuracy in classifying suicidal, mentally ill, and control groups based
on facial behaviors. In this paper, we explore combinations of different
discriminative models (multi-layer perceptron and SVMs) and data processing
methods to determine the best-performing classifier of suicide risk. We build
upon the previous research by extracting an extensive array of nonverbal
behaviors from head and eye movements. Then, we twice execute classification –
once with all features and once with just 10% selected through feature
selection and statistical analysis. Specifically, we chose 10% in order to
only select the most prominent features. If the classification results
obtained in both trials are similar, we deem the selected features to be
representative of the task and capable of explaining the model.

##  3 Methodology

Figure 1: The flowchart shows the general steps and their sub-steps start from
dataset acquisition, pre-processing of signals (cropping image, facial
landmark extraction, and basic feature calculation), low-level feature
extraction (EAR, eye gaze, head pose estimation), post-processing of signals
(window smoothing, signal normalization, thin-slicing), high-level feature
extraction through functional summaries, and ending with analysis and
interpretation of results through statistical analysis, features selection,
and classification.

The overall process including pre-processing, low-level features extraction,
post-processing and high-level feature summaries, and analysis is summarized
in Fig. 1 and described in the following subsections.

Table 1: Number of Subjects, Interview Duration and Total Segments in each Suicide Risk Level in the Sub-Dataset used in this Work Risk Level | # Subj | Interview Duration (min) | # of 2 min Segments
---|---|---|---
Low | 4 | 60.80 | 54
Medium | 4 | 88.00 | 63
High | 2 | 31.16 | 27
Total | 10 | 179.96 | 144

###  3.1 Suicide Dataset Collection

The dataset was collected in collaboration with Japan Broadcasting Corporation
(NHK), who conducted interviews with Japanese participants aged 15 to 25 who
are suffering from suicidal thoughts as part of their suicide awareness
project. The project was advertised through social media, which invited people
to share stories of their (or their loved ones’) suffering on the project’s
website “Face to Suicide.” 111http://www6.nhk.or.jp/heart-net/mukiau. People
who were willing to be further interviewed were recruited for the project. All
participants provided consent via email communication. The selected
participants were pre-interviewed (without camera recording) by two project
directors for screening and suicide risk assessment. During the pre-interview,
the participants were assessed using the “Suicidal Risk” subsection of the
M.I.N.I. (Mini-International Neuropsychiatric Interview) [24], which consists
of 6 questions about their negative thoughts (e.g., suicidal ideation and
intensity). Upon consultation with psychiatrists, another 5 questions were
added to the pre-interview about their history of self-harm and clinical
visits, environment (e.g., age, family, socioeconomic status), and positive
thoughts (e.g., trusted persons, hobbies, coping mechanisms). The two
directors then categorized each participant as a low, medium, or high suicide
risk individual. These assessments and potential conflicts were subsequently
cross-validated and were resolved via discussion that accompanied psychiatrist
consultation. A total of 14 participants were interviewed, once each; 4 of
them requested to have their faces blurred for the recording. This work uses
the 10 interviews with faces not blurred to study participants’ nonverbal
behaviors. Four participants were assessed as low risk, four at medium risk,
and two at high risk (see Table 1). The interviews were conducted in several
locations with different room layouts (e.g., window and door location,
furniture), but the relative position of the interviewer-interviewee were
consistent — both sat in chairs where they faced each other.

###  3.2 Feature Extraction

##### Pre-processing of feature extraction

High resolution videos were manually cropped to contain only the face region
of the subject. For each frame of the processed video, we used single-shot
face detection to locate a total of 68 facial landmarks, as proposed in [38].
Frames where the face is not fully captured were skipped. Additionally, using
the facial landmarks, cropped images containing each eye were obtained from
each valid frame, which served as inputs to calculate gaze direction (yaw and
roll).

##### Low Level Feature Extraction

For each frame, Eye Aspect Ratio (EAR) and gaze direction were extracted for
each eye, and head direction was extracted for the entire face image, as
described below. Proposed by Soukupova and Cech, EAR characterize the openness
of an eye as a scalar quantity in an image[7], where this ratio is normalized
against distance of the eye to the camera and eye size differences between the
subjects. A larger EAR value indicates a wider opening. A similar approach was
taken by [2] for the analysis of eye movement for depression detection. Using
the 6 eye-region landmark coordinates, EAR is computed based on the formula as
follows, where $p_{i}$ are the 2D landmark locations as depicted in Fig. 1
(EAR).

$\dfrac{\left\lVert p_{2}-p_{6}\right\rVert-\left\lVert
p_{3}-p_{5}\right\rVert}{2\left\lVert p_{1}-p_{4}\right\rVert}$ (1)

To describe each eye’s angle of gaze, two scalar quantities, pitch and yaw,
were extracted per frame. We used [40]’s multimodal CNN to estimate the eye
gaze. The inputs of this CNN are the facial landmarks, normalized head angle
vectors, and cropped eye images. This been trained on the MPIIGaze dataset
that contains 213,659 images with diversity in appearance and illumination
collected from everyday laptop usage of 15 participants [39].

Head pose was extracted in terms of three angle vectors — pitch, yaw, and roll
for each frame — to represent the direction the subject is facing. Our study
employed the technique outlined by [34] to estimate the 3D head pose using 6
facial landmarks — four eye corners and two mouth corners. By fitting generic
3D facial shape models onto our facial landmarks, we obtained angles of
rotation of the head. Lastly, a distance to the face was calculated from the
camera’s focal point, based on the apparent size of the face.

##### Post-processing of the Signals

Low-level feature signals were post-processed to eliminate outliers and noise.
For each low-level signal, a moving average was computed with a window size of
7 frames (window size 3 – 20 were attempted). The moving average window was
chosen empirically to reduce noise without eliminating actual movements. For
features that are applicable to both eyes (EAR, eye pitch, and eye yaw),
results obtained from the left and right eye were averaged to reduce the
variability and noise of the two signals. This final 7 low-level features
(avg. EAR, avg. eye pitch, avg. eye yaw, head distance, head pitch, head yaw,
and head roll) were used in further analysis. In order to eliminate inter-
subject variability including the participant’s personality, mannerism, and
appearance, we linearly scaled each signal from 0 to 1.

##### High Level Feature Extraction

We segmented our data into small windows following the physiological thin-
slicing theory [5], where a brief observation of a behavior (a thin slice) can
be indicative to the physiological outcome at levels similar to the full
observation. This process also serves as a mitigation for the small number of
subjects in our dataset. As Shah et al.’s experiment has shown, 2-minute
slices are just as effective as 5-minute slices in capturing features of the
video [33]. Hence, we segmented each subject’s signal into 2-minute slices,
with a one-minute overlap between two adjacent slices. The total number of
segments per risk group is listed in Table 1. A total of 210 statistical
features, i.e., “functionals”, were extracted to summarize the segments. For
each of the 7 post-process low-level features, 30 statistics were calculated,
which are:

  * •

The speed and acceleration at which the signal moves. This is computed by
taking the first and second derivatives with regard to the frame index.

  * •

Maximum, minimum, range, mean, variance, standard deviation, skewness,
kurtosis, number of peaks and valleys for the original signal and its
derivatives.

These functionals capture the frequency and velocity of the movement of each
signal, as well as the duration of continuous movement and their direction.
The compilations of statistical features from 2-minute segments across videos
from the same categories — a statistical summary — were used as samples for
our classification, feature selection, and statistical analysis. One study
employing a similar approach of statistical hypothesis testing and multimodal
predictive modeling yielded promising results in predicting suicide intent
using sliced videos from social media [33].

###  3.3 Feature Selection

A feature selection framework (FS) to systematically select the most
representative features that are correlated to an independent variable, i.e.
nonverbal behavior with suicide risks, was proposed in [4], which we
replicated in this work. While extracting a large number of features as an
exploration of behaviors is a common practice in the AI community, it is not
commonly practiced in the psychology field for diagnoses without confirming an
assumption or a hypothesis. Such confirmation is done through statistical
analysis that accounts for multi-test correction (e.g., Bonferroni
correction), which might not be ideal for large feature spaces and small
sample sizes. Therefore, the FS framework aimed to fill the gap by
systematically aggregating the results from several statistical analyses with
several methods of feature selection. The framework also serves as an
interpretation tool through narrowing the feature space to the most meaningful
features for a variable/class, not only by analyzing the features
independently, but also analyzing the relationship between features (e.g.,
removing redundant features, finding a combination of features that correlate
together, etc.). Given the two-step validation approach, i.e., within and
between methods validation, the sensitivity of any method to the sample size
is mitigated. That is, when a feature is not stable enough for the selection
process, the framework excludes it.

In this work, we selected up to top 10% behavioral features using 16 feature
selection methods (e.g., statistical-based, information theory-based) for the
multi-class suicide risk assessment – high, medium, low. For the cross-
validation method, we used 10-folds with two runs to measure both Jaccard
Index (JI) and Between Threshold Stability (BTS), which were then used to
validate and select the final feature set, as described in [4]. Notably, we
treated the low, medium, and high categories as nominal instead of ordinal
because we are more interested in categorical differences among the three
clusters, without assuming that measurements from medium-risk participants
always sit between those from low and high-risk counterparts. As such, we
chose not to execute correlations analysis where both variables are ordinal or
continuous.

###  3.4 Statistical Analysis

A one-way analysis of variance (ANOVA) test was performed for each functional
to determine whether a systematic difference between the three suicide risk
classes (low, medium, high) exists, followed by a post-hoc two-tailed t-test
when statistical significance was found. For both tests, we chose a
significance value threshold of 0.05 as alpha, and apply Bonferroni correction
to obtain a corrected significance level of $0.05/210=2.3809\cdot 10^{-4}$. We
selected the functionals for which at least one of the three post-hoc p-values
is within this corrected significance threshold. Additionally, we conducted a
multivariate analysis of variance (MANOVA) and a two-factor ANOVA to
understand subject dependence and selected for the most discriminating feature
of suicide risk. Specifically, since the between-subject groups are unbalanced
and there is an interaction between independent variables, type-III sums of
squares estimation was utilized for the two-factor test. As more than one
sample was taken from each video sample the due to the 2-minute segmentation,
we ran a repeated-measure ANOVA to determine features that are indicative of
suicide risk, accounting for subject-dependence. Although assumptions for
ANOVA and T-tests are not fully satisfied – normality and non-colinearity – as
our sample size is enlarged with video segmentation, we proceed executing the
tests with caution.

###  3.5 Classification

Our goal is to classify the statistical and selected features into three
classes for suicide risk: low-risk, medium-risk, high-risk. To select the best
model for this task, we tried variations on three discriminative models: a
multilayer perceptron (MLP) with one hidden layer and SVMs with linear and
radial kernels. To reduce potential bias caused by overrepresented classes, we
tested two different sampling methods: oversampling and undersampling. For
oversampling, we randomly duplicated samples from underpopulated classes, and
for undersampling, we randomly removed samples from overrepresented classes.
This is to ensure that all classes have the same number of samples for
training.

For each combination of model and sampling method, we performed 10 random
stratified trials using data from all 144 statistical summaries. In each
trial, the data was randomly split into a stratified training set and testing
set, with a split of around 75% training data and 25% test data. Both sets
were normalized based on the training data and scaled to values between 0 and
1. The training set was then either left alone, balanced via oversampling, or
balanced via undersampling before fitting the model.

Hyperparameter tuning was done via a grid search using 10-split cross-
validation on the training data. For MLP, our only hyperparameter was the
number of hidden units. We searched over an evenly spaced range of 10 values
slightly greater than the number of features to ensure that a good spread of
reasonable hypothesis classes were tested. For testing with 210 features, this
range was from 22 to 232 hidden units, and for the selected 12 features, the
range was 2 to 14. For SVM, we chose our $C$ value by iteratively performing
narrowing grid searches over intervals of values between 0 and 1, eventually
narrowing down to values 0.01 apart, then selecting the best one. For SVM with
radial kernels, we used several values for gamma, calculated using
1/(n_features * variance in training sample).

The final performance of each modeling process is represented by its
performance on the test data, measured by the mean balanced accuracy score
across all 10 trials. As false negatives are highly undesirable when
classifying suicide risk, we took measures to ensure classification quality
between classes. First, we selected hyperparameters based on balanced scores,
which account for the number of samples in each class, and calculated a
Matthews Correlation Coefficient (MCC) score for each trial. Accounting for
the differences between class sample sizes, MCC is regarded as one of the best
evaluators of classification quality in comparison with other measures like F1
score, since it is only high when good results are achieved from all classes
[8]. We used a generalized multiclass version of MCC as described by Gorodkin
[16].

We performed two additional randomized experiments. One experiment was done
using randomly shuffled feature vectors and the original labels, and another
with randomly shuffled labels and the original feature data. By comparing our
classification results to these results obtained by chance, we ensure that our
results are robust and meaningful. We report all of our results with the best-
performing combinations of models and sampling methods.

##  4 Results and Discussion

###  4.1 Interpretation of Nonverbal Behavior

####  4.1.1 Statistical Analysis

Table 2: Interpretations of the Features that Passed Corrected p-values by Statistical Analysis: 69 behavioral features passed ANOVA and post-hoc t-tests.  Behavioral Theme |  Statistical Features |  Direction
---|---|---
Deminished Eye and Head movement |  var. of speed of EAR changes, var. of EAR, max. of speed of eye pitch movement, mean of speed of eye pitch movement, rang. of speed of eye pitch movement, std. of speed of eye pitch movement, var. of speed of eye pitch movement, max. of acc. of eye pitch movement, rang. of acc. of eye pitch movement, std. of acc. of eye pitch movement, var of acc. of eye pitch movement, std. of head pitch, var. of head pitch |  L >M >H
std. of speed of EAR changes, std of acc. of EAR changes, max. of EAR, range of EAR, std. of EAR, mean of eye yaw |  L >H >M
rang. of head roll, std. of head roll, rang. of head yaw, std. of head yaw, var. of head yaw |  M >L >H
var. of head roll |  M >H >L
kurt. of speed of head pitch movement, skew of speed of head pitch movement, skew of head roll movement, kurt. of head roll movement |  H >L >M
min. of eye pitch movement, min. of acc. of eye pitch movement, kurt. of speed of head distance movement, skew of speed of head distance movement |  H >M >L
Anxiety-related involuntary behaviors |  skew of speed of eye yaw movement |  L >M >H
kurt. of head distance |  M >L >H
skew of EAR changes |  M >H >L
peaks of head roll, valys. of head roll |  H >L >M
Engagment in conversation (head movement to signify emotion, eye contact, head distance) |  kurt. of eye yaw, max. of eye pitch, mean of eye pitch, rang. of eye pitch, std. of eye pitch, var. of eye pitch, kurt. of speed of eye yaw movement, kurt. of acc. of eye yaw movement |  L >M >H
mean of head pitch |  L >H >M
rang. of eye yaw, std. of eye yaw, var. of eye yaw, std. of speed of eye yaw movement, var. of speed of eye yaw movement |  M >H >L
min. of head pitch |  H >L >M
peaks of head pitch, valys. of head pitch |  H >M >L
Miscellaneous |  min. of eye yaw, peaks of eye yaw, valys. of eye yaw |  L >H >M
min. of head roll, head_roll-d1_kurt |  H >L >M
var.: variance, kurt.: kurtosis, valys.: number of valleys, acc.: acceleration

ANOVA tests were performed for the 210 functionals collected over two-minute
windows, among which, 123 were significant. To further verify the
significance, we conducted post-hoc t-tests for each behavioral feature
between the three groups. From this process, 69 functionals were found to be
significant in at least one of the three intergroup comparisons (low/high,
med/high, low/med). Moreover, we used the t-values to determine the direction
of the correlations between the groups.

T-tests show that higher risk participants exhibit lower levels of ocular and
head activity, as well as heightened anxiety and depression-related symptoms,
including “fight-or-flight” movements and social withdrawal. Table 2
enumerates all 69 features, from which the most prominent ones are discussed
in this section. Higher risk individuals blink less often and have narrower
eye-openings and less sudden movements of the eyelids. The minimum of the EAR
is zero (when the eye is closed). As low-risk individuals have the highest
maximum, range, and standard deviation in EAR, their ocular activity is more
diverse. They have wider eye openings and more frequent blink. Additionally,
given that the speed and acceleration of the EAR is near-zero when the eyes
are open, low-risk participants’ high standard deviations in their EAR’s first
and second derivatives also signify they blink more and faster. Assuming all
interviewees have similar physical needs of blinking for lubricating the eye,
the reduction in eyelid activity among high-risk individuals allude to their
less robust nervous and muscular system around the eyes.

Low-risk subjects have higher levels of gaze movement, characterized by faster
and more frequent up-down glancing. Among all three groups, low-risk
individuals have the greatest mean, range, and standard deviation in their eye
pitch movement speeds, highlighting they have the most significant gaze shift.
The same statistics show that high-risk participant’s up-down gaze shifts have
much smaller magnitude. Similarly, low-risk individuals also have more jerky
movements, as evidenced by high max., range, std., and var. in the
acceleration of eye pitch. Analysis of head motions yields similar
findings—high-risk subjects have the least range of motion when moving their
heads in all directions. These trends are inferred from the low range and std.
of head roll and yaw among the high-risk participants, and are in line with
literature: the slower, diminished, and more sporadic movement of higher risk
individuals exhibit signs of fatigue and lethargy that can be attributed to
psycho-motor retardation — one of the most prominent identifiers of depression
[14, 2, 36, 3].

Statistical analysis also reveals that suicidal individuals have higher levels
of anxiety. Notably, the low skew on the speed of high-risk subjects’ eye yaw
indicates that they have more frequent left-right movement in their eyes, as
they spend more time move their eyes at higher speeds. This corroborates with
findings in [27, 35, 19, 22] that high risk subjects scan the room from left
to right and rarely make eye contact with the interviewer. Moreover, despite
having the smallest range in motion, high-risk group tilt, nod, and move their
head more than their lower-risk counterparts, as signified by the low range,
var. and std. of head roll and yaw.

This study observed augmented levels of social withdrawal among higher risk
individuals. Statistics pertaining to the left-right movement of the gaze
indicates that low-risk participants’ gaze is often focused on the
interviewer—they rarely look away, and even when they do so, their gaze would
quickly come back to the interviewer. Additionally, low-risk subjects’ gaze
and head pose direction are the most elevated. These findings corroborate with
the correlation between depression, anxiety, and eye contact avoidance found
by previous studies [27, 19, 22, 2, 36, 3].

Medium-risk participants’ features frequently sit between low and high-risk
groups in most behaviors. The level of their eyelid and gaze activity is
higher than high-risk individuals, but lower than low-risk individuals; their
vertical gaze direction is higher than high-risk, but lower than low-risk
subjects. These trends affirm that medium-risk participants experience
moderate drowsiness in movement and depression comparing to high and low-risk
participants. Other trends pertaining to medium-risk participants, however,
call for further investigation.They have the least front-back head movement,
as they seldom lean forward or back to show engagement in the conversation.
They blink the slowest, although the difference is not significant between
high and medium-risk groups. They tilt their head the least often compare to
low and high-risk group, which can be interpreted as either high level of
focus or severe psycho-motor retardation. When scanning the room, medium-risk
subjects’ gaze diverges the furthest away from the interviewer, although their
left-right gaze movement is sparse. This may be interpreted as fixation at a
spot away from the interviewer, either due to curious distraction or eye
contact avoidance. While these trends are insightful, additional information
is required for further interpretation.

Table 3: Interpretations of the 21 Behavioral Features Selected that Passed
the 2-Factor ANOVA

Behavioral Theme |  Functional Features | Direction
---|---|---
Engagement in conversation (head movement to signify emotion, eye contact, head distance) |  range of head pitch | L >M >H
skew of head distance | M >L >H
std. of head distance | M >H >L
skew of EAR | H >L >M
Room scanning (anxiety symptom) |  kurt. of acc. of eye pitch | L >H >M
min. of speed of head yaw | H >L >M
kurt. of head roll |
min. of speed of head yaw |
Diminished Eye and Head movement (depression symptom) |  skew of speed of eye pitch | L >H >M
range of head pitch | L >M >H
var. of speed of head pitch |
var. of acc. of head pitch |
|  var. of acc. of head roll |
|  var. of acc. of eye pitch |
|  min. of speed of EAR | M >H >L
|  skew of EAR | H >L >M
|  kurt. of head yaw |
|  min. of speed of head yaw |
Miscellaneous |  kurt. of speed of EAR | M >L >H
skew of head yaw | H >L >M
skew of head roll |
var.: variance, valys.: number of valleys, acc.: acceleration

Multivariate Analysis of Variance (MANOVA) determines no significant
difference in the extracted features for the three suicide levels. Roy’s
Greatest Root is the only statistic that yields a significant p-value
222Detailed MANOVA results are presented here: https://bit.ly/32h6CRa. This is
expected, as the feature space is especially large. Using the repeated-measure
ANOVA, we found suicide risk to have a significant effect in the number of
sudden movements in participants’ left-right eye gaze. T-tests subsequently
found the low-risk group have the highest number of valys. of eye yaw acc.,
hence the most sudden movements, while high-risk group to have the least. This
finding corroborates the depression-induced psycho-motor retardation
hypothesis.

Two-factor ANOVAs were conducted for each feature in order to further
understand subject and risk-level-based dependence. Specifically, we conducted
type-III ANOVA, which assumes dependence between independent variables – risk
level and identity of the participant – which are not independent. For all
features, we compute the probability of the distribution being identical for
both variables and their interaction, and selected 18 features that are both
significant in level-identity interaction and insignificant for identity
differences (alpha $<0.05$). As such, we deem the 18 features to be free of
subject-dependence. Among the 18 features, one was selected by features
selection, and one by t-tests.

Analysis of the 18 features using the group-wise t-statistic found remarkable
trends. In general, participants at higher suicide risk exhibit more
distinguishable depression symptoms, and those at lower risk, the least.
Specifically, since head pitch’s velocity, var. of acc., and range are high,
we deem the low-risk group to have the most frequent sudden up-down head
movements, the greatest range of head-nodding motion, and move their heads the
quickest when they do so. The low-risk group also have the most sudden head
tilts and fastest sudden up-down movements in their eye gaze, as signified by
the variance in acceleration in head roll and max. acc. in eye pitch.
Participants in the high-risk group display the least of these eye and head
activities. Additionally, high-risk participants rarely turn or tilt their
heads (high kurt. in head yaw and roll), and have low levels of eye openness
(high skew in EAR). Such ocular and head movement reduction can be explained
with depression-induced psycho-motor retardation and social disengagement. In
contrast, the low-risk group often lean forward as they engage in conversation
with the interviewer, as observed by the high std. of their head-distance.
This signal, in combination with low-risk participants’ higher activity level,
signify their higher level of social engagement.

High and medium-risk groups also exhibit anxiety symptoms. Especially, medium-
risk individuals have the quickest and most frequent head turns and tilts,
given the low skew in head yaw and roll. As the interviewees’ gaze is in the
direction of the interview most of the time, the medium-risk group’s least
uniform velocity in their eyes’ up-down movements signify that they frequently
look away, up-down scanning the room.

Some findings of 2-factor ANOVA lack explanation. Medium-risk group have the
highest kurt. of speed of EAR, meaning that the speed in their eyelid’s
movement is the most uniform. This implies they blink the least often. It was
also found that medium-risk group have the lowest skew in head yaw and roll,
and high-risk group have the highest. This implies that medium risk group tilt
their heads to the right more often while the high risk group, more to the
left. This may be due to fixation at a spot away from the interviewer, yet
further investigation is required for a more meaningful interpretation.

####  4.1.2 Feature Selection

Table 4: Interpretations of the Features Selected by Feature Selection
Framework: 12 behavioral features were narrowed down through the framework

Behavioral Theme |  Functional Features |  Direction
---|---|---
Engagement in conversation (head movement to signify emotion, eye contact, head distance) |  std. of head pitch |  L >M >H
avg. of eye gaze pitch |
std. of head yaw |  M >L >H
var. of head yaw |
min. of head distance |  M >H >L
std. of eye gaze yaw |
var. of eye gaze yaw |
Room Scanning (anxiety related) |  var. of the speed of eye gaze yaw shift |  M >H >L
Diminished Eye and Head movement |  std. of the speed of EAR change |  L >H >L
valys of acc. of eye gaze yaw shifts |  L >M >H
var. of acc. of head pitch change |
skewness of acc. of head roll change |  M >H >L
var.: variance, valys.: number of valleys, acc.: acceleration
Table 5: The Best Classification Results using Statistical Summaries Obtained from 2-Minute Windows Features | All features | Selected features |  Random features |  Shuffled labels
---|---|---|---|---
Performance | Best | 2nd | 3rd | Best | 2nd | 3rd | Best | Best
| Model | best | best | Model | best | best | Model | Model
sampling method |  none | none |  over-sample | none |  none |  over-sample |  under-sample |  over-sample
model |  SVM (linear) | MLP |  SVM (linear) | MLP |  SVM (radial) |  SVM (linear) |  SVM (radial) |  MLP
avg accuracy |  0.983 | 0.978 |  0.977 | 0.977 |  0.971 |  0.961 |  0.425 |  0.399
std. accuracy |  0.019 | 0.020 |  0.033 | 0.021 |  0.023 |  0.033 |  0.045 |  0.077
avg MCC |  0.966 | 0.970 |  0.957 | 0.966 |  0.974 |  0.943 |  0.138 |  0.106

A feature selection method developed by [4] was applied on our suicide data
set. Using the same 210 functionals’ statistical summaries as inputs, a total
of 12 features passed the FS framework. To investigate the direction of these
features in each risk group, the state value of the t-test analysis was
obtained as listed in Table 4.1.2. The results from FS align with those
obtained from statistical analysis. We observed that, in comparison to the low
and medium-risk participants, the high-risk group rarely nod and shake their
heads. Their heads’ range of up-down movements are the smallest; their sudden
left-right movement of the gaze is the least frequent; their eyes’ blinking
velocity is the most uniform — indicating less and slower blinks. These
observations lead us to conclude that, due to depression-induced psycho-motor
retardation, higher-risk individuals exhibit reduced activity in their heads
and eye regions [14].

Results from feature selection also shed light on the participants’ emotional
expression. The low mean in the eye-pitch and the high mean in the eye yaw
indicate that medium and high-risk subjects spend significantly more time
looking down and avoiding eye contact with the interviewer — strong signals of
social withdrawal [36, 14]. The low-risk group spent more time tilting their
heads and leaning forwards as they engage with the interviewer. The eye’s
left-right moving speed is the most uniform among low-risk subjects, which
parallels previous works — anxiety patients often have a hard time focusing on
the interviewer and scan the room. Given the prevalence of depression and
anxiety among suicidal patients, these results are in line with literature,
that depression leads to psycho-motor retardation and anxiety gives rise to
more frequent distraction from the conversation.

###  4.2 Classification Results

Table 5 shows the classification results using the best-performing
classifiers, measured by balanced accuracy333Full classification results are
presented here: https://bit.ly/3tnqcY1. From our top 3 best-performing
classifiers, we can see that all models performed similarly well. Still, the
best model accuracy-wise for all features was an SVM with linear kernels,
while the best for selected features was MLP. Even though the best balanced
accuracies were 98.3% and 97.7% with full and selected features, respectively,
the best MCC scores were obtained from our 2nd best models (0.970 and 0.974
for full and selected features respectively). Given that MCC accounts for the
imbalanced samples from each class, we can argue that our 2nd best models
performed better than the best (even though with small margin) since it has
less confusion between the classes. For both all features and selected
features, training using the original dataset worked far better than
undersampling (4.5% absolute difference across all trials) and slightly better
than oversampling (0.6% absolute difference).

Using all statistical features for training, our best model averaged 98%
accuracy with a small standard deviation. The mean MCC was 0.967 — almost
perfect test performance across all three classes. We achieved similar numbers
using the selected features with slightly greater standard deviation. Since we
were able to immensely reduce the feature space with the FS framework without
impacting classification accuracy, we conclude that the 12 selected behavioral
features can indicate suicide risk level as well as represent the model for a
behavioral interpretation.

For comparison, running the same tests using randomly switched labels resulted
in far worse accuracy (ranging from 30-40%), as did the random feature vectors
(also ranging from around 30-40%). Thus, we are confident that our results are
indicative of the discriminative power of our features, not of any algorithm
bias. In addition, achieving high accuracy with both MLP and SVM implies that
our features are easily separable between classes, so they would hold good
discriminative power regardless of the modeling method. Overall, our results
indicate that both the entire statistical summary and the selected features
are excellent at distinguishing between different levels of risk with high
accuracy.

###  4.3 Limitations

We acknowledge the small sample size of the dataset, and our results
necessitates further investigation upon features that cannot be explained.
Particularly, high-risk participants have the greatest head tilt to their
left, but the speed of their head tilt movement is the most uniform. Moreover,
while our results support the psycho-motor retardation theory, we found the
high-risk group to tilt and nod their head frequently but with the smallest
range of motion. We hypothesized that such contradicting results stem from the
association between suicide risk and bipolar disorder [21, 30]. Since bipolar
disorder patients experience both high and low energy episodes — which could
sufficiently interpret our findings — we hope to also explore the biomarkers
of bipolar disorder.Additionally, studies on eye gaze of people suffering from
mental disorders shows nonspecific gaze [32], anxious gaze aversion [29], or
simply gazing at the door during uncomfortable situations [17, 12]. As the
interviews were conducted in different locations, adding the room context to
the feature set will provide insight into what the subject is gazing at.

##  5 Conclusion

Suicide poses significant risk, to those at risk, their families, and our
society. Suicide is also often connected with depression, anxiety, and self-
harm, but none of these factors alone is deterministic enough for an objective
and effective suicide risk assessment. Additionally, due to privacy
constraints, biomarkers of suicide have received very limited attention in
academia and industry, as datasets that contain sensitive information are
rarely available.

In this work, we aim to extend and fill the gap by systematically and
objectively exploring nonverbal behaviors. We employ statistical analysis,
feature selection, and machine learning classification to analyze signals from
the eyes and head. Applying these three methods to our dataset, our conclusion
corroborates previous psychological studies. Particularly, high-risk suicidal
subjects exhibit lower activity levels in their eyes and head movements and
are subject to a higher degree of psycho-motor retardation. Their eye gaze
shows symptoms of anxiety and depression, including constant left-right
scanning, difficulty of maintaining eye contact, and angling down their gaze
and head pose.

Our classification results showed that these features are representative of
suicidal intent. We achieved classification accuracy ranging from 96-98% for
three risk levels across different modeling methods. We also highlight that
since the results from statistical analysis and feature selection lead to the
same conclusion, both are viable means of objective suicide risk assessment.
Yet, conclusions from this work must be tested and verified against a larger
and more diverse dataset. Future work will also extend to study body movements
and speech prosody as potential biomarkers of suicide intents.

## Acknowledgement

We thank and acknowledge the effort made by NHK, Nippon Hoso Kyokai (Japan
Broadcasting Corporation) for conducting, recording and providing the
interview dataset used in this work.

## References

  * [1] Abaei, N., Osman, H.A.: A hybrid model for bipolar disorder classification from visual information. In: ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 4107–4111 (2020). https://doi.org/10.1109/ICASSP40776.2020.9054648
  * [2] Alghowinem, S., Goecke, R., Wagner, M., Parker, G., Breakspear, M.: Eye movement analysis for depression detection. In: 2013 IEEE International Conference on Image Processing. pp. 4220–4224 (2013). https://doi.org/10.1109/ICIP.2013.6738869
  * [3] Alghowinem, S., Goecke, R., Wagner, M., Parkerx, G., Breakspear, M.: Head pose and movement analysis as an indicator of depression. In: 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction. pp. 283–288. IEEE (2013)
  * [4] Alghowinem, S.M., Gedeon, T., Goecke, R., Cohn, J., Parker, G.: Interpretation of depression detection models via feature selection methods. IEEE Transactions on Affective Computing 1(1), 1–1 (2020). https://doi.org/10.1109/TAFFC.2020.3035535
  * [5] Ambady, N., Rosenthal, R.: Thin slices of expressive behavior as predictors of interpersonal consequences: A meta-analysis. Psychological bulletin 111(2), 256 (1992)
  * [6] Bernert, R.A., Hilberg, A.M., Melia, R., Kim, J.P., Shah, N.H., Abnousi, F.: Artificial intelligence and suicide prevention: A systematic review of machine learning investigations. International Journal of Environmental Research and Public Health 17(16), 5929 (2020). https://doi.org/10.3390/ijerph17165929
  * [7] Cech, J., Soukupova, T.: Real-time eye blink detection using facial landmarks. Cent. Mach. Perception, Dep. Cybern. Fac. Electr. Eng. Czech Tech. Univ. Prague pp. 1–8 (2016)
  * [8] Chicco, D., Jurman, G.: The advantages of the matthews correlation coefficient (mcc) over f1 score and accuracy in binary classification evaluation. BMC Genomics 21(1) (2020). https://doi.org/10.1186/s12864-019-6413-7
  * [9] Chowdhury, G.: Trec: Experiment and evaluation in information retrieval. Online information review (2007)
  * [10] Cohn, J.F., Kruez, T.S., Matthews, I., Yang, Y., Nguyen, M.H., Padilla, M.T., Zhou, F., De la Torre, F.: Detecting depression from facial actions and vocal prosody. In: 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops. pp. 1–7 (2009). https://doi.org/10.1109/ACII.2009.5349358
  * [11] Eigbe, N., Baltrusaitis, T., Morency, L., Pestian, J.: Toward visual behavior markers of suicidal ideation. In: 2018 13th IEEE International Conference on Automatic Face Gesture Recognition (FG 2018). pp. 530–534 (2018). https://doi.org/10.1109/FG.2018.00085
  * [12] Eisenberg, N., Spinrad, T.L.: Emotion-related regulation: Sharpening the definition. Child Development 75(2), 334–339 (2004)
  * [13] Fonseka, T.M., Bhat, V., Kennedy, S.H.: The utility of artificial intelligence in suicide risk prediction and the management of suicidal behaviors. Australian & New Zealand Journal of Psychiatry 53(10), 954–964 (2019)
  * [14] Fossi, L., Faravelli, C., Paoli, M.: The ethological approach to the assessment of depressive disorders. Journal of Nervous and Mental Disease (1984)
  * [15] Gómez, J.M.: Language technologies for suicide prevention in social media. In: Proceedings of the Workshop on Natural Language Processing in the 5th Information Systems Research Working Days (JISIC). pp. 21–29 (2014)
  * [16] Gorodkin, J.: Comparing two k-category assignments by a k-category correlation coefficient. Computational Biology and Chemistry 28(5-6), 367–374 (2004). https://doi.org/10.1016/j.compbiolchem.2004.09.006
  * [17] Grandin, T.: How people with autism think. In: Learning and cognition in autism, pp. 137–156. Springer (1995)
  * [18] Guan, K., Fox, K.R., Prinstein, M.J.: Nonsuicidal self-injury as a time-invariant predictor of adolescent suicide ideation and attempts in a diverse community sample. Journal of consulting and clinical psychology 80(5), 842 (2012)
  * [19] Horley, K., Williams, L.M., Gonsalvez, C., Gordon, E.: Social phobics do not see eye to eye: A visual scanpath study of emotional expression processing. Journal of Anxiety Disorders 17(1), 33–44 (2003). https://doi.org/https://doi.org/10.1016/S0887-6185(02)00180-9, https://www.sciencedirect.com/science/article/pii/S0887618502001809
  * [20] Huang, Y.P., Goh, T., Liew, C.L.: Hunting suicide notes in web 2.0-preliminary findings. In: Ninth IEEE International Symposium on Multimedia Workshops (ISMW 2007). pp. 517–521. IEEE (2007)
  * [21] Jamison, K.R., et al.: Suicide and bipolar disorder. Journal of Clinical Psychiatry 61, 47–51 (2000)
  * [22] Jun, Y.Y., Mareschal, I., Clifford, C.W., Dadds, M.R.: Cone of direct gaze as a marker of social anxiety in males. Psychiatry Research 210(1), 193–198 (2013). https://doi.org/https://doi.org/10.1016/j.psychres.2013.05.020, https://www.sciencedirect.com/science/article/pii/S0165178113002795
  * [23] Laksana, E., Baltrušaitis, T., Morency, L., Pestian, J.P.: Investigating facial behavior indicators of suicidal ideation. In: 2017 12th IEEE International Conference on Automatic Face Gesture Recognition (FG 2017). pp. 770–777 (2017). https://doi.org/10.1109/FG.2017.96
  * [24] Lecrubier, Y., Sheehan, D.V., Weiller, E., Amorim, P., Bonora, I., Sheehan, K.H., Janavs, J., Dunbar, G.C.: The mini international neuropsychiatric interview (mini). a short diagnostic structured interview: Reliability and validity according to the cidi. European psychiatry 12(5), 224–231 (1997)
  * [25] Low, D.M., Bentley, K.H., Ghosh, S.S.: Automated assessment of psychiatric disorders using speech: A systematic review. Laryngoscope Investigative Otolaryngology 5(1), 96–116 (2020). https://doi.org/10.1002/lio2.354
  * [26] Matsubayashi, T., Sekijima, K., Ueda, M.: Government spending, recession, and suicide: Evidence from japan. BMC Public Health 20(1), 243 (Feb 2020). https://doi.org/10.1186/s12889-020-8264-1, https://doi.org/10.1186/s12889-020-8264-1
  * [27] Moukheiber, A., Rautureau, G., Perez-Diaz, F., Soussignan, R., Dubal, S., Jouvent, R., Pelissolo, A.: Gaze avoidance in social phobia: Objective measure and correlates. Behaviour Research and Therapy 48(2), 147–151 (2010). https://doi.org/https://doi.org/10.1016/j.brat.2009.09.012, https://www.sciencedirect.com/science/article/pii/S0005796709002265
  * [28] NIH: Suicide prevention. https://www.nimh.nih.gov/health/topics/suicide-prevention/index.shtml (2021), accessed: 2021-04-03
  * [29] Perez, J.E., Riggio, R.E.: Nonverbal social skills and psychopathology. In: Philippot, P., Feldman, R.S., Coats, E.J. (eds.) Series in Affective Science. Nonverbal Behavior in Clinical Settings, p. 17–44. Oxford University Press (2003)
  * [30] Plans, L., Barrot, C., Nieto, E., Rios, J., Schulze, T., Papiol, S., Mitjans, M., Vieta, E., Benabarre, A.: Association between completed suicide and bipolar disorder: a systematic review of the literature. Journal of affective disorders 242, 111–122 (2019)
  * [31] Runeson, B., Odeberg, J., Pettersson, A., Edbom, T., Jildevik Adamsson, I., Waern, M.: Instruments for the assessment of suicide risk: a systematic review evaluating the certainty of the evidence. PLoS one 12(7) (2017)
  * [32] Schelde, J.T.M.: Major depression: Behavioral markers of depression and recovery. The Journal of nervous and mental disease 186(3), 133–140 (1998)
  * [33] Shah, A.P., Vaibhav, V., Sharma, V., Al Ismail, M., Girard, J., Morency, L.P.: Multimodal behavioral markers exploring suicidal intent in social media videos. In: 2019 International Conference on Multimodal Interaction. pp. 409–413 (2019)
  * [34] Sugano, Y., Matsushita, Y., Sato, Y.: Learning-by-synthesis for appearance-based 3d gaze estimation. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. pp. 1821–1828 (06 2014). https://doi.org/10.1109/CVPR.2014.235
  * [35] Waern, M., Kaiser, N., Renberg, E.S.: Psychiatrists’ experiences of suicide assessment. BMC Psychiatry 16(1) (2016)
  * [36] Waxer, P.H.: Nonverbal cues for anxiety: An examination of emotional leakage. Journal of abnormal psychology 86(3), 306 (1977)
  * [37] WHO: Suicide, https://www.who.int/news-room/fact-sheets/detail/suicide
  * [38] Zhang, S., Zhu, X., Lei, Z., Shi, H., Wang, X., Li, S.Z.: S3fd: Single shot scale-invariant face detector. In: Proceedings of the IEEE international conference on computer vision. pp. 192–201 (2017)
  * [39] Zhang, X., Sugano, Y., Fritz, M., Bulling, A.: MPIIGaze: Real-world dataset and deep appearance-based gaze estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence 41(1), 162–175 (2019). https://doi.org/10.1109/TPAMI.2017.2778103
  * [40] Zhang, X., Sugano, Y., Fritz, M., Bulling, A.: Appearance-based gaze estimation in the wild. In: Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 4511–4520 (June 2015)