# Explainable Depression Detection via Head Motion Patterns

Monika Gahalawat  University of Canberra<EMAIL_ADDRESS>,
Raul Fernandez Rojas  University of Canberra
<EMAIL_ADDRESS>,  Tanaya Guha  University of Glasgow
<EMAIL_ADDRESS>,  Ramanathan Subramanian  University of Canberra
<EMAIL_ADDRESS>and  Roland Goecke  University of Canberra
<EMAIL_ADDRESS>
###### Abstract.

While depression has been studied via multimodal non-verbal behavioural cues,
head motion behaviour has not received much attention as a biomarker. This
study demonstrates the utility of fundamental head-motion units, termed
_kinemes_ , for depression detection by adopting two distinct approaches, and
employing distinctive features: (a) discovering kinemes from head motion data
corresponding to both depressed patients and healthy controls, and (b)
learning kineme patterns only from healthy controls, and computing statistics
derived from reconstruction errors for both the patient and control classes.
Employing machine learning methods, we evaluate depression classification
performance on the _BlackDog_ and _AVEC2013_ datasets. Our findings indicate
that: (1) head motion patterns are effective biomarkers for detecting
depressive symptoms, and (2) explanatory kineme patterns consistent with prior
findings can be observed for the two classes. Overall, we achieve peak F1
scores of 0.79 and 0.82, respectively, over BlackDog and AVEC2013 for binary
classification over episodic _thin-slices_ , and a peak F1 of 0.72 over videos
for AVEC2013.

Kinemes, Head-motion, Depression detection, Explainability

##  1\. Introduction

Clinical depression, a prevalent mental health condition, is considered as one
of the leading contributors to the global health-related burden (Greenberg et
al., 2015; Lépine and Briley, 2011), affecting millions of people worldwide
(Vos et al., 2016; of Health Metrics and Evaluation, 2021). As a mood
disorder, it is characterised by a prolonged (¿ two weeks) feeling of sadness,
worthlessness and hopelessness, a reduced interest and a loss of pleasure in
normal daily life activities, sleep disturbances, tiredness and lack of
energy. Depression can lead to suicide in extreme cases (Goldney et al., 2000)
and is often linked to comorbidities such as anxiety disorders, substance
abuse disorders, hypertensive diseases, metabolic diseases, and diabetes
(Steffen et al., 2020; Campayo et al., 2011). Although effective treatment
options are available, diagnosing depression through self-report and clinical
observations presents significant challenges due to the inherent subjectivity
and biases involved.

Over the last decade, researchers from affective computing and psychology have
focused on investigating objective measures that can aid clinicians in the
initial diagnosis and monitoring of treatment progress of clinical depression
(Cohn et al., 2018; Pampouchidou et al., 2019). A key catalyst to this
progress is the availability of relevant datasets, such as AVEC2013 and
subsequent challenges (Valstar et al., 2013). In recent years, research on
depression detection employing affective computing approaches has increasingly
focused on leveraging non-verbal behavioural cues such as facial expressions
(Bourke et al., 2010; de Melo et al., 2019), body gestures (Joshi et al.,
2013a), eye gaze (Alghowinem et al., 2016), head movements (Alghowinem et al.,
2013) and verbal features (Cummins et al., 2011; Huang et al., 2019) extracted
from multimedia data to develop distinctive features to classify individuals
as depressed or healthy controls, or to estimate the severity of depression on
a continuous scale.

In this study, we examine the utility of inherently interpretable head motion
units, referred to as _kinemes_ (Madan et al., 2021), for assessing
depression. Initially, we utilise data from both healthy controls and
depressed patients to discover a basis set of kinemes via the (_pitch_ , _yaw_
, and _roll_) head pose angular data obtained from short overlapping time-
segments (termed two-class kineme discovery or 2CKD). Further, we employ these
kinemes to generate features based on the frequency of occurrence of
distinctive, class-characteristic kinemes. Subsequently, we discover kineme
patterns solely from head pose data corresponding to healthy controls (Healthy
control kineme discovery or HCKD), and use them to represent both healthy and
depressed class segments. A set of statistical features are then computed from
the reconstruction errors between the raw and learned head-motion segments
corresponding to both the depressed and control classes (see Figure 1). Using
machine learning methodologies, we evaluate the performance of the features
derived from the two approaches. Our results show that head motion patterns
are effective behavioural cues for detecting depression. Additionally,
explanatory class-specific kinemes patterns can be observed, in alignment with
prior research.

Figure 1. Overview: We learn kinemes for the control class, and the
reconstruction errors between the raw and reconstructed head-motion segments,
obtained via kineme clustering, are computed for both the control and
depressed classes. Statistical descriptors over the yaw, pitch and roll
dimensions (a total of $8\times 3$ features) are utilized for depression
detection via machine learning techniques.

This paper makes the following research contributions:

  * •

A study of head movements as a biomarker for clinical depression, which so far
has been understudied.

  * •

Proposing the kineme representation of motion patterns as an effective and
explanatory means for depression analysis.

  * •

A detailed investigation of various classifiers for 2-class and 4-class
categorisation on the AVEC2013 and BlackDog datasets. We obtain peak F1-scores
of 0.79 and 0.82, respectively, on thin-slice chunks for binary classification
on the BlackDog and AVEC2013 datasets, which compare favorably to prior
approaches. Also, a video-level F1-score of 0.72 is achieved for 4-class
categorisation on AVEC2013.

The remainder of this paper is organised as follows. Section 2 provides an
overview of related work. Section 3 describes the kineme formulation, followed
by Section 4 that details the explainable kineme features used as a
representation of motion patterns. The methodology is presented in Section 5,
while Section 6 provides details of the datasets, experimental settings, and
classifiers used in this study. The experimental results are shown and
discussed in Section 7. Finally, the conclusions are drawn in Section 8.

##  2\. Related Work

In this section, we briefly review the literature focusing on (a) depression
detection as a classification problem, and (b) depression detection using head
motion patterns.

###  2.1. Depression Analysis as a Classification Task

Traditionally, depression detection has been approached as a supervised binary
classification task, with many studies relying on discriminative classifiers
to distinguish between _healthy controls_ and _patients_ (Alghowinem et al.,
2015; Cohn et al., 2018; Alghowinem et al., 2013). A typical recognition
accuracy of up to 80% demonstrates the promise of behavioural cues such as
eye-blink and closed-eye duration rate, statistical features computed over the
yaw, pitch and roll head-pose features, etc. to differentiate the two classes.
However, challenges involved in depression detection such as limited
clinically validated, curated data and a skewed data distributions have been
acknowledged in the literature (Alghowinem et al., 2013; Nasir et al., 2016).

Recent efforts have sought to learn patterns indicative of only the target
class and reformulate depression detection as a one-class classification
problem to mitigate the issues with imbalanced datasets (Opoku Asare et al.,
2019; Aguilera et al., 2021). Studies have attempted to learn features
associated with control participants and treat inputs that deviate from these
patterns as anomalous (Gerych et al., 2019; Mourão-Miranda et al., 2011).
Gerych _et. al._ (Gerych et al., 2019) formulate the task as anomaly detection
by leveraging autoencoders to learn features of the non-depressed class and
treating depressed user data as outliers. Similarly, Mourão-Miranda _et. al._
(Mourão-Miranda et al., 2011) employ a one-class SVM to classify patients as
outliers compared to healthy participants based on the fMRI responses to sad
facial expressions. Conversely, a few studies explore one-class classification
by learning features characterising the depressed class, and treating non-
depressed subjects as outliers (Aguilera et al., 2021; Opoku Asare et al.,
2019).

###  2.2. Depression Detection via Head Motion Cues

Many studies have focused on non-verbal behavioural cues, such as body
gestures (Joshi et al., 2013b, a), facial expressions (Bourke et al., 2010; He
et al., 2022; de Melo et al., 2019), their combination (Parekh et al., 2018)
and speech features (Cummins et al., 2011; Rejaibi et al., 2022; Huang et al.,
2019) as biomarkers for depression diagnosis and rehabilitation utilising
computational tools (Ringeval et al., 2019). Head motion patterns have
nevertheless received little attention. Psychological research on depression
assessment has identified head motion as a significant non-verbal cue for
depression with more pronounced behavioural changes in hand and head regions
as compared to other body parts for depressed patients (Pedersen et al.,
1988). Waxer _et. al._ (Waxer, 1974) found that depressed subjects are more
likely to keep their heads in a downward position and exhibit significantly
reduced head nodding compared to healthy subjects (Fossi et al., 1984).
Another study focusing on social interactions identified the reduced
involvement of depressed patients in conversations, where their behaviour was
characterised by lesser encouragement (head nodding and backchanneling while
listening) and fewer head movements (Hale III et al., 1997).

From a computational standpoint, only a few studies have employed head pose
and movement patterns for automatic depression detection. Alghowinem  _et al._
(Alghowinem et al., 2013) analysed head movements by modelling statistical
features extracted from the 2D Active Appearance Model (AAM) projection of a
3D face and demonstrated the efficacy of head pose as behavioural cue. Another
study (Joshi et al., 2013b) generated a histogram of head movements normalised
over time to highlight the diminished movements of depressed patients due to
psychomotor retardation, characterised by a more frequent occurrence of static
head positions than in healthy controls. Several studies (Song et al., 2020;
Dibeklioğlu et al., 2017; Cohn et al., 2018; Morales et al., 2017) explored
the utilisation of head motion as a complementary cue to other modalities to
enhance detection performance. For instance, several studies (Alghowinem et
al., 2016, 2020) combined head pose with speech behaviour and eye gaze to
develop statistical features for depression analysis. Generalisation across
different cross-cultural datasets was attempted in (Alghowinem et al., 2015)
by using head pose and eye gaze based temporal features. Kacem  _et. al._
(Kacem et al., 2018) encoded head motion dynamics with facial expressions to
classify depression based on severity, while Dibeklioglu _et. al._
(Dibeklioğlu et al., 2015) included vocal prosody in combination with head and
facial movements for depression detection.

###  2.3. Novelty of the Proposed Approach

From the literature review, it can be seen that while a number of studies have
employed head movements as a complementary cue in multimodal approaches, only
few studies have deeply explored head motion as a rich source of information.
Further, the explainability of behavioural features, especially head motion
features, for depression detection has not yet been explored in the
literature. This study (a) is the first to propose the use of kinemes as
depression biomarkers, (b) explores multimodal cues derived from head motion
behaviour as potential biomarkers for depression; specifically, we show that
kinemes learned for the depressed and control classes, or only the control
class enable accurate depression detection, and (c) the learned kinemes also
explain depressed behaviours consistent with prior observations.

##  3\. Kineme Formulation

This section describes our approach to discovering a set of elementary head
motion units termed _kinemes_ from 3D head pose angles. These head pose angles
are expressed as a time-series of short overlapping segments, which enables
shift invariance. The segments are then projected onto a lower-dimensional
space and clustered using a Gaussian Mixture Model (Samanta and Guha, 2017).

We extracted 3D head pose angles using the OpenFace tool (Baltrušaitis et al.,
2016) in terms of 3D Euler rotation angles, _pitch_ ($\theta_{p}$), _yaw_
($\theta_{y}$) and _roll_ ($\theta_{r}$). The head movement over a duration
$T$ is denoted as a time-series:
$\boldsymbol{\theta}=\\{\theta_{p}^{1:T},\theta_{y}^{1:T},\theta_{r}^{1:T}\\}$.
We ensure that the rotation angles remain non-negative by defining the range
in [0∘, 360∘].

For each video, the multivariate time-series $\boldsymbol{\theta}$ is divided
into short overlapping segments of length $\ell$ with overlap $\ell/2$, where
the $i^{th}$ segment is represented as a vector
$\mathbf{h}^{(i)}=[\theta_{p}^{i:i+\ell}\,\theta_{y}^{i:i+\ell}\,\theta_{r}^{i:i+\ell}]$.
Considering the total number of segments in any given video as $s$, the
characterisation matrix $\mathbf{H}_{\boldsymbol{\theta}}$ for this video is
defined as
$\mathbf{H}_{\boldsymbol{\theta}}=[\mathbf{h}^{(1)},\mathbf{h}^{(2)},\cdots,\mathbf{h}^{(s)}]$.
Thus, for a training set of $N$ samples, the head motion matrix is created as
$\mathbf{H}=[\mathbf{H}_{\boldsymbol{\theta}_{1}}|\mathbf{H}_{\boldsymbol{\theta}_{2}}|\cdots|\mathbf{H}_{\boldsymbol{\theta}_{N}}]$
with each column of $\mathbf{H}$ representing a single head motion time-series
for a given video sample. We decompose $\mathbf{H}\in\mathbb{R}_{+}^{m\times
n}$ into a basis matrix $\mathbf{B}\in\mathbb{R}_{+}^{m\times q}$ and a
coefficient matrix $\mathbf{C}\in\mathbb{R}_{+}^{q\times n}$ using Non-
negative Matrix Factorization (NMF) such that $m=3\ell$, $n=Ns$

(1) $\underset{\mathbf{B}\geq 0,\mathbf{C}\geq 0}{\text{
min}}\lVert{\mathbf{H}-\mathbf{B}\mathbf{C}}\rVert_{F}^{2}$

where $q\leq min(m,n)$ and $\lVert\textbf{ . }\rVert_{F}$ denotes the
Frobenius norm. Rather than clustering the raw head motion segments, we employ
a more interpretable and stable approach by clustering the coefficient vectors
in the transformed space. To this end, we learn a Gaussian Mixture Model (GMM)
using the columns of the coefficient matrix $\mathbf{C}$ to produce a
${\mathbf{C}^{*}}\in\mathbb{R}_{+}^{q\times k}$ where $k<<Ns$. These vectors
in the learned subspace are transformed back to the original head motion
subspace defined by the Euler angles using
$\mathbf{H}^{*}=\mathbf{B}\mathbf{C}^{*}$. The columns of matrix
$\mathbf{H}^{*}$ represent the set of $K$ kinemes as
$\\{\mathcal{K}_{i}\\}_{i=1}^{K}$.

Now, we can represent any head motion time-series $\theta$ as a sequence of
kinemes discovered from the input video set by associating each segment of
length $\ell$ from $\theta$ with one of the kinemes. For each $i^{th}$ segment
in the time-series, we compute the characterisation vector $\mathbf{h}^{(i)}$
and project it onto the transformed subspace defined by $\mathbf{B}$ to yield
$\mathbf{c}^{(i)}$ such that:

(2) $\hat{\mathbf{c}}=\underset{\mathbf{c}^{(i)}\geq 0}{\text{arg
min}}\lVert{\mathbf{h}^{(i)}-\mathbf{B}\mathbf{c}^{(i)}}\rVert_{F}^{2}$

We then maximise the posterior probability $P({K}|\hat{\mathbf{c}})$ over all
kinemes to map the $i^{th}$ segment with its corresponding kineme $K^{(i)}$.
In the same way, we compute the corresponding kineme label for each segment of
length $\ell$ to obtain a sequence of kinemes: $\\{K^{(1)}\cdots K^{(s)}\\}$,
where $K^{(j)}\in\mathcal{K}$ for all segments of time-series $\theta$.

##  4\. Explainable Kineme Features

Figure 2. Plots of kinemes that occur more frequently for the control (left)
and patient (right) cohorts in the BlackDog dataset.

Figure 3. Plots of kinemes that occur more frequently for the minimally
depressed (left) and patient (right) cohorts in AVEC2013.

We now examine kineme patterns obtained from the depression datasets, namely
BlackDog (Alghowinem et al., 2016) and AVEC2013 (Valstar et al., 2013)
(described in Sec. 6.1). Using the Openface (Baltrušaitis et al., 2016)
toolkit, we extracted yaw, pitch and roll angles per frame, and segmented each
video into 2s and 5s-long chunks with 50% overlap for the AVEC2013 and
BlackDog datasets, respectively. Considering $K=16$ (Samanta and Guha, 2017),
we extracted kinemes from both patient and healthy control segments, following
the procedure outlined in Sec. 3. We further examined the kinemes learned for
each dataset to identify the set of distinctive kinemes for the two classes.
To obtain the most discriminative kinemes, we computed the relative frequency
of occurrence for each kineme for the control and patient data, and selected
the top five kinemes per class based on their relative frequency difference
(see Sec. 5.1).

Selected kinemes corresponding to the maximal difference in their relative
frequency of occurrence for the control and patient classes are visualised in
Figures 2 (BlackDog) and 3 (AVEC2013). Examining the control-specific kinemes
in Figs. 2 and 3, we observe a greater degree of movement for healthy subjects
as compared to a predominantly static head pose conveyed by the depressed
patient-specific kinemes. Head nodding, characterised by pitch oscillations,
and considerable roll angle variations can be noted for at least one control-
class kineme; conversely, patient-specific kinemes exhibit relatively small
changes over all head pose angular dimensions. These findings are reflective
of reduced head movements in the depressed cohort compared to healthy
individuals, which is consistent with observations made in past studies (Hale
III et al., 1997; Alghowinem et al., 2013).

##  5\. Classification Methodology

In this section, we outline the methodology for discovering kinemes from short
overlapping video segments. Initially, we discovered kinemes utilising data
segments from both control and patient classes (two-class kineme discovery or
2CKD approach; see Section 4). Subsequently, we learned kinemes solely from
the healthy control cohort and utilised them to represent the head pose data
of depressed patients (denoted as healthy control kineme discovery or HCKD)
approach.

Given a time series $\boldsymbol{\theta}$, we divide it into short overlapping
segments of uniform length, with a segment duration of $5s$ for the BlackDog
and $2s$ for the AVEC dataset. These segment lengths were empirically chosen
and provided the best results from among segment lengths spanning $2s$ to $7s$
for both datasets. For both approaches, a total of $K=16$ kinemes are learned
from the two datasets as per the procedure outlined in Section 3.

###  5.1. Kineme Discovery from Two-class Data

To examine whether the kinemes discovered from head pose angles of both
classes are effective cues for depression detection, we learn kinemes from
segments corresponding to both patient and control videos. Upon discovering
the kineme values, the _relative frequency_ $\eta_{K_{i}}$ of each kineme
$K_{i}$ is computed over the two classes as:

(3) $\mathbf{\eta}_{K_{i}}=\frac{\emph{f }(K_{i})}{\sum_{i=1}^{16}\emph{f
}(K_{i})}$

where $\emph{f}(K_{i})$ represents the frequency of occurrence of the kineme
$K_{i}$ for a particular class. We then compute the relative frequency
difference for each kineme between the two classes to identify the ten most
differentiating kinemes (four kinemes per class are depicted in Figs. 2, 3).
Next, we generate a feature set by extracting the frequencies of the selected
kinemes over the thin-slice chunks considered for analysis. Thus, we obtain a
10-dimensional feature vector representing kineme frequencies for each chunk.

###  5.2. Kineme Discovery from Control Data

Here, we learn kinemes representing head motion solely from the control
cohort. Subsequently, head pose segments from both the patient and control
classes are represented via the discovered kinemes, and reconstruction errors
computed. Let the raw head pose vector $\mathbf{h}^{(i)}$ for the $i^{th}$
segment in the original subspace be denoted as:

(4)
$\mathbf{h}^{(i)}=[\theta_{p}^{i:i+\ell}\,\theta_{y}^{i:i+\ell}\,\theta_{r}^{i:i+\ell}]$

Let the kineme value associated with this segment be $K^{(i)}$. Based on the
kinemes discovered from the control cohort alone, we calculate the
reconstructed kineme for the $i^{th}$ segment as $\tilde{\mathbf{h}}^{(i)}$.
The reconstructed vector for each kineme is determined by converting the GMM
cluster centre for each kineme from the learned space to the original _pitch_
-_yaw_ -_roll_ space. The reconstructed head pose vector for the segment is:

(5)
$\tilde{\mathbf{h}}^{(i)}=[\tilde{\theta}_{p}^{i:i+\ell}\,\tilde{\theta}_{y}^{i:i+\ell}\,\tilde{\theta}_{r}^{i:i+\ell}]$

To compute the reconstruction error for both depressed patients and healthy
controls, we compute the signed difference between the two vectors for each
segment to account for the difference between raw head pose vector and the GMM
cluster centres. We calculate the difference vector $\mathbf{d}^{(i)}$ for
each $i^{th}$ segment as:

(6)
$\mathbf{d}^{(i)}=\mathbf{h}^{(i)}-\tilde{\mathbf{h}}^{(i)}=[d_{p}^{i:i+\ell}\,d_{y}^{i:i+\ell}\,d_{r}^{i:i+\ell}]$

These signed differences values are added over each angular dimension of pitch
(_p_) , yaw (_y_), and roll (_r_) for the segment.

(7) ${s_{e}}^{(i)}=\sum_{n=1}^{\ell}d_{e}^{i:i+n}$

where each ${s_{e}}^{(i)}$ is calculated for each angular dimension
$e\in\\{p,y,r\\}$ over all segments of both classes. Depending on the thin-
slice chunk duration considered for classification, we compute different
descriptive statistics to generate the feature set. Considering number of
elementary kineme chunks in the considered time-window to be $n_{c}$, we
obtain the following feature vector for each angle $e\in\\{p,y,r\\}$:

(8)
$\mathbf{as_{e}}=[\lvert{s_{e}}^{(1)}\rvert,\lvert{s_{e}}^{(2)}\rvert,\cdots,\lvert{s_{e}}^{(n_{c})}\rvert]$

where $\lvert\cdot\rvert$ represents the absolute value. We then calculate
eight statistical features from the above vectors, namely, _minimum_ ,
_maximum_ , _range_ , _mean_ , _median_ , _standard deviation_ , _skewness_ ,
and _kurtosis_ (total of $8\times 3$ features over the yaw, pitch, roll
dimensions).

##  6\. Experiments

We perform binary classification on the BlackDog and AVEC2013 datasets, plus
4-class classification on AVEC2013. This section details our datasets,
experimental settings and learning algorithms.

###  6.1. Datasets

We examine two datasets in this study: clinically validated data collected at
the Black Dog Institute – a clinical research facility focusing on the
diagnosis and treatment of mood disorders such as anxiety and depression
(referred to as _BlackDog_ dataset) – and the _Audio/Visual Emotion Challenge_
(AVEC2013) depression dataset.

BlackDog Dataset (Alghowinem et al., 2016): This dataset comprises responses
from healthy controls and depression patients selected as per the criteria
outlined in the Diagnostic and Statistic Manual of Mental Disorders (DSM-IV).
Healthy controls with no history of mental illness and patients diagnosed with
severe depression were carefully selected (Alghowinem et al., 2016). For our
analysis, we focus on the structured interview responses in (Alghowinem et
al., 2016), where participants answered open-ended questions about life
events, designed to elicit spontaneous self-directed responses, asked by a
clinician. In this study, we analyse video data from 60 subjects (30 depressed
patients and 30 healthy controls), with interview durations ranging from
$183-1200s$.

AVEC2013 Dataset (Valstar et al., 2013): Introduced for a challenge in 2013,
this dataset is a subset of the audio-visual depressive language corpus (AViD-
corpus) comprising 340 video recordings of participants performing different
PowerPoint guided tasks detailed in (Valstar et al., 2013). The videos are
divided into three nearly equal partitions (training, development, and test)
with videos ranging from $20-50min$. Each video frame depicts only one
subject, although some participants feature in multiple video clips. The
participants completed a multiple-choice inventory based on the Beck
Depression Index (BDI) (Beck et al., 1996) with scores ranging from 0 to 63
denoting the severity of depression. For binary classification, we dichotomise
the recordings into the non-depressed and depressed cohorts as per the BDI
scores. Subjects with a BDI score $\leq 13$ are categorised as _non-depressed_
, while the others are considered as _depressed_.

AVEC2013 Multi-Class Classification: For fine-grained depression detection
over the AVEC2013 dataset, we categorise the dataset based on the BDI score
into four classes as detailed below:

  * •

Nil or minimal depression: BDI score 0 - 13

  * •

Mild depression: BDI score 14 - 19

  * •

Moderate depression: BDI score 20 - 28

  * •

Severe depression: BDI score 29 - 63

###  6.2. Experimental Settings

Implementation Details: For binary classification, we evaluate performance for
the smaller _BlackDog_ dataset via 5-repetitions of 10-fold cross-validation
(10FCV). For the AVEC2013, the pre-partitioned train, validation and test sets
are employed. We utilise the validation sets for fine-tuning classifier
hyperparameters.

Chunk vs Video-level Classification: The videos from both datasets are
segmented into smaller chunks of $15s-135$s length, to examine the influence
of _thin-slice_ chunk duration on the classifier performance. We repeated the
video label for all chunks and metrics are computed over all chunks for chunk-
level analysis. Additionally, video-level classification results are obtained
by computing the majority label over all video chunks in the test set.

Performance Measures: For the BlackDog dataset, results are shown as
$\mu\pm\sigma$ values over 50 runs ($5\times$ 10FCV repetitions). For
AVEC2013, performance on the test set is reported. For both, we evaluate
performance via the accuracy (Acc), weighted F1 (F1), precision (Pr), and
recall (Re) metrics. The weighted F1-score denotes the mean F1-score over the
two classes, weighted by class size.

###  6.3. Classification Methods

Given that our proposed features do not model spatial or temporal
correlations, we employ different machine learning models for detecting
depression as described below:

  * •

Logistic Regression (LR), a probabilistic classifier that employs a sigmoid
function to map input observations to binary labels. We utilise extensive
grid-search to fine-tune parameters such as penalty $\in\\{l1,l2,None\\}$ and
regulariser $\lambda\in\\{1e^{-6},\cdots,1e^{3}\\}$.

  * •

Random Forest (RF), where multiple decision trees are generated from training
data whose predictions are aggregated for labelling. Fine-tuned parameters
include the number of estimators $N\in[2,\cdots,8]$, maximum depth
$\in[3,\cdots,7]$, and maximum features in split $\in[3,\cdots,7]$.

  * •

Support Vector Classifier (SVC), a discriminative classifier that works by
transforming training data to a high-dimensional space where the two classes
can be linearly separated via a hyperplane. For SVC, we examine different
kernels $\in\\{rbf,poly,sigmoid\\}$ and fine-tune regularisation parameter
$C\in\\{0.1,1,10,100\\}$ and kernel coefficient
$\gamma\in\\{0.0001,\cdots,1,scale,auto\\}$.

  * •

Extreme Gradient Boosting (XGB), a model built upon a gradient boosting
framework, and focused on improving a series of weak learners by employing the
gradient descent algorithm in a sequential manner. The fine-tuned
hyperparameters include the number of estimators $\\{50,100,150\\}$, maximum
depth $\in[3,\cdots,7]$ of the tree and learning rate
$\in[0.0005,\cdots,0.1]$.

  * •

Multi Layer Perceptron (MLP), where we employed a feed-forward neural network
with two hidden dense layers comprising 12 and 6 neurons, resp., with a
rectified linear unit (ReLU) activation. For training, we employ categorical
cross-entropy as the loss function and fine-tune the following
hyperparameters: learning rate $\in\\{1e^{-4},1e^{-3},1e^{-2}\\}$, and batch
size $\in\\{16,24,32,64\\}$. We utilise the Adam optimiser for updating the
network weights during training.

##  7\. Results and Discussion

Table 1. Chunk and Video-level classification results on the BlackDog dataset with the 2CKD and HCKD approaches. Accuracy (Acc), F1, Precision (Pr) and Recall (Re) are tabulated as ($\mu\pm\sigma$) values.  Condition | Classifier | Chunk-level | Video-level
---|---|---|---
|  | Acc | F1 | Pr | Re | Acc | F1 | Pr | Re
2CKD | LR | 0.60±0.15 | 0.61±0.14 | 0.67±0.22 | 0.65±0.22 | 0.60±0.20 | 0.59±0.21 | 0.55±0.30 | 0.65±0.33
RF | 0.58±0.13 | 0.60±0.12 | 0.67±0.21 | 0.61±0.21 | 0.61±0.19 | 0.62±0.19 | 0.59±0.32 | 0.59±0.32
SVC | 0.60±0.15 | 0.62±0.15 | 0.68±0.25 | 0.62±0.25 | 0.62±0.19 | 0.63±0.19 | 0.61±0.32 | 0.59±0.33
XGB | 0.55±0.17 | 0.54±0.16 | 0.63±0.21 | 0.71±0.22 | 0.53±0.17 | 0.50±0.20 | 0.54±0.23 | 0.79±0.21
MLP | 0.53±0.15 | 0.52±0.17 | 0.60±0.22 | 0.71±0.21 | 0.51±0.20 | 0.47±0.21 | 0.53±0.27 | 0.74±0.32
HCKD | LR | 0.77±0.13 | 0.78±0.12 | 0.85±0.19 | 0.74±0.21 | 0.79±0.16 | 0.78±0.17 | 0.81±0.30 | 0.66±0.31
RF | 0.71±0.13 | 0.73±0.12 | 0.75±0.25 | 0.71±0.20 | 0.76±0.15 | 0.76±0.16 | 0.75±0.26 | 0.82±0.25
SVC | 0.78±0.14 | 0.79±0.13 | 0.87±0.18 | 0.74±0.20 | 0.80±0.18 | 0.80±0.19 | 0.83±0.30 | 0.70±0.31
XGB | 0.72±0.13 | 0.72±0.12 | 0.75±0.18 | 0.81±0.15 | 0.78±0.17 | 0.78±0.17 | 0.74±0.27 | 0.82±0.27
MLP | 0.75±0.13 | 0.76±0.12 | 0.78±0.21 | 0.81±0.21 | 0.76±0.16 | 0.76±0.16 | 0.74±0.29 | 0.77±0.28
Table 2.  Chunk and Video-level classification results on the AVEC2013 dataset with the 2CKD and HCKD approaches. Accuracy (Acc), F1, Precision (Pr) and Recall (Re) are tabulated as ($\mu\pm\sigma$) values. Condition | Classifier | Chunk-level | Video-level
---|---|---|---
|  | Acc | F1 | Pr | Re | Acc | F1 | Pr | Re
2CKD | LR | 0.58 | 0.58 | 0.54 | 0.65 | 0.61 | 0.61 | 0.57 | 0.71
RF | 0.61 | 0.61 | 0.57 | 0.59 | 0.72 | 0.72 | 0.67 | 0.82
SVC | 0.61 | 0.61 | 0.57 | 0.63 | 0.64 | 0.64 | 0.61 | 0.65
XGB | 0.59 | 0.58 | 0.57 | 0.44 | 0.67 | 0.67 | 0.65 | 0.65
MLP | 0.56 | 0.56 | 0.52 | 0.60 | 0.58 | 0.58 | 0.65 | 0.65
HCKD | LR | 0.80 | 0.80 | 0.77 | 0.81 | 0.94 | 0.94 | 0.94 | 0.94
RF | 0.78 | 0.78 | 0.78 | 0.75 | 1.00 | 1.00 | 1.00 | 1.00
SVC | 0.82 | 0.82 | 0.83 | 0.77 | 1.00 | 1.00 | 1.00 | 1.00
XGB | 0.80 | 0.80 | 0.79 | 0.77 | 1.00 | 1.00 | 1.00 | 1.00
MLP | 0.81 | 0.80 | 0.80 | 0.77 | 1.00 | 1.00 | 1.00 | 1.00
Table 3. Comparison with prior works for the two datasets. Dataset | Methods | Features | Evaluation metrics
---|---|---|---
|  |  | Acc | F1 | Pr | Re
BlackDog | Alghowinem et al. (Alghowinem et al., 2013) | Head movement | - | - | - | 0.71
Joshi et al. (Joshi et al., 2013b) | Head movement | 0.72 | - | - | -
Ours (Chunk-level) | Kinemes | 0.75 | 0.76 | 0.78 | 0.81
Ours (Video-level) | Kinemes | 0.80 | 0.80 | 0.83 | 0.70
AVEC2013 | Senoussaoui et al. (AVEC2014) (Senoussaoui et al., 2014) | Video features | 0.82 | - | - | -
Al-gawwam et al. (AVEC2014 - Northwind) (Al-gawwam and Benaissa, 2018) | Eye Blink | 0.85 | - | - | -
Al-gawwam et al. (AVEC2014 - Freeform) (Al-gawwam and Benaissa, 2018) | Eye Blink | 0.92 | - | - | -
Ours (AVEC2013 at chunk-level) | Kinemes | 0.82 | 0.82 | 0.83 | 0.87
| Ours (AVEC2013 at Video-level) | Kinemes | 1.00 | 1.00 | 1.00 | 1.00
Table 4.  Video-level 4-class categorization results on the AVEC dataset obtained with the HCKD approach. Accuracy (Acc), F1, Precision (Pre) and Recall (Re) are tabulated. Condition | Classifier | Video-level
---|---|---
|  | Acc | F1 | Pr | Re
HCKD | LR | 0.71 | 0.72 | 0.73 | 0.71
RF | 0.74 | 0.72 | 0.80 | 0.74
SVC | 0.74 | 0.72 | 0.75 | 0.74
XGB | 0.71 | 0.69 | 0.68 | 0.71
MLP | 0.69 | 0.66 | 0.64 | 0.69

Table 1 shows the classification results obtained for the _BlackDog_ dataset
with the 2CKD and HCKD approaches (Section 5). Table 2 presents the
corresponding results for the _AVEC2013_ dataset. These tables present
classification measures obtained at the _chunk-level_ (best results achieved
over $15-135s$-long chunks for the two datasets are presented), and the
_video-level_ (label derived upon computing the mode over the chunk-level
labels). Based on these results, we make the following observations:

  * •

It can be noted from Tables 1 and 2 that relatively lower accuracies and F1
scores are achieved for both datasets using the 2CKD approach, implying that
while class-characteristic kinemes are explanative as seen from Figs. 2 and 3,
they are nevertheless not discriminative enough to effectively distinguish
between the two classes.

  * •

In comparison, we note far superior performance with the HCKD method over all
classifiers. As a case in point, we obtaine peak chunk-level F1-scores of 0.79
and 0.62, resp., for HCKD and 2CKD on BlackDog, while the corresponding
F1-scores are 0.82 and 0.61, resp., on AVEC. This observation reveals
considerable and distinguishable differences in the reconstruction errors for
the patient and control classes, and convey that patient data are
characterised as _anomalies_ when kinemes are only learned from the control
cohort.

  * •

Examining the HCKD precision and recall measures for both datasets, we note
higher precision than recall at the chunk-level for the BlackDog dataset.
Nevertheless, higher recall is achieved at the video-level with multiple
classifiers. Likewise, higher chunk-level precision is noted for AVEC, even if
ceiling video-level precision and recall are achieved.

  * •

Comparing HCKD chunk and video-level F1-scores for both datasets, similar or
higher video-level F1 values can be seen in Table 1. F1-score differences are
starker in Table 2, where video-level scores are considerably higher than
chunk-level scores. These results suggest that aggregating observations over
multiple thin-slice chunks is beneficial and enables more precise predictions
as shown in (Madan et al., 2021).

  * •

Examining measures achieved with the different classifiers, the support vector
classifier achieves the best chunk-level F1-score on both datasets, with the
LR classifier performing very comparably. All classifiers achieve very similar
performance when video-level labels are compared.

###  7.1. Comparison with the state-of-the-art

Our best results are compared against prior classification-based depression
detection studies in Table 3. For the BlackDog dataset, Alghowinem _et al._
(Alghowinem et al., 2013) analysed statistical functional features extracted
from a 2D Active Appearance Model, whereas Joshi _et al._ (Joshi et al.,
2013b) computed a histogram of head movements by estimating the displacement
of fiducial facial points. Compared to N-average recall of 0.71 reported in
(Alghowinem et al., 2013), and an accuracy of 0.72 noted in (Joshi et al.,
2013b), our kineme-based approach achieves better chunk and video-level
accuracies (0.75 and 0.80, resp.), and superior chunk-level recall (0.81). As
most previous studies on the AVEC2013 dataset focus on continuous prediction,
we compare our model’s performance with the AVEC2014 (Valstar et al., 2014)
results examining visual features.

AVEC2014 used the same subjects as AVEC2013, but with additional, specific
task data (_Northwind_ , _Freeform_) extracted from the AViD videos. For video
analysis, Senoussaoui et al. (Senoussaoui et al., 2014) extracted LGBP-TOP
features from frame blocks to obtain an accuracy of 0.82 using an SVM
classifier. On the other hand, Al-gawwam et al. (Al-gawwam and Benaissa, 2018)
extracted eye-blink features from video data using a facial landmark tracker
to achieve an accuracy of 0.92 for the _Northwind_ task and 0.88 for the
_Freeform_ task. Comparatively, our work achieves an accuracy of 0.82 at the
chunk-level and 1.00 at the video-level. The next section will detail the
performance of a more fine-grained 4-class categorisation on the AVEC2013
dataset.

###  7.2. AVEC2013 Multi-class Classification

Table 4 depicts video-level 4-class classification results achieved on the
AVEC2013 dataset via the HCKD approach. The 4-class categorisation was
performed to further validate the correctness of the HCKD approach, which
produces ceiling video-level F1, Precision and Recall measures on AVEC2013 in
binary classification. Results are reported on the test set, upon fine-tuning
the classifier models on the development set. Reasonably good F1-scores are
achieved even with 4-class classification, with a peak F1 of 0.72 obtained
with the LR, RF and support vector classifiers. Cumulatively, our empirical
results confirm that kinemes encoding atomic head movements are able to
effectively differentiate between (a) the patient and control classes, and (b)
different depression severity bands.

Figure 4. Chunk vs video-level performance comparison for the BlackDog (left)
and AVEC2013 (right) datasets.

Figure 5. Comparing F1 scores with different descriptors for the BlackDog
(left) and AVEC2013 (right) datasets across classifiers.

###  7.3. Ablative Analysis over Thin Slices

Tables 1 and 2 evaluate detection performance over (_thin-slice_) chunks or
short behavioural episodes, and over the entire video, on the BlackDog and
AVEC2013 datasets. We further compared labelling performance at the chunk and
video-levels using chunks spanning $15-135s$. The corresponding results are
presented in Figure 4. For both plots presented in the figure, the dotted
curves denote video-level F1-scores, while solid curves denote chunk-level
scores obtained for different classifiers.

For the BlackDog dataset (Fig. 4 (left)), longer time-slices (of length
$75-105s$) achieve better performance than shorter ($15-60s$ long) ones at
both the chunk and video-levels across all classifiers; these findings are
consistent with the finding that more reliable predictions can be achieved
with longer observations in general (Madan et al., 2021). However, a
performance drop is noted for very long chunk-lengths of $120-135s$ duration.
Decoding results on the AVEC2013 dataset, consistent with Table 3 results, a
clear gap is noted between the chunk and video-level results, with the latter
demonstrating superior performance. Very similar F1-scores are observed across
classifiers for various chunk lengths. No clear trends are discernible from
video-level F1-scores obtained with different chunk-lengths, except that the
performance in general decreases for all classifiers with very long chunks.

###  7.4. Ablative Analysis over Angular Dimensions

To investigate the impact of the head pose angular dimensions on chunk-level
binary depression detection performance, we perform detection utilising
($8\times 1$) statistical features over each of the (pitch, yaw, and roll)
angular dimensions, and concatenate features ($8\times 2$) for the dimensional
pairs to evaluate which angular dimension(s) are more informative.

Figure 5 presents F1-scores obtained with the different classifiers for uni-
dimensional and pairwise-dimensional features. On the BlackDog dataset, a
combination of the pitch and yaw-based descriptors produce the best
performance across all models, while roll-specific descriptors perform worst.
For the AVEC2013 dataset, pitch-based descriptors achieve excellent
performance across models. The F1-scores achieved with these features are very
comparable to the pitch + yaw and pitch + roll combinations. Here again, roll-
specific features achieve the worst performance. Cumulatively, these results
convey that pitch is the most informative head pose dimension, with roll being
the least informative. With respect to combinations, the pitch + yaw
combination in general produces the best results. These results again confirm
that responsiveness in social interactions, as captured by pitch (capturing
actions such as head nodding) and yaw (capturing head shaking), provides a
critical cue for detecting depression, consistent with prior studies (Hale III
et al., 1997; Alghowinem et al., 2013).

##  8\. Conclusion

In this paper, we demonstrate the efficacy of elementary head motion units,
termed _kinemes_ , for depression detection by utilising two approaches: (a)
discovering kinemes from data of both patient and control cohorts, and (b)
learning kineme patterns solely from the control cohort to compute statistical
functional features derived from reconstruction errors for the two classes.
Apart from effective depression detection, we also identify explainable kineme
patterns for the two classes, consistent with prior research.

Our study demonstrates the utility of head motion features for detecting
depression, but our experiments are restricted to classification tasks
involving a discretisation of the depression scores. In the future, we will
investigate (a) the utility of kinemes for continuous prediction (regression)
of depression severity, (b) the cross-dataset generalisability of models
trained via kinemes, and (c) the development of multimodal methodologies
combining kinemes with other behavioural markers, and evaluating their
efficacy.

## References

  * (1)
  * Aguilera et al. (2021) Juan Aguilera, Delia Irazú Hernández Farías, Rosa María Ortega-Mendoza, and Manuel Montes-y Gómez. 2021\.  Depression and anorexia detection in social media as a one-class classification problem.  _Applied Intelligence_ 51 (2021), 6088–6103.
  * Al-gawwam and Benaissa (2018) Sarmad Al-gawwam and Mohammed Benaissa. 2018.  Depression detection from eye blink features. In _2018 IEEE international symposium on signal processing and information technology (ISSPIT)_. IEEE, 388–392.
  * Alghowinem et al. (2015) Sharifa Alghowinem, Roland Goecke, Jeffrey F Cohn, Michael Wagner, Gordon Parker, and Michael Breakspear. 2015\.  Cross-cultural detection of depression from nonverbal behaviour. In _2015 11th IEEE International conference and workshops on automatic face and gesture recognition (FG)_ , Vol. 1. IEEE, 1–8.
  * Alghowinem et al. (2016) Sharifa Alghowinem, Roland Goecke, Michael Wagner, Julien Epps, Matthew Hyett, Gordon Parker, and Michael Breakspear. 2016.  Multimodal depression detection: Fusion analysis of paralinguistic, head pose and eye gaze behaviors.  _IEEE Transactions on Affective Computing_ 9, 4 (2016), 478–490.
  * Alghowinem et al. (2013) Sharifa Alghowinem, Roland Goecke, Michael Wagner, Gordon Parkerx, and Michael Breakspear. 2013.  Head pose and movement analysis as an indicator of depression. In _2013 Humaine Association Conference on Affective Computing and Intelligent Interaction_. IEEE, 283–288.
  * Alghowinem et al. (2020) Sharifa Mohammed Alghowinem, Tom Gedeon, Roland Goecke, Jeffrey Cohn, and Gordon Parker. 2020\.  Interpretation of depression detection models via feature selection methods.  _IEEE transactions on affective computing_ (2020).
  * Baltrušaitis et al. (2016) Tadas Baltrušaitis, Peter Robinson, and Louis-Philippe Morency. 2016\.  OpenFace: An open source facial behavior analysis toolkit. In _2016 IEEE Winter Conference on Applications of Computer Vision (WACV)_. 1–10.  https://doi.org/10.1109/WACV.2016.7477553
  * Beck et al. (1996) Aaron T Beck, Robert A Steer, Roberta Ball, and William F Ranieri. 1996.  Comparison of Beck Depression Inventories-IA and-II in psychiatric outpatients.  _Journal of personality assessment_ 67, 3 (1996), 588–597.
  * Bourke et al. (2010) Cecilia Bourke, Katie Douglas, and Richard Porter. 2010\.  Processing of facial emotion expression in major depression: a review.  _Australian & New Zealand Journal of Psychiatry_ 44, 8 (2010), 681–696.
  * Campayo et al. (2011) Antonio Campayo, Carlos H Gómez-Biel, and Antonio Lobo. 2011\.  Diabetes and depression.  _Current psychiatry reports_ 13, 1 (2011), 26–30.
  * Cohn et al. (2018) Jeffrey F Cohn, Nicholas Cummins, Julien Epps, Roland Goecke, Jyoti Joshi, and Stefan Scherer. 2018\.  Multimodal assessment of depression from behavioral signals.  _The Handbook of Multimodal-Multisensor Interfaces: Signal Processing, Architectures, and Detection of Emotion and Cognition-Volume 2_ (2018), 375–417.
  * Cummins et al. (2011) Nicholas Cummins, Julien Epps, Michael Breakspear, and Roland Goecke. 2011.  An investigation of depressed speech detection: Features and normalization. In _Twelfth Annual Conference of the International Speech Communication Association_.
  * de Melo et al. (2019) Wheidima Carneiro de Melo, Eric Granger, and Abdenour Hadid. 2019\.  Combining global and local convolutional 3d networks for detecting depression from facial expressions. In _2019 14th ieee international conference on automatic face & gesture recognition (fg 2019)_. IEEE, 1–8.
  * Dibeklioğlu et al. (2017) Hamdi Dibeklioğlu, Zakia Hammal, and Jeffrey F Cohn. 2017\.  Dynamic multimodal measurement of depression severity using deep autoencoding.  _IEEE journal of biomedical and health informatics_ 22, 2 (2017), 525–536.
  * Dibeklioğlu et al. (2015) Hamdi Dibeklioğlu, Zakia Hammal, Ying Yang, and Jeffrey F Cohn. 2015.  Multimodal detection of depression in clinical interviews. In _Proceedings of the 2015 ACM on international conference on multimodal interaction_. 307–310.
  * Fossi et al. (1984) Luciano Fossi, C Faravelli, and M Paoli. 1984\.  The ethological approach to the assessment of depressive disorders.  _The Journal of nervous and mental disease_ 172, 6 (1984), 332–341.
  * Gerych et al. (2019) Walter Gerych, Emmanuel Agu, and Elke Rundensteiner. 2019\.  Classifying depression in imbalanced datasets using an autoencoder-based anomaly detection approach. In _2019 IEEE 13th International Conference on Semantic Computing (ICSC)_. IEEE, 124–127.
  * Goldney et al. (2000) Robert D Goldney, David Wilson, Eleonora Dal Grande, Laura J Fisher, and Alexander C McFarlane. 2000\.  Suicidal ideation in a random community sample: attributable risk due to depression and psychosocial and traumatic events.  _Australian & New Zealand Journal of Psychiatry_ 34, 1 (2000), 98–106.
  * Greenberg et al. (2015) Paul E Greenberg, Andree-Anne Fournier, Tammy Sisitsky, Crystal T Pike, and Ronald C Kessler. 2015\.  The economic burden of adults with major depressive disorder in the United States (2005 and 2010).  _The Journal of clinical psychiatry_ 76, 2 (2015), 5356\.
  * Hale III et al. (1997) William W Hale III, Jaap HC Jansen, Antoinette L Bouhuys, Jack A Jenner, and Rutger H van den Hoofdakker. 1997.  Non-verbal behavioral interactions of depressed patients with partners and strangers: The role of behavioral social support and involvement in depression persistence.  _Journal of affective disorders_ 44, 2-3 (1997), 111–122.
  * He et al. (2022) Lang He, Chenguang Guo, Prayag Tiwari, Hari Mohan Pandey, and Wei Dang. 2022.  Intelligent system for depression scale estimation with facial expressions and case study in industrial intelligence.  _International Journal of Intelligent Systems_ 37, 12 (2022), 10140–10156.
  * Huang et al. (2019) Zhaocheng Huang, Julien Epps, and Dale Joachim. 2019\.  Investigation of speech landmark patterns for depression detection.  _IEEE Transactions on Affective Computing_ 13, 2 (2019), 666–679.
  * Joshi et al. (2013a) Jyoti Joshi, Abhinav Dhall, Roland Goecke, and Jeffrey F Cohn. 2013a.  Relative body parts movement for automatic depression analysis. In _2013 Humaine association conference on affective computing and intelligent interaction_. IEEE, 492–497.
  * Joshi et al. (2013b) Jyoti Joshi, Roland Goecke, Gordon Parker, and Michael Breakspear. 2013b.  Can body expressions contribute to automatic depression analysis?. In _2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG)_. IEEE, 1–7.
  * Kacem et al. (2018) Anis Kacem, Zakia Hammal, Mohamed Daoudi, and Jeffrey Cohn. 2018\.  Detecting depression severity by interpretable representations of motion dynamics. In _2018 13th ieee international conference on automatic face & gesture recognition (fg 2018)_. IEEE, 739–745.
  * Lépine and Briley (2011) Jean-Pierre Lépine and Mike Briley. 2011.  The increasing burden of depression.  _Neuropsychiatric disease and treatment_ 7, sup1 (2011), 3–7.
  * Madan et al. (2021) Surbhi Madan, Monika Gahalawat, Tanaya Guha, and Ramanathan Subramanian. 2021.  Head Matters: Explainable Human-Centered Trait Prediction from Head Motion Dynamics. In _Proceedings of the 2021 International Conference on Multimodal Interaction_ (Montréal, QC, Canada) _(ICMI ’21)_. Association for Computing Machinery, New York, NY, USA, 435––443.  https://doi.org/10.1145/3462244.3479901
  * Morales et al. (2017) Michelle Morales, Stefan Scherer, and Rivka Levitan. 2017\.  A cross-modal review of indicators for depression detection systems. In _Proceedings of the fourth workshop on computational linguistics and clinical psychology—From linguistic signal to clinical reality_. 1–12.
  * Mourão-Miranda et al. (2011) Janaina Mourão-Miranda, David R Hardoon, Tim Hahn, Andre F Marquand, Steve CR Williams, John Shawe-Taylor, and Michael Brammer. 2011.  Patient classification as an outlier detection problem: an application of the one-class support vector machine.  _Neuroimage_ 58, 3 (2011), 793–804.
  * Nasir et al. (2016) Md Nasir, Arindam Jati, Prashanth Gurunath Shivakumar, Sandeep Nallan Chakravarthula, and Panayiotis Georgiou. 2016\.  Multimodal and multiresolution depression detection from speech and facial landmark features. In _Proceedings of the 6th international workshop on audio/visual emotion challenge_. 43–50.
  * of Health Metrics and Evaluation (2021) Institute of Health Metrics and Evaluation. 2021.  Global Health Data Exchange (GHDx).
  * Opoku Asare et al. (2019) Kennedy Opoku Asare, Aku Visuri, and Denzil ST Ferreira. 2019\.  Towards early detection of depression through smartphone sensing. In _Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers_. 1158–1161.
  * Pampouchidou et al. (2019) Anastasia Pampouchidou, Panagiotis G. Simos, Kostas Marias, Fabrice Meriaudeau, Fan Yang, Matthew Pediaditis, and Manolis Tsiknakis. 2019.  Automatic Assessment of Depression Based on Visual Cues: A Systematic Review.  _IEEE Transactions on Affective Computing_ 10, 4 (2019), 445–470.  https://doi.org/10.1109/TAFFC.2017.2724035
  * Parekh et al. (2018) Viral Parekh, Pin Sym Foong, Shengdong Zhao, and Ramanathan Subramanian. 2018.  AVEID: Automatic Video System for Measuring Engagement In Dementia. In _23rd International Conference on Intelligent User Interfaces_ (Tokyo, Japan) _(IUI ’18)_. 409–413.  https://doi.org/10.1145/3172944.3173010
  * Pedersen et al. (1988) Jesper Pedersen, JTM Schelde, E Hannibal, K Behnke, BM Nielsen, and M Hertz. 1988\.  An ethological description of depression.  _Acta psychiatrica scandinavica_ 78, 3 (1988), 320–330.
  * Rejaibi et al. (2022) Emna Rejaibi, Ali Komaty, Fabrice Meriaudeau, Said Agrebi, and Alice Othmani. 2022.  MFCC-based recurrent neural network for automatic clinical depression recognition and assessment from speech.  _Biomedical Signal Processing and Control_ 71 (2022), 103107.
  * Ringeval et al. (2019) Fabien Ringeval, Björn Schuller, Michel Valstar, Nicholas Cummins, Roddy Cowie, Leili Tavabi, Maximilian Schmitt, Sina Alisamir, Shahin Amiriparian, Eva-Maria Messner, et al. 2019\.  AVEC 2019 workshop and challenge: state-of-mind, detecting depression with AI, and cross-cultural affect recognition. In _Proceedings of the 9th International on Audio/visual Emotion Challenge and Workshop_. 3–12.
  * Samanta and Guha (2017) Atanu Samanta and Tanaya Guha. 2017.  On the role of head motion in affective expression. In _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2886–2890.
  * Senoussaoui et al. (2014) Mohammed Senoussaoui, Milton Sarria-Paja, João F Santos, and Tiago H Falk. 2014.  Model fusion for multimodal depression classification and level detection. In _Proceedings of the 4th International Workshop on Audio/Visual Emotion Challenge_. 57–63.
  * Song et al. (2020) Siyang Song, Shashank Jaiswal, Linlin Shen, and Michel Valstar. 2020.  Spectral representation of behaviour primitives for depression analysis.  _IEEE Transactions on Affective Computing_ 13, 2 (2020), 829–844.
  * Steffen et al. (2020) Annika Steffen, Julia Nübel, Frank Jacob, Jörg Bätzing, and Jakob Holstiege. 2020\.  Mental and somatic comorbidity of depression: a comprehensive cross-sectional analysis of 202 diagnosis groups using German nationwide ambulatory claims data.  _BMC Psychiatry_ (March 2020).
  * Valstar et al. (2014) Michel Valstar, Björn Schuller, Kirsty Smith, Timur Almaev, Florian Eyben, Jarek Krajewski, Roddy Cowie, and Maja Pantic. 2014\.  Avec 2014: 3d dimensional affect and depression recognition challenge. In _Proceedings of the 4th international workshop on audio/visual emotion challenge_. 3–10.
  * Valstar et al. (2013) Michel Valstar, Björn Schuller, Kirsty Smith, Florian Eyben, Bihan Jiang, Sanjay Bilakhia, Sebastian Schnieder, Roddy Cowie, and Maja Pantic. 2013.  Avec 2013: the continuous audio/visual emotion and depression recognition challenge. In _Proceedings of the 3rd ACM international workshop on Audio/visual emotion challenge_. 3–10.
  * Vos et al. (2016) Theo Vos, Christine Allen, and Megha Arora. 2016\.  Global, regional, and national incidence, prevalence, and years lived with disability for 310 diseases and injuries, 1990–2015: a systematic analysis for the Global Burden of Disease Study 2015\.  _The Lancet_ 388, 10053 (2016), 1545–1602.  https://doi.org/10.1016/S0140-6736(16)31678-6
  * Waxer (1974) Peter Waxer. 1974\.  Nonverbal cues for depression.  _Journal of Abnormal Psychology_ 83, 3 (1974), 319\.


# I-SplitEE: Image classification in Split Computing DNNs with Early Exits

Divya J. Bajpai, Aastha Jaiswal, and Manjesh K. Hanawal
Email: {20i190003, 22n0463<EMAIL_ADDRESS>Indian Institute of
Technology Bombay, India

###### Abstract

The recent advances in Deep Neural Networks (DNNs) stem from their exceptional
performance across various domains. However, deploying these networks on
resource-constrained devices—like edge, mobile, and IoT platforms—is hindered
by their inherent large size. Strategies have emerged, from partial cloud
computation offloading (split computing) to integrating early exits within DNN
layers. Our work presents an innovative unified approach merging early exits
and split computing. We determine the ’splitting layer,’ the optimal depth in
the DNN for edge device computations, and whether to infer on edge device or
be offloaded to the cloud for inference considering accuracy, computational
efficiency, and communication costs. Also, Image classification faces diverse
environmental distortions, influenced by factors like time of day, lighting,
and weather. To adapt to these distortions, we introduce I-SplitEE, an online
unsupervised algorithm ideal for scenarios lacking ground truths and with
sequential data. Experimental validation using Caltech-256 and Cifar-10
datasets subjected to varied distortions showcases I-SplitEE’s ability to
reduce costs by a minimum of $55\%$ with marginal performance degradation of
at most $5\%$.

##  I Introduction

In recent years, the scale of Deep Neural Networks (DNNs) has undergone
substantial expansion, leading to remarkable performance achievements [1],
especially for computer vision tasks such as image classification [2]. This
increased scale has increased computational demands, rendering their
deployment on resource-constrained platforms like mobile and edge devices. To
mitigate the computational challenges associated with deploying DNNs on these
devices, various strategies have been introduced. These include approaches
like Split Computing, Early Exits, and offloading to cloud [3].
Split Computing.  Edge devices, often constrained by limited processing
capabilities, can opt to offload their data to the cloud, which boasts
superior computational power. This approach involves running the full-fledged
Deep Neural Network (DNN) in the cloud, subsequently returning the inferences
to the edge device. However, this offloading of data to the cloud incurs
latency due to communication delays between the edge device and the cloud. To
harness the combined resources available on both mobile devices and the cloud,
a strategy known as ”split computing” has been introduced. In split computing,
the DNN is bifurcated into two segments: a subset of initial layers is
deployed on the edge device, while the remaining layers are hosted in the
cloud. The portion deployed on the edge device serves as an encoder for
processing input images, reducing their size and, consequently, mitigating
latency costs. This hybrid approach facilitates edge-cloud co-inference.
Nonetheless, it’s important to note that in split computing, all samples still
need to be inferred at the final layer on the cloud, which, while reducing the
computational burden on edge devices, maintains significant communication
costs, as every sample is offloaded [3].
Early Exits (EE-DNNs). Several previous methods, including BranchyNet [4],
SPINN [5], and LGViT [6], enable the classification of input images by adding
classifiers at intermediate layers within DNNs. In these early exit DNNs, the
decision to classify the image at an intermediate stage is contingent on the
confidence of the prediction surpassing a predefined threshold. Exiting the
network at its early layers demands less computational resources but may lead
to less precise predictions. Conversely, deeper exits offer greater precision
but at the cost of higher computational demands. Therefore, determining the
best exit point within the network is important to model the accuracy-
efficiency trade-off.

In our approach, we introduce a hybrid strategy that combines both split
computing and early exits to optimize resource usage on both edge devices and
the cloud. We define the ”splitting layer” as the final layer on the edge
device. At this layer, we attach an exit point for making inferences locally
on the edge device. Before an input image is transferred to the cloud, it
undergoes classification on the edge device. If the prediction confidence
exceeds a predetermined threshold, the process is terminated, and the image is
classified on the edge. In cases where the confidence falls below the
threshold, the image is offloaded to the cloud for further processing.

While previous methods [7, 8] traditionally employ a fixed splitting layer on
the edge device, our approach advocates for the adaptive selection of the
splitting layer. This adaptability is particularly crucial in real-world
scenarios where input images often exhibit various distortions [9]. For
instance, edge devices frequently encounter noisy or blurred images, with
different levels of distortion. Pacheco et al. [10] empirically illustrate the
influence of noise on confidence values in images. The distortion in images
changes the latent distribution of the dataset and significantly impacts the
confidence values and inference results [7]. During inference, the data
arrives in an online (sequential order) fashion and the ground truth values
for the image sample are not available. Hence the question then arises, how to
adaptively choose the optimal splitting layer to model the accuracy-efficiency
trade-off when the underlying latent distribution of incoming data changes?

To address the challenge of selecting an optimal splitting layer when the test
data has a different distribution as compared to the dataset used for
training, we introduce an online learning algorithm called I-SplitEE. This
algorithm leverages the Multi-Armed Bandit framework [11], utilizing the
confidence in prediction as a surrogate for prediction accuracy at the exit
point. The computational cost is the cost of running the DNN on edge devices,
while the offloading cost is the cost of communication delays between edge and
cloud. The primary objective is to maximize prediction confidence while
minimizing computational and offloading costs. I-SplitEE dynamically
determines the optimal splitting layer for an incoming sample based on past
observations from the dataset with a given latent data distribution. I-SplitEE
model could be adapted to different edge devices and communication networks as
well using user-specific inputs.

For our backbone model, we adopt the lightweight MobileNet V2 [12] model. This
choice is an ideal testbed for our algorithm, given its efficiency and
competitive accuracy compared to state-of-the-art models. We conduct
experiments under varying levels of noise (distortion) in the dataset, as
detailed in Section V. Our experimental results on Caltech-256 [13] and
Cifar-10 [14] datasets demonstrate that the optimal splitting layer differs as
there are changes in the distribution of data. Specifically, I-SplitEE
achieves a significant reduction in the cost ($>55\%$) with only a minimal
drop in performance ($\leq 5\%$) when compared to the scenario where all
samples are processed at the final layer.

Our key contributions encompass: 1) We integrate split computing and early
exiting to enable early inferences for ”easy” samples on edge devices. 2) We
develop an online learning algorithm I-SplitEE to dynamically select the
optimal splitting layer based on context in an online and unsupervised manner.
3) I-SplitEE optimizes the resource utilization across edge devices and the
cloud. 4) We experimentally validate that I-SplitEE minimizes performance
degradation while significantly reducing costs compared to the final exit
scenario.

Figure 1: Split computing combined with early exits, where the DNN is split
into two parts with an additional advantage of inference on mobile device.

##  II Related works

In this section, we discuss the previous works on Split computing and Early
Exit to use DNNs on mobile devices.

Split computing for DNNs. Neurosurgeon, as introduced in [15], explores
strategies for optimizing the splitting of DNNs based on the cost
considerations associated with selecting a specific splitting layer. In a
similar vein, BottleNet [16] incorporates a bottleneck mechanism within split
computing. This approach entails deploying a segment of the DNN on an edge
device to encode the input sample into a more compact representation before
transmitting it to the cloud for further processing. Various training
methodologies have been proposed for training the encoder situated on the edge
device. Notably, BottleNet++ employs cross-entropy-based training approaches
in the context of split computing.
Early-Exit DNNs. Early-exit DNNs have found applications across diverse tasks.
In the context of image classification, BranchyNet [4] and several preceding
studies utilize classification entropy metrics at different intermediate
layers to determine whether early inference can be made with sufficient
confidence. Approaches like SPINN [5] incorporate early exits into DNN
architectures, primarily aimed at handling service disruptions. Beyond image
classification, early exits have found relevance in various domains, spanning
natural language processing tasks (NLP) [17, 18].
DNNs on mobile device. AdaEE [7] employs a combination of early-exit DNNs and
DNN partitioning to facilitate offloading data from mobile devices using
early-exit DNNs. LEE (Learning Early Exit) [19], DEE (Dynamic Early exit) [20]
and UEE-UCB (Unsupervised early exits using Upper Confidence Bound) [21]
leverage the multi-armed bandit framework to determine optimal exit points.
However, they lack the offloading and primarily focus on performing operations
on edge devices. LEE and DEE are specifically designed for efficient edge
device inference, particularly in cases of service disruptions, employing
utility functions that require access to ground-truth labels. Notably, UEE-UCB
specializes in learning the optimal exit strategy for the ElasticBERT [18]
backbone, primarily tailored for text classification tasks.

Our approach distinguishes itself from prior methods in several key aspects.
1) We consider not only accuracy but also computational and communication
costs when determining the optimal splitting layer. 2) Our method operates in
an online and unsupervised setup, offering a novel perspective. 3) We leverage
contextual information, such as confidence scores, to dynamically determine
the splitting layer for each sample during edge device processing.

##  III Problem Setup

In our scenario, we train a deep neural network (DNN) consisting of $L$
layers. We introduce exit points into this backbone after specific layers. We
denote the set of layers as $[L]=\\{1,2,\ldots,L\\}$. The set of target
classes for classification is denoted as $\mathcal{C}$. For a given image,
$x$, and an exit attached to layer $i$, we define $\hat{P}_{i}(c)$ as the
estimated probability that $x$ belongs to class $c$, where $c$ belongs to
$\mathcal{C}$. The confidence in the estimated probability class, defined as
$C_{i}=\max_{c\in\mathcal{C}}\hat{P}_{i}(c)$, is the maximum among these
estimated probabilities.

The DNN processes the image $x$, and the point at which the DNN splits can
occur at any layer $i$ where exit is attached, where the layers $1$ through
$i$ reside on the edge device, and the remaining layers $i+1$ through $L$ are
processed on the cloud. In our setup, each image requires two-stage decisions:
1) Determining the optimal splitting point in the DNN, and 2) Deciding whether
to exit at the chosen splitting layer or offload the task. Notably, the choice
of the splitting layer relies on the collective context rather than individual
samples. The choice to exit or offload depends on individual samples and
follows this process: When the DNN splits at the $i$th layer, we calculate
$C_{i}(x)$ and compare it with a predefined threshold, denoted as $\alpha$. If
the confidence $C_{i}(x)\geq\alpha$, the image sample is inferred locally on
the edge device and does not require offloading. Otherwise, it is offloaded to
the cloud for inference at the final layer incurring communication costs.

The cost of using the DNN up to layer $i$ encompasses the computational cost
of processing the image sample up to that layer on the edge and performing
inference. We represent the computational cost associated with splitting at
the $i$th layer as $\gamma_{i}$, where $\gamma_{i}$ is proportional to the
layer depth $i$. Additionally, we introduce the cost of offloading from the
edge device to the cloud, denoted as $o$, which depends on factors like
transmission type (e.g., 3G, 4G, 5G, or Wi-Fi). We define the reward function
for splitting at layer $i\in[L]$ as

$r(i)=\left\\{\begin{array}[]{ll}C_{i}-\mu\gamma_{i}&\textit{if}\quad
C_{i}\geq\alpha\text{ or }i=L\\\
C_{L}-\mu(\gamma_{i}+o)&\textit{otherwise,}\end{array}\right.$ (1)

where $\mu$ serves as a conversion factor, enabling us to represent the cost
in terms of confidence. The user-defined value of $\mu$ reflects the user’s
preference and captures the balance between accuracy and cost.

The reward function’s interpretation is as follows: when the DNN generates a
confident prediction at the splitting layer, the reward is calculated as the
confidence at the splitting layer minus the cost associated with processing
the sample up to the $i$th layer and performing inference. In the event of
lower confidence, the sample is offloaded to the cloud, incurring offloading
costs and inferring at the final layer, with the confidence of $C_{L}$. If $i$
is equal to $L$, the sample is processed entirely on the edge device,
eliminating the need for offloading. For any splitting layer $i\in[L]$, the
expected reward can be expressed as

$\mathbb{E}[r(i)]=\mathbb{E}[C_{i}-\mu\gamma_{i}|C_{i}\geq\alpha]\cdot
P[C_{i}\geq\alpha]\\\ +\mathbb{E}[C_{L}-\mu(\gamma_{i}+o)|C_{i}<\alpha]\cdot
P[C_{i}<\alpha]$ (2)

and for the last layer $L$, it is a constant given as
$\mathbb{E}(r(L))=C_{L}-\mu\gamma_{L}$. The goal is to find the optimal
splitting layer $i^{*}$ defined as $i^{*}=\arg\max_{i\in[L]}\mathbb{E}[r(i)]$.

Recall (section 1) that the problem of choosing optimal thresholds is in an
online and unsupervised setup. In the context of online learning problems, the
Multi-Armed Bandit (MAB) [11] setup plays a pivotal role in addressing
challenges related to dynamic decision-making. Hence, we employ an MAB
framework to find the optimal splitting layer based on user-specific
arguments. In this framework, we establish our action set as the set of layer
indices where exits are attached in the DNN denoted as $E$, where each choice
corresponds to splitting the DNN at an exit point. In MAB terminology, these
choices are referred to as ”arms.” Within this framework, we devise a policy
denoted as $\pi$ that selects arm $i_{t}$ at each time step $t$ based on prior
observations. We define the cumulative regret of policy $\pi$ over $T$ rounds
as

$R(\pi,T)=\sum_{t=1}^{T}\mathbb{E}[r(i^{*})-r(i_{t})]$ (3)

where the expectation is with respect to the randomness in the arm selection
caused by previous samples. A policy $\pi^{*}$ is said to be sub-linear if the
average cumulative regret vanishes, i.e. $R(\pi^{*},T)/T\rightarrow 0$.

##  IV Algorithm

In this section, we develop an algorithm named I-SplitEE. The algorithm is
based on the ’optimism in the face of uncertainty principle’ and utilizes the
upper confidence bounds.

Algorithm 1 I-SplitEE

1:Input: $\alpha\text{ (threshold)},\beta\geq 1,E,\gamma_{i}\text{ }\forall
i\in E$, $o$

2:Initialize: $Q(i)\leftarrow 0,N(i)\leftarrow 0$.

3:Initialize by playing each arm once.

4:for $t=|E|+1,|E|+2,\ldots$ do

5: Observe an instance $x_{t}$

6: $i_{t}\leftarrow\arg\max_{i\in
E}\left(Q(i)+\beta\sqrt{\frac{\ln(t)}{N(i)}}\right)$

7: Pass $x_{t}$ till layer $i_{t}$, use threshold $\alpha$ and observe
$C_{i_{t}}$

8: if $C_{i_{t}}\geq\alpha$ then

9: Infer at layer $i_{t}$ and exit

10: $r_{t}(i_{t})\leftarrow C_{i_{t}}(x_{t})-\mu\gamma_{i_{t}}$,
$N_{t}(i_{t})\leftarrow N_{t-1}(i_{t})+1$

11:
$Q_{t}(i_{t})\leftarrow\sum_{j=1}^{t}r_{j}(k)\mathbbm{1}_{\\{k=i_{t}\\}}/N_{t}(i_{t})$

12: else

13: Offload to the last layer. Observe $C_{L}$

14: $r_{t}(i_{t})\leftarrow C_{L}(x_{t})-\mu(\gamma_{i_{t}}+o)$

15: $N_{t}(i_{t})\leftarrow N_{t-1}(i_{t})+1$

16:
$Q_{t}(i_{t})\leftarrow\sum_{j=1}^{t}r_{j}(k)\mathbbm{1}_{\\{k=i_{t}\\}}/N_{t}(i_{t})$

17: Infer at the last layer

18: end if

19:end for

Figure 2: Effect of different offloading costs over the total cost. The cost
is mostly linear but with some deviations. Figure 3: Effect of different
offloading costs over the accuracy. The accuracy remains constant, however,
the changes might be due to randomness in the dataset.

The input to the algorithm is the exploration parameter $\beta$, confidence
threshold $\alpha$, number of exits $E$, computational cost $\gamma_{i}\text{
}\forall i\in E$, and the offloading cost $o$. The pseudo-code of the
algorithm is given in Algorithm 1. The working of the algorithm could be
explained as follows: It plays each arm once to obtain rewards $Q(i)$ and
counters $N(i)$ for each layer with an attached exit. After observing each arm
once, it plays an arm $i_{t}$ that maximizes the UCB index (line 6) in
succeeding rounds. UCB index comprises the weighted sum of empirical averages
of rewards $Q(i)$ and the confidence bonuses. The sample is then processed
till the splitting layer $i_{t}$ and if the confidence is above the threshold
$\alpha$, the sample exits the DNN; else the sample is offloaded to the cloud
with additional cost $o$.

From the analysis of UCB1 [22], one can easily verify that I-SplitEE achieves
a sub-linear regret. The observed regret is
$\mathcal{O}\left(\sum_{i\in\mathcal{A}\backslash
i^{*}}\frac{log(n)}{\Delta_{i}}\right)$ where $\Delta_{i}=r(i^{*})-r(i)$.

##  V Experiments

TABLE I: In this table, we provide the $\%$ reduction in cost as well as accuracy with different distortion levels $\sigma$ on Caltech-256 dataset. Model/Distortion | $\sigma=0.0$ | $\sigma=0.5$ | $\sigma=1$ | $\sigma=1.5$ | $\sigma=2$ | $\sigma=3$
---|---|---|---|---|---|---
| Acc | Cost | Acc | Cost | Acc | Cost | Acc | Cost | Acc | Cost | Acc | Cost
Final layer | 78.5 | 100 | 75.2 | 100 | 71.6 | 100 | 69.8 | 100 | 67.8 | 100 | 64.1 | 100
Random exit | -4.1 | -33.5% | -3.4 | -20.1% | -5.1 | -39.7% | -7.2 | -41.3% | -6.2 | -37.8% | -9.5 | -43.9%
AdaEE | -3.3 | -45.8% | -4.2 | -47.1% | -4.6 | -47.4% | -5.8 | -49.7% | -5.7 | -53.0% | -5.2 | -50.2%
Early-Exit | -5.3 | -39.2% | -9.1 | -38.5% | -9.9 | -38.5% | -10.5 | -38.8% | -10.1 | -38.4% | -10.8 | -37.8%
I-SplitEE | -2.0 | -66.3% | -2.1 | -65.9% | -3.9 | -66.7% | -5.0 | -67.4% | -4.8 | -66.4% | -4.7 | -65.7%

In this section, we provide all the experimental details of the paper. There
are two key phases in our experimental setup. The source code is available at
https://anonymous.4open.science/r/I-SplitEE-5DE5/README.md.
1) Training the backbone:  To evaluate I-SplitEE, we train the MobileNet V2
model with $6$ task-specific exits at intermediate points and the final exit.
The placement of exits is strategically chosen as described in some previous
works [23, 7]. The dataset used for training is the Caltech-256 [13] and
Cifar-10 [14] datasets. We closely follow the training procedure as described
in the paper [7, 19]. The Caltech-256 dataset contains undistorted images of
objects such as bikes, bears, camels etc. The dataset has a total of 257
different classes. The Cifar-10 dataset consists of 10 classes and also has a
classification task. For the training of the backbone, we split the dataset
into $80\%$ for training, $10\%$ for validation and remaining as the test set.
Note that after this step no training happens. Except in table II, all the
given results are from the Caltech-256 dataset due to space constraints.
2) Unsupervised learning of Splitting layer based on underlying distribution:
The change in distribution can be defined by multiple environmental effects,
however, we focus on the change as image distortion. Following the training of
the DNN with attached exits, we apply Gaussian noise as image distortion on
each image from the test set. Gaussian noise occurs when the camera goes out
of focus or the image is from a foggy environment. To make an image noisy, we
add Gaussian noise to an image with zero mean and standard deviation $\sigma$
to an undistorted image. More standard deviation adds more noise to an image.
We vary the standard deviation as $\sigma\in\\{0.5,1,1.5,2,2.5,3\\}$.

After adding the noise to the image we employ I-SplitEE to find the optimal
splitting layer based on the change in distortion present in the image. The
predefined threshold is learned on the validation set of the model and is set
to $0.6$ for Caltech and $0.9$ for Cifar-10 as learned by the model. We also
have two types of costs: 1) Computational cost: As explained in section III,
the computational cost is directly proportional to the number of layers being
processed on the edge device i.e. $\gamma_{i}=\lambda i$ where $\lambda$ is
the computational cost for a single layer. $\lambda$ is user-defined and
depends on the processing power of the user’s device. We experimented with
different processing cost values but due to space constraints, the results are
not provided. However, we set $\lambda=0.1$ thus normalizing the cost for
direct comparison. 2) Offloading cost is also user-defined and depends on the
communication network used (e.g. 3G, 4G, 5G and Wi-Fi). Hence, we experiment
with different offloading costs from the set $o\in\\{0.2,0.4,\ldots,1.0\\}$
and show (see subsection V-C) that I-SplitEE gets better results from other
baselines making it robust to different communication networks. For the main
results in table I, we fix the offloading cost as $o=1.0$ which is the worst-
case scenario. More details on how to compute the offloading cost can be found
in [24].

We run all the experiments $10$ times and average the results. The action set
is the set of attached exits $\mathcal{A}=\\{3,6,\ldots 18,20\\}$. We set the
trade-off factor $\mu=1$ which could be varied based on the user’s preferences
for accuracy and cost. Smaller values of $\mu$ will provide accurate results
but with higher costs and vice versa. All the experiments are performed on a
single NVIDIA RTX 2070 GPU. The training requires $\sim 10$ hours of GPU
runtime for Caltech-256 and $\sim 7$ hours for Cifar-10 with all the exits.
Learning of the optimal splitting layer could be executed quickly using the
CPU.

###  V-A Baselines

1) Final-layer: We form this as our main baseline as it is similar to the
MobileNet’s inference. However, added exits might marginally deteriorate the
performance.
2) Random exit: We randomly choose a splitting layer and then exploit that
layer for the complete dataset. We choose the random splitting layer for $10$
times and then average the results provided in table I.
3) AdaEE:  fixes the splitting layer and then adaptively chooses the threshold
as given in the paper [7]. There is an option to offload but the splitting
layer is fixed. All the hyperparameters were kept the same as in the paper.
4) Early-Exit: is similar to the classic early exiting setup where all the
samples are inferred at the edge device and based on the confidence values
being above a given threshold. The sample passes through the backbone until it
obtains sufficient confidence.

###  V-B Impact of change in distortion

To gain a deeper understanding of the influence of image distortion, we
conducted an analysis of the confidence values associated with the final layer
of our model.

Figure 4: Effect of added noise on confidence. These confidence values are the
final layer.

As depicted in Figure 4, we present boxplots that showcase the confidence
values at the final layer, considering various levels of image distortion, as
well as undistorted images. The visual representation in this figure serves as
a compelling illustration of the significant impact that image distortion
exerts on the model’s confidence. There is a loss in confidence as the
distortion in images increases and might affect the model if a fixed splitting
layer is used. This observation compels us to adaptively choose the splitting
layer as with a change in the underlying distribution of the dataset the
distribution of confidence values also changes at different exits including
the final exit.

###  V-C Analysis over different overheads

Given that the offloading cost is a user-defined parameter, we conducted a
comprehensive analysis by varying this cost within the set of values
$\\{0.2,0.4,0.6,0.8,1.0\\}$ in figure 2. This exploration focused on examining
the behaviour of both accuracy and cost under different offloading cost
scenarios. As the offloading cost increases, it is unsurprising that the
overall cost also rises. Notably, this increase in the offloading cost does
not result in a significant impact on accuracy. This observation suggests the
robustness of I-SplitEE to variations in offloading cost, ensuring that
accuracy remains relatively stable. However, we observe that the choice of the
optimal splitting layer adjusts with shifts in the offloading cost. When the
offloading cost is elevated, I-SplitEE opts for a deeper splitting layer,
aligning with expectations, as the computational cost becomes relatively
smaller compared to the offloading cost. Consequently, the algorithm’s design
prompts the sample to be locally inferred on the edge device. Remarkably, this
decision does not compromise the accuracy of the model, due to offloading of
samples with low confidence on the edge device. As a result, every prediction
maintains a high level of confidence, resulting in only a marginal reduction
in accuracy.

Moreover, our analysis reveals that even from a cost perspective, our model
outperforms the early-exit baseline. This finding strengthens the claims
regarding the efficacy of I-SplitEE and its ability to strike a compelling
balance between accuracy and cost.

##  VI Results

TABLE II: Similar results as in Table I for Cifar-10. Observe that reduction in accuracy and cost is in $\%$ Model/Distortion |  $\sigma$ = 0.0 |  $\sigma$ = 0.5 |  $\sigma$ = 1
---|---|---|---
| Acc | Cost | Acc | Cost | Acc | Cost
Final layer | 92.5 | 100 | 89.9 | 100 | 78.3 | 100
Random exit | -0.9 | -37.2 | -9.1 | -41.8 | -8.7 | -31.5
AdaEE | -0.3 | -65.9 | -4.5 | -67.2 | -6.3 | -41.5
Early-Exit | -0.7 | -64.4 | -8.9 | -65.6 | -9.5 | -28.7
I-SplitEE | -0.05 | -74.2 | -1.8 | -72.9 | -5.1 | -55.8

In Table I, II, we present the significant outcomes of I-SplitEE, providing
insights into accuracy and cost metrics for images exhibiting varying levels
of distortion. Notably, our proposed method, I-SplitEE, demonstrates its
remarkable performance by yielding the smallest performance drop ($\leq 5\%$)
and the most substantial cost reduction ($>65\%$) in comparison to the final
exit. The cost values in the table are computed as the cumulative sum of both
computational and offloading costs over the entire dataset. It is worth noting
that both the ”Final Layer” and ”Early Exit” baselines entail executing all
computations on the mobile device, resulting in a significant computational
cost that impacts the efficiency of the edge device. The accuracy loss
observed in the ”Early Exit” scenario primarily stems from misclassifications
made by intermediate classifiers, which are subsequently rectified in the
deeper layers. On the other hand, both ”Random Exit” and ”AdaEE” provide an
option to offload, but they have fixed splitting layers, preventing dynamic
adaptation of the splitting point. ”AdaEE,” however, possesses the advantage
of adjusting thresholds based on contextual information.

Comparing our method with other baselines, I-SplitEE showcases a remarkable
reduction in cost by efficiently optimizing resource utilization on both edge
and cloud platforms. The distinct advantage of I-SplitEE lies in its adaptive
nature, enabling it to select the optimal splitting layer based on past
samples. It accounts for a minimal reduction in accuracy due to offloading to
the cloud. Overall, I-SplitEE effectively optimizes resource utilization
across both mobile devices and cloud infrastructure by balancing the
computational as well as communication costs.

##  VII Conclusion

We address the problem of executing the DNNs on edge devices by considering
the resources available on the edge. We propose a method that combines early
exits and split computing to reduce the computational cost of inference on
edge devices as well as communication costs to the cloud. The developed
algorithm I-SplitEE splits the DNN based on the resources available where the
initial part of the DNN is deployed on the edge device with an exit attached
at the last layer on the edge device. If the sample gains sufficient
confidence on the edge device, it is inferred locally, else it is offloaded to
the cloud for inference on the final layer of DNN. I-SplitEE adapts to changes
in the distribution of the test data when it data arrives in an online and
unsupervised. Notably, I-SplitEE has a minimal drop in accuracy even when cost
changes making it more robust to different user-specific devices.

Our work could be extended by simultaneously adapting to the optimal threshold
as well as the splitting layer. Also, our method finds the optimal splitting
layer based on all the samples, however, the splitting layer could be learned
based on the difficulty level of the individual samples.

## References

  * [1] K. Han, Y. Wang, H. Chen, X. Chen, J. Guo, Z. Liu, Y. Tang, A. Xiao, C. Xu, Y. Xu _et al._ , “A survey on vision transformer,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 45, no. 1, pp. 87–110, 2022.
  * [2] K. Bochie, M. S. Gilbert, L. Gantert, M. S. Barbosa, D. S. Medeiros, and M. E. M. Campista, “A survey on deep learning for challenged networks: Applications and trends,” _Journal of Network and Computer Applications_ , vol. 194, p. 103213, 2021.
  * [3] Y. Matsubara, M. Levorato, and F. Restuccia, “Split computing and early exiting for deep learning applications: Survey and research challenges,” _ACM Computing Surveys_ , vol. 55, no. 5, pp. 1–30, 2022.
  * [4] S. Teerapittayanon, B. McDanel, and H.-T. Kung, “Branchynet: Fast inference via early exiting from deep neural networks,” in _2016 23rd International Conference on Pattern Recognition (ICPR)_. IEEE, 2016, pp. 2464–2469.
  * [5] S. Laskaridis, S. I. Venieris, M. Almeida, I. Leontiadis, and N. D. Lane, “Spinn: synergistic progressive inference of neural networks over device and cloud,” in _Proceedings of the 26th annual international conference on mobile computing and networking_ , 2020, pp. 1–15.
  * [6] G. Xu, J. Hao, L. Shen, H. Hu, Y. Luo, H. Lin, and J. Shen, “Lgvit: Dynamic early exiting for accelerating vision transformer,” in _Proceedings of the 31st ACM International Conference on Multimedia_ , 2023, pp. 9103–9114.
  * [7] R. G. Pacheco, F. D. Oliveira, and R. S. Couto, “Early-exit deep neural networks for distorted images: Providing an efficient edge offloading,” in _2021 IEEE Global Communications Conference (GLOBECOM)_. IEEE, 2021, pp. 1–6.
  * [8] J. Shao and J. Zhang, “Bottlenet++: An end-to-end approach for feature compression in device-edge co-inference systems,” in _2020 IEEE International Conference on Communications Workshops (ICC Workshops)_. IEEE, 2020, pp. 1–6.
  * [9] S. Dodge and L. Karam, “Understanding how image quality affects deep neural networks,” in _2016 eighth international conference on quality of multimedia experience (QoMEX)_. IEEE, 2016, pp. 1–6.
  * [10] R. G. Pacheco, F. D. V. R. Oliveira, and R. S. Couto, “Early-exit deep neural networks for distorted images: providing an efficient edge offloading,” in _IEEE Global Communications Conf. (GLOBECOM)_ , 2021, pp. 1–6.
  * [11] P. Auer, N. Cesa-Bianchi, and P. Fischer, “Finite-time analysis of the multiarmed bandit problem,” _Machine learning_ , vol. 47, no. 2, pp. 235–256, 2002.
  * [12] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “Mobilenetv2: Inverted residuals and linear bottlenecks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 4510–4520.
  * [13] G. Griffin, A. Holub, and P. Perona, “Caltech-256 object category dataset,” 2007.
  * [14] A. Krizhevsky, V. Nair, and G. Hinton, “The cifar-10 dataset,” _online: http://www. cs. toronto. edu/kriz/cifar. html_ , vol. 55, no. 5, 2014.
  * [15] Y. Kang, J. Hauswald, C. Gao, A. Rovinski _et al._ , “Neurosurgeon: Collaborative intelligence between the cloud and mobile edge,” in _ACM Computer Architecture News_ , vol. 45, 2017, pp. 615–629.
  * [16] A. E. Eshratifar, A. Esmaili, and M. Pedram, “Bottlenet: A deep learning architecture for intelligent mobile cloud computing services,” in _2019 IEEE/ACM International Symposium on Low Power Electronics and Design (ISLPED)_. IEEE, 2019, pp. 1–6.
  * [17] J. Xin, R. Tang, J. Lee, Y. Yu, and J. Lin, “Deebert: Dynamic early exiting for accelerating bert inference,” _arXiv preprint arXiv:2004.12993_ , 2020.
  * [18] X. Liu, T. Sun, J. He, L. Wu, X. Zhang, H. Jiang, Z. Cao, X. Huang, and X. Qiu, “Towards efficient NLP: A standard evaluation and A strong baseline,” 2021. [Online]. Available: https://arxiv.org/abs/2110.07038
  * [19] W. Ju, W. Bao, D. Yuan, L. Ge, and B. B. Zhou, “Learning early exit for deep neural network inference on mobile devices through multi-armed bandits,” in _2021 IEEE/ACM 21st International Symposium on Cluster, Cloud and Internet Computing (CCGrid)_. IEEE, 2021, pp. 11–20.
  * [20] W. Ju, W. Bao, L. Ge, and D. Yuan, “Dynamic early exit scheduling for deep neural network inference through contextual bandits,” in _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_, 2021, pp. 823–832.
  * [21] H. N. N U, M. K. Hanawal, and A. Bhardwaj, “Unsupervised early exit in dnns with multiple exits,” in _Proceedings of the Second International Conference on AI-ML Systems_ , ser. AIMLSystems ’22. New York, NY, USA: Association for Computing Machinery, 2023.
  * [22] P. Auer _et al._ , “Finite-time analysis of the multiarmed bandit problem,” _Machine Learning_ , vol. 47, pp. 235–256, 2002.
  * [23] M. Wang, J. Mo, J. Lin, Z. Wang, and L. Du, “Dynexit: A dynamic early-exit strategy for deep residual networks,” in _2019 IEEE International Workshop on Signal Processing Systems (SiPS)_. IEEE, 2019, pp. 178–183.
  * [24] Z. Kuang, L. Li, J. Gao, L. Zhao, and A. Liu, “Partial offloading scheduling and power allocation for mobile edge computing systems,” _IEEE Internet of Things Journal_ , vol. 6, no. 4, pp. 6774–6785, 2019.