# Explainable AI for survival analysis: a median-SHAP approach

Lucile Ter-Minassian  Sahra Ghalebikesabi  Karla Diaz-Ordaz  Chris Holmes

###### Abstract

With the adoption of machine learning into routine clinical practice comes the
need for Explainable AI methods tailored to medical applications. Shapley
values have sparked wide interest for locally explaining models. Here, we
demonstrate their interpretation strongly depends on both the summary
statistic and the estimator for it, which in turn define what we identify as
an ’anchor point’. We show that the convention of using a mean anchor point
may generate misleading interpretations for survival analysis and introduce
median-SHAP, a method for explaining black-box models predicting individual
survival times.

Interpretable ML, Healthcare

##  1 Introduction

Recent years have seen a rapid growth in the Explainable AI (XAI) literature.
Given the great variety of explanation models and the diversity of
interpretations they can generate (Sundararajan & Najmi, 2020), decision-
makers are increasingly demanding that XAI methods be designed for a single
application. Contrasting with one-size-fits all solutions, such tailored XAI
techniques would better fulfill the precise requirements of the task they’re
meant for (Arrieta et al., 2020). Here, we introduce a specific method for
explaining black-box survival analysis models, which is based on Shapley
values.

Shapley values (SV) have become a gold standard for local model
explainability. They compute feature attributions by quantifying the change in
model output when dropping certain feature values and sampling them from a
reference distribution. The choice of a reference probability distribution on
feature values has been the subject of multiple debates (Aas et al., 2019;
Janzing et al., 2020; Frye et al., 2020). In contrast, we argue that a key
challenge of SV arises from the convention of both using (i) the expectation
as a summary statistic for the distribution of model outcomes and (ii) the
sample mean to estimate it, and that this question has been overlooked. Most
importantly, we show these parameters jointly induce a mean ’anchor point’
against which comparisons are made, and how such an anchor point isn’t adapted
the time-to-event models. Ultimately, we demonstrate that using the original
formulation of Shapley values can generate misleading interpretations when the
distribution of the value function is right-skewed, which is the case for
survival analysis models as some individuals may not be experiencing the event
throughout the study (Rao & Schoenfeld, 2007; Ying et al., 1995).

To resolve this problem, we introduce median-SHAP, which uses the median as
the summary statistic and for estimation. We demonstrate the benefits of this
approach for explaining survival analysis models that output predicted
survival times from a set of features. Note that such models differ from
inferential methods (e.g. the Cox proportional hazards model (Lin & Wei,
1989)) for understanding the drivers of survival. For point prediction models,
the median predicted survival time is more commonly reported than the mean.

To the best of our knowledge, our method is the first additive feature
attribution method tailored to survival analysis. In contrast, SurvLIME (Utkin
et al., 2020; Kovalev et al., 2020) is a local linear approximation based on
the Cox proportional hazards model.

##### Contributions

The gaps in the existing literature motivate the following contributions of
this paper:

  1. 1.

We shed a new light on the discussion around SV by interrogating the notions
of summary statistics and mean estimators within Shapley values and the
subsequent ’anchor points’ they induce. We identify the anchor point as the
instance our observation of interest is compared against, and explain why it
is essential for the interpretation of SV.

  2. 2.

We show how using a mean anchor point can generate misleading explanations of
survival analysis models

  3. 3.

We introduce median-SHAP, an explanation model specific to survival analysis.
Our method is based on observational SV and uses the median as a summary
statistic. We experimentally show that our method has improved
interpretability and robustness compared with the original SV.

##  2 Shapley Values

We consider a model $f:\mathbb{R}^{m}\rightarrow\mathbb{R}^{l}$ and aim to
explain the prediction of an instance $x\in\mathbb{R}^{m}$, given only black-
box access to the model. Shapley values quantify the contribution of input
features $\\{1,\ldots,m\\}$ to the prediction of a complex model
$f:\mathbb{R}^{m}\rightarrow\mathbb{R}^{l}$ at an instance $x$ as follows:

$\displaystyle f(x)=\phi^{f}_{0}+\sum_{i=1}^{M}\phi^{f}_{i}(x)$ (1)

where $\phi^{f}_{j}(x)$ is the Shapley value of feature $j$ to $f(x)$ and
$\phi^{f}_{0}=\mathbb{E}[f(X)]$ is the prediction averaged over the observed
data distribution. The attribution of a feature $j$ is computed from the
difference in value function $v_{f}$ comparing when the feature $j$ is equal
to the value $x_{j}$ with when it is removed from the coalition. We denote
this difference $\Delta v_{f}(S,j,x)=v_{f}(S\cup\\{j\\},x)-v_{f}(S,x)$. When a
feature is included in the coalition its value is set to the observed instance
value $x_{S}$. When a feature is not in the coalition its value is sampled
from a reference distribution $r(X^{*}\ |\ x_{S})$.

$v_{f}({{S}})=\operatorname*{\mathbb{E}}_{r(X^{*}\ |\
x_{S})}[f({x_{S},X^{*}_{\overline{S}}})]$

for $\overline{S}:=\\{1,\ldots,m\\}\backslash S$ with
$({x_{S},x_{\overline{S}}})$ denoting the concatenation of its two arguments.
To account for the dependence with other features, one takes the difference in
value function $v$ averaged over all possible coalitions $S$ of features
excluding feature $j$. Ultimately, a binomial weight
$\frac{{|S|!(m-|S|-1)!}}{m!}$ is added to recover the original Shapley values
which account for all possible orderings. All in all, the Shapley values of
feature $j$ is defined as follows:

$\displaystyle\phi_{j}^{f}(x)$
$\displaystyle=\frac{1}{m}\sum_{\begin{subarray}{c}|S|\in\\\
\\{0,\ldots,m-1\\}\end{subarray}}\left[\frac{1}{\binom{m-1}{|S|}}\sum_{\begin{subarray}{c}S\subseteq\\{1,\ldots,m\\}/j\;\\\
\text{with }|S|=j\end{subarray}}\Delta v_{f}(S,j,x)\right]$
$\displaystyle=\operatorname*{\mathbb{E}}_{S}\left[\operatorname*{\mathbb{E}}_{r(X^{*}\
|\ x_{S})}[f({x_{S},X^{*}_{\overline{S}}})]\right]$

Further, note that the two expectations can be taken in whichever order. We
refer to the _expectation as a summary statistic_ to describe the fact that
the distribution of model outcomes (whether over coalitions or reference
points) is summarised using the central location.

The choice of a reference distribution for SV has been subject to recent
debates (Aas et al., 2019; Janzing et al., 2020; Merrick & Taly, 2020).
Interventional SV (Lundberg & Lee, 2017; Janzing et al., 2020) define
$r(X^{*}\ |\ x_{S}):=p({X^{*}})$ where $p$ denotes the marginal data
distribution. Observational Shapley values (Aas et al., 2019) set the
reference distribution equal to the conditional distribution given $x_{S}$
$r(X^{*}\ |\ x_{S}):=p({X^{*}|X^{*}_{S}=x_{S}})$. Observational SV are often
described as ’true to the data’ because they do not break the correlations
between features. Moreover, this makes them robust to adversarial attacks, as
their computation does not involve evaluations on out-of-distribution
instances i.e. concatenated inputs which were built by sampling from a
marginal distribution.

At this stage, it is important to note that, in reality, SV assess the
importance of a feature _value_ , and not a feature. In other words, the
central question of SV is ’By how much did the feature $j$ shift the local
model’s prediction $f(x)$ away (in any direction) from the average prediction
$\phi_{0}^{f}$, once feature $j$ was set to $x_{j}$?’. By implementing
Equation 1 – also known as the Efficiency property – we are able to derive
bespoken drift from the SV, e.g. ’mode shift’. Consequently, as feature
attributions result from comparisons between $f(x)$ and $\phi_{0}^{f}$, our
instance $x$ is being compared with a synthetic input for which the model
prediction is equal to $\phi^{f}_{0}(x)$. We call this synthetic input the
_anchor point_ in the following. Such comparisons are not readily
interpretable as the anchor point is not an actual realisation of the
population.

Finally, given that Shapley values are Monte Carlo approximated in practice,
the expectation over the reference distribution is estimated by the mean over
a finite set of $L$ observations $\\{x^{*}_{i,\overline{S}}\\}_{i=1}^{L}$ from
the reference distribution, henceforth called ’reference points’ $r(X^{*}_{S}\
|\ x_{S})$,

$\displaystyle\widehat{\phi}(j)$
$\displaystyle=\frac{1}{m}\sum_{S\subseteq\\{1,\ldots,m\\}/j}[\frac{1}{\binom{m-1}{|S|}}(\frac{1}{L}\sum_{i=1}^{L}f(x_{S\cup
j},x^{*}_{i,\overline{S}/j})$
$\displaystyle-\frac{1}{L}\sum_{i=1}^{L}f(x_{S},x^{*}_{i,\overline{S}\cup
j})].$

We will refer to this as using the _sample mean as an estimator_ of the
expectation, or central location.

##  3 Summary statistics, estimators, and anchor points

In the following, we differentiate between the drawbacks of the expectation as
a summary statistic, the sample mean over reference points as its estimator
and the previously defined an anchor point.

###  3.1 Drawbacks of the expectation as a summary statistic

When averaging the difference in value function $\Delta v_{f}$ over all
coalitions, the original Shapley formulation summarises the distribution of
the model outcomes of concatenated inputs by taking their expectation. First,
taking the expectation to summarise central location is problematic when the
distribution is skewed. Second, the value of the expectation – and not its
relative position within the sample distribution – can lead to misleading
interpretations. For instance, if the single reference values are centred
around zero, this may give the impression that the corresponding feature has a
negligible impact on the model outcome. For instance, consider a model

$f({x})=1000x_{1}+x_{2}$

at observation $x=(0,1)$ and where marginally $x_{1}\sim\mathrm{Normal}(0,1)$
and $x_{2}\sim\mathrm{Normal}(0,1)$. Adding $x_{1}$ to the empty coalition
does not, on average, make a difference in terms of model outcome as on the
one hand we marginalise over $x_{1}$ which is centered on 0, and on the other
hand we specify $x_{1}=0$. Similarly, model outcomes are be equal in
expectation for the coalitions $\\{x_{2}\\}$ and $\\{x_{1},x_{2}\\}$. (Fryer
et al., 2021) argue this is why Shapley values should not be used for feature
selection. We then have Shapley values:

$\phi^{f}_{X_{1}}(x_{1},x_{2})=0$ $\phi^{f}_{X_{2}}(x_{1},x_{2})=1$

which can be misleading. Challenges can also arise from taking the expectation
over feature coalitions, as shown by Merrick et.al (2020). To solve this
problem, the authors propose to cluster the single reference Shapley values
and return the cluster means. Nonetheless, multidimensional clustering
algorithms used on one-dimensional data have several shortcomings, including
the fact that two clusters might be of considerably different size. Instead we
remark that since we are interested in summarising a distribution, we should
thus use summary statistics that are robust to distribution shifts, such as
quantiles.

###  3.2 Drawbacks of the mean estimator

Using the sample mean over reference points as an estimator of central
location is challenging when in the presence of heavy-tailed data
distributions and outliers. Consider, for instance a black-box model

$f(x)=x+\epsilon$

where $\epsilon$ is Cauchy noise. Here, the Shapley value at $x$ should be
$\infty$ in theory. In practical applications, an adversary might return
extreme values for unlikely observations, i.e.
$f(x)=-x-1000\cdot\mathbb{I}(x>0.99),x\sim\mathrm{Uniform}[0,1]$. The extreme
values with small probability push the mean change in model outcome of
including the feature to the positive at any observation with $x$ outside of
$[0.99,1]$. Such an effect is all the more likely to happen for marginal
Shapley values, as the model estimator can suffer under high variance.

###  3.3 Drawbacks of a global mean ’anchor point’

Taking the expectation as a summary statistic and the sample as its estimator
together imply that comparisons are made w.r.t. the global mean, which acts as
an anchor point. This has negative impacts on the interpretability of the
resulting attributions as the average model outcome over reference points does
not link to any single particular individual. Put differently, in a clinical
example we would compute the respective SV for a patient comparing them with
an ’artificial’ patient whose model outcome would be equal to the global mean
$\phi_{0}$. Further, when this distribution is marginal, the anchor point
might not relate to a plausible instance. This poses a problem for clinical
management, as patients may want to understand the influence of modifiable
features (e.g. smoking, physical activity) to lengthen their survival.

##  4 median-SHAP for predictions of median survival times

Our proposed method is a local explanation model based on observational
Shapley values tailored to black-box models that predict a median survival
time. median-SHAP takes the median of the expected change in model outcome for
a coalition $S$, that is

$\displaystyle\widehat{\phi}(j)^{med}=\frac{1}{m}\sum_{S\subseteq\\{1,\ldots,m\\}/j}[\frac{1}{\binom{m-1}{|S|}}$
$\displaystyle(med(\\{f(x_{S\cup
j},x^{*}_{i,\overline{S}/j})\\}_{i=1}^{L})-med(\\{f(x_{S},x^{*}_{i,\overline{S}})\\}_{i=1}^{L})]$

Here, like in the original Shapley, each $\phi_{j}$ may be interpreted as the
contribution of feature value $x_{j}$ to the shift $f(x)-\phi_{0}$. However in
median-SHAP $\phi_{0}$ is the median prediction from $f$ in the reference
population. Therefore comparing $f(x)$ with $\phi_{0}$ implies we’re comparing
our individual $x$ with the median _predicted_ individual i.e. the individual
with median prediction in the cohort. Ultimately, with median-SHAP the anchor
point is the median individual, and thus an observed data point. Using a
conditional reference distribution further ensures that our black-box isn’t
evaluated off the data manifold i.e. concatenated instances aren’t out of
distribution. Contrastively, using a marginal reference distribution would
jeopardise the interpretation by breaking the correlation and defining an out-
of-distribution anchor point. Ultimately, using the median over references
makes our estimation robust to outliers and skewed distribution compared to
using the sample mean.

Given the clinical focus of our targeted application, we chose observational
Shapley values to preserve the robustness to adversarial attacks and thus
maintain the high safety standards needed for medical purposes. Further, note
that in the case of a binary classification problems (e.g. having a survival
time above/below a threshold) and assuming that the data set $\\{f(x_{S\cup
j},x^{*}_{i,\overline{S}/j})\\}_{i=1}^{L}$ is balanced, the median model
outcome will be a realisation at the decision boundary. Such an approach is
easier (but also more restricted) than a linear search for the decision
boundary with e.g. $b$ perturbations used for LIME (White & Garcez, 2019). In
contrast to $l1$ and $l2$ regularisation, additivity to $f(x)$ of the Shapley
values is preserved with median-SHAP. Moreover, while Shapley values lose
their intuition when regularisation is added, the median Shapley values can be
interpreted as the expected change in median model outcome when a feature is
added.

##  5 Experiments

We demonstrate the benefits of using median-SHAP by doing the following
experiment on two datasets. We use the entire set as a training set, and a
reference distribution.

  1. 1.

Training a regression model $f$ on $(X,Y)$ where $X$ is a set of features and
$Y$ is a median survival time. We denote $med=\text{Median}[f(X)]$.

  2. 2.

Using SHAP and median-SHAP to generate feature attributions for the outcome of
a given individual $x$ .

  3. 3.

’Re-labeling’ the outcome of all individuals in the dataset to create a
classification problem, such that: $Y^{\prime}=1$ if $f(x)>med$ and
$Y^{\prime}=0$ otherwise.

  4. 4.

Training a classification model $g$ on $(X,Y^{\prime})$.

  5. 5.

Using SHAP to compute feature attributions explanations for the classification
of $x$ by our model $g$

We repeat this experiment for randomly sampled individuals and report the
relative differences (i) between median-SHAP for $f$ and SHAP for $g$ (ii)
between SHAP for $f$ and SHAP for $g$. This process may attest if median-SHAP
captures which are the most important features for predicting how an
individual _ranks_ within the cohort, w.r.t. a model’s predictions. In other
words, if a feature is important for model $g$ it means that it is important
to predict how our individual ranks w.r.t. the median individual. Therefore,
if median-SHAP for $f$ computes feature attributions similar to the Shapley
values for $g$, this confirms our method captures how the instance of interest
compares with the anchor point, or median individual here. Details regarding
the experiments and the datasets are shown in the Supplements B.

###  5.1 The Worcester Heart Attack Study

The Worcester Heart Attack Study (Whas) is a longitudinal study on on acute
myocardial infarction which looks at patient features at presentation
(Goldberg et al., 2000). A Random Survival Forest (Ishwaran et al., 2008) –
which is a classification tree method for the analysis of right-censored
survival data – was trained on the entire set (N=500) to predict time of
death, which occurred for 215 patients (43.0%).

Shapley type | Age | BMI | Chf | Heart Rate
---|---|---|---|---
SHAP on $f$ vs SHAP on $g$ | 7.84 | 2.1 | -0.54 | 6.51
median-SHAP on $f$ vs SHAP on $g$ | 1.87 | -0.1 | -1.03 | 1.22

Table 1: Mean difference of scaled feature attributions for the Whas data
comparing (i) mean Shapley values on $f$ vs mean Shapley values on the
classification model $g$, (ii) median-SHAP on $f$ vs mean Shapley values on
the classification model $g$.

Here, feature attributions should be interpreted comparing each instance of
interest with the median patient, who is a 59 year old individual with a BMI
of 29.3 $kg/m^{2}$, a heart rate of 117 bpm and who has experienced congestive
heart failure. Table 1 shows the results of our experiment on the Whas
dataset.

###  5.2 The breast cancer survival data

The breast cancer survival data (Desmedt et al., 2007) is a longitudinal study
which explores the time dependence between a genetic prognostic signature and
the occurrence of distant metastases. A Random Survival Forest was trained on
198 individuals to predict the time of potential metastases, which occurred
for 51 patients (25.8%). Resulting feature attributions are described in table
2, and should be interpreted as comparisons with the median individual who has
the following features: `X202240 = 6.96`, `X214806 = 7.32`, `X219724_s = 6.33`
and `X203306_s = 10.51` and has a predicted median survival time of 51 time
units.

For both experiments, feature attributions computed using median-SHAP are
closer to the ones for the classification task than the original mean Shapley
values. This further demonstrates that median-SHAP is able to capture which
features are predominant when predicting how our individual compares with the
rest of the cohort, and more specifically here how it compares with the median
individual. This further illustrates the applicability of our method in this
context. Ultimately, note that the median is more appropriate as an estimator
for such examples with a small sample size, due to its robustness to outliers.

Shapley type | X202240 | X219724_s | X214806 | X203306_s
---|---|---|---|---
SHAP on $f$ vs SHAP on $g$ | 0.04 | -3.32 | -2.54 | 10.51
median-SHAP on $f$ vs SHAP on $g$ |  $\leq$ 0.001 | -1.41 | -3.57 | -1.28

Table 2: Mean difference of scaled feature attributions for the breast cancer
survival data comparing (i) mean Shapley values on $f$ vs mean Shapley values
on the classification model $g$, (ii) median-SHAP on $f$ vs mean Shapley
values on the classification model $g$.

##  6 Concluding remarks

We introduce median-SHAP, a specific explanation method for survival models.
In addition to the robustness of the median estimator w.r.t. outliers and its
higher likelihood for skewed distributions, median-SHAP offers increased
interpretability as comparisons are made with an actual individual from the
reference population. Ultimately, we advocate that median-SHAP has improved
clinical applicability as it allows physicians to illustrate their
recommendations using person-centric explanations.

## References

  * Aas et al. (2019) Aas, K., Jullum, M., and Løland, A.  Explaining individual predictions when features are dependent: More accurate approximations to shapley values.  _arXiv preprint arXiv:1903.10464_ , 2019.
  * Arrieta et al. (2020) Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al.  Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai.  _Information fusion_ , 58:82–115, 2020.
  * Desmedt et al. (2007) Desmedt, C., Piette, F., Loi, S., Wang, Y., Lallemand, F., Haibe-Kains, B., Viale, G., Delorenzi, M., Zhang, Y., d’Assignies, M. S., et al.  Strong time dependence of the 76-gene prognostic signature for node-negative breast cancer patients in the transbig multicenter independent validation series.  _Clinical cancer research_ , 13(11):3207–3214, 2007.
  * Frye et al. (2020) Frye, C., de Mijolla, D., Cowton, L., Stanley, M., and Feige, I.  Shapley-based explainability on the data manifold.  _arXiv preprint arXiv:2006.01272_ , 2020.
  * Fryer et al. (2021) Fryer, D., Strümke, I., and Nguyen, H.  Shapley values for feature selection: The good, the bad, and the axioms.  _arXiv preprint arXiv:2102.10936_ , 2021.
  * Goldberg et al. (2000) Goldberg, R. J., Yarzebski, J., Lessard, D., and Gore, J. M.  Decade-long trends and factors associated with time to hospital presentation in patients with acute myocardial infarction: the worcester heart attack study.  _Archives of Internal Medicine_ , 160(21):3217–3223, 2000.
  * Ishwaran et al. (2008) Ishwaran, H., Kogalur, U. B., Blackstone, E. H., and Lauer, M. S.  Random survival forests.  _The annals of applied statistics_ , 2(3):841–860, 2008.
  * Janzing et al. (2020) Janzing, D., Minorics, L., and Blöbaum, P.  Feature relevance quantification in explainable ai: A causal problem.  In _International Conference on Artificial Intelligence and Statistics_ , pp. 2907–2916. PMLR, 2020.
  * Kovalev et al. (2020) Kovalev, M. S., Utkin, L. V., and Kasimov, E. M.  Survlime: A method for explaining machine learning survival models.  _Knowledge-Based Systems_ , 203:106164, 2020.
  * Lin & Wei (1989) Lin, D. Y. and Wei, L.-J.  The robust inference for the cox proportional hazards model.  _Journal of the American statistical Association_ , 84(408):1074–1078, 1989.
  * Lundberg & Lee (2017) Lundberg, S. and Lee, S.-I.  A unified approach to interpreting model predictions.  _arXiv preprint arXiv:1705.07874_ , 2017.
  * Merrick & Taly (2020) Merrick, L. and Taly, A.  The explanation game: Explaining machine learning models using shapley values.  In _International Cross-Domain Conference for Machine Learning and Knowledge Extraction_ , pp. 17–38. Springer, 2020.
  * Rao & Schoenfeld (2007) Rao, S. R. and Schoenfeld, D. A.  Survival methods.  _Circulation_ , 115(1):109–113, 2007.
  * Sundararajan & Najmi (2020) Sundararajan, M. and Najmi, A.  The many shapley values for model explanation.  In _International Conference on Machine Learning_ , pp. 9269–9278. PMLR, 2020.
  * Utkin et al. (2020) Utkin, L. V., Kovalev, M. S., and Kasimov, E. M.  Survlime-inf: A simplified modification of survlime for explanation of machine learning survival models.  _arXiv preprint arXiv:2005.02387_ , 2020.
  * White & Garcez (2019) White, A. and Garcez, A. d.  Measurable counterfactual local explanations for any classifier.  _arXiv preprint arXiv:1908.03020_ , 2019.
  * Ying et al. (1995) Ying, Z., Jung, S.-H., and Wei, L.-J.  Survival analysis with median regression models.  _Journal of the American Statistical Association_ , 90(429):178–184, 1995.

## Supplementary Material

##  Appendix A Shaley axioms for qSHAP

Shapley values have been shown to satisfy the following axioms.

  * •

According to the _Dummy_ axiom, a feature $j$ receives a zero attribution if
it has no possible contribution, i.e. $v(S\cup j)=v(S)$ for all
$S\subseteq\\{1,...,m\\}$.

  * •

According to the _Symmetry_ axiom, two features that always have the same
contribution receive equal attribution, i.e. $v(S\cup i)=v(S\cup j)$ for all
$S$ not containing $i$ or $j$ then $\phi_{i}(v)=\phi_{j}(v)$.

  * •

According to the _Efficiency_ axiom, the attributions of all features sum to
the total value of all features. Formally,
$\sum_{j}\phi_{j}(v)=v(\\{1,..,m\\})$.

  * •

According to the _Linearity_ axiom, for any value function $v$ that is a
linear combination of two other value functions $u$ and $w$ (i.e. $v(S)=\alpha
u(S)+\beta w(S)$), the Shapley values of $v$ are equal to the corresponding
linear combination of the Shapley values of $u$ and $w$ (i.e.
$\phi_{i}(v)=\alpha\phi_{i}(u)+\beta\phi_{i}(w)$).

Since all these axioms have been defined conditionally on the value function
$v$, they hold if the value function is changed. We have expressed qSHAP, and
thus median-SHAP, as a change in the value function of standard Shapley
values. As such they also adhere to the axioms.

##  Appendix B Further experimental results

Black-box models were trained using the default parameters from `sklearn` and
`scikit-survival` for the Multi-layer Perceptron regressor (MLP regressor) and
the Random Survival Forest (RSF) respectively. For all experiments, we
computed feature importances using the Mean Decrease Accuracy from the ELI5
framework (https://github.com/mayur29/Machine-Learning-Model-Interpretation.
Only the top 4 features were kept and the model was retrained using this
restricted set of features.

####  B.0.1 The Worcester Heart Attack dataset

The dataset has 500 samples and 14 features. The endpoint is death, which
occurred for 215 patients (43.0%). A Random Survival Forest was trained on all
samples. The model fit was measured with the out of bag score, which was equal
to 0.761 for the entire model and 0.892 for the parsimonious model. The
training set was used as a reference population for Shapley values
computation. The URL for this dataset is: https://scikit-
survival.readthedocs.io/en/stable/api/generated/sksurv.datasets.load_whas500.html#sksurv.datasets.load_whas500

####  B.0.2 The breast cancer survival dataset

The dataset has 198 samples and 80 features. The endpoint is the presence of
distance metastases, which occurred for 51 patients (25.8%). A Random Survival
Forest was trained on all samples. The model fit was measured with the out of
bag score, which was equal to 0.761 for the entire model and 0.844 for the
parsimonious model. The training set was used as a reference population for
Shapley values computation. The URL for this dataset is: https://scikit-
survival.readthedocs.io/en/latest/api/generated/sksurv.datasets.load_breast_cancer.html

The median individual has `X202240 = 6.96`, `X214806 = 7.32`, `X219724_s =
6.33` and `X203306_s = 10.51` and has a predicted median survival time of 51
time units.


# Exploring Facial Biomarkers for Depression through Temporal Analysis of
Action Units

Aditya Parikh1, Misha Sadeghi123, Björn Eskofier 123  1 University of
Erlangen–Nuremberg  2 Machine Learning and Data Analytics Lab (MaD Lab)  3
Department Artificial Intelligence in Biomedical Engineering (AIBE)

###### Abstract

Depression is characterized by persistent sadness and loss of interest,
significantly impairing daily functioning and now a widespread mental
disorder. Traditional diagnostic methods rely on subjective assessments,
necessitating objective approaches for accurate diagnosis. Our study
investigates the use of facial action units (AUs) and emotions as biomarkers
for depression. We analyzed facial expressions from video data of participants
classified with or without depression. Our methodology involved detailed
feature extraction, mean intensity comparisons of key AUs, and the application
of time series classification models. Furthermore, we employed Principal
Component Analysis (PCA) and various clustering algorithms to explore the
variability in emotional expression patterns. Results indicate significant
differences in the intensities of AUs associated with sadness and happiness
between the groups, highlighting the potential of facial analysis in
depression assessment.

###### Index Terms:

depression, action units, emotion, facial analysis, biomarkers, clustering,
time-series

##  I Introduction

The World Health Organization (WHO) estimates that depression affects millions
of people worldwide, making it one of the most common mental health
conditions[2]. Depression is a condition that considerably reduces everyday
operations and quality of life. It is characterized by continuous sorrow, lack
of interest in activities, and a variety of mental and physical difficulties
[3]. A timely and accurate diagnosis is essential for managing and treating
conditions effectually. Traditional diagnostic techniques frequently depend on
subjective self-reported questionnaires and clinical interviews like the Beck
Depression Inventory (BDI) [11], the Hamilton Depression Rating Scale (HDRS)
[4], and the Patient Health Questionnaire (PHQ-8 and PHQ-9) [5]. Several
factors could influence these evaluations.

More objective, trustworthy, and quantitative diagnostic tools are needed to
supplement these established techniques. One promising approach involves the
analysis of facial expressions and emotions, as these can provide objective
and non-invasive indicators of mental health status [10]. Previous studies
have demonstrated that certain facial expressions and feelings are more
prevalent in individuals with depression [9].

Using facial expression analysis to find biomarkers linked to depression is
one potential method. Facial expressions, which are governed by the Facial
Action Coding System (FACS)’s definition of facial action units (AUs), are
widely used markers of emotional states and can reveal important information
about a person’s mental state [8]. The FACS, proposed by Ekman and Friesen
[8], provides an exhaustive framework for categorizing facial movements
associated with specific emotions. This system allows for the detailed
analysis of facial action units, which are the fundamental actions of
individual facial muscles [10]. Additionally, people with depression often
exhibit a higher prevalence of sadness and lower frequencies of happiness
compared to healthy individuals. However, there is a need for more refined and
quantitative analyses to understand better the relationship between AUs,
emotions, and depression.

This study aims to investigate the potential of facial AUs as biomarkers for
depression through a detailed temporal analysis. Temporal analysis refers to
examining how AUs change and interact over time. Instead of merely looking at
static facial expressions, we analyze the dynamic sequences and patterns of
these expressions to understand their temporal characteristics. By examining
the dynamic patterns of facial expressions, we seek to identify specific AUs
that are indicative of depression and to develop predictive models that can
accurately distinguish between individuals with and without depression. Our
study contributes to a thorough examination of the temporal dynamics of facial
expressions in depression, adding to the expanding corpus of research on
objective mental health screening instruments. The results of this research
could improve the precision of diagnoses and facilitate the creation of
automated, non-invasive depression screening instruments, which would
eventually improve patient outcomes and accelerate intervention times.

##  II Related Work

The relationship between facial expressions, depression, and emotions has been
studied broadly, providing significant insights into the possible applications
of face analysis in mental health evaluations. This section reviews key
studies that have identified dominant facial AUs and emotions in people with
depression, as well as the techniques used to conduct these analyses.

Jones et al. (2018) [6] conducted an extensive study on the facial expressions
of depressed and healthy individuals using the FACS. Their analysis revealed
that depressed patients exhibited shows higher frequencies of AU1 (inner brow
raiser), AU4 (brow lowerer), and AU15 (lip corner depressor), which are
commonly associated with sadness and distress. Also, the study found that
these patients displayed lower frequencies of AU12 (lip corner puller), which
is related to expressions of happiness.

Li et al. (2020) [12] explored the use of machine learning models to classify
depression based on facial expression data. They extracted a wide range of AUs
and employed Principal Component Analysis (PCA) to reduce dimensionality
before applying support vector machines (SVM) for classification. Their
findings showed that AUs associated with sadness (e.g., AU1, AU4) and reduced
expressions of happiness (e.g., AU12, AU25 - lips part) were significant
predictors of depression. The study demonstrated the feasibility of using
automated facial analysis and machine learning for depression detection.

Zhang et al. (2022) [13] proposed a hybrid model combining convolutional
neural networks (CNNs) and recurrent neural networks (RNNs) to capture both
spatial and temporal features of facial expressions in depressed patients. By
leveraging long short-term memory (LSTM) networks, their model effectively
captured the temporal dynamics of facial expressions, leading to a significant
improvement in depression classification performance. According to their
research, temporal information must be taken into account to reliably identify
depression from facial cues.

A more recent study by Wang et al. (2023) [14] utilized a multi-modal deep
learning approach, integrating facial expression analysis with audio and
textual data to detect depression. Their multi-modal model showed superior
performance compared to traditional models, highlighting the importance of
combining different types of behavioral data for comprehensive depression
assessment. This study provided plausible evidence for the potential of multi-
modal deep learning frameworks in mental health diagnostics.

These studies collectively highlight the importance of specific facial AUs and
emotions in identifying depression. They provide a foundation for our
research, which further investigates the dominance of certain AUs and emotions
in depression versus people without depression.

##  III Methodology

###  III-A Data Collection

Data collection was part of the EMPKINS Subproject D02 - Empatho-Kinaesthetic
Sensor Technology for Biofeedback in Depressed Patients[21]. The goal of this
sub-project is to train smartphone-based reappraisal training for
psychological assessment using facial expressions as biomarkers. The project
aims to understand the relationship between cognition, facial expressions, and
affect as underlying mechanisms for the development and maintenance of
depression, addressing the lack of empirical studies on facial expressions and
the need for quantifying and modifying them.

Each participant was provided with a mobile application designed for the
study. Participants were asked questions and expressed emotions were recorded
using their smartphones and a camera setup. This process was conducted in the
presence of a psychologist, encompassing multiple phases to capture a diverse
range of emotional expressions.

###  III-B Feature Engineering

The recorded video data for this study were processed and exported to CSV
files using the OpenDBM tool [15]. These CSV files contained detailed frame-
wise information about facial expressivity and AUs. Specifically, OpenDBM
provided features such as the intensity and presence of individual AUs, as
well as composite metrics for overall facial expressivity. The videos were
recorded at a frame rate of 30 frames per second (fps).

To handle the large volume of data efficiently, a Python script was developed
to batch process the videos on a high-performance computing (HPC) cluster.
This script automated the extraction of relevant features and ensured
consistent processing across all video files.

####  III-B1 Data Preprocessing

The exported files included two key files for each video as CSVs: one for the
emotions and AUs. These files contained frame-wise data on facial expressivity
and AU metrics, respectively. AU presence is a binary indicator that denotes
whether a specific AU is active (1) or not (0) in a given frame. AU intensity,
on the other hand, provides a more nuanced measure of emotional expression by
indicating the strength of the AU activation on a continuous scale. The
intensity values typically range from 0 to 1, where 0 means the AU is not
present, and 1 indicates the maximum intensity of the AU.

For this study, we focused on AU intensity values as they provide a more
subtle measure of emotional expression, capturing the gradations in facial
movements that are crucial for identifying patterns associated with
depression.

Given the structure of our experiments, we managed the timing information
provided by the mobile application to trim the CSV files. This mobile app
recorded the start and end times of the experiment as well as the timestamps
for different experimental phases. For this study, we specifically analyzed
data from the ‘emotional induction’ (EI) phase. This phase involved inducing
negative moods and emotions in participants using negative statements. By
analyzing participants’ expressive behavior under these standardized
conditions, we gathered critical data on how their facial expressions
responded to negative emotional stimuli.

####  III-B2 Depression-Indicative Action Units

Based on previous studies, several AUs have been identified as significantly
associated with depression. These AUs were found to be more prominent or
frequently activated in depressed patients compared to non-depressed patients
[16] [17].

TABLE I: Top 5 Action Units Associated with Depression Rank | Action Unit | Description
---|---|---
1 | AU1 | Inner Brow Raiser
2 | AU4 | Brow Lowerer
3 | AU15 | Lip Corner Depressor
4 | AU6 | Cheek Raiser
5 | AU10 | Upper Lip Raiser

According to Table I, the top 5 action units associated with depression
include AU1 (Inner Brow Raiser), AU4 (Brow Lowerer), AU15 (Lip Corner
Depressor), AU6 (Cheek Raiser), and AU10 (Upper Lip Raiser) [16, 17]. These
AUs have been identified in previous research as being more frequently
activated or prominent in individuals with depression compared to non-
depressed individuals.

Figure 1: Example of AU1: Inner Brow Raiser [18] Figure 2: Example of AU6: Cheek Raiser [18] Figure 3: Example of AU15: Lip Corner Depressor [18] TABLE II: Mapping of AUs to Emotional Expressions [19] Emotion | Combination of AUs
---|---
Happiness | AU6 + AU12
Sadness | AU1 + AU4 + AU15
Surprise | AU1 + AU2 + AU5B + AU26
Fear | AU1 + AU2 + AU4 + AU5 + AU20 + AU26 + AU28
Disgust | AU9 + AU15 + AU16
Anger | AU4 + AU5 + AU7 + AU23

Studies already mentioned in Section II have demonstrated that AUs mentioned
in II are more frequently observed or have higher intensity in individuals
with depression. For instance, AU1 (Inner Brow Raiser) and AU4 (Brow Lowerer)
are associated with sadness and distress, while AU15 (Lip Corner Depressor) is
directly linked to expressions of sadness and despair.

Using FACS, we mapped specific combinations of AUs to distinct emotional
expressions. This mapping allowed us to quantify the presence and intensity of
various emotions based on the AU intensity values recorded in the exported
files. Table II outlines the combinations of AUs used to identify each
emotion. For each frame, we calculated the intensity of these emotions by
summing the intensities of the corresponding AUs. This approach provided a
continuous measure of emotional expression over time.

####  III-B3 Feature Extraction

The primary features extracted for analysis included:

  * •

AU Intensity Values: The intensity levels of individual AUs recorded for each
frame of the video data.

  * •

Emotional Expressivity: Calculated as the cumulative intensity of specific
combinations of AUs associated with distinct emotional states.

These features were pivotal in capturing both subtle and pronounced changes in
facial expressions, crucial for distinguishing between depressed and healthy
individuals during the emotional induction phase.

###  III-C Mean Intensity Comparison

To compare the differences in emotional expressions between depressed and
healthy participants, we conducted a mean intensity comparison across selected
emotions. This involved averaging AU intensity values over all frames during
the phase for each participant.

Statistical tests were applied to compare the mean intensity values of AUs
associated with emotions such as happiness and sadness, particularly between
the different patient groups. Significant differences were identified to
highlight distinctive emotional expression patterns.

###  III-D PCA Clustering

Principal Component Analysis (PCA) was employed to reduce the dimensionality
of AU intensity data. This technique transformed the original set of
correlated AUs into a smaller set of uncorrelated principal components (PCs).
The first few PCs, which captured the highest variance in the data, were
selected for subsequent clustering analysis.

Various clustering methods, including K-means clustering, Agglomerative
clustering, and Gaussian Mixture Model (GMM), was applied to these principal
components. These methods enabled the identification of coherent clusters of
facial expression patterns, facilitating an unsupervised understanding of
emotional expression variability among participants.

###  III-E Silhouette Score Analysis

The quality of clusters generated by K-means clustering was assessed using the
silhouette score [22], a metric that evaluates the coherence and separation of
clusters. For each data point, the silhouette score computed how similar it
was to its cluster compared to neighboring clusters.

For each data point, the silhouette score was calculated as follows:

$s(i)=\frac{b(i)-a(i)}{\max(a(i),b(i))}$

where $a(i)$ is the average distance from the data point $i$ to the other
points in the same cluster, and $b(i)$ is the average distance from $i$ to
points in the nearest different cluster.

The overall silhouette score was calculated as the mean of individual scores
across all data points. Higher silhouette scores indicated well-defined
clusters, providing validation of the clustering approach and insight into the
distinctiveness of facial expression patterns associated with depression.

###  III-F Time Series Classification Performance

Given the temporal nature of facial expression data, we explored the use of
time series classification to distinguish between depressed and healthy
patients. Each video sequence was treated as a time series of AU intensity
values.

A crucial step before applying any classification model is data pre-
processing. This typically involves splitting the data into training (80% of
the whole dataset) and testing sets (20% of the whole dataset). Splitting
strategies include random splitting and stratified splitting (maintaining
class proportions in both sets). Scaling the data to a common range (e.g., 0-1
or standard deviation) to ensure all features contribute equally during model
training.
We employed several classification models tailored for time series analysis:

  * •

ROCKET + LogisticRegression: This method involves applying random
convolutional kernels to the time series to extract features efficiently.
After feature extraction, a logistic regression classifier is trained on the
transformed features [23].

  * •

ROCKET + RidgeClassifierCV: Similar to Logistic Regression but uses ridge
regression for classification, which can handle multicollinearity among
features better [23].

  * •

InceptionTime: A deep learning architecture designed specifically for time
series classification, leveraging inception modules similar to those used in
image classification [24].

  * •

LSTM: A type of recurrent neural network (RNN) well-suited for sequence data,
capable of learning long-term dependencies in time series [25].

  * •

XGBoost Classifier: An ensemble learning method known for its efficiency and
effectiveness in various machine learning tasks, including time series
classification [26].

Performance was evaluated using accuracy to assess the model’s ability to
classify sequences correctly.

##  IV Results and Discussion

###  IV-A Descriptive Statistics

Figure 4: Distribution of participants across three categories in the EmpkinS
D02 dataset: Depressed ($n=46$), Healthy ($n=43$), and Sub-clinical ($n=9$).

The bar plot in Figure 4 shows the distribution of participants in our study
across three categories: Depressed, Healthy, and Sub-clinical. A smaller group
of participants (9 individuals) were categorized as Sub-clinical, indicating
that they exhibited some signs of depression but did not meet the full
criteria for a depressed classification.

To ensure a more comprehensive analysis, these Sub-clinical participants were
included in the Depressed category for subsequent analysis. This decision was
made based on their demonstrated symptoms of depression, which justifies their
inclusion in the depressed group to better understand the spectrum of
depressive symptoms and their impact on facial expressivity and emotional
expression metrics. This reclassification resulted in a combined Depressed
group of 57 participants, providing a more robust dataset for our analysis.

###  IV-B Mean Intensity Comparison of the Dominant Action Units

In this series of experiments, we compared the mean intensity of the dominant
AUs associated with depression between depressed and healthy patients which
are known to be significantly associated with facial expressions related to
depression, offering a measurable basis for comparing facial expressivity
between the two groups, specifically, Inner Eyebrow Raise (AU1), Brow Lowerer
(AU4), and Lip Corner Depressor (AU15). The comparison of these AUs provides
insight into the facial expressivity differences between the two groups over
time during the target phase.

Figure 5: Comparison of Average Intensity of Inner Eyebrow Raise (AU1) Over
Time Between Depressed and Non-Depressed Patients. Figure 6: Comparison of
Average Intensity of Cheek Raiser (AU6) Over Time Between Depressed and Non-
Depressed Patients. Figure 7: Comparison of Average Intensity of Lip Corner
Depressor (AU15) Over Time Between Depressed and Non-Depressed Patients.

The plots 5, 6 and 7 include a horizontal dashed line representing the overall
mean intensity of happiness for both groups, as well as shaded areas depicting
the standard deviation. These additions provide a clear visual representation
of the variability and central tendency of the data.

As illustrated in Figures 5, 6, and 7, there are notable differences in the
mean intensity of specific AUs between depressed and healthy patients.

  * •

Inner Eyebrow Raise (AU1): Figure 5 shows that the average intensity of AU1 is
generally higher in depressed patients compared to healthy patients. This
difference is consistent over time, suggesting that depressed patients exhibit
more pronounced inner eyebrow raises, often associated with sadness and
distress.

  * •

Cheek Raiser (AU6): Figure 6 demonstrates that the average intensity of AU6 is
generally higher in depressed patients compared to healthy patients. This
difference is noticeable over time, indicating that depressed patients exhibit
more pronounced brow lowering, which is associated with expressions of sadness
and concern.

  * •

Lip Corner Depressor (AU15): As shown in Figure 7, the average intensity of
AU15 is significantly higher in depressed patients. This action unit is
associated with expressions of sadness and despair, aligning with the
emotional state of depressed individuals.

Action Units | AU1 | AU6 | AU15
---|---|---|---
Depressed | 0.087 | -0.077 | 0.100
Healthy | -0.081 | 0.071 | -0.092
Difference | 0.167 | -0.148 | 0.192
TABLE III: Overall Mean Intensity of Dominant AUs in Depressed and Healthy
Patients.

The analysis of mean intensity comparisons of key action units (AU1, AU4,
AU15) in Table III reveals significant differences in facial expressivity
between depressed and healthy patients. Depressed individuals consistently
exhibit higher intensities of these action units, corresponding to expressions
of sadness, distress, and concern. These findings support the hypothesis that
specific facial expressions can serve as biomarkers for depression, providing
a potential avenue for more objective and automated psychological assessments.

###  IV-C Mean Intensity Comparison of Dominant Expressions: Sadness and
Happiness

In this section, we analyze the mean intensity of dominant expressions,
specifically sadness, and happiness, between depressed and healthy patients.
This comparison highlights the differences in emotional expressivity between
the two groups, providing further insight into their emotional states.

Figure 8: Comparison of Average Intensity of Happiness Over Time Between
Depressed and Healthy Patients. Figure 9: Comparison of Average Intensity of
Sadness Over Time Between Depressed and Healthy Patients.

Figures 8 and 9 illustrate the mean intensity of expressions of happiness and
sadness, respectively, over time.

  1. 1.

Happiness: As shown in Figure 8, the average intensity of happiness is
consistently lower in depressed patients compared to healthy patients. This
suggests that depressed individuals exhibit fewer and less intense expressions
of happiness, indicative of their reduced positive affect.

  2. 2.

Sadness: Figure 9 shows that the average intensity of sadness is markedly
higher in depressed patients compared to healthy patients. This consistent
difference underscores the prevalence of negative affect and emotional
distress in depressed individuals.

The analysis of dominant expressions reveals that depressed patients show
significantly lower intensities of happiness and higher intensities of
sadness. These findings align with the expected emotional profiles of
depressed individuals, reinforcing the potential of using facial expressions
as objective indicators for assessing depression.

###  IV-D PCA and Clustering Analysis

####  IV-D1 Explained Variance by Principal Components

Figure 10 illustrates the cumulative explained variance with respect to the
number of principal components, demonstrating that the first 20 components
explain approximately 95% of the variance.

Figure 10: Cumulative variance explained by PCA with 20 components selected to
retain 95% of variance.

####  IV-D2 Clustering Analysis

In this section, we perform K-means clustering on PCA-transformed data to
analyze the clustering patterns of emotion intensities. We applied K-Means
clustering with 2 clusters on the PCA-transformed data. The silhouette score
was computed to evaluate the clustering performance and is noted in Section
IV-D3.

Figure 11: K-Means Clustering of Happiness Emotion Intensities. Figure 12:
K-Means Clustering of Sadness Emotion Intensities.

In Figure 11 & 12 we see the results of K-means clustering applied to PCA-
transformed data. It visualizes how data points are grouped into clusters
based on the similarity of emotion intensity features. The clusters are
reasonably well-separated and most data points are correctly assigned to their
clusters, with some overlap near cluster boundaries.

We then perform Agglomerative Clustering on PCA-transformed data to analyze
the clustering patterns of emotion intensities. It works from the
dissimilarities between the objects to be grouped.

Figure 13: Agglomerative Clustering of Happiness Emotion Intensities. Figure
14: Agglomerative Clustering of Sadness Emotion Intensities.

In Figure 13 & 14 we see the results of Agglomerative clustering applied to
the transformed data. The clusters are well-separated from each other, data
points in one cluster are distinctly different from those in neighboring
clusters where the cluster for healthy patients indeed contains significantly
fewer points than the other.

We utilize the Gaussian Mixture Model (GMM) on PCA-transformed data to analyze
the clustering patterns of emotion intensities.

Figure 15: GMM Clustering of Happiness Emotion Intensities. Figure 16: GMM
Clustering of Sadness Emotion Intensities.

In Figure 15 & 16 we see the results of Agglomerative clustering applied to
the transformed data. The clusters are distinguishable from each other, but
there is some overlap or ambiguity in the assignment of a few data points.

####  IV-D3 Evaluating Silhouette Scores

Table IV compares the silhouette scores for happiness and sadness across
different clustering algorithms.

Clustering Algorithm | Score (Happiness) | Score (Sadness)
---|---|---
K-means | 0.478 | 0.417
Agglomerative | 0.446 | 0.709
GMM | 0.437 | 0.431
TABLE IV: Comparison of Silhouette Scores for Different Clustering Algorithms

Based on the comparison of silhouette scores for happiness and sadness
intensities across different clustering algorithms (Table IV), Agglomerative
Clustering emerges as the optimal method, achieving the highest score of 0.709
and 0.446. It demonstrates a moderately well-defined clustering structure. A
score in this range suggests that the clusters are sufficiently distinct and
separated from each other, although there may still be some overlap or
ambiguity at the boundaries between clusters. These results underline the
effectiveness of Agglomerative Clustering in capturing distinct emotional
patterns, particularly for sadness, thereby showcasing its suitability for
such clustering tasks.

###  IV-E Time Series Classification Performance

This section evaluates the performance of various classification techniques on
our dataset, which consists of emotion intensities related to happiness and
sadness. We compare the accuracy scores achieved by each method and discuss
key aspects of their implementation.

We utilized the learning rate finder from the ’tsai’ framework [20], a leading
deep learning library for time series and sequences, to ascertain the optimal
learning rate for training data across all classification techniques employed.

Figure 17: Example - Learning Rate Finder for InceptionTime.

A comparison of accuracy scores achieved by each method on the dataset is
presented in Table V.

Method | Accuracy Score
---|---
ROCKET + Logistic Regression | 0.70
ROCKET + RidgeClassifierCV | 0.66
InceptionTime | 0.60
LSTM | 0.51
XGBoost Classifier | 0.54
TABLE V: Comparison of accuracy scores using various time series
classification techniques. Figure 18: Plot for predicted class probability per
true class for ROCKET + LogisticRegression.

Based on the results in Table V, ROCKET (Random Convolutional Kernel
Transform) combined with Logistic Regression demonstrates superior performance
in our classification task potentially due to:

  * •

Effective Feature Extraction: Utilizes random convolutional kernels to capture
diverse time-dependent patterns, providing crucial features for accurate
classification.

  * •

Dimensionality Reduction: Reduces high-dimensional feature space through
techniques like random projections and pooling, preserving essential
information and enhancing generalization.

  * •

Efficient Training: Logistic Regression efficiently learns linear decision
boundaries in the transformed feature space, enabling faster training compared
to complex models.

##  V Conclusion

This study demonstrates the efficacy of using facial action units (AUs) and
emotions as objective biomarkers for detecting depression. The analysis of
facial action units, specifically comparing their mean intensities, supports
the hypothesis that these metrics effectively evaluate a patient’s condition.
The results also highlight sadness and happiness as the predominant emotions
observed during patient evaluations.

By applying time series classification to facial expression data, significant
differences in the intensity of specific AUs between depressed and healthy
individuals were identified. Moving forward, future research should
concentrate on refining these models and exploring multi-modal approaches that
integrate facial expression analysis with other behavioral data sources such
as voice and text. This integration aims to enhance diagnostic accuracy and
reliability, offering a more comprehensive understanding of an individual’s
mental health status and enabling personalized and timely interventions.

The findings suggest that automated facial analysis can complement traditional
diagnostic methods, providing a more objective and non-invasive approach to
mental health assessment. Future studies should continue to refine these
models and explore multi-modal frameworks to further improve diagnostic
efficacy and reliability in clinical settings.

## References

  * [1] J. M. Girard, J. F. Cohn, M. A. Mahoor, S. M. Mavadati, and D. P. Rosenwald. ”Social Risk and Depression: Evidence from Facial Expressions of Emotion,” _IEEE Transactions on Affective Computing_ , vol. 5, no. 4, pp. 324-333, Oct.-Dec. 2014.
  * [2] World Health Organization. (2020). Depression. Retrieved from https://www.who.int/news-room/fact-sheets/detail/depression
  * [3] World Health Organization, ”Depression,” World Health Organization, 2021. [Online]. Available: https://www.who.int/news-room/fact-sheets/detail/depression. [Accessed: 25-Jun-2024].
  * [4] Hamilton, M. (1960). A rating scale for depression. Journal of Neurology, Neurosurgery, and Psychiatry, 23(1), 56-62.
  * [5] K. Kroenke, R. L. Spitzer, and J. B. W. Williams. (2001). ”The PHQ-9: Validity of a Brief Depression Severity Measure,” Journal of General Internal Medicine, vol. 16, no. 9, pp. 606-613.
  * [6] A. Jones, H. Q. Ngo, and R. L. Miller. (2018). ”Facial Action Units and Depression: Evidence from a Large Clinical Sample,” _Journal of Affective Disorders_ , vol. 245, pp. 65-72, Feb. 2018.
  * [7] U. Çiftçi and F. Akçay. (2013). ”Recognition of Emotional States in Depressed Patients Using Facial Action Coding System,” _Computers in Biology and Medicine_ , vol. 43, no. 12, pp. 2260-2268, Dec. 2013.
  * [8] Ekman, P.,& Friesen, W. V. (1978). Facial Action Coding System: A Technique for the Measurement of Facial Movement. Palo Alto: Consulting Psychologists Press.
  * [9] F. Ekman and E. Rosenberg. (2005). What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS), 2nd ed. Oxford University Press.
  * [10] Girard, J. M., Cohn, J. F., Mahoor, M. H., Mavadati, S. M., & Rosenwald, D. P. (2013). Social Risk and Depression: Evidence from Manual and Automatic Facial Action Unit Analysis. In Proceedings of the 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), 1-7.
  * [11] Beck, A. T., Steer, R. A., & Brown, G. K. (1996). Beck Depression Inventory–II. San Antonio, TX: Psychological Corporation.
  * [12] Li, M., Yang, Y., Shi, W., & Wang, B. (2020). Classification of depression based on facial expressions using machine learning. Journal of Affective Disorders, 276, 263-271. doi:10.1016/j.jad.2020.07.012
  * [13] Zhang, Y., Liu, X., Wang, Z., & Li, H. (2022). Hybrid model combining CNNs and RNNs for depression detection through temporal analysis of facial expressions. IEEE Transactions on Affective Computing, 13(2), 325-335. doi:10.1109/TAFFC.2022.3141123
  * [14] Wang, J., Chen, L., Zhang, X., & Yang, S. (2023). Multi-modal deep learning for depression detection: Integrating facial expression, voice, and text analysis. Journal of Affective Disorders, 321, 246-255. doi:10.1016/j.jad.2023.01.112
  * [15] Baltrusaitis, T., Zadeh, A., Lim, Y. C., & Morency, L.-P. (2018). OpenFace 2.0: Facial behavior analysis toolkit. In 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018) (pp. 59-66). IEEE. doi:10.1109/FG.2018.00019
  * [16] Li, X., Huang, S., & Sui, J. (2020). Facial action units and machine learning for depression detection: A review. IEEE Transactions on Affective Computing, 11(3), 432-444. doi:10.1109/TAFFC.2020.2979262
  * [17] Zhang, W., Lin, H., Liu, Z., & Yu, J. (2022). Hybrid deep learning models for capturing spatiotemporal features of facial expressions in depression detection. Journal of Affective Disorders, 295, 897-904. doi:10.1016/j.jad.2021.09.089
  * [18] T. Kanade, J. F. Cohn and Y. Tian. (2000). Comprehensive database for facial expression analysis, Proc. of FG00, pages 46-53.
  * [19] Cohn, J. F., & Ekman, P. (2005). Measuring facial action by manual coding, facial EMG, and automatic facial image analysis. In J. A. Harrigan, R. Rosenthal, & K. R. Scherer (Eds.), The New Handbook of Methods in Nonverbal Behavior Research (pp. 9-64). Oxford University Press.
  * [20] Ignacio Oguiza. (2023). tsai - A state-of-the-art deep learning library for time series and sequential data, Github.
  * [21] Keinert, M., Schindler-Gmelch, L., Rupp, L.H., Sadeghi, M., Capito, K., Hager, M., Fahimi, F., Richer, R., Egger, B., Eskofier, B.M., Berking, M. (2024). Facing Depression: Evaluating the Efficacy of the EmpkinS-EKSpression Reappraisal Training augmented with Facial Expressions – Protocol of a Randomized Controlled Trial. Manuscript submitted for publication
  * [22] J. Rousseeuw. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. (1987) J Comput Appl Math, 20 (1987), pp. 53-65
  * [23] Dempster, A., Schmidt, D. F., & Webb, G. I. (2019). ROCKET: Exceptionally fast and accurate time series classification using random convolutional kernels. Data Mining and Knowledge Discovery, 33(6), 2066-2095.
  * [24] Fawaz, H. I., Forestier, G., Weber, J., Idoumghar, L., & Muller, P. A. (2019). InceptionTime: Finding AlexNet for time series classification. arXiv preprint arXiv:1909.04939.
  * [25] Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735-1780.
  * [26] Chen, T., & Guestrin, C. (2016). XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’16), 785-794.