# Improving Molecular Force Fields Across Configurational Space by Combining
Supervised and Unsupervised Machine Learning

Gregory Fonseca  Department of Physics and Materials Science, University of
Luxembourg, L-1511 Luxembourg  Igor Poltavsky  Department of Physics and
Materials Science, University of Luxembourg, L-1511 Luxembourg  Valentin
Vassilev-Galindo  Department of Physics and Materials Science, University of
Luxembourg, L-1511 Luxembourg  Alexandre Tkatchenko
<EMAIL_ADDRESS>Department of Physics and Materials Science,
University of Luxembourg, L-1511 Luxembourg

###### Abstract

The training set of atomic configurations is key to the performance of any
Machine Learning Force Field (MLFF) and, as such, the training set selection
determines the applicability of the MLFF model for predictive molecular
simulations. However, most atomistic reference datasets are inhomogeneously
distributed across configurational space (CS), thus choosing the training set
randomly or according to the probability distribution of the data leads to
models whose accuracy is mainly defined by the most common close-to-
equilibrium configurations in the reference data. In this work, we combine
unsupervised and supervised ML methods to bypass the inherent bias of the data
for common configurations, effectively widening the applicability range of
MLFF to the fullest capabilities of the dataset. To achieve this goal, we
first cluster the CS into subregions similar in terms of geometry and
energetics. We iteratively test a given MLFF performance on each subregion and
fill the training set of the model with the representatives of the most
inaccurate parts of the CS. The proposed approach has been applied to a set of
small organic molecules and alanine tetrapeptide, demonstrating an up to two-
fold decrease in the root mean squared errors for force predictions of these
molecules. This result holds for both kernel-based methods (sGDML and GAP/SOAP
models) and deep neural networks (SchNet model). For the latter, the developed
approach simultaneously improves both energy and forces, bypassing the
compromise to be made when employing mixed energy/force loss functions.

##  I Introduction

With the enormous rise in computational power and molecular simulation methods
in the last decades, atomistic modeling is increasingly becoming the method of
choice Hollingsworth and Dror (2018); Phillips  _et al._ (2005); Hansson,
Oostenbrink, and van Gunsteren (2002); Abraham  _et al._ (2015); Rapaport
(2004); Plattner and Noé (2015); Krylova  _et al._ (2020); Wu  _et al._
(2019); Wolf  _et al._ (2019). Applications range from the study and
prevention of corrosion Kallikragas and Svishchev (2018); DorMohammadi  _et
al._ (2018); Obot, Haruna, and Saleh (2019) to protein folding Best (2012),
unfolding Xiao  _et al._ (2018), and self-assembly Meneksedag-Erol and
Rauscher (2019).

Efficient molecular modelling can be done using empirical force-fields (FF):
computing interactions between atoms and molecules using predefined functional
forms. Unfortunately, when it comes to accuracy, such simulations leave a lot
to be desired; they are, for example, not able to compute any chemical changes
or capture many-body effects in a reliable way. When both accuracy and
efficiency are a concern, machine learning force fields (MLFF) are becoming
the method of choice.

In contrast to empirical FF, ML models can potentially reproduce any
functional form of interatomic and intermolecular interactions, leading to
reliable descriptions of potential energy surfaces (PES) of arbitrary
complexity. Many successes have been found in this domain in the recent years,
with a multitude of methods being able to predict the behaviour of small to
medium sized molecules and more Noé  _et al._ (2020); Chmiela  _et al._
(2018); Schütt  _et al._ (2017a); Montavon  _et al._ (2013); Behler (2016);
Schütt  _et al._ (2017b); Hansen  _et al._ (2015); Mardt  _et al._ (2018)
These methods were used to calculate the stability of molecules with chemical
accuracy Bartók  _et al._ (2017), predict the formation energy of crystals at
the level of density functional theory Faber  _et al._ (2016), or even
reconstruct phase diagrams Artrith, Urban, and Ceder (2018), to name a few
examples.

Despite those achievements, the data-driven nature of ML has its downsides:
the quality of the ML results are at the mercy of the availability of “good”
initial data. Collecting such data and choosing “good” training points is a
nontrivial problem requiring a deep understanding of the nature of the data,
which relies on human intuition. This puts into question the unbiased nature
of the ML approaches, eliminating one of their main advantages over the human-
designed FFs. For instance, for applications in molecular dynamics (MD)
simulations, the training data are generally parts of molecular trajectories
extracted from a reference ab initio simulation with the desired level of
accuracy. ML models are then frequently trained to have the best overall
prediction across the entire dataset. This however skews the ML models toward
more common (close-to-equilibrium) molecular configurations, as poorly
predicted but rare (out-of-equilibrium) configurations hardly impact overall
statistics. Hence, the usage of such ML models is unreliable and unpredictable
in long MD simulations where out-of-equilibrium configurations are
significantly more important for constructing the partition function. This
will always be the case when the data distributions in the target simulations
are different from those in the reference datasets. Examples include: studying
nuclear quantum effects (such as proton transport) using the MLFF trained on
classical MD trajectories, simulating phase transitions based on the
information collected only in stable phases, computing reaction rates with ML
models trained on meta-dynamics, etc. In all these cases, minimizing the
prediction error on the reference dataset does not guarantee good prediction
quality across all important configurations, making the results of the
simulations questionable.

In this work, we address the issue outlined above by “flattening the error” of
ML models: i.e. we ensure that the predictive accuracy of the MLFF is equally
reliable for out-of-equilibrium structures or rare events as for common
configurations, thus enhancing the stability of the model regardless of its
use case. To accomplish this, we propose a novel method to optimise the
training of ML models, leading to unbiased molecular FFs with almost constant
accuracy across the entire reference dataset. This method is equally
applicable to any ML model and is available in our free open-source MLFF
package Gregory (2020). We showcase its application on small organic molecules
(uracil, salicylic acid, ethanol, toluene) as well as a larger molecule
(alanine tetrapeptide) using GAP Bartók and Csányi (2015) models with the SOAP
Bartók, Kondor, and Csányi (2013) descriptor and sGDML Chmiela  _et al._
(2018) as a representative kernel-based approach and SchNet Schütt  _et al._
(2017a) as a representative neural-network-based approach. Comparing our
improved models to default models of equal training set size reveals an error
reduction on rare/out-of-equilibrium configurations by a factor of up to 2 for
a negligible sacrifice in mean error.

While most standard approaches involve calculating an average error across an
entire dataset, we are able to compute the prediction error of a model for
different regions of the reference data (configurational space for our
examples), leading to a detailed view on the domain of applicability of the
model. The presented approach can also be applied as an outlier detection
method, effectively finding rare processes or out-of-equilibrium mechanisms
inside of a dataset in an automated way. As an example, the developed approach
enabled us to reveal the fingerprints of the proton transfer mechanism between
the hydroxyl and the carboxylic group in the salicylic acid molecule database
Chmiela  _et al._ (2018) generated using classical MD simulations. This
process is represented only by a few hundreds of configurations within more
than 320k molecular geometries. Nevertheless, the developed approach is
sensitive enough to separate this subset of geometries into an individual
cluster.

The structure of the article is the following. In the Theory section we
explain the developed methodology for outlier detection and improved training
technique. The Practical Application section contains the best practice
example, where we explain in detail how to use the proposed method for the
outlier detection and improved training on the example of the salicylic acid
molecule. In the Results section we apply our method to reconstruct the FF of
small organic molecules and alanine tetrapeptide, which serves as a
representative case for how the method performs on larger molecules. The
Conclusions section presents a summary and an outlook.

##  II Theory

The method described in this section allows for an in-depth error analysis of
a ML model, outlier detection, as well as an improved training technique
resulting in “equally” reliable predictions for all parts of the entire
reference dataset. We apply the developed approach for constructing accurate
MLFF for molecules consisting of a few tens of atoms, but it can be
generalized for any regression problem. An overview of the methodology can be
found in Figure 1.

Figure 1: Overview of the improved learning method. A dataset is clustered
into subsets and the error of an initial ML model is assessed on each
individual cluster. High-error clusters are re-clustered finely and
representative configurations are extracted from each and added to the
training set. This is repeated until a given number of training points is
reached.

The method can be subdivided into three main steps. In the “Initial
Clustering" step, we split molecular configurations into groups based on
similarities in geometric and energetic properties using a combination of
clustering techniques. Namely, we employ agglomerative clustering Ward and Jr.
(1963) to group up configurations of similar geometries and further split the
groups in different energy brackets using KMeans Sculley (2010).

We then apply a given ML model to each individual cluster and compute the
respective mean prediction error, as illustrated in the “Outlier detection"
step. Clusters with high prediction errors represent regions of
configurational space (CS) the model is ill-adapted to, restricting its range
of applicability. This poor prediction can arise from two main sources:
significant differences in physical/chemical properties compared to common
configurations, and/or poor representation for specific regions of CS in the
training set.

We address the latter in the “Improving model" step, where the combination of
all poorly predicted areas is considered and subdivided into a larger amount
of clusters, providing a fine grid of the problematic regions. The numerous
clusters allow us to filter out well-predicted configurations that the initial
clustering had previously misrepresented, as well as find problematic clusters
on a finer scale. Extracting representative geometries (largest error, cluster
centroid, random,…) from poorly predicted fine clusters and adding them to the
initial training set improves the model’s performance on the problematic
regions of CS. Repeating the described procedure with re-trained models
results in a final model with an optimised training set of a given size,
capable of producing comparable errors across all reference data.

In order to subdivide tens of thousands of unlabeled data points into just a
handful of broad initial clusters, similarities between molecular
configurations were defined based on the descriptors of CS used for training
the ML models. In this paper, pairwise atomic distances is the descriptor of
choice. Differences between configurations were defined using the Euclidean
distance in the descriptor space. An agglomerative approach was chosen to
cluster the dataset into configurations with similar geometries, as the
algorithm avoids merging rare but geometrically unique configurations with
large groups of common ones.

With $\vec{x_{i}^{a}}$ the Euclidean position of atom $i\in[1,N]$ of data
point $a\in[1,M]$, the descriptor $\vec{z^{a}}$ is given by:

$\displaystyle\vec{z^{a}}$ $\displaystyle=[...,z^{a}_{i,j},...],\;j<i$ (1)
$\displaystyle z^{a}_{i,j}$
$\displaystyle=||\vec{x_{i}^{a}}-\vec{x_{j}^{a}}||_{2}$ (2)

with $||\cdot||_{2}$ a simple Euclidean distance. Distances in the descriptor
space are then:

$\displaystyle d(\vec{z^{a}},\vec{z^{b}})=||\vec{z^{a}}-\vec{z^{b}}||_{2}$ (3)

Since Euclidean distances are not a natural metric of our chosen descriptor
space, clusters produced this way often contained large variations in
potential energy. To avoid this problem, a further distinction between
different energy levels was done using a KMeans method. The combination of
both clustering techniques helped distinguish between possible degenerate
states as well as geometrically ‘similar’ configurations with significant
energy differences.

After successfully splitting the dataset both by geometries and energies, an
initial ML model was applied to every individual cluster and the average error
on all configurations therein was computed between the predicted and actual
forces. The root mean squared error (RMSE) was chosen as a way to emphasize
large differences. Ordering the clusters by their average prediction error led
to a simple way to identify outliers for the given dataset and model. Poor
predictions on specific clusters were commonly caused by the training set
containing too few examples from the relevant region of CS. Often, this arose
as a simple consequence of a non-optimal training set choice: out-of-
equilibrium geometries are naturally rarer and thus less represented in
datasets born from physical simulations. As such, a random choice of training
points — even if according to some statistical distribution — is very unlikely
to contain those important out-of-equilibrium points. In other cases, clusters
contained configurations whose physiochemical properties deviate from the rest
of the dataset. In such cases, even small changes in geometry can lead to
large differences in forces, hence the need to include a sizable contribution
of outlying configurations to the training set for accurate predictions.

The outlier detection described above also enabled an improved method to
choose the training set. To this end, poorly predicted initial clusters were
recombined and re-clustered more finely by applying the agglomerative approach
as before but with a larger amount of clusters. This increased the resolution
in which problematic regions of CS were identified, allowing for (a) filtering
out of well-predicted configurations, previously buried in overly broad
clusters, and (b) a finer distinction between all sub-regions of CS that
include the configurations problematic for our initial model. Systematically
adding data points from the worst predicted fine clusters to the model’s
training set ensured that all subregions of CS were sufficiently represented.

Several methods were explored to choose which data points to add from the fine
clusters to the training set. Selecting random points from the clusters
already lead to improvements, but to a lesser extent than centroids (in the
descriptor space) or points with the highest prediction error within their
cluster. Both of the latter methods performed similarly, however, as previous
steps already required the computation of prediction errors for every point,
the highest-prediction error criterion proved to be more efficient and is the
default method for this paper.

An alternative to the above scheme could be considered: to skip the initial
clustering and simply continue the process using all configurations whose
prediction error exceeds some minimum value. However, this would require
calculating the prediction for every single data point, whereas our method
allows calculating only a subset of each cluster as a representative error for
the whole, saving on valuable computational cost. As the datasets in this work
are of limited size, the latter was not necessary, but will become important
when scaling to larger systems. Furthermore, many well-predicted clusters
still contain singular configurations associated with high errors. As opposed
to poorly predicted configurations inside poorly predicted geometry clusters,
the former are victims to the limitations of the ML model rather than that of
the training set. Including those in the training set in an attempt to improve
their prediction comes at a significant cost in accuracy in their otherwise
well-predicted cluster. Including entire clusters rather than singular points
of high error anchors the aforementioned exceptions to their respective low-
error cluster, thus giving room to the algorithm to favour configurations that
are representative of entire poorly-predicted subregions of the CS.

One could also think of skipping the initial clustering step and immediately
proceed to creating fine clusters from which to extract new training points.
However, our clustering method of choice—agglomerative clustering—has a time
complexity of $O(n^{3})$ and memory requirement of $O(n^{2})$, making it
inadequate for handling large datasets in one go; instead in a first step, a
subset of the data is chosen and clustered. The remaining (unclustered) data
points are then iteratively added to an existing cluster based on smallest
average distance, mimicking agglomerative clustering while bypassing
computational limitations. This approach reduces the quality of the clustering
scheme, but is still able to exclude well-predicted regions of CS in broad
strokes. The combination of all remaining clusters represent a subset much
smaller than the original dataset, thus most, if not all of the remaining data
points can now be clustered in a single agglomerative step, leading to fine
clusters of higher quality.

Thanks to the fine grid of problematic configurations provided by our
clustering algorithm, new data points could be added to the training set such
as to address the model’s poor predictions in targeted way. The complete
training set for given data was created in an iterative manner, successively
computing prediction errors, targeting problematic configurations to add to
the training set, and re-training ML models. In the end, resulting models were
trained on all the necessary data points to produce comparable prediction
errors across all of CS within a dataset. This extends the MLFF application
range beyond near-equilibrium simulations, providing reliable results even for
out-of-equilibrium computations such as finding reaction rates or transition
pathways.

##  III Practical application of clustering algorithm to salicylic acid ML
force field

In this section we describe in detail each step of the outlier detection and
improved training process on the example of the salicylic acid molecule
Chmiela  _et al._ (2018). All the results shown within this article were
obtained using exactly the same procedure and settings as explained here
unless specified otherwise.

As a first step, the atomic positions of each reference configuration are
converted to the more appropriate pairwise distance descriptor. This
descriptor is used to split the dataset into 10 clusters through agglomerative
clustering with Ward Ward and Jr. (1963) linkage and Euclidean metric. Then,
an additional clustering step is performed on each individual cluster using
the KMeans algorithm with Euclidean metric and kmeans++ initialization Arthur
and Vassilvitskii (2007). This step splits each previous cluster into 5 for a
total of 50 clusters.

For purposes of outlier detection, an sGDML model is trained on the salicylic
acid database Chmiela  _et al._ (2018) with 1000 training points using the
default training scheme implemented within the sGDML package. This model is
subsequently used to predict forces for all 320k configurations of the
reference dataset. Then, force prediction errors are computed for each
individual cluster, and clusters are rearranged based on the cluster RMSE (see
Figure 2). This outlier detection can be done automatically for sGDML using
our MLFF software Gregory (2020) with the following call on default settings:

    python run.py cluster_error -d <dataset_file> -i <model_file>

Figure 2: Force prediction root mean squared error (bars) on all 50 clusters
(x-axis) of the salicylic acid dataset, ordered by ascending error. Relative
population of each cluster is also indicated (solid blue line, arbitrary
units). A representative structure of the highest-error cluster is shown (red
box).

The above will provide the user with a graph similar to Figure 2 as well as
the indices of every cluster and the prediction errors on each one
respectively. It is worth noting that each cluster corresponds to a
qualitatively different set of configurations. Hence the proposed scheme
detects poorly predicted regions of CS for a given model. An example geometry
from the worst-predicted cluster is shown in the figure below: this
configuration has a clear fingerprint of shared hydrogen between the
carboxylic and hydroxyl groups. This process is a rare event in the reference
database obtained by employing classical MD simulations and can be easily
missed by visualization of the trajectory or other human analyses. In
contrast, the proposed clustering approach can easily separate such nontrivial
configurations (a few hundred) from the overwhelming number (above 300
thousand) of simple fluctuations around the equilibrium geometry.

In order to create improved models, we instead start with a smaller sGDML
model trained on only 200 configurations using the default training scheme.
The same error analysis is performed, giving us a rough idea of which part of
CS the current model is struggling with. Every cluster whose error exceeds a
factor of the overall error (factor of $1.1$ here) is merged and re-clustered
to distinguish between finer subparts of CS. The fine clusterisation created a
total of 200 clusters based solely on pairwise atomic distances, using the
same agglomerative approach as for the initial clusterisation. Finally, one
representative configuration is extracted from half the fine clusters
(prioritising high-error clusters). The extracted configuration corresponds to
the highest error within the respective cluster. Overall, 100 points are
extracted to be added to the training set before the sGDML model is re-trained
using the new combined training set (containing 300 configurations in total).
The model’s errors are then re-assessed on the same initial broad clusters,
after which new fine clusters are created and a new subset of 100 training
points is extracted. This process is repeated 8 times in total, resulting in
an optimized training set of 1000 reference molecular configurations and the
corresponding improved sGDML model. Using our software, one can simply execute
the following command:

    python run.py train -d <dataset_file> -n 8 -i 200 -s 100

Figure 3: Force prediction root mean squared error (solid lines) on all 50
clusters (x-axis) of the salicylic acid dataset, ordered by ascending error.
Different colours correspond to varying sizes of training set, using the
default sGDML training method (left) and the improved method (right).

The above command will provide the improved model with 1000 training points as
per the procedure described above, as well as the indices of the broad
clusters. The results are shown in Figure 3 for the initial, final, and one
intermediate step. For comparison we add the results of the default training
scheme as implemented in the sGDML package for models of equal size. One can
see a noticeable flattening of the error curve with every iteration step of
the improved training method. In contrast, the increase in the number of
training points for the default sGDML model leads mainly to an overall
decrease in prediction errors, keeping entire parts of the CS poorly
predicted.

##  IV Results

The developed methodology was used to perform a detailed error analysis of
three state-of-the-art MLFF models, namely sGDML Chmiela  _et al._ (2018), GAP
Bartók and Csányi (2015) using the SOAP Bartók, Kondor, and Csányi (2013)
descriptor and SchNet Schütt  _et al._ (2017a). The reference datasets used
are of ethanol, salicylic acid and uracil Chmiela  _et al._ (2017). From
Figure 4 (note the different scales of the ordinate axes), it is clear that
these different models show varying performance on the molecular datasets,
however all of them are consistently inaccurate for out-of-equilibrium
geometries. We show that by employing default training techniques, the
prediction error on some physically relevant configurations can exceed the
overall root mean squared error (RMSE) by up to a factor of 3. On the flip
side, our improved training method alleviates the problem by creating models
with significantly flattened errors across all configurations. This was
applied to the previously mentioned datasets as well as toluene Chmiela  _et
al._ (2017) and an alanine tetrapeptide dataset.

In many cases, the RMSE of initially poorly predicted configuration clusters
is reduced twofold, while well-predicted clusters suffer a marginal increase
in RMSE, largely within the error margin of the original dataset. Furthermore
for most molecule/model combinations, our improved learning methods result in
an overall RMSE decrease despite the focus of the method on rarer
configurations.

###  IV.1 Outlier detection

Before improving the models, it is necessary to find and show the underlying
problems in the default training methods. For this, we used our outlier
detection methods on multiple datasets for different molecules. The MLFF
models of choice were sGDML, SchNet and GAP/SOAP: we applied each model to the
same molecular datasets. We cluster a dataset of salicylic acid, uracil and
ethanol into 50 different regions of CS and compute the mean squared force
prediction error for every cluster. The results are plotted in Figure 4. A
very large disparity between the errors in clusters and the mean squared error
can be observed, with some clusters presenting an error 3 times higher than
the mean. The difference between them and the cluster of lowest error is of
course even higher. It is clear that in these cases, a single MSE is a very
bad metric to quantify how good the ML model works for out-of-equilibrium
geometries. This is in direct contradiction with the idea of the MLFF being
comparable to the underlying ab initio method, as entire regions of CS present
an accuracy significantly worse than that of the reference calculations.

There are two possible reasons for the observations above. The trivial one is
that the poorly predicted regions contain large fluctuations of molecular
geometries, which are not well represented in the training set. This lack of
information then renders the ML models unable to learn them. Many applications
mainly deal with close-to-equilibrium molecular configurations where the PES
of the molecule is constrained within a given region of CS. There, higher
prediction errors out-of-equilibrium might not significantly impact the
results of simulations. However, for studying molecular stability,
configurational changes or chemical reactions to name a few, large prediction
errors on such fluctuations may affect the final results greatly.

The second reason is non-trivial and can have a significant impact on the
reliability of the MLFF. The poorly predicted areas of the CS can represent
physics or chemistry missing in the majority of the configurations in the
reference dataset. This is showcased in our previous example of salicylic
acid, where the cluster with the most significant error corresponds to a
shared proton between the carboxylic and hydroxyl group (for the details see
the Practical Application section). An accurate simulation of this process
would require proper account of nuclear quantum effects, hence the
corresponding configurations are a negligible minority (a few hundred) in the
salicylic acid dataset containing over 320k molecular geometries of a
classical MD run. Even if corresponding reference data were added to the MD
dataset, those would mainly be ignored within the standard training schemes
due to their relatively high energies. All in all, this points to default MLFF
being inapplicable for studying the proton sharing effect for our given
dataset. In contrast, the developed method is designed to alleviate this
problem by widening the model’s applicability range to the fullest capability
of the dataset.

Figure 4: Force prediction RMSE for sGDML, SchNet and GAP/SOAP (with 12 radial
and 6 angular functions) models on the same ethanol, uracil and salicylic acid
datasets (y-axis, scale adapted for each model for better visibility), split
into 50 clusters of similar configurations (x-axis) ordered by ascending
error. RMSE (bars) is given on a per-cluster basis in contrast to the RMSE
over the entire dataset (solid horizontal black line). Relative cluster
populations are also indicated (solid blue line, arbitrary units).

###  IV.2 Improved models

Figure 5: Force prediction RMSE for sGDML and SchNet default models compared
to the improved models (orange/blue bars, y-axis scale adapted for each model
for better visibility). RMSE was computed on a per-cluster basis on ethanol,
uracil and salicylic acid datasets, split into 50 clusters of similar
configurations (x-axis) ordered by ascending error.

To resolve the problem with nonuniform prediction accuracy, we applied the
improved training techniques developed in this work using both sGDML and
SchNet as our FF models once again. The molecules explored include all the
datasets from the previous subsection as well as toluene. First, we performed
the outlier detection by computing the root mean squared force prediction
error on 50 clusters for an initial model with 200 training points. After
that, 100 training points were added at every step for a total of 8 steps,
resulting in models of a total of 1000 training points each. All the details
of the improved training procedure can be found in the Practical Application
section. The comparison between the default and improved models of the same
size is shown in Figure 5. In contrast to the default models, the improved
versions present a more constant accuracy, with most models reaching a maximal
cluster error of less than half that of their default counterpart. The results
shown in Figure 5 represent the quality of MD simulations performed with the
default and the improved models with respect to the reference method. Since
forces are the variables entering the equations of motion, their errors are
directly related to the deviations between the reference and ML trajectories
(more so than the energies). Of course, computing properties that are mainly
defined by the most common configurations in the reference dataset—such as
average energies at reasonably low temperatures—both types of models would
lead to nearly identical results. On the other hand, processes involving broad
parts of the PES or regions underrepresented in the reference dataset will be
much better described using the proposed improved ML models.

The goal of the improved models is to present a more stable prediction error
across all of configurational space. They do so by explicitly including more
out-of-equilibrium/rare configurations in their training set, at the expense
of the more common/in-equilibrium configurations in the dataset. Despite this,
the overall RMSE across the entire dataset does not change significantly, and
even sees some decrease for many of the molecules shown below (see Table 1).
This further highlights the importance and usefulness of choosing the training
set in a careful and meticulous way beyond just RMSE.

Table 1: Overall RMSE for sGDML and SchNet models, comparing default and improved versions. All numbers are given in $kcal/\left(mol\,\AA\right)$ molecule | def. sGDML | imp. sGDML | def. SchNet | imp. SchNet
---|---|---|---|---
uracil | 0.38 | 0.32 | 0.77 | 0.65
salicylic acid | 0.44 | 0.39 | 0.99 | 1.03
toluene | 0.21 | 0.20 | 0.78 | 0.67
ethanol | 0.51 | 0.50 | 0.57 | 0.47

###  IV.3 Application to larger molecule

So far, we applied the developed methods only on rather small molecules,
demonstrating significant improvements in the resulting MLFFs. In this
subsection, we extend the applications to noticeably larger molecules, using
as an example an alanine tetrapeptide (AcAla3NHMe). This peptide is large
enough to exhibit several incipient secondary structure motifs akin to
biological peptides and proteins. It is important to note that, especially for
larger molecules, our training method can only lead to improvements if the
base model has acceptable accuracy in the first place, hence we use SchNet as
our model in this subsection, since SchNet can be easily employed with much
larger datasets than kernel-based methods (sGDML or GAP).

Our reference dataset was constructed via ab initio molecular dynamics at 500
K with the FHI-aims software Blum  _et al._ (2009) wrapped with the i-PI
package Kapil  _et al._ (2019), using the Perdew-Burke-Ernzerhof (PBE)
exchange-correlation functional Perdew, Burke, and Ernzerhof (1996) with tight
settings and the Many-Body Dispersion (MBD) method Ambrosetti  _et al._
(2014); Tkatchenko  _et al._ (2012) to account for van der Waals interactions.
The dataset contains over 80k data points and covers at least three local
minima. Our ML models were trained on 6k points, a number large enough to
fully represent all physically different configurations given the intrinsic
correlation between neighboring steps within MD trajectories. Figure 6 shows
the performance of two equal-size well-converged SchNet models trained using
the default and improved training schemes for identical architectures. The
details of the improved training procedure are similar to those in the
Practical Application section with two main differences: first, the number of
training points added at each step is 500. Secondly, 150 fine clusters are
created and several training points are extracted from each, where the number
of points is weighted by the size of the fine cluster.

Figure 6: Energy (left) and force (right) prediction RMSE for a AcAla3NHMe
SchNet default models on different clusters: default model (orange bars)
compared to the improved model (blue bars). Each model consists of 6000
training points with identical training procedures and architecture.

Importantly, our method concentrates only on the improvement of the force
predictions, where the RMSE of the default model for the worst clusters is
about twice as large as that for the best one. The minor improvement in energy
demonstrated in Figure 6 is a nice accompanying effect which we were not
aiming for. Although the overall improvement in RMSE for forces and energy
drops only from $0.89$ to $0.8$ $kcal/(mol\,\AA)$ and $0.54$ to $0.49$
$kcal/mol$ respectively, the flattening of the errors for the force prediction
can have noticeable results in practice. Improving the force predictions along
with learning the energy within the SchNet model (or any other ML model) would
usually require the employment of mixed loss functions, where the errors for
energy and forces are minimized together. Such mixed loss functions are less
efficient than pure ones, since optimizing two competing functions leads to
sub-optimal results for each component. In contrast, the proposed method
decouples the problem of improving the predictions for energy and forces. We
use an energy-based loss function in the SchNet model while the forces are
improved by manipulating the training set. As a result, we have a ML model
with equally reliable energies and forces. Moreover, the errors in energy
predictions also decrease, which would not happen if mixed loss functions were
used. Alternatively, if our method instead focused on flattening the energy
predictions, the improvements in the latter would be significantly more
noticeable. However, this would come at the expense of force predictions,
which would largely worsen as a result.

Figure 6 also reveals challenges for the developed method when applied to
large systems. The cluster force predictions of the default model does not
present a variance quite as large as the previously explored molecules. The
main reason is that high-dimensional space (AcAla3NHMe has 42 atoms, i.e. 861
pairwise atomic distances) makes our clustering algorithms significantly
weaker. Any distance metric loses meaning as the number of dimensions
increases, and clustering algorithms rely on the latter to subdivide datasets.
As a consequence, our clusters are ill-defined and contain larger overlaps
between qualitatively different configurations. This reduces the resolution
between well and poorly predicted parts of CS, decreasing the efficiency of
the proposed method. We expect that reducing the size of descriptors by making
use of dimensionality reduction techniques (such as kernel principal component
analysis) would improve the efficiency of the clustering schemes and in turn
make the developed approach reliable for systems containing hundreds and
thousands of atoms.

Nevertheless, even without the aforementioned additional step, significant
improvement can be found when applying our training methods to alanine
tetrapeptide: both energy and force prediction errors are reduced for almost
every cluster. Importantly, achieving the same improvement within the default
training scheme would require adding more reference data to the training set.
Figure 7 shows the energy and forces prediction accuracy of the SchNet models
trained with different sizes of the training set. Depending on the size of the
training set, one can observe three qualitatively different behaviors of the
resulting SchNet models: (a) Whenever the training set contains insufficient
data (the model with 3k points), the constructed MLFF demonstrates low
accuracy across the entire CS for both energy and forces. In this limit, the
force-based improved training method proposed in this work does little to
improve the FF since the starting model cannot distinguish between poorly and
well-predicted areas of the CS, (b) The training set contains enough data for
the ML model to accurately learn the PES, but the forces are poorly predicted
across CS (the model with 6k points), akin to previous examples (see the
Improved Models subsection). This is precisely the scenario for which the
proposed improved training technique has been developed. By comparing the
default and improved models with 6k training points, one can see a significant
boost in accuracy for forces accompanied by a slight improvement of the PES
reconstruction (here, an improved model of 6k points is comparable to a
default model of 7.5-8k points). As such, the proposed training method gives
an optimal compromise between data-efficiency and accuracy of ML models, (c)
Finally, a training set overloaded with reference data (the model with 9k
points) leaves little room for improvement. Indeed in this case, the training
contains all relevant configurations in the dataset (and by extension the
validation set), such that the choice of training points becomes
insignificant.

Figure 7: Energy (left) and force (right) prediction RMSE for a AcAla3NHMe
SchNet default (orange) and improved (blue) models on different clusters.
Comparing different size of training sets: 3000, 6000, 9000 for default and
6000 for improved.

To compare the performance of the improved and default models, we ran
constant-temperature MD simulations at 300 K and 400 K using the SchNet FF
model. The time step was set to 0.5 fs to accurately reproduce fast hydrogen
fluctuations in the molecule. Due to the size and high flexibility of the
peptide, obtaining well-converged average energies requires MD trajectories of
more than four million steps, equivalent to two nanoseconds. Simulations of
this size come at prohibitively expensive computational costs for any accurate
ab initio method; MLFFs are the only way to perform them in practice. Note
that our improved training procedure does come with higher computational costs
(due to training the model multiple times), but the time spent on training is
still very low compared to that of actually running the MD.

At 300 K both models converge without any issues with a difference in average
total energies of only 0.5 $kcal/mol$. The latter is within the accuracy of
the ML models, see Figure 7, meaning that both simulations give identical
results. This is exactly what should be expected for a well trained ML model
in its zone of comfort. At 400 K the situation changes drastically: the
average total energy as a function of simulation time is shown in Figure 8.
One can see that while the improved model remains stable (as a zero energy
level, we use the lowest potential energy in the reference dataset), the
default one fails to reproduce the dynamics of the molecule at 400 K. The
monotonic decay of the red curve advocates for the unreliability of the
default training scheme for high-temperature simulations. As a result of wrong
predictions, the molecule escapes the applicability range of the MLFF and we
observe nonphysical results. Note that the training set was generated at 500
K, and thus contains all the information needed for 400 K MD simulations.
Importantly, increasing the temperature to generate new reference data would
require broader sampling of parts of CS with computationally expensive ab
initio methods — an unacceptable scenario for growing molecule sizes. Hence,
the developed improved training scheme not only leads to quantitatively better
predictions, but also qualitatively increases the applicability range of the
ML models by boosting their reliability.

Figure 8: Average total energy as a function of simulation time for the
default (red) and the improved (blue) SchNet models for the AcAla3NHMe
molecule. The constant-temperature MD simulations have been done at 400 K with
0.5 fs time step.

##  V Conclusions

By leveraging supervised and unsupervised ML, we proposed a new strategy for
improved training set selection for the construction of molecular machine
learning force fields. We developed an automatic outlier detection method that
exposed a noticeable bias in the predictive accuracy of the models towards
common/in-equilibrium configurations at the expense of rarer/out-of-
equilibrium ones, leading to entire regions of CS with significantly higher-
than-average prediction errors. Our procedure is able to extract tiny subsets
of molecular configurations representing nontrivial physical or chemical
processes from an overwhelming amount of reference data. For example, a few
hundred configurations with fingerprints of a shared proton in the salicylic
acid molecule were found within 300k+ classical fluctuations around the
equilibrium state.

The developed error analysis helped us optimise the training set choice,
resulting in largely improved accuracy of ML models across all of
CS—effectively “flattening" the prediction error curve throughout input space.
During the training process, we iteratively selected poorly predicted training
points from different parts of CS to add to the training set. This ensured
that it contained sufficient representation from every qualitatively different
type of configurations in the reference dataset. Models born from this
approach proved more reliable than those with training sets in-line with the
dataset’s inherent distributions and guarantee “chemical accuracy" for the
entire sampled CS. With the examples of small organic molecules and an alanine
tetrapeptide we demonstrated that the developed training method leads to an
optimal compromise between data-efficiency and accuracy of MLFFs, avoiding the
need to generate extensive amounts of computationally expensive highly-
accurate reference data for training sets. Along with quantitative reductions
in prediction errors, the ML models trained on the optimised training sets
offer qualitative improvements in reliability for practical applications. This
is demonstrated on the example of high-temperature MD simulations for the
alanine tetrapeptide. Future plans include combining the developed approach to
dimensionality reduction techniques to extend the applicability range to
systems consisting of hundreds and thousands of atoms.

While this paper focused on improving three specific ML models (GAP with SOAP
descriptors and sGDML as kernel based approaches and SchNet as a neural
network), all methods can easily be extended to any ML field presenting
similar training problems. The code for the outlier detection and improved
training is available in the open-source software MLFF on Github. Gregory
(2020)

###### Acknowledgements.

We acknowledge financial support from the Luxembourg National Research (FNR)
under the AFR project 14593813 and C19/MS/13718694/QML-FLEX, FNR DTU-PRIDE
MASSENA, and the European Research Council (ERC-CoG grant BeStMo).

##  VI Data availability

The data that support the findings of this study are openly available on the
sGDML Chmiela  _et al._ (2017, 2018) official website and in the supplementary
material.

## References

  * Hollingsworth and Dror (2018) S. A. Hollingsworth and R. O. Dror, “Molecular Dynamics Simulation for All,” Neuron 99, 1129–1143 (2018).
  * Phillips  _et al._ (2005) J. C. Phillips, R. Braun, W. Wang, J. Gumbart, E. Tajkhorshid, E. Villa, C. Chipot, R. D. Skeel, L. Kalé, and K. Schulten, “Scalable molecular dynamics with NAMD,” J. Comput. Chem. 26, 1781–1802 (2005).
  * Hansson, Oostenbrink, and van Gunsteren (2002) T. Hansson, C. Oostenbrink, and W. van Gunsteren, “Molecular dynamics simulations,” Curr. Opin. Struct. Biol. 12, 190–196 (2002).
  * Abraham  _et al._ (2015) M. J. Abraham, T. Murtola, R. Schulz, S. Páll, J. C. Smith, B. Hess, and E. Lindahl, “GROMACS: High performance molecular simulations through multi-level parallelism from laptops to supercomputers,” SoftwareX 1-2, 19–25 (2015).
  * Rapaport (2004) D. C. Rapaport,  _The Art of Molecular Dynamics Simulation_ (Cambridge University Press, 2004).
  * Plattner and Noé (2015) N. Plattner and F. Noé, “Protein conformational plasticity and complex ligand-binding kinetics explored by atomistic simulations and Markov models,” Nat. Commun. 6, 7653 (2015).
  * Krylova  _et al._ (2020) K. A. Krylova, J. A. Baimova, I. P. Lobzenko, and A. I. Rudskoy, “Crumpled graphene as a hydrogen storage media: Atomistic simulation,” Physica B Condens. Matter 583, 412020 (2020).
  * Wu  _et al._ (2019) Y. Wu, H. Sun, L. Wu, and J. D. Deetz, “Extracting the mechanisms and kinetic models of complex reactions from atomistic simulation data,” J. Comput. Chem. 40, 1586–1592 (2019).
  * Wolf  _et al._ (2019) S. Wolf, M. Amaral, M. Lowinski, F. Vallée, D. Musil, J. Güldenhaupt, M. K. Dreyer, J. Bomke, M. Frech, J. Schlitter, and K. Gerwert, “Estimation of Protein-Ligand Unbinding Kinetics Using Non-Equilibrium Targeted Molecular Dynamics Simulations,” J. Chem. Inf. Model. 59, 5135–5147 (2019).
  * Kallikragas and Svishchev (2018) D. T. Kallikragas and I. M. Svishchev, “Atomistic simulations of corrosion related species in nano-cracks,” Corros. Sci. 135, 255–262 (2018).
  * DorMohammadi  _et al._ (2018) H. DorMohammadi, Q. Pang, L. Árnadóttir, and O. Burkan Isgor, “Atomistic simulation of initial stages of iron corrosion in pure water using reactive molecular dynamics,” Comput. Mater. Sci. 145, 126–133 (2018).
  * Obot, Haruna, and Saleh (2019) I. B. Obot, K. Haruna, and T. A. Saleh, “Atomistic Simulation: A Unique and Powerful Computational Tool for Corrosion Inhibition Research,” Arab. J. Sci. Eng. 44, 1–32 (2019).
  * Best (2012) R. B. Best, “Atomistic molecular simulations of protein folding,” Curr. Opin. Struct. Biol. 22, 52–61 (2012).
  * Xiao  _et al._ (2018) H. F. Xiao, B. Huang, G. Yao, W. B. Kang, S. Gong, H. Pan, Y. Cao, J. Wang, J. Zhang, and W. Wang, “Atomistic simulation of the coupled adsorption and unfolding of protein GB1 on the polystyrenes nanoparticle surface,” Science China: Physics, Mechanics and Astronomy 61, 038711 (2018).
  * Meneksedag-Erol and Rauscher (2019) D. Meneksedag-Erol and S. Rauscher, “Atomistic Simulation Tools to Study Protein Self-Aggregation,” Methods Mol. Biol. 2039, 243–262 (2019).
  * Noé  _et al._ (2020) F. Noé, A. Tkatchenko, K.-R. Müller, and C. Clementi, “Machine Learning for Molecular Simulation,” Annu. Rev. Phys. 71, 361–390 (2020).
  * Chmiela  _et al._ (2018) S. Chmiela, H. E. Sauceda, K.-R. Müller, and A. Tkatchenko, “Towards exact molecular dynamics simulations with machine-learned force fields,” Nat. Commun. 9, 3887 (2018).
  * Schütt  _et al._ (2017a) K. T. Schütt, P.-J. Kindermans, H. E. Sauceda, S. Chmiela, A. Tkatchenko, and K.-R. Müller, “SchNet: A continuous-filter convolutional neural network for modeling quantum interactions,” Adv. Neural Inf. Process. Syst. 30, 992–1002 (2017a).
  * Montavon  _et al._ (2013) G. Montavon, M. Rupp, V. Gobre, A. Vazquez-Mayagoitia, K. Hansen, A. Tkatchenko, K.-R. Müller, and O. A. von Lilienfeld, “Machine learning of molecular electronic properties in chemical compound space,” New J. Phys. 15, 95003 (2013).
  * Behler (2016) J. Behler, “Perspective: Machine learning potentials for atomistic simulations,” J. Chem. Phys. 145, 170901 (2016).
  * Schütt  _et al._ (2017b) K. T. Schütt, F. Arbabzadah, S. Chmiela, K. R. Müller, and A. Tkatchenko, “Quantum-chemical insights from deep tensor neural networks,” Nat. Commun. 8, 13890 (2017b).
  * Hansen  _et al._ (2015) K. Hansen, F. Biegler, R. Ramakrishnan, W. Pronobis, O. A. von Lilienfeld, K.-R. Müller, and A. Tkatchenko, “Machine Learning Predictions of Molecular Properties: Accurate Many-Body Potentials and Nonlocality in Chemical Space,” J. Phys. Chem. Lett. 6, 2326–2331 (2015).
  * Mardt  _et al._ (2018) A. Mardt, L. Pasquali, H. Wu, and F. Noé, “VAMPnets for deep learning of molecular kinetics,” Nat. Commun. 9, 5 (2018).
  * Bartók  _et al._ (2017) A. P. Bartók, S. De, C. Poelking, N. Bernstein, J. R. Kermode, G. Csányi, and M. Ceriotti, “Machine learning unifies the modeling of materials and molecules,” Sci. Adv. 3, e1701816 (2017).
  * Faber  _et al._ (2016) F. A. Faber, A. Lindmaa, O. A. von Lilienfeld, and R. Armiento, “Machine Learning Energies of 2 Million Elpasolite $(AB{C}_{2}{D}_{6})$ Crystals,” Phys. Rev. Lett. 117, 135502 (2016).
  * Artrith, Urban, and Ceder (2018) N. Artrith, A. Urban, and G. Ceder, “Constructing first-principles phase diagrams of amorphous LixSi using machine-learning-assisted sampling with an evolutionary algorithm,” J. Chem. Phys. 148, 241711 (2018).
  * Gregory (2020) F. Gregory, “MLFF,”  (2020).
  * Bartók and Csányi (2015) A. P. Bartók and G. Csányi, “Gaussian approximation potentials: A brief tutorial introduction,” Int. J. Quantum Chem. 115, 1051–1057 (2015).
  * Bartók, Kondor, and Csányi (2013) A. P. Bartók, R. Kondor, and G. Csányi, “On representing chemical environments,” Phys. Rev. B 87, 1–19 (2013).
  * Ward and Jr. (1963) J. H. Ward and J. H. W. Jr., “Hierarchical Grouping to Optimize an Objective Function,” J. Am. Stat. Assoc. 58, 236–244 (1963).
  * Sculley (2010) D. Sculley, “Web-Scale k-Means Clustering,” in  _Proceedings of the 19th International Conference on World Wide Web_, WWW ’10 (Association for Computing Machinery, New York, NY, USA, 2010) pp. 1177–1178.
  * Arthur and Vassilvitskii (2007) D. Arthur and S. Vassilvitskii, “K-Means++: The Advantages of Careful Seeding,” in  _Proc. of the Annu. ACM-SIAM Symp. on Discrete Algorithms_, Vol. 8 (2007) pp. 1027–1035.
  * Chmiela  _et al._ (2017) S. Chmiela, A. Tkatchenko, H. E. Sauceda, I. Poltavsky, K. T. Schütt, and K.-R. Müller, “Machine learning of accurate energy-conserving molecular force fields,” Sci. Adv. 3, e1603015 (2017).
  * Blum  _et al._ (2009) V. Blum, R. Gehrke, F. Hanke, P. Havu, V. Havu, X. Ren, K. Reuter, and M. Scheffler, “Ab initio molecular simulations with numeric atom-centered orbitals,” Comput. Phys. Commun. 180, 2175–2196 (2009).
  * Kapil  _et al._ (2019) V. Kapil, M. Rossi, O. Marsalek, R. Petraglia, Y. Litman, T. Spura, B. Cheng, A. Cuzzocrea, R. H. Meißner, D. M. Wilkins, B. A. Helfrecht, P. Juda, S. P. Bienvenue, W. Fang, J. Kessler, I. Poltavsky, S. Vandenbrande, J. Wieme, C. Corminboeuf, T. D. Kühne, D. E. Manolopoulos, T. E. Markland, J. O. Richardson, A. Tkatchenko, G. A. Tribello, V. Van Speybroeck, and M. Ceriotti, “i-PI 2.0: A universal force engine for advanced molecular simulations,” Comput. Phys. Commun. 236, 214–223 (2019).
  * Perdew, Burke, and Ernzerhof (1996) J. P. Perdew, K. Burke, and M. Ernzerhof, “Generalized Gradient Approximation Made Simple,” Phys. Rev. Lett. 77, 3865–3868 (1996).
  * Ambrosetti  _et al._ (2014) A. Ambrosetti, A. M. Reilly, R. A. DiStasio, and A. Tkatchenko, “Long-range correlation energy calculated from coupled atomic response functions,” J. Chem. Phys. 140, 18A508 (2014).
  * Tkatchenko  _et al._ (2012) A. Tkatchenko, R. A. DiStasio, R. Car, and M. Scheffler, “Accurate and Efficient Method for Many-Body van der Waals Interactions,” Phys. Rev. Lett. 108, 236402 (2012).


11institutetext: Université Paris-Est Créteil (UPEC), LISSI, Vitry sur Seine,
94400, France
11email<EMAIL_ADDRESS>
$\spadesuit$ Corresponding author: 11email<EMAIL_ADDRESS>
# An Ambient Intelligence-based Approach For Longitudinal Monitoring of Verbal
and Vocal Depression Symptoms

Alice Othmani$\spadesuit$ 1122  0000-0002-3442-0578 Muhammad Muzammel 11

###### Abstract

Automatic speech recognition (ASR) technology can aid in the detection,
monitoring, and assessment of depressive symptoms in individuals. ASR systems
have been used as a tool to analyze speech patterns and characteristics that
are indicative of depression. Depression affects not only a person’s mood but
also their speech patterns. Individuals with depression may exhibit changes in
speech, such as slower speech rate, longer pauses, reduced pitch variability,
and decreased overall speech fluency. Despite the growing use of machine
learning in diagnosing depression, there is a lack of studies addressing the
issue of relapse. Furthermore, previous research on relapse prediction has
primarily focused on clinical variables and has not taken into account other
factors such as verbal and non-verbal cues. Another major challenge in
depression relapse research is the scarcity of publicly available datasets. To
overcome these issues, we propose a one-shot learning framework for detecting
depression relapse from speech. We define depression relapse as the similarity
between the speech audio and textual encoding of a subject and that of a
depressed individual. To detect depression relapse based on this definition,
we employ a Siamese neural network that models the similarity between of two
instances. Our proposed approach shows promising results and represents a new
advancement in the field of automatic depression relapse detection and mental
disorders monitoring.

###### Keywords:

Ambient Intelligence Automatic speech recognition (ASR) one-shot learning
depression relapse clinical depression.

##  1 Introduction

Major Depressive Disorder (MDD) is a mood disorder that has detrimental
effects on an individual’s cognition, emotions, and daily functioning. It is
primarily characterized by persistent feelings of sadness, anger, and loss of
interest in activities. MDD is among the most prevalent mental disorder,
impacting over 300 million people globally [14]. Furthermore, recent studies
have indicated a significant rise in mental health issues, including anxiety,
stress, and depression, during the COVID-19 pandemic [27].

Depression relapse refers to the re-occurrence of depressive symptoms after a
period of partial or complete remission. It means that a person who previously
experienced an episode of depression and showed improvement or recovery from
their symptoms subsequently experiences a return or worsening of those
symptoms [16]. Relapse can happen during or after treatment for depression,
and it is often characterized by a recurrence of the emotional, cognitive, and
behavioral manifestations associated with depression. Relapse and recurrence
rates are high in MDD patients with a percentage of $60\%$ after 5 years,
$67\%$ after 10, and $85\%$ after 15 [10]. Thus, there is a pressing need for
automatic monitoring systems which can be employed to detect depression
relapse or recurrence at an early stage, thereby facilitating timely
intervention.

Despite the increasing utilization of machine learning (ML) for analyzing
Major Depressive Disorder (MDD) and other mental health issues, and its
potential to enhance decision-making for mental health practitioners, [19, 6,
7, 28], there is a noticeable lack of studies utilizing ML to address the
issue of depression relapse. Additionally, previous research on relapse
prediction primarily relies on clinical variables such as age, gender,
medication types, number of episodes, symptom severity, cognitive markers, and
medical image data [1, 4, 26, 25]. However, these approaches have overlooked
other important attributes, such as the analysis of speech patterns and facial
expressions.

On the other hand, a big limitation to depression research studies is the lack
of public datasets. Moreover, due to privacy, safety, expense, and ethical
concerns [29], acquiring examples to train a model for depression relapse is a
difficult task. This motivates research for approaches that take into account
the data scarcity problem in this area. Few-shot learning is a subfield of
machine learning that deals with the challenge of learning new concepts or
tasks with limited labeled training data. In traditional machine learning
approaches, a large amount of labeled data is typically required to train
models effectively. However, in few-shot learning, the goal is to develop
algorithms that can generalize and learn from only a few examples or instances
of a particular class or task.

In this paper, we propose a robust approach that deals with depression relapse
data scarcity. The proposed approach is based on one-shot learning. We define
depression relapse as the closeness of the speech encoding of a subject to
that of a depressed subject. By modeling the similarity between two instances,
Siamese neural network is chosen as a suitable candidate for depression
relapse detection following the proposed definition. The proposed approach
investigates the predictive power of audio and textual encodings of the speech
for depression relapse prediction using Siamese neural network architecture.
Three Siamese networks built using (1) MFCC audio features, (2) VGGish audio
features, and (3) audio-textual fusion features are compared and investigated
for the relapse identification task. To our knowledge, this work is the first
to tackle the prediction of depression relapse based on verbal and non-verbal
cues in a speech, and to employ one-shot learning for this task.

The paper is structured as follows. In the following section, a literature
review is conducted to identify major previous works done to predict
depression relapse and recurrence. An overview of n-shot learning and Siamese
networks is also included (section 2). In section 3, the different steps of
our methodology are described. Finally, the obtained results and discussion of
the performed tests are presented in section 4.

##  2 Related work

Relapse and recurrence can be attributed to various factors. It is suggested
that recurrence is ascribed to a genetic vulnerability [2]. Likewise, some
findings suggest that inflammation can issue a possible mechanism for
recurrent depression [13]. Another factor can be neuropsychological
functioning [25]. Factors triggering depression relapse may vary among
individuals. Moreover, in many cases it is difficult to underpin the trigger
factors leading to a depression relapse episode. This motivates the
development of automatic monitoring systems capable of detecting relapse.

Researcher for predicting depression relapse often relies on n clinical
variables such as medication type, episode number, symptom severity, cognitive
markers, and occasionally medical imaging data. For example, Borges et al.
(2018) [1] have explored the use of machine learning algorithms in predicting
relapse in bipolar patients based on their clinical data, with Random Forests
showing promising results (68% for the Relapse Group and 74% for the No
Relapse Group). Another study by Chanda et al. (2020) [4] aimed to develop a
recurrence classification platform based on gender, age, medication, and
treatment time, where K-Nearest Neighbor outperformed SVM and RF with an 83%
accuracy. Emotional biases were investigated investigated by Ruhe et al.
(2019) [25], as potential biomarkers for depression relapse, resulting in a
linear SVM model with a 75% accuracy. Moreover, a multimodal approach proposed
by Cearns et al. (2019) [3] incorporating various modalities, including
clinical, blood-biomarker, genetic, bio-electrical impedance,
electrocardiography, and structural imaging, achieved an accuracy of 65.72%
using SVM. Further, a holistic approach involving the analysis of depression
scores trajectories and predicting individual treatment outcomes was proposed
using smoothing splines, K-means clustering, and collaborative modeling [12].

Just recently, three research studies have emerged that propose utilizing
video data for the prediction of depression relapse [22, 17, 21]. In [22]
along with CNN, a Model of Normality (MoN) was utilize to detect depression
and relapse. While, the other approach [17] proposed a preliminary study based
one-shot learning due to capacity to learn instantly. Both of these state of
the art modalities relies on audio and visual features for monitoring
depression relapse. In [21], a deep learning-based approach is proposed for
depression recognition and depression relapse prediction using videos of
clinical interviews. Their approach is based on a correlation-based anomaly
detection framework and a measure of similarity to depression where depression
relapse is detected when the deep audiovisual patterns of a depression-free
subject become close to the deep audiovisual patterns of depressed subjects.
Thus, the correlation between the audiovisual encoding of a test subject and a
deep audiovisual representation of depression is computed and used for
monitoring depressed subjects and for predicting relapse after depression.

##  3 Methodology

Depression relapse is defined in this work as the similarity (dissimilarity)
between the audio-textual speech encoding of a subject and the speech encoding
of a diagnosed depressed (non-depressed) subject. One-shot learning based
Siamese network is chosen in this work for modeling depression relapse, as it
models the similarity (dissimilarity) between two samples. The proposed
framework is composed of four stages: (1) pre-processing audio data
augmentation (section 3.1), (3) audio-textual features extraction (section
3.2), and (4) one-shot learning-based depression relapse detection (section
3.3). Each of these steps are detailed in the following.

Figure 1: Proposed multimodal-based Siamese networks for depression relapse
detection.

###  3.1 Pre-processing & Data Augmentation

####  3.1.1 Audio Preprocessing

In the pre-processing step, unvoiced segments are first filtered and the
speech of the subject is extracted from the audio signal. The speech signal is
then divided into $n=7.6$ seconds speech segments. To increase the number of
data samples and the system’s robustness to noise, data augmentation is
performed. The signals are perturbed using two audio augmentation techniques
[23, 18, 20]:

1) Noise Injection: the audio signal is perturbed through the injection of
random noise. Let $x$ be an audio signal and $\alpha$ a noise factor, the
noise perturbed signal $x_{N}$ is given by:
$x_{N}=x-{{\alpha}\times{rand(x)}}$, with $\alpha$ = $0.01$, $0.02$ and
$0.03$.

2) Pitch Augmentation: pitch is lowered by factors of $0.5$, $2$, and $2.5$
(in semitones).

####  3.1.2 Text Preprocessing

The clinical interviews are recorded and accompanied by transcriptions of the
conversations between the participant and the interviewer. Unlike the audio
data, we analyze both the transcriptions of the interviewer and the
participant. This is because the verbal reactions of the interviewer following
the participant’s responses can contain valuable information about the
participant’s emotions. For example, when the participant responds negatively,
the interviewer may express phrases like ”that sucks” or ”I’m sorry to hear
that,” which provide insight into the participant’s depressive state. Our
decision to focus solely on the audio patterns of the participant is based on
the fact that the interviewer’s audio patterns do not exhibit signs of
depression. However, the words used by the interviewer can indicate sympathy
when the patient is depressed.

###  3.2 Audio-Textual Features Extraction

Following data augmentation, audio-textual features are extracted. These
constitute MFCC and Vggish audio features, and textual word2Vec features.

####  3.2.1 MFCC Features Extraction from Audio Signal

MFCC features represent the audio cepstrum in the non-linear Mel scale. Such
representation is said to approximate the human auditory system. Below is the
detailed description of the MFCC feature extraction steps.

Windowing: The signal is split into $60$ milliseconds frames. A Hamming window
is then applied on each frame to taper the signal towards the frame
boundaries. Given a signal $s[n]$ of length $n$ and a hamming window $w[n]$,
the sliced frame is given by:

$x[n]=w[n]s[n]\quad with\quad
w[n]=\alpha-\beta\cos\left(2\pi\frac{n}{N-1}\right)$ (1)

where $\alpha=0.54$, $\beta=0.46$, $N$ is the window length such that $0\leq
n\leq N$.

DFT spectrum: Discrete Fourier Transform (DFT) is then performed on each
windowed frame to get the magnitude spectrum:

$X[k]=\sum_{n=0}^{N-1}{x[n]}e^{-j\frac{2\pi}{N}kn};\quad 0\leq k\leq N-1$ (2)

Mel spectrum: Triangular Mel-scale filter banks are then multiplied by the
magnitude spectrum to compute the Mel spectrum.

$Y_{t}[m]=\sum_{k=0}^{N-1}{W_{m}[k]|X_{t}[k]|^{2}};\quad 0\leq m\leq M-1$ (3)

where $W_{m}$ represents the $m^{th}$ triangular Mel-scale filter bank, $M$ is
total number of filters, and $k$ corresponds to the DFT bin number.

Discrete cosine transform (DCT): The Mel spectrum is then represented on a log
scale and DCT is applied generating a set of cepstral coefficients. MFCC is
thus calculated as:

$c[n]=\sum_{m=0}^{M-1}{log_{10}\left(Y_{t}[m]\right)cos\left(n(m-0.5)\frac{\pi}{M}\right)}$
(4)

For this work, 60 dimensional MFCC features are extracted leading to a matrix
of $378\times 60$ for each $7.6$ seconds signal.

####  3.2.2 Vggish Features Extraction from Audio signal

In this work, VGGish features [11] are extracted from the audio segments.
VGGish converts audio features into semantically significant compact high-
level 128 dimensional embedding. This embedding can be then fed to a shallow
or deep classification model.

To compute the VGGish features, first a one-sided Short-Time Fourier Transform
(STFT) is applied to the audio clip using a 25 ms periodic Hann window with 10
ms hop, and 512-point DFT. The complex spectral values are then converted to
magnitude and phase information is discarded. The one-sided magnitude spectrum
are passed to a 64-band mel-spaced filter bank, and the magnitudes in each
band are then summed. The obtained mel spectrograms are then converted into a
log scale and buffered into overlapped segments consisting of 96 spectrums
each. These mel spectrograms are passed through VGGish network to obtain a
14x128 matrix [8, 11].

The VGGish feature matrix contained zero values. To resolve this issue, the
VGGish model was released with a precomputed principal component analysis
(PCA) matrix [8, 11]. First, we subtract the precomputed 1x128 PCA mean vector
from the 14x128 feature matrix, and then premultiply the result by the
precomputed 128x128 PCA matrix.

####  3.2.3 Textual Features Extraction

For each audio segment ($7.6sec$) of the clinical interviews, the words spoken
by both the participant and the interviewer are converted into sequences of
vectors using word embedding as a textual features. Word embedding is a
technique that transforms words into dimensional vectors, where words with
similar meanings or words that frequently occur together in context are
represented by vectors that are close to each other in the dimensional space.
In particular, each frame transcript is represented by a matrix
$E=(e_{1},\dots,e_{k},\dots e_{nw})$ where $e_{k}$ is the word vector
corresponding to the $k^{th}$ word and $nw$ is the number of words in the
frame transcript.

For word embedding, we utilize the fastText pretrained network [15], which was
trained on Common Crawl 111https://commoncrawl.org/2017/06 and incorporates
sub-word information. This network produces word vectors of size $300$. In
cases where certain words are not present in the pretrained model, we replace
them with their synonyms. Additionally, the resulting word vector matrix for
each transcript frame is resized to $60\times 9$, where $60$ represents the
size of MFCC coefficients and $9$ represents the minimum number of words
present in a single frame.

###  3.3 One-Shot Learning Framework for Depression Relapse Detection

In this work, 1D convolutional Siamese networks are proposed for depression
relapse detection based on audio-textual cues. The proposed architectures are
summarized in Fig. 1.

####  3.3.1 One-shot learning

One-shot learning refers to the classification task where the number of
available instances per class is limited. In some cases, a class may have only
a single example during the model training phase. The objective of one-shot
learning is to train models that can determine the similarity between a pair
of inputs and identify whether they belong to the same class or not.

Siamese neural networks, also known as twin neural networks, are a type of
one-shot learning model that aim to learn a distance function between pairs of
input vectors. In the architecture of a Siamese neural network, two identical
sub-networks with shared weights are employed. Each sub-network takes in a
distinct pair of inputs and produces output representations. The network then
computes a distance metric, such as Euclidean distance or cosine similarity,
between the outputs of the sub-networks. This distance metric reflects the
similarity between the two input vectors [5]. The training can be done on
different pairs of inputs from all possible classes. To obtain a prediction
for an instance, a comparison between the instance and reference instances of
all classes should be performed. The final prediction is then obtained based
on the different similarity scores.

For training a Siamese network, dataset pre-processing is required. A pairing
procedure is performed where pairs of data points are created: similar samples
pair, and dissimilar samples pair. Similar samples are then assigned positive
labels, while dissimilar samples are assigned negative labels. The pairs are
then fed to the Siamese architecture. The pairing procedure employed in this
paper is detailed in section 4.1.

####  3.3.2 Proposed Siamese Architectures

We propose two Siamese networks built using (1) MFCC audio features, (2)
VGGish audio features, and (3) audio-textual fusion features.

Audio based Siamese Network: The proposed MFCC and VGGish Siamese models have
similar architectures. The architecture consists of two blocks of
convolutional, Relu, dropout, and flatten layers. Two fully connected layers
are used to compute encodings and an euclidean distance is measured between
the two encodings. In other words, both Net-1 (A and B) from Fig. 1 are
connected with Net-3 through the fully connected layers. A last layer of size
two is added with a sigmoid activation function for binary prediction of
relapse.

Audio-Textual based Siamese Network: In Fig. 1, the 60 MFCC features are fed
to a Convolutional Neural Network composed of two blocks of 1D convolutional
and Relu layers, followed by dropout and flatten layers. Similarly 14 VGGish
features are fed to a similar Convolutional Neural Network consisting of two
blocks of 1D convolutional and Relu layers, followed by dropout and flatten
layers. These high level MFCC and VGGish features are then concatenated with
the textual features into a features vector of size $540$ and fed to two fully
connected layers to obtain the audio-textual encoding. Afterwords, an
euclidean distance is computed between the two encodings: one of non-diagnosed
subject and one of a reference depressed subject. A last layer of size two is
also added for binary prediction of relapse.

##  4 Results and Discussion

Predicted |  | Actual |
---|---|---|---
| NS | S | total
NS |  356 (33.46) |  187 (17.58) |  65.56 34.44
S |  176 (16.54) |  345 (32.42) |  66.22 33.78
total |  66.92 33.08 |  64.85 35.15 |
(a)

|  | Actual |
---|---|---|---
| NS | S | total
NS |  358 (33.65) |  183 (17.20) |  66.17 33.83
S |  174 (16.35) |  349 (32.80) |  66.73 33.27
|  67.29 32.71 |  65.60 34.40 |
(b)

|  | Actual |
---|---|---|---
| NS | S | total
NS |  404 (37.97) |  158 (14.85) |  71.89 28.11
S |  128 (12.03) |  374 (35.15) |  74.50 25.50
|  75.94 24.06 |  70.30 29.70 |
(c)

Figure 2: Confusion matrices of Siamese networks for depressive state
similarity detection. NS: Non-Similar, S: Similar.

###  4.1 Dataset

We use the Distress Analysis Interview Corpus Wizard-of-Oz dataset (DAIC- WOZ)
([9]). This dataset was introduced in AVEC 2017 challenge [24]. $189$ subjects
took part in the data collection, where they were interviewed by a virtual
agent manipulated in a wizard-of-oz setting. The subject’s speech and
interviews transcripts were saved and publicly made available for research.
The average length of the audio samples is $15$ minutes, sampled at $16kHz$.
The dataset includes depression scores in two formats: binary and severity
level. Binary depression and severity labels were collected via a self-report
Patient Health Questionnaire (PHQ) [30]. Only $182$ subjects were used in the
evaluation. We use a percentage split strategy where the dataset is randomly
divided into $80\%$ training, $10\%$ validation, and $10\%$ testing. Out of
the $182$ subjects, the training dataset included $146$ subjects, while the
validation and test sets included $18$ subjects each.

Pairing Features pairs are then generated to train, test and validate the one-
shot learning model. Samples from the train set are randomly paired with each
others to form the training pairs. No sample was paired with itself. To
generate the test and validation pairs, samples from these sets are paired
with the samples from the training set. The binary depression score provided
in DAIC-WOZ consists of two classes: depressed and non-depressed. Binary
depression relapse is defined to be the paired feature groups of the couple
(Non-Depressed, Depressed). Also, in both classes (i.e., Depressed, Non-
Depressed) 50% features matrix groups are paired with the same class, while
the remaining $50\%$ features matrix groups are paired with the other class.
Also pairs are created for PhQ-score provided in DAIC-WOZ dataset. For all 24
classes $50\%$ features matrix groups are paired with the same class, while
the remaining $50\%$ features matrix groups are paired with the other classes.

###  4.2 Network Implementation Details

A ReLU activation function with $64$ filters is used for the convolutional
layers in the presented CNN architectures. The stride and filter sizes are set
to $1$ and $3$, respectively. The dropout fraction value is set to $0.01\%$.
The dense layers’ sizes are set to $1024$ and hyperbolic tangent (tanh) is
used as an activation function. To predict the similarity between two feature
sets for depression relapse detection based on PHQ-binary, the output layer is
a dense layer with a sigmoid activation function and a size of $2$. To predict
the similarity based on PHQ-score pairs, the output dense layer has a size of
$25$. To train the models, an initial learning rate of $10^{-5}$ and a decay
of $10^{-6}$ is used. The batch size and epochs are set to $100$ and $300$,
respectively. For both models, Root Mean Square Error is used as a loss
function and trained with RMSProp optimizer. An early stopping is used if the
loss function stops decreasing after 10 epochs.

Table 1: Performance of proposed Siamese networks for depressive state similarity in terms of accuracy, Root Mean Square Error (RMSE) and CC Network | Acc. (%) | RMSE | CC
---|---|---|---
MFCC Siamese | 65.88 | 0.4841 | 0.3177
VGGish Siamese | 66.45 | 0.4792 | 0.3290
Audio-Textual Siamese | 73.12 | 0.4585 | 0.4631

###  4.3 Performance Analysis of Siamese Networks

The proposed Siamese networks are evaluated using accuracy, Root Mean Square
Error (RMSE), and Pearson Correlation Coefficient (CC) metrics. The proposed
framework reaches an accuracy of $65.88\%$ and $66.45\%$ when using only the
MFCC features and VGGish features, respectively. The fusion of MFCC, VGGish
and textual features notably increases the performance where an accuracy of
$73.12\%$ is obtained. A minor decrease in RMSE and a minor increase in CC is
noted for the VGGish model compared to the MFCC based one. Further, the CC
value of MFCC and VGGish based Siamese networks are $0.3177$ and $0.3290$,
which increases to $0.4631$ with the fusion of audio and textual features.

Fig. 2 shows the confusion matrices of MFCC, VGGish and Audio-textual based
Siamese networks. From the figure, one can notice that the fusion of textual
features with audio features network adequately improves the performance of
one shot learning Siamese network. For non-similar feature pairs
classification, the audio-textual based fusion network achieved $9.02\%$ and
$8.65\%$ better results compared to MFCC and VGGish based networks. Also for
non-similar feature pairs, a notable decrement in false positives has been
reported for audio-textual based fusion network.

For similar feature pairs (i.e., when both features sets in pair belong to the
same class) classification the Audio-Textual based fusion network obtained
$5.45\%$ and $4.70\%$ better results compared to MFCC and VGGish based
networks. Also a considerable decrement in false positives has been reported
for Audio-Textual based fusion network compared to MFCC and VGGish based
Siamese networks. Furthermore, We also investigate the pair matching for PhQ-
Score using multi-class Audio-Textual based Siamese network and we obtained an
RMSE value of 4.025 (Normalized RMSE of 0.161).

##  5 Conclusion and Future Work

In this work, an ASR framework is proposed for depression relapse detection,
modeling the similarity of audio and textual speech encoding between a new
subject and a diagnosed depressed subject using one-shot learning. The
proposed model gave reliable results using the speech’s audio and textual
cues. The fusion of audio and textual features enhanced the one-shot learning
model performance, which made it reliable for detecting depression relapse.
Further, the proposed ASR system could help depression patients to monitor
their recovery. Lastly, in future work we plan to consider also visual cues in
the proposed framework.

##  6 ACKNOWLEDGMENTS

This work is funded under grant number IF040-2021 (MATCH2021: Malaysia France
Bilateral Research Grant).

## References

  * [1] Borges-Júnior, R., Salvini, R., Nierenberg, A.A., Sachs, G.S., Lafer, B., Dias, R.S.: Forecasting depressive relapse in bipolar disorder from clinical data. In: 2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). pp. 613–616. IEEE (2018)
  * [2] Burcusa, S.L., Iacono, W.G.: Risk for recurrence in depression. Clinical psychology review 27(8), 959–985 (2007)
  * [3] Cearns, M., Opel, N., Clark, S., Kaehler, C., Thalamuthu, A., Heindel, W., Winter, T., Teismann, H., Minnerup, H., Dannlowski, U., et al.: Predicting rehospitalization within 2 years of initial patient admission for a major depressive episode: a multimodal machine learning approach. Translational psychiatry 9(1), 1–9 (2019)
  * [4] Chanda, K., Bhattacharjee, P., Roy, S., Biswas, S.: Intelligent data prognosis of recurrent of depression in medical diagnosis. In: 2020 8th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions)(ICRITO). pp. 840–844. IEEE (2020)
  * [5] Chicco, D.: Siamese neural networks: An overview pp. 73–94
  * [6] Dwyer, D.B., Falkai, P., Koutsouleris, N.: Machine learning approaches for clinical psychology and psychiatry. Annual review of clinical psychology 14, 91–118 (2018)
  * [7] Gao, S., Calhoun, V.D., Sui, J.: Machine learning in major depression: From classification to treatment outcome prediction. CNS neuroscience & therapeutics 24(11), 1037–1052 (2018)
  * [8] Gemmeke, J.F., Ellis, D.P., Freedman, D., Jansen, A., Lawrence, W., Moore, R.C., Plakal, M., Ritter, M.: Audio set: An ontology and human-labeled dataset for audio events. In: 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 776–780. IEEE (2017)
  * [9] Gratch, J., Artstein, R., Lucas, G.M., Stratou, G., Scherer, S., Nazarian, A., Wood, R., Boberg, J., DeVault, D., Marsella, S., et al.: The distress analysis interview corpus of human and computer interviews. In: Proceedings of the International Conference on Language Resources and Evaluation. pp. 3123–3128 (2014)
  * [10] Hardeveld, F., Spijker, J., De Graaf, R., Nolen, W., Beekman, A.: Prevalence and predictors of recurrence of major depressive disorder in the adult population. Acta Psychiatrica Scandinavica 122(3), 184–191 (2010)
  * [11] Hershey, S., Chaudhuri, S., Ellis, D.P., Gemmeke, J.F., Jansen, A., Moore, R.C., Plakal, M., Platt, D., Saurous, R.A., Seybold, B., et al.: Cnn architectures for large-scale audio classification. In: 2017 ieee international conference on acoustics, speech and signal processing (icassp). pp. 131–135. IEEE (2017)
  * [12] Lin, Y., Huang, S., Simon, G.E., Liu, S.: Analysis of depression trajectory patterns using collaborative learning. Mathematical biosciences 282, 191–203 (2016)
  * [13] Liu, C.H., Zhang, G.Z., Li, B., Li, M., Woelfer, M., Walter, M., Wang, L.: Role of inflammation in depression relapse. Journal of neuroinflammation 16(1), 1–11 (2019)
  * [14] Marcus, M., Yasamy, M.T., van van Ommeren, M., Chisholm, D., Saxena, S.: Depression: A global public health concern (2012). https://doi.org/10.1037/e517532013-004, https://doi.org/10.1037/e517532013-004
  * [15] Mikolov, T., Grave, E., Bojanowski, P., Puhrsch, C., Joulin, A.: Advances in pre-training distributed word representations. In: Proceedings of the International Conference on Language Resources and Evaluation (2018)
  * [16] Monroe, S.M., Harkness, K.L.: Recurrence in major depression: a conceptual analysis. Psychological review 118(4), 655 (2011)
  * [17] Muzammel, M., Othmani, A., Mukherjee, H., Salam, H.: Identification of signs of depression relapse using audio-visual cues: A preliminary study. In: 2021 IEEE 34th International Symposium on Computer-Based Medical Systems (CBMS). pp. 62–67. IEEE (2021)
  * [18] Muzammel, M., Salam, H., Hoffmann, Y., Chetouani, M., Othmani, A.: Audvowelconsnet: A phoneme-level based deep cnn architecture for clinical depression diagnosis. Machine Learning with Applications 2, 100005 (2020)
  * [19] Muzammel, M., Salam, H., Othmani, A.: End-to-end multimodal clinical depression recognition using deep neural networks: A comparative analysis. Computer Methods and Programs in Biomedicine 211, 106433 (2021)
  * [20] Othmani, A., Kadoch, D., Bentounes, K., Rejaibi, E., Alfred, R., Hadid, A.: Towards robust deep neural networks for affect and depression recognition from speech. In: ICPR International Workshops and Challenges. ICPR 2021. Lecture Notes in Computer Science. vol. 12662 (2021)
  * [21] Othmani, A., Zeghina, A.O.: A multimodal computer-aided diagnostic system for depression relapse prediction using audiovisual cues: A proof of concept. Healthcare Analytics 2, 100090 (2022)
  * [22] Othmani, A., Zeghina, A.O., Muzammel, M.: A model of normality inspired deep learning framework for depression relapse prediction using audiovisual data. Computer Methods and Programs in Biomedicine 226, 107132 (2022)
  * [23] Rejaibi, E., Komaty, A., Meriaudeau, F., Agrebi, S., Othmani, A.: Mfcc-based recurrent neural network for automatic clinical depression recognition and assessment from speech. arXiv preprint arXiv:1909.07208 (2019)
  * [24] Ringeval, F., Schuller, B., Valstar, M., Gratch, J., Cowie, R., Scherer, S., Mozgai, S., Cummins, N., Schmitt, M., Pantic, M.: Avec 2017: Real-life depression, and affect recognition workshop and challenge. In: Proceedings of the 7th Annual Workshop on Audio/Visual Emotion Challenge. pp. 3–9. ACM (2017)
  * [25] Ruhe, H.G., Mocking, R.J., Figueroa, C.A., Seeverens, P.W., Ikani, N., Tyborowska, A., Browning, M., Vrijsen, J.N., Harmer, C.J., Schene, A.H.: Emotional biases and recurrence in major depressive disorder. results of 2.5 years follow-up of drug-free cohort vulnerable for recurrence. Frontiers in psychiatry 10, 145 (2019)
  * [26] Sato, J.R., Moll, J., Green, S., Deakin, J.F., Thomaz, C.E., Zahn, R.: Machine learning algorithm accurately detects fmri signature of vulnerability to major depression. Psychiatry Research: Neuroimaging 233(2), 289–291 (2015)
  * [27] Shah, S.M.A., Mohammad, D., Qureshi, M.F.H., Abbas, M.Z., Aleem, S.: Prevalence, psychological responses and associated correlates of depression, anxiety and stress in a global population, during the coronavirus disease (covid-19) pandemic. Community mental health journal 57(1), 101–110 (2021)
  * [28] Su, C., Xu, Z., Pathak, J., Wang, F.: Deep learning in mental health outcome research: a scoping review. Translational Psychiatry 10(1), 1–26 (2020)
  * [29] Thelisson, E., Sharma, K., Salam, H., Dignum, V.: The general data protection regulation: An opportunity for the hci community? In: Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems. pp. 1–8 (2018)
  * [30] Williams, L.S., Brizendine, E.J., Plue, L., Bakas, T., Tu, W., Hendrie, H., Kroenke, K.: Performance of the phq-9 as a screening tool for depression after stroke. stroke 36(3), 635–638 (2005)