# Automated voice- and text-based classification of neuropsychiatric
conditions in a multidiagnostic setting

Lasse Hansen<EMAIL_ADDRESS>Department of Affective Disorders, Aarhus
University Hospital - Psychiatry, Aarhus, Denmark Department of Clinical
Medicine, Aarhus University, Aarhus, Denmark  Center for Humanities Computing,
Aarhus University, Aarhus, Denmark  Roberta Rocca  Interacting Minds Centre,
Aarhus University, Aarhus, Denmark  Arndis Simonsen  Interacting Minds Centre,
Aarhus University, Aarhus, Denmark  Psychosis Research Unit, Aarhus University
Hospital - Psychiatry, Aarhus, Denmark  Alberto Parola  Interacting Minds
Centre, Aarhus University, Aarhus, Denmark  Department of Linguistics,
Cognitive Science and Semiotics, Aarhus University, Aarhus, Denmark  Vibeke
Bliksted  Department of Clinical Medicine, Aarhus University, Aarhus, Denmark
Interacting Minds Centre, Aarhus University, Aarhus, Denmark  Psychosis
Research Unit, Aarhus University Hospital - Psychiatry, Aarhus, Denmark
Nicolai Ladegaard  Department of Affective Disorders, Aarhus University
Hospital - Psychiatry, Aarhus, Denmark Department of Clinical Medicine, Aarhus
University, Aarhus, Denmark  Dan Bang  Wellcome Centre for Human Neuroimaging,
University College London, WC1N 3BG London, United Kingdom  Department of
Experimental Psychology, University of Oxford, OX2 6GG Oxford, United Kingdom
Center of Functionally Integrative Neuroscience, Aarhus University, Aarhus,
Denmark  Kristian Tylén  Interacting Minds Centre, Aarhus University, Aarhus,
Denmark  Department of Linguistics, Cognitive Science and Semiotics, Aarhus
University, Aarhus, Denmark  Ethan Weed  Interacting Minds Centre, Aarhus
University, Aarhus, Denmark  Department of Linguistics, Cognitive Science and
Semiotics, Aarhus University, Aarhus, Denmark  Søren Dinesen Østergaard
Department of Affective Disorders, Aarhus University Hospital - Psychiatry,
Aarhus, Denmark Department of Clinical Medicine, Aarhus University, Aarhus,
Denmark  Riccardo Fusaroli  Interacting Minds Centre, Aarhus University,
Aarhus, Denmark  Department of Linguistics, Cognitive Science and Semiotics,
Aarhus University, Aarhus, Denmark  Linguistic Data Consortium, University of
Pennsylvania, Philadelphia, Pennsylvania, USA

Abstract: Speech patterns have been identified as potential diagnostic markers
for neuropsychiatric conditions. However, most studies only compare a single
clinical group to healthy controls, whereas clinical practice often requires
differentiating between multiple potential diagnoses (multiclass setting). To
address this, we assembled a dataset of repeated recordings from 420
participants (67 with major depressive disorder (MDD), 106 with schizophrenia
and 46 with autism spectrum disorder (ASD), as well as matched controls), and
tested the performance of conventional machine learning models and advanced
Transformer models on both binary and multiclass classification, based on
voice and text features.

While binary models performed comparably to previous research (F1 scores
between 0.54-0.75 for ASD; 0.67-0.92 for MDD; and 0.71-0.83 for
schizophrenia); when differentiating between multiple diagnostic groups
performance decreased markedly (F1 scores between 0.35-0.44 for ASD, 0.57-0.75
for MDD, 0.15-0.66 for schizophrenia, and 0.38-0.52 macro F1). Combining voice
and text-based models increased performance, suggesting that they capture
complementary diagnostic information.

Our results indicate that models trained on binary classification might rely
on markers of generic differences between clinical and non-clinical
populations, or markers of clinical features that overlap across conditions,
rather than identifying markers specific to individual conditions. We provide
recommendations for future research in the field, suggesting increased focus
on developing larger transdiagnostic datasets that include more fine-grained
clinical features, and that can support the development of models that better
capture the complexity of neuropsychiatric conditions and naturalistic
diagnostic assessment.

##  1 Introduction

Machine learning is increasingly used to identify voice and text-based markers
that complement current methods for screening and monitoring neuropsychiatric
conditions [1, 2]. However, while clinical assessment often requires the
identification of one diagnosis amongst many with partially overlapping
clinical features, virtually all machine learning studies of voice and text-
based markers focus on simpler binary outcomes, i.e., patients with one
specific diagnosis versus controls without any neuropsychiatric conditions [3,
4, 5, 6, 7]. This raises concerns that the markers identified by studies in
one-diagnosis settings may not have diagnostic specificity – instead capturing
general differences between clinical and non-clinical populations.

In this paper, we briefly review the rationale behind using voice and text-
based markers of neuropsychiatric disorders, and the techniques available to
do so (including the most recent deep learning Transformer model). We then
investigate their potential to discriminate between multiple diagnoses at the
same time – a more challenging, but more clinically relevant task than
deciding if a person is best classified with one particular psychiatric
diagnosis or not – using a novel speech dataset including patients with autism
spectrum disorder (ASD), major depressive disorder (MDD) and schizophrenia.

###  1.1 Previous work on vocal and linguistic markers of neuropsychiatric
conditions

Early descriptions of individuals with neuropsychiatric conditions tend to
include remarks about atypical communicative patterns [8, 9]. Individuals with
ASD are often described as producing stereotyped or robotic-like speech [6,
10, 11], individuals with schizophrenia tend to produce more monotone,
fragmented and less coherent speech [12, 13], and depression has been
associated with producing sentences with longer pauses and more negative
affect [14, 5, 15]. Accordingly, variations in what people say and how they
say it are directly or indirectly used as part of the diagnostic criteria for
mood disorders, schizophrenia, and ASD [16, 17].

Traditionally, voice-based classification of neuropsychiatric conditions has
relied on intuitive handcrafted features that were easy to measure, such as
pitch, number and duration of pauses, and response latency [18]. Today,
advances in natural language processing (NLP) and speech processing
methodologies allow for largely automated extraction of a broader range of
features, such as formants, energy, and spectral and cepstral descriptors
(e.g., mel-frequency cepstral coefficients (MFCCs)) from audio, relying on
standard software libraries such as OpenSMILE and Covarep [19, 20]. Texts and
transcribed speech can also be quantified in terms of features such as
relative frequencies of words, sentiment expressed, and semantic embeddings.
These features are motivated by general research on linguistic production and
clinical observations of specific patient groups (e.g. longer pauses in
individuals with alogia - a symptom of schizophrenia - which can be measured
from voice recordings, and more negative emotional words in individuals with
depression, which can be measured by applying text-based tools to
transcripts). Classifiers relying on these human-defined features have often
reported promising performance in discriminating diagnosed individuals (e.g.,
with schizophrenia) from controls without any neuropsychiatric condition, with
accuracies between 60% and 90% (see [6, 5, 3, 11] for an overview of existing
studies). Furthermore, several studies have sought to combine the two
modalities (speech, text) and found them to contain complementary information
for the classification of neuropsychiatric conditions [21, 1]. This
complementarity may be due to the ability of text models to capture variations
in word usage that are not captured by acoustic models, and the ability of
acoustic models to identify variations in prosody (e.g. tone, pitch, or
rhythm) that are not expressed in text.

Recent approaches to the encoding of text and audio, which use deep neural
networks and self-attention [22], have eschewed traditional feature
engineering [23, 24]. Briefly, self-attention allows the model to selectively
focus on different parts of the input sequence, enabling it to effectively
incorporate contextual information from the entire input into its predictions
[22]. Models like Transformers, which leverage self-attention to produce
encodings of words and acoustic patterns modulated by the context surrounding
them [22], outperform other approaches across fields ranging from automated
speech recognition to natural language understanding [25, 26]. These models
are generally pre-trained on generic language tasks using large amounts of
data, and can be deployed with minimal additional fine-tuning on novel data
and novel predictive tasks. This approach, known as transfer learning, has
been highly successful in many natural language processing tasks [27], but it
has not been widely tested in the context of neuropsychiatric assessments
based on vocal and text markers [7].

###  1.2 Multidiagnostic challenges

Previous attempts at identifying voice and text-based markers of
neuropsychiatric conditions have primarily focused on clear binary contrasts:
given a set of clearly diagnosed individuals, could voice markers discriminate
them from controls with no symptoms or diagnoses [4, 5]? Yet, clinical
practice is much more likely to involve discrimination between multiple
diagnoses with partially overlapping clinical features [16, 17]. This more
clinically relevant task is also likely to be a much more challenging
classification since some vocal markers are shared across conditions (e.g.
decreased pitch variability in both MDD and schizophrenia; or longer pauses in
both ASD and schizophrenia) [18, 12, 5].

###  1.3 Aims of the study

Given the clinical limitations of binary classification, this study aimed to
evaluate the performance of voice- and text-based machine learning models of
neuropsychiatric conditions in multiclass classification settings. Compared to
traditional binary classification tasks, which contrast diagnosed individuals
with matched controls, multiclass classification better mirrors the complexity
of clinical assessment and provides a better understanding of the potential
clinical utility of machine learning models. Therefore, we established a
multidiagnosis dataset consisting of individuals with MDD, ASD and depression
to answer the following research questions:

  * •

How well can we distinguish individuals with MDD, ASD, or schizophrenia from
matched controls based on speech patterns in a binary classification setting?

  * •

How good are traditional machine learning models at identifying diagnosis-
specific speech-based markers? In other words, how does multi-class
performance compare to binary performance?

  * •

How do voice- vs text-based models compare? Does combining the two modalities
yield better predictions?

  * •

How do pre-trained Transformer models perform compared to more traditional
feature-based models?

##  2 Methods

In this study, we trained binary and multi-class models on voice and
transcripts from Danish adult individuals with a diagnosis of schizophrenia,
MDD, or ASD as well as healthy matched controls performing the Animated
Triangles (Frith-Happé) task [28]. We first assessed whether we can achieve
binary classification performance comparable to existing studies. We then
evaluated how well the same architectures performed in a multi-class setup. In
order to extensively assess these issues, we used both voice recordings and
text-based properties of transcribed speech, and contrasted more traditional
machine learning approaches with pre-trained state-of-the-art Transformer
models. The following sections provide details on the data and modelling
strategies.

###  2.1 Pipeline

For training, we split the data into a training set, a validation set, and a
balanced held-out test set. We ensure that the models are trained (fitted) and
optimized on one subset of the data (the training + validation dataset), while
their performance is assessed on a different subset of the data (the held-out
test dataset). Figure 1 provides an overview of the full pipeline, which is
discussed in detail below.

Figure 1: Processing and training pipelines for feature-based text, voice, and ensemble models (see section 2 and Glossary for details on each processing step). For transformer-based text models, we use the same pre- and postprocessing step, while feature extraction and predictive modeling are performed end-to-end within the transformer architecture. Table 1: Overview of demographic and clinical information available for each study. | Autism Spectrum Disorder | Major Depressive Disorder | Schizophrenia
---|---|---|---
| ASD | Controls | MDD | Controls | SCZ | Controls
| (N=46) | (N=41) | (N=67) | (N=42) | (N=106) | (N=118)
Age
Mean (SD) | 28.0 (7.23) | 26.1 (6.42) | 34.5 (11.1) | 33.5 (12.0) | 27.1 (9.79) | 27.1 (9.78)
Sex
Female | 19 (41.3%) | 17 (41.5%) | 50 (74.6%) | 30 (71.4%) | 44 (41.5%) | 50 (42.4%)
Male | 26 (56.5%) | 24 (58.5%) | 15 (22.4%) | 11 (26.2%) | 60 (56.6%) | 66 (55.9%)
Education (years)
Mean (SD) | 12.2 (3.40) | 13.8 (2.82) | 12.7 (2.12) | 12.9 (1.97) | 15.2 (2.72) | 14.9 (2.52)
HDRS-17
Mean (SD) | - | - | 22.5 (3.71) | 1.62 (1.36) | - | -
ADOS
Mean (SD) | 32.5 (7.03) | 13.8 (9.05) | - | - | - | -
Uses antipsychotics
Yes | - | - | - | - | 0 (0%) | 28 (26.4%)
Top-level grouping specifies which study the participants come from.
Note that counts for sex might not add up to the total number of participants
due to incomplete data.
HDRS-17: The total score on the 17-item Hamilton Depression Rating Scale.
ADOS: The total score on the Autism Diagnostic Observation Schedule.

###  2.2 Data

The dataset consists of speech recordings and corresponding word-level
transcriptions from 420 individuals with MDD (N=67), ASD (N=46), and
schizophrenia (N=106) along with controls matched on sex, age and socio-
economic status (N=201). The data were collected in previous studies for other
purposes [29, 30, 31, 32, 33]. For demographic, clinical, and cognitive
information about the participants, see Table 1. See Appendix Glossary for a
description of the original studies.

All participants performed the Animated Triangles task [28]: they watched 8-10
short videos of geometric figures moving on the screen and after each video
produced a description of it (for more details on the task, see Appendix A).
The task was chosen as it generates relatively free speech production - shown
to produce more distinctive voice patterns in patient populations than, for
instance, reading aloud a text [14, 3, 34] \- but still pertaining to a common
domain (e.g., all will mention triangles).

Initial data collection was approved by the IRB at the Central Denmark Region
(reference numbers 2007-58-0010, 2009-0035, 2010-0161, 264/2014), which also
deemed the current analysis exempt from the need of further ethical approval.

Table A.1 and Table A.2 in the Appendix present the number of participants and
size of the audio and transcript data, respectively. Some participants did not
complete all animations and some recordings were excluded due to poor audio
quality.

####  2.2.1 Data collection and preprocessing

Recording equipment and procedures were consistent across studies: quiet but
not soundproof rooms, and tabletop microphones. To prepare the recordings for
analysis, we manually identified and fixed issues with sound quality, tagged
segments with participant speech, and transcribed them after removing
background noise. Recordings with fewer than 3 seconds of participant speech
were removed, and the remaining ones were split into 5-second segments with a
1-second sliding window. Transcripts were manually checked for errors and
inconsistencies. See Figure 1(B) for a visualization, and Appendix B for more
details on preprocessing.

####  2.2.2 Data Splits

Validation and test sets - on which we test the performance of the machine
learning models - were constructed to equally represent each diagnosis and
sex. Both validation and test set included 36 participants (for each
diagnostic group, 6 randomly chosen participants with diagnosis and 6 matched
controls). The training data included the remaining participants. No
participant was included in multiple splits.

###  2.3 Models

For voice and text separately, we created two types of predictive models:
models relying on more "traditional" machine learning pipelines (henceforth:
baseline models) and Transformer models with added classification heads
(henceforth: Transformer models). Each model was trained on three binary
classification tasks (for each diagnosis, predicting whether or not the
participant is diagnosed with the condition) and on multi-class
classification, that is, on predicting the correct diagnostic group (MDD, ASD,
schizophrenia, or control) in a setup where data from all other diagnostic
groups and their matched controls are included.

####  2.3.1 Baseline Models

We created four baseline audio models using different sets of features as
inputs, respectively relying on the eGeMAPS and ComParE feature sets
(openSMILE Python library v2.4.1 [19]), 40 MFCCs (torchaudio v0.11.0), and
representations from the final layer of a non-contextual deep learning model
pre-trained on speaker recognition (X-Vector, [35] [36]). Baseline audio
models consisted of shallow classifiers with one intermediate layer of size
1024, with input dimensionality defined by the number of input features (88
for eGeMAPS, 6373 for ComParE, 40 for MFCCs, and 512 for X-Vector).

For text, we trained 28 baseline models: one for each of the basic statistical
descriptors defined in TextDescriptives v.1.0.7 Python library [37]; a model
including all of these descriptors as input features simultaneously; three TF-
IDF models (term frequency - inverse document frequency, [38]) of varying
dimensionality (100, 1000 and 10000) implemented in Keras (v.2.7.0); and one
model using pre-trained fasttext word embeddings [39]. See Table D.2 for an
overview of all features and models. Note that, as the focus of the present
paper is a comprehensive evaluation of the predictive performance of
automatically extractable feature sets in a multiclass vs. binary setting, we
do not delve into an in-depth qualitative discussion of predictive performance
of individual features.

Text baseline models were trained using XGBoost, which implements regularised
gradient boosted decision trees that are robust to heterogeneous feature sets
[40]. XGBoost combines bagging and boosting to achieve optimal bias-variance
trade-off, and to reduce overfitting. XGBoost hyperparameters for baseline
models were found via randomized grid search over the parameter grid defined
in Table D.3, and optimized on the validation set. An overview on the baseline
models and the features used in each is presented in Table D.2 and Table D.1.

####  2.3.2 Transformer Models

We relied on 3 pre-trained audio Transformer models - wav2vec2-base-da, [41],
multilingual XLS-R [42], a Danish version of XLS-R, [43] \- and 4 text-based
models - ScandiBERT [44], XLM-RoBERTa-base [45], Multilingual MiniLM Sentence
Transformer [46], and Danish Electra (Ælectra) [47].

We optimized the hyperparameters for our text-based Transformers and fine-
tuned them both by freezing the encoder and only training the classification
head and by fine-tuning the whole model (see subsection C.2). We then selected
the best-performing model from each of the 4 pre-trained model classes by
evaluating them on the validation set. Models were fed transcripts from
individual trials as input, yielding one prediction per trial. Class
probabilities for multiple transcripts from the same individual were
aggregated to produce one prediction per individual during evaluation.

For audio models – which are significantly more computationally demanding – we
froze the weights of the entire model and trained the classification head
only. We tuned the learning rate, and selected the best-performing model based
on the performance on the evaluation set. As with baseline models, we
aggregated the extracted features over each 5-second window of recording using
the mean before feeding them to the classification layer. To get a single
prediction per recording, we aggregated the model output from all respective
5-second windows using the mean. Full details on models and training
procedures (including data augmentation) are provided in the appendix, but see
also Figure 1.

Figure 2: Binary and multiclass performance per group and classification
setup. The x-axis indicates the three different diagnostic groups, as well as
overall performance in the multiclass setting. The y-axis indicates
performance measured in terms of F1-scores for group vs matched controls (left
panel) or group vs all other participants (including other diagnostic groups,
right panel). The best 5 baseline models per group are in grey, the best 5
Transformer models in red. The grey horizontal bar indicates the performance
of a highly conservative dummy baseline which always predicts "control".

####  2.3.3 Multi-modal Combination

To estimate whether text and voice models provided overlapping or
complementary information, we created ensemble models that combined
predictions from the best performing models, in terms of macro F1 on the test
set, from each modality. The F1 score is a useful metric for comparison, as it
incorporates both the sensitivity and the positive predictive value of a
classification model. Macro F1 is obtained by taking the unweighted average of
the per-class F1 score. We created two ensemble models, one combining the two
best-performing baselines for audio and text, and one combining the best-
performing Transformer models. For each trial, the ensemble models selected
the output of the most confident model Figure 1. Cross-trial aggregation was
performed analogously to text- and voice-based models.

##  3 Results

###  3.1 Binary performance

When trained on identifying individuals with a diagnosis versus matched
controls in a unimodal binary classification setting, the 5 best baseline
models and transformer models achieved F1 classification performance of
0.54-0.75 for ASD, 0.67-0.92 for MDD, and 0.71-0.83 for schizophrenia (see
Figure 2). These ranges are comparable to those reported in previous
literature [6, 3, 5]. The best Transformer models performed better than
baseline models for MDD and ASD, and were on par with baselines models for
schizophrenia (see Figure 2). See Table E.3, Table E.4 and Table E.2 for
further details.

###  3.2 Multi-diagnostic performance

When trained on predicting diagnoses in a multi-class setting, overall F1
scores (F1-macro) were markedly lower than binary F1 scores, ranging between
0.38 and 0.57 for the top 5 baselines models, and between 0.41 and 0.46 for
the top 5 Transformers (see Figure 2). Class-specific F1 scores were
comparable to binary scores only for MDD, and were lower for ASD and
schizophrenia. Transformers’ performance was not generally better than
baseline models’. Class-specific F1 scores were slightly higher for
transformers compared to baselines for MDD, they were lower than baseline
models for ASD, and markedly worse for schizophrenia. See Table E.1 for
further details.

###  3.3 Voice vs Transcripts

Figure 3: Detailed performance overview for the 10 best multiclass models.
Y-axis indicates model details (with models ordered by ascending performance).
X-axis indicates performance, quantified as F1 macro, on the the training,
validation and test set. Model names on the y-axis are tagged by modality.
Large gaps between training and test performance indicates overfitting on the
training data.

As seen in Figure 3, the best voice-based model (X-Vector) and text-based
model (100-dimensional TF-IDF) performed on par, but they were outperformed by
an ensemble model pooling the outputs of the two (Baseline Ensemble).
Transformers also feature among the top 10 models, but only text ones, and
their performance is not better than the best baseline models. Transformer-
based ensemble pooling does not improve performance over the best text
Transformer.

###  3.4 Error Analysis

As shown in the confusion matrices in Figure 4, the error patterns for the
best models from each modality were slightly different. On the test set, the
best voice model (X-Vector) over-predicted the healthy control group. The best
text model (100-dimensional TF-IDF) tended to over-diagnose MDD and
schizophrenia, while only rarely predicting ASD. As shown in Figure E.1, age,
gender and symptom severity did not seem to be systematically associated with
misclassifications.

##  4 Discussion

This study assessed the performance of machine learning models of voice- and
text-based markers of neuropsychiatric conditions in a multi-diagnostic
setting. We used a novel dataset including audio recordings from individuals
with schizophrenia, MDD, and ASD, as well as matched controls for each group.
We trained a variety of machine learning models on both binary classification
(specific disorder vs. matched controls) and multiclass classification
(identifying the correct diagnosis) to assess performance in problem settings
that are closer to naturalistic clinical assessment. To comprehensively
evaluate the potential of state-of-the-art speech-based models, we trained and
evaluated models leveraging different input modalities (voice-based models,
text-based models, and cross-modal ensembles), as well as architectures
relying on both traditional feature-based approaches and state-of-the-art deep
learning models (i.e., Transformers), which are designed to better capture
contextual dependencies and the temporal dynamics of the input.

Figure 4: Confusion matrices of the performance of the best multiclass models
on the test set. The large number indicates the number of participants
identified to belong to the specific cell. The small number shows the number
of trials (that is, recordings or transcripts) falling into the cell, with the
number in parenthesis showing the total number of positive trials. For
instance, in Panel A, 33 trials (recordings) were correctly classified as
being from individuals with depression out of a total 55 recordings from
individuals with depression. A: X-Vector. B: TF-IDF-100.

###  4.1 Binary vs. multi-diagnostic performance of voice and text models

Our findings indicate that, generally, voice and transcripts contain
information relevant to identifying patients vs matched controls in binary
settings. In line with previous work leveraging clean contrasts between
patients and non-psychiatric controls, we were able to train binary models
that performed better than highly conservative dummy baselines. The only
exception was ASD, the smallest diagnostic group, where the best model
performed only slightly better than our baseline. However, performance dropped
for models trained in a multi-class context, where models are asked to
discriminate between multiple diagnostic groups and against a more varied
group of controls. The only notable exception to this was MDD, where
performance was still fairly high.

These findings suggest that, in binary classification settings, models may not
be learning diagnosis-specific and generalizable markers of neuropsychiatric
conditions, but rather develop heuristics that: a) apply to multiple
conditions; and/or b) only apply to samples drawn from specific matched socio-
demographic groups.

Concerning the first point, high performance in binary classification settings
may be observed if models pick up on auditory and linguistic patterns that are
found in multiple clinical conditions, or if the model identifies markers
related to clinical features that are present in multiple groups. To provide
an example of the first scenario, both MDD and schizophrenia present decreased
pitch variability in relation to negative and depressive symptoms. This is a
useful feature to learn in a binary classification setting, but it does not
help classify diagnoses in a multiclass setting that includes both groups.
Analogously, anxiety is a common co-morbidity of many psychiatric conditions,
including MDD, schizophrenia, and ASD; and has been shown to be detectable
from speech [48].

Secondly, voice or text features that might seem relevant to discriminating
between patients and controls in matched samples might be confounded with
markers of socio-demographic factors such as age, gender, and education, thus
failing to classify diagnoses in more varied samples. For instance, slow
speech, increased pauses and simpler syntax have been found in schizophrenia,
but they also characterize older adults. As a consequence, models fitting to
these markers might not perform well in multi-diagnostic settings with a more
varied population. To develop better heuristics, models may need to learn more
complex features that require significantly larger datasets.

Importantly, there are currently no large-scale multidiagnostic datasets that
include the fine-grained information on patients’ symptoms and clinical
characteristics required to test these hypotheses. In our view, developing
these resources should be a major focus for future research, as assessing the
clinical utility of speech-based models of neuropsychiatric condition relies
on thorough assessment of these scenarios. We will expand upon this point in
the next sections.

###  4.2 Model performance across modalities and model types

Text models generally performed better than audio models on multi-class
classification, with 7 out of 10 among the best-performing models being text
models (see Figure 3). The only exception to this pattern is a classifier
based on embeddings from an X-Vector model, pretrained on speaker verification
[36].

An important finding from the present study is that voice and text-based
features provide complementary information, which can be combined to improve
multi-diagnostic performance. Indeed, we found that a simple ensemble pooling
predictions of the best-performing audio and text baseline models outperformed
all other models.

Contextual embeddings from Transformer models did not seem to yield
performance gains compared to more traditional features. While generally well-
performing, Transformers were rarely the top-performing models, possibly due
to the size of current datasets not being sufficient to adequately fine-tune
these large architectures (see subsection 4.3 for a more in-depth discussion).

###  4.3 Limitations

Our findings provide valuable insight on the specificity and generalizability
of current machine learning approaches to speech markers in psychiatry.
However, in line with the general landscape of the field, the study presents
some limitations which must be overcome before we can develop robust,
generalizable, and clinically useful markers of neuropsychiatric conditions.
First, the novel dataset we developed for the present study includes 3
diagnostic groups, 420 participants and more than 3000 recordings. This is
larger and more heterogeneous than most analogous datasets, usually involving
fewer than 100 participants and clean binary contrasts [4]. Nevertheless, our
dataset does not capture the full complexity of psychiatric conditions and
their overlaps. More diagnostic groups, and more participants would be
required, especially in the light of the overfitting (large performance
differences between training and unseen test sets) we observe for many of our
models. Furthermore, our data lacks more fine-grained meta-data to better
characterize clinical heterogeneity and overlaps between diagnostic groups,
for instance, cognitive abilities, anxiety and depression scores for all
participants, and test the scenarios outlined in subsection 4.1.

Limitations in size and heterogeneity of the training data are likely to
especially penalize Transformer-based models. Transformer models are
significantly larger than feature-based models, and adequate fine-tuning of
the full architecture requires more data than those required to train feature-
based classifiers. This may be the reason we only observe limited gains
compared to feature-based approaches. As we will outline in the next section,
larger and more heterogeneous datasets, and domain-specific pretrained models
may help address these limitations.

Approximately a quarter (26.4%) of the individuals with schizophrenia included
in this study were receiving treatment with antipsychotic medications, which
may affect speech [49]. This might make it harder for models to learn to
identify vocal patterns of schizophrenia.

Finally, in line with existing work, we focus on training models that predict
diagnoses. But, diagnoses are not monolithic syndromes, and the "ontology" of
psychiatric diagnoses itself is very much an open question [50]. Individuals
with the same diagnosis may exhibit radically different clinical profiles,
and, conversely, different diagnoses may have substantial overlap in terms of
symptoms and cognitive profiles, and comorbidities are far from infrequent.
Different prediction targets might be preferable.

###  4.4 Future directions

In light of these findings and of the above-mentioned limitations, we advance
a series of suggestions for future directions.

####  4.4.1 Larger and more heterogeneous datasets

Voice- and text-based models trained on binary classification of psychiatric
diagnoses may lack the ability to learn specific and generalizable markers of
psychiatric conditions, which limits their robustness and potential for real-
world deployment. Machine learning research in this domain should therefore
strive to tackle predictive problems that more adequately capture the
complexity, heterogeneity and overlap of neuropsychiatric conditions, and that
better mirror naturalistic application scenarios.

At the moment, however, the lack of large datasets including data from
multiple diagnostic groups is a major obstacle to this endeavour. Developing
datasets that include speech samples from multiple diagnostic groups and
report fine-grained clinical information for individual participants is a
crucial step towards more robust speech-based psychiatric assessment.

Availability of larger and more heterogeneous datasets would also make it
possible to experiment with more sophisticated architectures. In this study,
for example, our models are trained and evaluated on generating separate
diagnostic predictions for each recording. With more data, models could be
trained on aggregated outputs from multiple recordings, allowing them to learn
dependencies between participants’ linguistic behavior across recordings and
to formulate more informed predictions. Furthermore, here we have trained the
simplest possible ensemble models by pooling estimates from audio and text
models. Larger amounts of data would allow us to implement more sophisticated
cross-modal pooling strategies, such as training classifiers on concatenated
voice-and-text representations, or training ensemble Transformers.

####  4.4.2 Better domain-specific models

Against our expectations, pretrained text and audio Transformers did not
outperform baseline models. Beyond the limited sample size available, an
additional possible explanation for this is that encodings from models pre-
trained on generic language modeling tasks, such as next-token (forward
language modeling) and missing token (masked language modeling) prediction may
not encode information suited to inferring psychiatric traits. Further fine-
tuning of Transformer models on non-clinical data using self-supervised tasks
[51], or supervised tasks (e.g., prediction of psychological traits) closer to
the target domain, may yield performance improvements when pre-trained models
are deployed for psychiatric applications.

####  4.4.3 Better targets of prediction

As anticipated in subsection 4.3, a limitation of current work on speech
markers of neuropsychiatric condition is its almost exclusive focus on
predicting diagnoses from voice and text. As diagnoses are heterogeneous and
often overlapping, shifting the focus from predicting overall diagnostic
groups towards predicting more fine-grained clinically relevant features (e.g.
affect, cognitive load, etc.) could be a more robust and clinically useful
endeavor [52]. Clinical assessment overwhelmingly relies on thorough testing
of a range of such intermediate descriptors - more than of final outcomes such
as diagnosis - and machine learning models able to complement and support this
assessment could make a substantial contribution and be of more immediate use.

Being able to predict fine-grained clinical features could also guide
practitioners beyond assessment, allowing them to monitor time-varying
symptoms, to select between alternative treatments, or provide them with
prognostic priors on, for instance, the probability of relapse. If predicting
diagnoses is a worthy first step for an emerging field (with limited data),
the largest impact will be given by focusing on predictive targets that can
support and scale, rather than simply replicate, human decision-making.

##  5 Conclusion

We investigated the performance of a wide range of speech-based machine
learning models of neuropsychiatric conditions in multi-class classification
settings. We assessed the performance of voice-, transcript-based model, and
cross-modal ensemble models, as well as the performance of traditional
feature-based machine learning approaches and advanced state-of-the-art
Transformer models. Models contrasting one specific diagnostic group with
matched non-psychiatric controls performed largely in line with the
literature. However, in multi-class contexts that better mirror naturalistic
differential diagnosis, performance decreased quite drastically. Voice and
text-based features complemented each other. Transformer models did not
generally outperform baseline models. In order to develop models that can
capture more robust and clinically relevant markers, we advocate for the field
to: 1) develop larger and more heterogeneous datasets, better reflecting the
complexity of neuropsychiatric conditions; 2) develop domain-specific
Transformer models for the task at hand; 3) re-focus predictions from
diagnoses to more fine-grained clinically relevant features, such as symptoms
and cognitive abilities.

Author statement: LH: Conceptualization; Methodology; Data Curation; Software;
Formal Analysis; Validation; Writing - Original Draft; Writing - Review and
Editing; Visualization; Interpretation; RR: Conceptualization; Methodology;
Data Curation; Software; Formal Analysis; Validation; Writing - Original
Draft; Writing - Review and Editing; Visualization; Interpretation; AS:
Investigation; Data Curation; Methodology; Writing - Review and Editing; AP:
Writing - Review and Editing; Interpretation; VB: Data Curation;
Investigation; Resources; Writing - Review and Editing; NL: Data Curation;
Investigation; Resources; Writing - Review and Editing; Interpretation; DB:
Data Curation; Investigation; Resources; Writing - Review and Editing;
Interpretation; KT: Investigation; Software; Writing - Review and Editing;
Interpretation; EW: Investigation; Software; Writing - Review and Editing;
Interpretation; SDØ: Writing - Review and Editing; Interpretation; RF:
Conceptualization; Methodology; Data Curation; Formal Analysis; Validation;
Writing - Original Draft; Writing - Review and Editing; Visualization;
Supervision; Funding Acquisition; Project Administration; Interpretation

Acknowledgements: We acknowledge seed funding from the Interacting Minds
Centre ("Clinical voices"). RR is partly supported by funding from the
Volkswagen Stiftung. DB is supported by a Sir Henry Wellcome Postdoctoral
Fellowship from the Wellcome Trust (213630/Z/18/Z). The Wellcome Centre for
Human Neuroimaging is supported by core funding from the Wellcome Trust
(203147/Z/16/Z).

Conflicts of interest SDØ received the 2020 Lundbeck Foundation Young
Investigator Prize. Furthermore, SDØ owns/has owned units of mutual funds with
stock tickers DKIGI, IAIMWC, SPIC25KL and WEKAFKI, and has owned units of
exchange traded funds with stock tickers BATE, TRET, QDV5, QDVH, QDVE, SADM,
IQQH, USPY, EXH2, 2B76 and EUNL.

## References

  * [1] Heather MacFarlane et al. “Combining voice and language features improves automated autism detection” _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/aur.2733 In _Autism Research_ , 2022 DOI: 10.1002/aur.2733
  * [2] Lang He et al. “Deep learning for depression recognition with audiovisual cues: A review” In _Information Fusion_ 80, 2022, pp. 56–86 DOI: 10.1016/j.inffus.2021.10.012
  * [3] Alberto Parola, Arndis Simonsen, Vibeke Bliksted and Riccardo Fusaroli “Voice patterns in schizophrenia: A systematic review and Bayesian meta-analysis” In _Schizophrenia Research_ 216, 2020, pp. 24–40 DOI: 10.1016/j.schres.2019.11.031
  * [4] Daniel M. Low, Kate H. Bentley and Satrajit S. Ghosh “Automated assessment of psychiatric disorders using speech: A systematic review” Publisher: Wiley Online Library In _Laryngoscope Investigative Otolaryngology_ 5.1, 2020, pp. 96–116
  * [5] Sanne Koops et al. “Speech as a Biomarker for Depression” In _CNS & neurological disorders drug targets_ 22.2, 2023, pp. 152–160 DOI: 10.2174/1871527320666211213125847
  * [6] Riccardo Fusaroli et al. “"Is voice a marker for Autism spectrum disorder? A systematic review and meta-analysis"” In _Autism Research: Official Journal of the International Society for Autism Research_ 10.3, 2017, pp. 384–407 DOI: 10.1002/aur.1678
  * [7] Lasse Hansen et al. “A Generalizable Speech Emotion Recognition Model Reveals Depression and Remission” Publisher: Jossey-Bass In _Acta Psychiatrica Scandinavica_ 145.2, 2022, pp. 186–199 DOI: 10.1111/acps.13388
  * [8] Emil Kraepelin “Manic-depressive insanity and paranoia” E. & S. Livingstone, 1921
  * [9] M. Hamilton “A rating scale for depression” In _Journal of Neurology, Neurosurgery, and Psychiatry_ 23, 1960, pp. 56–62 DOI: 10.1136/jnnp.23.1.56
  * [10] Riccardo Fusaroli et al. “Toward a cumulative science of vocal markers of autism: A cross-linguistic meta-analysis-based investigation of acoustic markers in American and Danish autistic children” Publisher: Wiley Online Library In _Autism Research_ 15.4, 2022, pp. 653–664
  * [11] Astrid Rybner et al. “Vocal markers of autism: Assessing the generalizability of machine learning models” Publisher: Wiley Online Library In _Autism Research_ , 2022
  * [12] Alberto Parola et al. “Speech disturbances in schizophrenia: Assessing cross-linguistic generalizability of NLP automated measures of coherence” In _Schizophrenia Research_ , 2022 DOI: 10.1016/j.schres.2022.07.002
  * [13] Alberto Parola et al. “Voice patterns as markers of schizophrenia: building a cumulative generalizable approach via cross-linguistic and meta-analysis based investigation” Publisher: Cold Spring Harbor Laboratory Press In _medRxiv_ , 2022
  * [14] Nicholas Cummins et al. “A review of depression and suicide risk assessment using speech analysis” Publisher: Elsevier In _Speech Communication_ 71, 2015, pp. 10–49
  * [15] Thin Nguyen et al. “Affective and content analysis of online depression communities” Publisher: IEEE In _IEEE Transactions on Affective Computing_ 5.3, 2014, pp. 217–226
  * [16] World Health Organization “The ICD-10 classification of mental and behavioural disorders : diagnostic criteria for research” World Health Organization, 1993 URL: https://apps.who.int/iris/handle/10665/37108
  * [17] American Psychiatric Association “Diagnostic and statistical manual of mental disorders (DSM-5®)” American Psychiatric Pub, 2013
  * [18] Louise Nyholdm Jensen and Nicole Dwenger “MetaVoice”, 2022 URL: https://metavoice.au.dk/index.html
  * [19] Florian Eyben and Björn Schuller “openSMILE:): the Munich open-source large-scale multimedia feature extractor” In _ACM SIGMultimedia Records_ 6.4, 2015, pp. 4–13
  * [20] Gilles Degottex et al. “COVAREP—A collaborative voice analysis repository for speech technologies” In _Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Conference on_ IEEE, 2014, pp. 960–964
  * [21] Alban E. Voppel et al. “Semantic and Acoustic Markers in Schizophrenia-Spectrum Disorders; a Combinatory Machine Learning Approach” In _Schizophrenia Bulletin_ , 2022, pp. sbac142 DOI: 10.1093/schbul/sbac142
  * [22] Ashish Vaswani et al. “Attention Is All You Need” arXiv: 1706.03762 In _arXiv:1706.03762 [cs]_ , 2017 URL: http://arxiv.org/abs/1706.03762
  * [23] Yann LeCun, Yoshua Bengio and Geoffrey Hinton “Deep learning” Number: 7553 Publisher: Nature Publishing Group In _Nature_ 521.7553, 2015, pp. 436–444 DOI: 10.1038/nature14539
  * [24] Nicholas Cummins, Alice Baird and Björn W. Schuller “Speech analysis for health: Current state-of-the-art and the increasing impact of deep learning” In _Methods_ 151, Health Informatics and Translational Data Analytics, 2018, pp. 41–54 DOI: 10.1016/j.ymeth.2018.07.007
  * [25] Alexei Baevski, Henry Zhou, Abdelrahman Mohamed and Michael Auli “wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations” arXiv: 2006.11477 In _arXiv:2006.11477 [cs, eess]_ , 2020 URL: http://arxiv.org/abs/2006.11477
  * [26] Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” arXiv: 1810.04805 In _arXiv:1810.04805 [cs]_ , 2019 URL: http://arxiv.org/abs/1810.04805
  * [27] Sebastian Ruder, Matthew E. Peters, Swabha Swayamdipta and Thomas Wolf “Transfer Learning in Natural Language Processing” In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Tutorials_ Minneapolis, Minnesota: Association for Computational Linguistics, 2019, pp. 15–18 DOI: 10.18653/v1/N19-5004
  * [28] Frances Abell, Frances Happe and Uta Frith “Do triangles play tricks? Attribution of mental states to animated shapes in normal and abnormal development” Publisher: Elsevier In _Cognitive Development_ 15.1, 2000, pp. 1–16
  * [29] Nicolai Ladegaard, Paul H. Lysaker, Erik R. Larsen and Poul Videbech “A comparison of capacities for social cognition and metacognition in first episode and prolonged depression” In _Psychiatry Research_ 220.3, 2014, pp. 883–889 DOI: 10.1016/j.psychres.2014.10.005
  * [30] Nicolai Ladegaard, Erik Roj Larsen, Poul Videbech and Paul H. Lysaker “Higher-order social cognition in first-episode major depression” In _Psychiatry Research_ 216.1, 2014, pp. 37–43 DOI: 10.1016/j.psychres.2013.12.010
  * [31] Katrine Ingeman Beck et al. “Cross-cultural comparison of theory of mind deficits in patients with schizophrenia from China and Denmark: different aspects of ToM show different results” In _Nordic Journal of Psychiatry_ 74.5, 2020, pp. 366–373 DOI: 10.1080/08039488.2020.1723687
  * [32] Vibeke Bliksted et al. “Hyper-and hypomentalizing in patients with first-episode schizophrenia: fMRI and behavioral studies” Publisher: Oxford University Press US In _Schizophrenia bulletin_ 45.2, 2019, pp. 377–385
  * [33] Vibeke Bliksted et al. “Social cognition and neurocognitive deficits in first-episode schizophrenia” In _Schizophrenia Research_ 153.1, 2014, pp. 9–17 DOI: 10.1016/j.schres.2014.01.010
  * [34] Konstantinos Sechidis et al. “A machine learning perspective on the emotional content of Parkinsonian speech” Publisher: Elsevier In _Artificial Intelligence in Medicine_ 115, 2021, pp. 102061
  * [35] David Snyder et al. “X-vectors: Robust dnn embeddings for speaker recognition” In _2018 IEEE international conference on acoustics, speech and signal processing (ICASSP)_ IEEE, 2018, pp. 5329–5333
  * [36] Speechbrain “speechbrain/spkrec-xvect-voxceleb · Hugging Face”, 2022 URL: https://huggingface.co/speechbrain/spkrec-xvect-voxceleb
  * [37] Lasse Hansen and Kenneth Enevoldsen “TextDescriptives: A Python package for calculating a large variety of statistics from text” arXiv:2301.02057 [cs] arXiv, 2023 DOI: 10.48550/arXiv.2301.02057
  * [38] Karen Sparck Jones “A statistical interpretation of term specificity and its application in retrieval” Publisher: MCB UP Ltd In _Journal of documentation_ , 1972
  * [39] Piotr Bojanowski, Edouard Grave, Armand Joulin and Tomas Mikolov “Enriching Word Vectors with Subword Information”, 2016 URL: http://arxiv.org/abs/1607.04606
  * [40] Tianqi Chen et al. “Xgboost: extreme gradient boosting” In _R package version 0.4-2_ 1.4, 2015, pp. 1–4
  * [41] Alvenir “Alvenir/wav2vec2-base-da · Hugging Face”, 2022 URL: https://huggingface.co/Alvenir/wav2vec2-base-da
  * [42] Arun Babu et al. “XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale” arXiv:2111.09296 [cs, eess] type: article, 2021 DOI: 10.48550/arXiv.2111.09296
  * [43] Lasse Hansen “chcaa/xls-r-300m-danish · Hugging Face”, 2022 URL: https://huggingface.co/chcaa/xls-r-300m-danish
  * [44] Vésteinn Snæbjarnarson “vesteinn/ScandiBERT · Hugging Face”, 2022 URL: https://huggingface.co/vesteinn/ScandiBERT
  * [45] Alexis Conneau et al. “Unsupervised Cross-lingual Representation Learning for Speech Recognition” arXiv: 2006.13979 In _arXiv:2006.13979 [cs, eess]_ , 2020 URL: http://arxiv.org/abs/2006.13979
  * [46] Nils Reimers and Iryna Gurevych “Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation” arXiv:2004.09813 [cs] type: article, 2020 DOI: 10.48550/arXiv.2004.09813
  * [47] Malte Højmark-Bertelsen “Ælæctra - A Step Towards More Efficient Danish Natural Language Processing”, 2021 URL: https://github.com/MalteHB/-l-ctra/
  * [48] Jinni A. Harrigan, Mary Ann Larson and Catherine J. Pflum “The Role of Auditory Cues in the Detection of State Anxiety 1” Publisher: Wiley Online Library In _Journal of Applied Social Psychology_ 24.22, 1994, pp. 1965–1983
  * [49] A. Thompson “Pharmacological agents with effects on voice” In _American Journal of Otolaryngology_ 16.1, 1995, pp. 12–18 DOI: 10.1016/0196-0709(95)90003-9
  * [50] Eiko I. Fried “Studying mental disorders as systems, not syndromes” Publisher: PsyArXiv, 2021
  * [51] Roberta Rocca and Tal Yarkoni “Language as a fingerprint: Self-supervised learning of user encodings using transformers”, 2022, pp. 1701–1714 URL: https://aclanthology.org/2022.findings-emnlp.123
  * [52] Thomas Insel et al. “Research domain criteria (RDoC): Toward a new classification framework for research on mental disorders” Place: US Publisher: American Psychiatric Assn In _The American Journal of Psychiatry_ 167, 2010, pp. 748–751 DOI: 10.1176/appi.ajp.2010.09091379
  * [53] Nicolai Ladegaard, Poul Videbech, Paul H. Lysaker and Erik R. Larsen “The course of social cognitive and metacognitive ability in depression: Deficit are only partially normalized after full remission of first episode major depression” In _The British Journal of Clinical Psychology_ 55.3, 2016, pp. 269–286 DOI: 10.1111/bjc.12097
  * [54] Vibeke Bliksted, Poul Videbech, Birgitte Fagerlund and Chris Frith “The effect of positive symptoms on social cognition in first-episode schizophrenia is modified by the presence of negative symptoms” In _Neuropsychology_ 31.2, 2017, pp. 209–219 DOI: 10.1037/neu0000309
  * [55] Lotte Veddum, Heine Lund Pedersen, Anna-Sofie Lose Landert and Vibeke Bliksted “Do patients with high-functioning autism have similar social cognitive deficits as patients with a chronic cause of schizophrenia?” Publisher: Taylor & Francis In _Nordic Journal of Psychiatry_ 73.1, 2019, pp. 44–50
  * [56] iZotope “RX 6 Elements” iZotope, 2017 URL: https://www.izotope.com/en/products/repair-and-edit/rx/rx-elements.html
  * [57] Arsha Nagrani, Joon Son Chung and Andrew Zisserman “Voxceleb: a large-scale speaker identification dataset” In _arXiv preprint arXiv:1706.08612_ , 2017
  * [58] Joon Son Chung, Arsha Nagrani and Andrew Zisserman “Voxceleb2: Deep speaker recognition” In _arXiv preprint arXiv:1806.05622_ , 2018
  * [59] Lasse Hansen, Rasmus Arpe Fogh Jensen and Martin Carsten Nielsen “xls-r-300m-danish”, 2022 URL: https://huggingface.co/chcaa/xls-r-300m-danish
  * [60] William Falcon and The PyTorch Lightning Team “PyTorch Lightning”, 2022 URL: https://www.pytorchlightning.ai
  * [61] Thomas Wolf et al. “Transformers: State-of-the-art natural language processing” In _Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations_ , 2020, pp. 38–45
  * [62] Ilya Loshchilov and Frank Hutter “Decoupled Weight Decay Regularization” arXiv:1711.05101 [cs, math] type: article, 2019 DOI: 10.48550/arXiv.1711.05101
  * [63] Iver Jordal et al. “asteroid-team/torch-audiomentations: v0.10.1” Zenodo, 2022 DOI: 10.5281/zenodo.6381721
  * [64] Luis Perez and Jason Wang “The effectiveness of data augmentation in image classification using deep learning” In _arXiv preprint arXiv:1712.04621_ , 2017
  * [65] Jason Wei and Kai Zou “EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks” In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ Hong Kong, China: Association for Computational Linguistics, 2019, pp. 6382–6388 DOI: 10.18653/v1/D19-1670
  * [66] Daniel S. Park et al. “SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition” arXiv:1904.08779 [cs, eess, stat] In _Interspeech 2019_ , 2019, pp. 2613–2617 DOI: 10.21437/Interspeech.2019-2680
  * [67] James Traer and Josh H. McDermott “Statistics of natural reverberation enable perceptual separation of sound and space” Publisher: National Acad Sciences In _Proceedings of the National Academy of Sciences_ 113.48, 2016, pp. E7856–E7865
  * [68] Arjit Jain et al. “SpliceOut: A Simple and Efficient Audio Augmentation Method” arXiv:2110.00046 [cs, eess] type: article, 2021 DOI: 10.48550/arXiv.2110.00046
  * [69] Fabian Pedregosa et al. “Scikit-learn: Machine learning in Python” In _Journal of machine learning research_ 12.Oct, 2011, pp. 2825–2830

## Glossary

ASD

    Autism Spectrum Disorder
CV

    Cross-validation
MDD

    Major Depressive Disorder
NLP

    Natural Language Processing
TF-IDF

    Term Frequency–Inverse Document Frequency

##  Appendix A Original studies

The depression corpus was collected by [30] for a study on changes in social
cognition in first-episode depression. The corpus consists of 42 patients with
first-episode Major Depressive Disorder (MDD), 42 pairwise matched controls on
age, sex, and educational background, and 25 patients with chronic depression
[53]. Patients were recruited from general practitioners in the Central
Denmark Region, and healthy controls were recruited via newspaper
advertisements and offered monetary compensation if included in the study
(1000 DKK). All participants were native speakers of Danish and met the
following inclusion criteria: 1) first-episode major depression was the
primary diagnosis, 2) the severity of depression was moderate to severe as
measured by the 17-item Hamilton Rating Scale for Depression (HamD-17 > 17),
3) patients were psychotropic drug-naïve, 4) no psychiatric comorbidity other
than anxiety disorders. Patients with head trauma, neurological illness, or
substance use disorders were not permitted to the study. Exclusion criteria
for healthy controls were the same as for depressed patients.

The schizophrenia corpus comes from [54, 55] and consists of recordings from
104 patients with schizophrenia and 116 healthy controls pairwise matched on
age, sex, level of education, handedness, race, community of residency,
parental social economic status, and expected parental income. Patients were
recruited from the OPUS Clinic for young people with schizophrenia and were
diagnosed with first-episode schizophrenia according to ICD-10 and received
less than 3 months of antipsychotic medication prior to the diagnostic
interview. Exclusion criteria included history of neurological disorder or
severe head trauma, ICD-10 diagnosis of drug- or alcohol dependency, estimated
premorbid IQ < 70, and not being able to understand spoken Danish sufficiently
to comprehend the testing procedures. Healthy controls were recruited via
newspaper advertisements.

Table A.1: Descriptive statistics of the audio data. Rows are paired to indicate the patients and corresponding matched controls.  Diagnosis | N participants | N recordings | Mean duration (s) | Total duration (h)
---|---|---|---|---
ASD | 46 | 424 | 40.0 | 4.7
CN | 41 | 382 | 29.1 | 3.1
MDD | 66 | 548 | 13.3 | 2.0
CN | 41 | 372 | 12.7 | 1.3
SCZ | 105 | 821 | 19.2 | 4.4
CN | 117 | 969 | 19.1 | 5.2
Total | 416 | 3516 | 21.2 | 20.7
Table A.2: Descriptive statistics of the transcriptions data. Rows are paired to indicate the patients and corresponding matched controls. Diagnosis | N participants | N transcripts | Mean number of words | Total number of words
---|---|---|---|---
ASD | 21 | 162 | 65.9 | 10,691
CN | 25 | 200 | 73.5 | 14,711
MDD | 67 | 585 | 28.9 | 16,959
CN | 42 | 408 | 34.6 | 14,124
SCZ | 105 | 873 | 43.4 | 37,917
CN | 116 | 990 | 48.5 | 48,112
Total | 376 | 3218 | 44.2 | 142,514

The Autism Spectrum Disorder (ASD) corpus consists of 46 patients, and 41
pairwise matched controls and was not otherwise published before this study.
Patients were included based on a self-reported diagnosis of autism spectrum
disorder.

##  Appendix B Animated Triangles Task

The Animated Triangles task is commonly used to measure theory of mind and
involves eight to twelve video clips of interactions between animated
triangles. In four of the clips, the triangles are moving randomly and
unintentionally (e.g. bouncing off walls). In the remaining clips, the
triangles are interacting intentionally; in four of them, the triangles are
trying to influence the mental state of one another (e.g., a larger triangle
trying to convince a small one to leave a closure), and in the remaining four
the triangles are performing an activity together. Each animation has a
duration of approximately 40 seconds, after which participants are asked to
provide an interpretation of what was happening in the animation. This
interpretation was recorded and forms the basis for this study.

##  Appendix C Data Processing

###  C.1 Audio data preprocessing

All recordings were first manually processed to identify issues with sound
quality, and remove long periods of silence and interviewer speech.
Subsequently, background noise was removed using iZotope Elements RX6 [56].
This was done to reduce the chance of models fitting to group-specific
patterns of background noise. Recordings containing less than 3 seconds of
speech were removed, as they are unlikely to contain enough signal to be
useful. Recordings were resampled to 16kHz as that is required by models such
as Wav2vec2 [25] and XLS-R [42], and audio samples were normalized to lie
between -1.0 and 1.0.

Before training, all recordings were split into 5 second segments with a 1
second striding window. That is, a recording with a duration of 7 seconds was
split into three separate recordings: 0-5 seconds, 1-6 seconds, and 2-7
seconds. Windowing was done to increase the amount of training data, and
serves as a form of augmentation. Additionally, it is computationally costly
to process audio, so short windows allowed us to use a larger batch size. See
Figure 1(B) for a visualization.

###  C.2 Transcript data preprocessing

All audio files with sufficient audio quality were manually transcribed by
research assistants. Transcripts were manually checked for errors and
consistency. For instance, some transcribers, due to software issues,
transcribed æ, ø, å as ae, oe, and aa, respectively, which was standardized.
Analogously, transcribers included comments in square parentheses (e.g.
’[sic]’), which were also excluded.

##  Appendix D Models

The following two subsections give a detailed explanation of the audio and
text models. For each modality, multiple types of models were created;
baselines and Transformer models. Each model type was trained to predict the
correct disorder in a multiclass setting, as well as separate binary models
that were trained to discriminate each disorder from their matching controls.

###  D.1 Audio Models

The following section describes the development of baseline and Transformer
models for audio and outlines the training procedure, hyperparameters, and
augmentation strategy. The training setup and procedure were the same across
all audio models to aid comparability: the only varying factor was the
features used.

####  D.1.1 Baseline Models

For consistency with previous studies of inferring mental disorder from voice,
two baseline models were created using the eGeMAPS and ComParE feature sets
extracted using the openSMILE Python library v2.4.1 [19]. A third baseline
model was created using 40 MFCCs extracted with torchaudio v0.11.0. A final
baseline model was trained using features extracted from an X-vector model
[35] pretrained on the multilingual VoxCeleb1 [57] and VoxCeleb2 [58] datasets
for speaker recognition [36]. The X-vector model is based on temporal/1D
convolutional neural networks (CNN) to utilize both temporal and acoustic
information from speech. We include the X-vector model to provide a comparison
to a different type of pretrained model that is CNN instead of Transformer
based.

The extracted baseline features were aggregated over each audio window using
the mean, to obtain a single feature embedding for each window. The number of
features extracted from each baseline feature method is shown in Table D.1.

Table D.1: Number of features per feature set Feature set | N. features
---|---
MFCCs | 40
eGeMAPS | 88
X-Vector | 512
Wav2vec2 | 1024
ComParE | 6377

####  D.1.2 Transformer Models

At present there are three pretrained Transformer models for Danish speech
publicly available; a Danish Wav2vec2 base model trained by Alvenir,
wav2vec2-base-da, [41], the multilingual XLS-R trained by Facebook [42], and a
Danish continued pretraining of XLS-R [59]. wav2vec2-base-da has approximately
95 million trainable parameters and was pretrained on 1,300 hours of Danish
speech, mainly from audiobooks. XLS-R was pretrained on a total of 436
thousand hours of speech from 53 different languages and has approximately 300
million trainable parameters. Danish makes up approximately 14 thousand hours
of the training data, and mainly stems from speeches held in the European
Parliament in Danish. The Danish XLS-R warmstarted from the XLS-R checkpoint
and further pretrained on 141,000 hours of Danish radio.

We use all three models as feature extractors by freezing the weights of the
entire model and training a single classification layer that is added on top.
As with the baseline models, we aggregate the extracted features over each
recording using the mean before feeding them to the classification layer.

####  D.1.3 Model Training

Audio models were trained using the PyTorch Lightning v1.6.3 [60] and
transformers v4.19.2 [61] Python libraries on a single NVIDIA Tesla V100 32GB
GPU. For both baselines and transformers, model training progressed in the
following steps:

  1. 1.

Load batch of windowed data.

  2. 2.

Randomly augment the data to increase or decrease gain, add coloured noise
(brown, white, pink, or blue), add an impulse response, and/or use SpliceOut.

  3. 3.

Extract features from the augmented data using either a baseline feature
extractor or a Transformer model.

  4. 4.

Pass features to a shallow neural network consisting of n_features input
units, a hidden layer with 1024 units, followed by an output layer with 2 or 4
units for the binary and multiclass task respectively.

  5. 5.

Update weights of the shallow neural network using the AdamW optimizer. [62].
If multiclass, loss was weighted by the number of examples from each class to
reduce the chance of over-predicting healthy controls.

  6. 6.

Iterate for 20 epochs.

The batch size was set to 32 and models were trained to minimize cross-
entropy. The learning rate was set to $0.00005$ for all models after initial
experimentation.

Augmentation was done using torch_audiomentations v0.10.1 [63]. The idea of
data augmentation is to inject noise or change the input in subtle ways to
expose the models to more diverse input representations. Data augmentation has
been successfully used in both Cross-validation (CV) [64], Natural Language
Processing (NLP) [65], and speech analysis [66], to create more robust and
generalizable models.

On each iteration, each example had a 50% chance of being augmented by each of
the augmentations described above. The gain augmentation changes the gain of
the recording by a number of decibels uniformly sampled between -18 and 6.
Coloured noise adds noise with peaks at different frequencies depending on
colour. The impulse responses were acquired from [67], and are a collection of
271 impulse responses from places encountered by 7 participants during their
daily life. An impulse response is the reverberation profile of a location and
can vary greatly. For instance, the reverberation in a church is very
different from in a forest. SpliceOut was introduced by [68] and is a
computationally more efficient variation of SpecAugment [66], which masks
random time intervals of the input audio. The reason for training a shallow
neural classifier on the feature sets instead of an XGBoost model (as for
text), was to enable the use of varying augmentations per epoch.

###  D.2 Text Models

The following section describes the development of the text models and
outlines the training procedure, hyperparameters, and feature sets. Baseline
models were trained using XGBoost, as the performance was found to be poor
using a shallow neural network in a similar fashion as for the audio models
and text Transformers.

####  D.2.1 Baseline Models

Multiple types of baseline models were constructed. One type using statistical
descriptors of text, extracted using the TextDescriptives v.1.0.7 Python
library [37], one using Term Frequency–Inverse Document Frequency (TF-IDF)
features, and one using pretrained fasttext word embeddings [39]. Table D.2
displays the different features used. Separate models were trained for each
feature, as well as a model using all the descriptive, readability, syntactic
complexity, and sentiment features.

####  D.2.2 Transformer Models

We trained four publicly available Transformer models for Danish: ScandiBERT
[44], XLM-RoBERTa-base [45], Multilingual MiniLM Sentence Transformer [46],
and Danish Electra [47]. ScandiBERT was pretrained on Danish, Norwegian, and
Swedish text, XLM-RoBERTa-base and Multilingual MiniLM Sentence Transformer
were pretrained for 100 different languages, and Danish Electra was trained
only on Danish text.

Table D.2: Features used for text baselines.

Descriptive
---
Sentence length mean and std
Token length mean and std
Proportion unique tokens
Number of unique tokens
Number of characters
Number of sentences
Readability
Flesch reading ease
Flesch-Kincaid grade
Lix
Rix
Coleman-Liau Index
Syntactic complexity
Dependency distance mean and std
Proportion adjacent dependency relations mean and std
Sentiment
Polarity
TF-IDF
100 features
1.000 features
10.000 features
Fasttext
Danish 300-dimensional Fasttext word vectors

####  D.2.3 Model Training

Baseline models were trained using scikit-learn v1.1.1 [69], and Transformers
using transformers v4.19.2 [61] and PyTorch Lightning v1.6.3 [60]. XGBoost
hyperparameters for baseline models were found via a grid search of the
parameters in Table D.3 and optimized on the validation set.

Table D.3: Grid for XGBoost hyperparameter search.

Parameter | Values
---|---
learning_rate | [0.001, 0.01, 0.1, 0.5]
min_child_weight | [1, 3, 5, 10]
gamma | [0, 0.5, 1, 2]
subsample | [0.6, 0.8, 1]
colsample_bytree | [0.25, 0.5, 0.75, 1]
max_depth | [1, 3, 5]
reg_alpha | [0, 0.1, 1, 5]
reg_lambda | [0.1, 1, 5]
n_estimators | [20, 100, 500]

Text transformers were trained by adding a classification layer on top of the
base Transformer architectures in a similar manner as for the audio models
(i.e. a linear layer feeding to an output layer with 2 or 4 outputs for the
binary and multiclass task, respectively). Transformer models were trained
both with a frozen base model, i.e. by only updating the parameters of the
newly initialized classification layer, and by updating the entire
Transformer. The learning rate for the Transformer models was searched for
among values of 1 to the negative power of 2 to 6, and optimized on the
validation set. After parameter optimization, all Transformer models were
trained with a batch size of 16 for 100 epochs.

Similarly to audio models, the loss of all text models was weighted by the
number of examples from each class.

###  D.3 Multi-modal Combination

To estimate whether additional performance could be gained by combining text
and audio models, we combined predictions by choosing the output of the most
confident model. Two ensembles were created for each task: one created from
the best performing baseline model from each modality, and one from the best
performing Transformer model from each modality. The best-performing models
were chosen based on F1-macro on the test set. This was done to eliminate the
need to create all possible combinations of models, but rather to see if the
combination of two good models yielded an improvement.

##  Appendix E Performances

Tables and figures reporting performance metrics for all models, as well as
details on our error analysis, are displayed in the next pages.

Table E.1: Performances for all multiclass models

F1 Macro F1 - ASD F1 - MDD F1 - SCZ Test Validation Train Test Validation
Train Test Validation Train Test Validation Train Baseline Ensemble ensemble
0.570 0.476 0.814 0.250 0.286 0.845 0.667 0.400 0.829 0.667 0.571 0.764
X-Vector audio 0.521 0.497 0.789 0.286 0.286 0.812 0.600 0.444 0.777 0.500
0.571 0.756 TF-IDF, 100-d text 0.511 0.432 0.642 0.444 0.167 0.615 0.667 0.500
0.654 0.400 0.500 0.630 Transformer Ensemble ensemble 0.461 0.274 0.811 0.286
0.214 0.773 0.667 0.500 0.894 0.375 0.286 0.828 Sentence Transformer text
0.461 0.455 0.869 0.286 0.286 0.947 0.667 0.588 0.894 0.375 0.333 0.832 XLM-
RoBERTa text 0.452 0.499 0.643 0.444 0.400 0.545 0.750 0.462 0.736 0.000 0.500
0.575 FastText text 0.450 0.354 0.597 0.400 0.333 0.545 0.625 0.444 0.687
0.375 0.200 0.648 ScandiBERT text 0.444 0.450 0.698 0.444 0.400 0.571 0.667
0.533 0.780 0.167 0.250 0.701 Ælectra text 0.413 0.421 0.607 0.444 0.400 0.545
0.571 0.545 0.675 0.154 0.222 0.620 All descriptives text 0.377 0.425 0.613
0.250 0.250 0.667 0.300 0.500 0.560 0.444 0.400 0.522 TF-IDF, 1000-d text
0.375 0.374 0.592 0.364 0.182 0.552 0.632 0.400 0.645 0.133 0.400 0.604 MFCCs
audio 0.373 0.337 0.669 0.167 0.250 0.644 0.462 0.182 0.680 0.250 0.375 0.627
Proportion Adjacent Dependencies (mean) text 0.368 0.305 0.455 0.222 0.182
0.409 0.250 0.222 0.486 0.364 0.167 0.262 Number of characters text 0.334
0.273 0.385 0.250 0.167 0.378 0.476 0.364 0.472 0.000 0.000 0.082 Flesch
reading ease text 0.333 0.410 0.461 0.250 0.167 0.400 0.182 0.588 0.408 0.286
0.400 0.411 Proportion unique okens text 0.316 0.169 0.543 0.000 0.200 0.560
0.364 0.100 0.505 0.400 0.000 0.475 RIX text 0.316 0.329 0.306 0.000 0.000
0.154 0.429 0.429 0.421 0.286 0.333 0.040 Sentence length (mean) text 0.313
0.369 0.576 0.000 0.000 0.667 0.400 0.222 0.497 0.267 0.588 0.440 Syllables
per token (std) text 0.303 0.309 0.526 0.333 0.267 0.360 0.364 0.400 0.573
0.000 0.235 0.479 Token length (std) text 0.289 0.312 0.270 0.182 0.182 0.092
0.500 0.471 0.397 0.000 0.000 0.040 Dependency distance (mean) text 0.289
0.261 0.618 0.286 0.000 0.485 0.000 0.286 0.660 0.222 0.333 0.608 Proportion
Adjacent Dependencies (std) text 0.286 0.277 0.324 0.182 0.375 0.203 0.400
0.000 0.419 0.000 0.250 0.216 Alvenir audio 0.281 0.116 0.216 0.375 0.263
0.211 0.462 0.200 0.560 0.000 0.000 0.000 Number of tokens text 0.276 0.286
0.243 0.353 0.375 0.111 0.381 0.286 0.465 0.000 0.000 0.021 Flesch-Kincaid
grade text 0.273 0.312 0.394 0.118 0.154 0.203 0.182 0.353 0.416 0.222 0.286
0.315 Gunning Fog index text 0.265 0.343 0.408 0.154 0.000 0.296 0.167 0.471
0.434 0.200 0.286 0.277 Number unique tokens text 0.253 0.246 0.402 0.182
0.118 0.412 0.300 0.273 0.477 0.000 0.286 0.100 TF-IDF, 10000-d text 0.251
0.374 0.474 0.000 0.167 0.387 0.571 0.400 0.560 0.125 0.500 0.490 Lix text
0.229 0.332 0.451 0.200 0.200 0.391 0.167 0.375 0.463 0.000 0.182 0.286
Syllables per token (mean) text 0.228 0.372 0.277 0.348 0.333 0.103 0.000
0.286 0.275 0.000 0.222 0.237 Automated readability index text 0.219 0.263
0.658 0.000 0.000 0.593 0.182 0.364 0.636 0.143 0.125 0.667 Number sentences
text 0.217 0.174 0.154 0.375 0.273 0.088 0.387 0.320 0.397 0.000 0.000 0.021
eGeMAPS audio 0.212 0.278 0.374 0.182 0.200 0.475 0.000 0.286 0.389 0.000
0.000 0.000 Coleman-Liau index text 0.203 0.404 0.587 0.000 0.250 0.545 0.154
0.471 0.516 0.143 0.308 0.575 Mean token length text 0.202 0.214 0.401 0.267
0.125 0.232 0.000 0.143 0.437 0.000 0.000 0.317 XLS-R audio 0.202 0.189 0.085
0.364 0.312 0.191 0.000 0.000 0.000 0.000 0.000 0.021 Sentence polarity text
0.187 0.175 0.171 0.261 0.200 0.105 0.000 0.000 0.000 0.000 0.000 0.000 Dummy
baseline 0.167 0.167 0.163 0.000 0.000 0.000 0.167 0.167 0.163 0.167 0.167
0.163 Sentence length (std) text 0.136 0.300 0.210 0.273 0.300 0.116 0.273
0.400 0.439 0.000 0.500 0.134 Dependency distance (std) text 0.129 0.122 0.129
0.214 0.276 0.107 0.300 0.211 0.410 0.000 0.000 0.000 Gjallarhorn audio 0.116
0.097 0.097 0.263 0.222 0.155 0.000 0.000 0.000 0.200 0.167 0.235 ComParE
audio 0.071 0.071 0.106 0.000 0.000 0.000 0.000 0.000 0.000 0.286 0.286 0.426

Table E.2: Binary performances for MDD models

F1 - MDD Test Train Validation Ælectra text 0.923 1.000 0.444 Transformer
Ensemble ensemble 0.923 1.000 0.444 ScandiBERT text 0.800 1.000 0.800 Baseline
Ensemble ensemble 0.714 0.713 0.667 XLM-RoBERTa text 0.714 0.800 0.400
Sentence Transformer text 0.714 1.000 0.600 Sentence length (std) text 0.714
0.688 0.571 MFCCs audio 0.714 0.714 0.667 eGeMAPS audio 0.667 0.848 0.750
ComParE audio 0.667 0.788 0.667 XLS-R audio 0.667 0.788 0.667 Token length
(std) text 0.667 0.633 0.250 Gjallarhorn audio 0.667 0.779 0.667 Dummy
baseline 0.667 0.786 0.667 Automated readability index text 0.615 0.729 0.667
Dependency distance (mean) text 0.615 0.718 0.500 TF-IDF, 100-d text 0.615
0.800 0.250 Lix text 0.545 0.638 0.667 X-Vector audio 0.545 0.821 0.667 Flesch
reading ease text 0.545 0.788 0.667 TF-IDF, 1000-d text 0.533 0.982 0.667
Sentence polarity text 0.533 0.696 0.714 TF-IDF, 10000-d text 0.500 0.472
0.286 RIX text 0.500 0.713 0.615 Flesch-Kincaid grade text 0.500 0.745 0.615
Number unique tokens text 0.462 0.674 0.364 Proportion Adjacent Dependencies
(std) text 0.444 0.421 0.400 Syllables per token (std) text 0.400 0.841 0.400
All descriptives text 0.400 0.739 0.667 Sentence length (mean) text 0.400
0.800 0.667 FastText text 0.400 0.972 0.727 Alvenir audio 0.364 0.617 0.286
Number of tokens text 0.364 0.755 0.600 Gunning Fog index text 0.364 0.747
0.667 Coleman-Liau index text 0.222 0.652 0.667 Proportion Adjacent
Dependencies (mean) text 0.222 0.596 0.571 Proportion unique okens text 0.200
0.622 0.400 Mean token length text 0.200 0.535 0.667 Syllables per token
(mean) text 0.000 0.164 0.000 Number sentences text 0.000 0.000 0.000 Number
of characters text 0.000 0.568 0.286 Dependency distance (std) text 0.000
0.000 0.000

Table E.3: Binary performances for schizophrenia models

F1 - SCZ Test Validation Train Baseline Ensemble ensemble 0.833 0.800 0.887
TF-IDF, 1000-d text 0.833 0.600 0.816 Dependency distance (std) text 0.800
0.400 0.613 Alvenir audio 0.769 0.533 0.619 Gjallarhorn audio 0.769 0.462
0.554 XLM-RoBERTa text 0.750 0.667 0.984 ScandiBERT text 0.727 0.500 1.000
X-Vector audio 0.727 0.800 0.873 Transformer Ensemble ensemble 0.727 0.500
0.989 Number sentences text 0.714 0.444 0.519 Sentence polarity text 0.667
0.667 0.641 Syllables per token (mean) text 0.667 0.667 0.633 Dummy baseline
0.667 0.667 0.641 Dependency distance (mean) text 0.667 0.545 0.568 ComParE
audio 0.667 0.667 0.639 XLS-R audio 0.667 0.667 0.639 Proportion Adjacent
Dependencies (mean) text 0.615 0.667 0.652 All descriptives text 0.600 0.571
0.626 MFCCs audio 0.600 0.833 0.798 eGeMAPS audio 0.571 0.429 0.654 Sentence
length (std) text 0.571 0.615 0.625 TF-IDF, 10000-d text 0.545 0.727 0.897
Proportion unique okens text 0.545 0.462 0.643 FastText text 0.545 0.667 0.750
Sentence length (mean) text 0.545 0.727 0.642 Flesch reading ease text 0.545
0.500 0.656 Syllables per token (std) text 0.545 0.615 0.578 TF-IDF, 100-d
text 0.545 0.444 0.660 Ælectra text 0.500 0.727 0.793 Automated readability
index text 0.500 0.769 0.705 Number of characters text 0.500 0.615 0.641
Flesch-Kincaid grade text 0.500 0.600 0.695 Mean token length text 0.462 0.545
0.641 Number unique tokens text 0.444 0.545 0.565 Number of tokens text 0.400
0.500 0.560 Gunning Fog index text 0.400 0.545 0.741 Token length (std) text
0.364 0.462 0.750 Coleman-Liau index text 0.364 0.545 0.685 RIX text 0.364
0.727 0.653 Sentence Transformer text 0.250 0.500 0.578 Proportion Adjacent
Dependencies (std) text 0.250 0.600 0.437 Lix text 0.200 0.667 0.674

Table E.4: Binary performances for ASD models

F1 - ASD Test Validation Train Alvenir audio 0.750 0.706 0.716 Gjallarhorn
audio 0.667 0.667 0.701 XLS-R audio 0.667 0.667 0.701 Transformer Ensemble
ensemble 0.667 0.444 0.810 TF-IDF, 10000-d text 0.667 0.462 0.667 Dependency
distance (std) text 0.667 0.400 0.667 Dummy baseline 0.667 0.667 0.581
FastText text 0.667 0.444 0.941 Automated readability index text 0.615 0.545
0.824 Number of characters text 0.600 0.500 0.706 Number unique tokens text
0.600 0.500 0.750 Proportion unique okens text 0.571 0.500 0.667 Proportion
Adjacent Dependencies (std) text 0.545 0.615 0.444 Sentence Transformer text
0.545 0.444 1.000 RIX text 0.545 0.462 0.667 Number of tokens text 0.545 0.500
0.706 Flesch reading ease text 0.545 0.545 0.889 Coleman-Liau index text 0.533
0.571 0.667 Mean token length text 0.500 0.727 0.526 Sentence length (std)
text 0.462 0.462 0.636 ScandiBERT text 0.462 0.444 0.778 All descriptives text
0.462 0.500 0.667 XLM-RoBERTa text 0.444 0.500 1.000 MFCCs audio 0.429 0.769
0.853 Lix text 0.400 0.364 0.625 Gunning Fog index text 0.400 0.444 0.625
Dependency distance (mean) text 0.400 0.545 0.667 Syllables per token (std)
text 0.364 0.727 0.778 Sentence length (mean) text 0.364 0.667 0.800 Flesch-
Kincaid grade text 0.364 0.364 0.842 TF-IDF, 1000-d text 0.364 0.400 0.700 TF-
IDF, 100-d text 0.333 0.364 0.737 Proportion Adjacent Dependencies (mean) text
0.333 0.533 0.667 Ælectra text 0.286 0.500 1.000 Syllables per token (mean)
text 0.200 0.545 0.462 eGeMAPS audio 0.200 0.727 0.688 Token length (std) text
0.182 0.615 0.471 Number sentences text 0.000 0.000 0.000 ComParE audio 0.000
0.000 0.000 Sentence polarity text 0.000 0.000 0.000 Baseline Ensemble
ensemble 0.000 0.000 0.000

Figure E.1: Performance of the X-vector and TFIDF-100 model on the test set
stratified by sex, age, and Hamilton score.