# They Look Like Each Other: Case-based Reasoning for Explainable Depression
Detection on Twitter using Large Language Models

Mohammad Saeid Mahdavinejad
Kansas State University
<EMAIL_ADDRESS>Peyman Adibi
University of Isfahan
<EMAIL_ADDRESS>Amirhassan Monadjemi
National University of Singapore
<EMAIL_ADDRESS>Pascal Hitzler
Kansas State University
<EMAIL_ADDRESS>
###### Abstract

Depression is a common mental health issue that requires prompt diagnosis and
treatment. Despite the promise of social media data for depression detection,
the opacity of employed deep learning models hinders interpretability and
raises bias concerns. We address this challenge by introducing ProtoDep, a
novel, explainable framework for Twitter-based depression detection. ProtoDep
leverages prototype learning and the generative power of Large Language Models
to provide transparent explanations at three levels: (i) symptom-level
explanations for each tweet and user, (ii) case-based explanations comparing
the user to similar individuals, and (iii) transparent decision-making through
classification weights. Evaluated on five benchmark datasets, ProtoDep
achieves near state-of-the-art performance while learning meaningful
prototypes. This multi-faceted approach offers significant potential to
enhance the reliability and transparency of depression detection on social
media, ultimately aiding mental health professionals in delivering more
informed care.

They Look Like Each Other: Case-based Reasoning for Explainable Depression
Detection on Twitter using Large Language Models

Mohammad Saeid Mahdavinejad Kansas State University<EMAIL_ADDRESS>Peyman Adibi
University of Isfahan<EMAIL_ADDRESS>

Amirhassan Monadjemi National University of Singapore<EMAIL_ADDRESS>Pascal Hitzler Kansas State University<EMAIL_ADDRESS>

##  1 Introduction

Depression is a common mental health disorder that affects a significant
number of people worldwide. According to the National Institute of Mental
Health, roughly 22.8% of adults in the U.S. experience a diagnosable mental
illness annually (of Mental Health, 2023). Timely diagnosis and intervention
are crucial, as untreated or inadequately managed depression can lead to
severe consequences, including suicide and chronic, risky behaviors like
substance abuse (Goodwin et al., 2022).

Traditional methods for depression detection, heavily reliant on self-reported
information through online questionnaires, are often hampered by low
participation rates and potential selection bias. This has spurred the
exploration of alternative approaches, with social media platforms emerging as
a promising avenue (Chancellor and De Choudhury, 2020; Culotta, 2014; De
Choudhury and De, 2014; Guntuku et al., 2017; Paul and Dredze, 2011).

While deep learning models exhibit significant potential in detecting
depression on social media, their inherent "black box" nature presents a
significant challenge (Ji et al., 2021; Nguyen et al., 2022). This opacity
hinders practitioners’ ability to assess the validity of model predictions and
raises concerns about potential biases or errors within the models themselves.

Recent efforts toward more explainable models for mental health assessment
have primarily focused on two key approaches: post-hoc methods and
interpretable models. Post-hoc methods aim to explain the predictions of pre-
trained models retrospectively. However, they rely on approximations of a
model’s internal decision-making process, failing to explain why specific
input features are crucial (Nguyen et al., 2021).

In contrast, interpretable models are inherently designed to be transparent by
restricting their complexity. However, current state-of-the-art models
primarily rely on attention weights for explanation (Han et al., 2022). The
validity of these weights as reliable explanations remains under debate (Bibal
et al., 2022). Furthermore, these explanations often focus on low-level input
features, like individual posts or tweets, which may not align with the
higher-level concepts, such as symptoms, used by professionals.

This paper introduces ProtoDep, a novel explainable framework for depression
detection on Twitter. It leverages prototype learning, utilizing
representative data points (prototypes) to classify new instances. This
framework facilitates the identification of key factors contributing to users’
depressive behavior on social media through three distinct levels of
explanations. Firstly, by harnessing the generative power of Large Language
Models (LLMs), ProtoDep generates symptom-level explanations for each tweet
and user, resembling human-readable concepts employed in mental health
assessments. These explanations express the likelihood of each depression
symptom being present for the specific tweet and user. Secondly, the framework
provides a case-based explanation for the final prediction by comparing the
user profile to the top k most representative users within the dataset.
Finally, ProtoDep’s classification weights reveal its transparent decision-
making process.

Our evaluation on five benchmark datasets demonstrates that ProtoDep not only
achieves near state-of-the-art performance compared to "black-box" models but
also learns meaningful prototypes. The key contributions of this framework are
as follows:

  1. 1.

Novel explainable framework for mental health assessment on social media.

  2. 2.

Multi-level explanations: (i) explaining underlying user symptoms, (ii)
identifying similar users, and (iii) transparent decision-making.

  3. 3.

Maintains performance on five benchmark datasets.

  4. 4.

Leverages LLMs to learn meaningful prototypes.

By addressing the critical gap in explainability through its multi-faceted
approach, ProtoDep holds the potential to significantly improve the
reliability and transparency of depression detection on social media,
ultimately aiding mental health professionals in providing more informed care.

##  2 Related Work

The rise of social media has created exciting opportunities for mental health
research. Its real-time nature and extensive archives offer unique advantages
over traditional, retrospective studies. Researchers can track and potentially
predict risk factors over time, enabling timely interventions for vulnerable
communities (Kruzan et al., 2022; Livingston et al., 2014; Ridout and
Campbell, 2018).

This potential has driven exploration into using social media data to identify
and predict mental health challenges like anxiety (Ahmed et al., 2022;
Saifullah et al., 2021; Shen and Rudzicz, 2017) and depression (De Choudhury
et al., 2013a, b; Park et al., 2013; Tsugawa et al., 2015; Xu et al., 2021).

Initial research employed basic methods for analyzing social media data to
glean mental health insights (Coppersmith et al., 2014; De Choudhury et al.,
2013a; Coppersmith et al., 2014). Subsequently, the focus shifted towards
developing feature engineering techniques and machine learning models for
prediction (Birnbaum et al., 2017; Moreno et al., 2011; Nguyen et al., 2014;
Rumshisky et al., 2016; Tsugawa et al., 2015). For instance, De Choudhury et
al. (2013b) extracted linguistic features to build an SVM model for depression
prediction.

The advent of deep learning further revolutionized the field, eliminating the
need for hand-crafted features (Ji et al., 2018; Sawhney et al., 2018).
Tadesse et al. (2019) leveraged an LSTM-CNN model with word embeddings to
identify suicide ideation on Reddit, showcasing the power of deep learning
approaches.

Pre-trained language models (PLMs) have gained significant traction in natural
language processing (NLP) tasks, including mental health prediction (Han et
al., 2022; Ji et al., 2021; Nguyen et al., 2022). Jiang et al. (2020) utilized
BERT’s contextual representations for mental health issue detection, while
Otsuka et al. (2023) evaluated BERT-based models in clinical settings.
Additionally, multi-task learning approaches have been explored to predict
multiple mental health conditions simultaneously (Benton et al., 2017). Sarkar
et al. (2022) trained a multi-task model for predicting both depression and
anxiety, demonstrating the potential for joint prediction. However, these
multi-task models often lack the flexibility to adapt to new tasks.

Despite these advancements, explainability remains a critical challenge in
computational mental health prediction. While feature importance methods like
SHAP (Shapley et al., 1953; Datta et al., 2016; Lundberg and Lee, 2017) and
LIME (Ribeiro et al., 2016) provide insights into feature contributions, they
lack explanations for why specific features are crucial. Recent studies
suggest that example-based explanations, often utilizing Case-Based Reasoning
(CBR) techniques, resonate more with human users (Nguyen et al., 2021).
However, this approach is limited by the underlying model architecture.

Building upon these developments, our work introduces a novel framework to
depression detection on Twitter using prototype learning. Our key focus is to
provide interpretable explanations at multiple levels, addressing the critical
gap in explainability for mental health prediction on social media.

##  3 Method

This section presents ProtoDep, a novel framework for transparent reasoning
about mental health in social media. ProtoDep uses prototypes to represent
symptoms and users, enabling more interpretable explanations. Given a labeled
user with its tweets, ProtoDep performs classification in five steps
illustrated in Figure 1. Step 1: Embedding user tweets. Step 2: Learning
symptom prototypes. Step 3: Encoding the user. Step 4: Learning user
prototypes. Step 5: Performing classification.

Figure 1: Overview of the Proto-Dep

###  3.1 Preliminaries

Our objective is to determine if a specific user $u$ is depressed. Each user
has a collection of tweets $T$ where $T=\\{t_{1},t_{2},...,t_{n}\\}$ and $n$
is number of tweets. Once the model is trained, the aim is to predict a binary
label $\hat{y}$ for the user $u$, where $\hat{y}\in\\{0,1\\}$. If $\hat{y}=1$,
the user is identified as depressed. Subsequently, $\hat{y}$ is matched
against $y$, which is the ground truth.

###  3.2 Step 1: Embedding User’s Tweets

Given a set of tweets $T$ for a user $u$, first, we obtain an embedding of $T$
using a pre-trained sentence encoder. $E$ will be an embedding matrix of $T$.
Formally,

$E=\textnormal{TweetEncoder}(T)$ (1)

###  3.3 Step 2: Learning Symptom Prototypes

This step focuses on training symptom prototypes that faithfully capture the
essence of each depression symptom while maintaining close alignment with
actual tweets from the dataset. In simpler terms, we aim to develop
representations of symptoms that are both accurate and grounded in the
language used by individuals describing their experiences. However, due to the
absence of information about individual user symptoms or specific tweets
(limited to user-level labels), we propose a supervised initialization
strategy for prototypes, coupled with a specific loss function, to achieve
this objective. This step encompasses two sub-steps: A. Symptom Space Creation
and B. Symptom Space Optimization. Figure 2 $(a)$ illustrates a general schema
for this layer.

Figure 2: (a) Symptom Prototype Layer (b) User Prototype Layer

A. Symptom Space Creation: The first step in creating an embedding space for
symptom prototypes is identifying the underlying concepts or symptoms within
the space. We use the Patient Health Questionnaire - 9
(PHQ-9111https://www.apa.org/depression-guideline/patient-health-
questionnaire.pdf), one of the most widely used questionnaires to assess
depression (Kroenke et al., 2010), as a reference for defining the concepts.
The PHQ-9 is a self-administered questionnaire that measures the presence and
severity of nine depressive symptoms over two weeks. It has been validated by
multiple studies and is regarded as a reliable and accurate measure of
depression (Levis et al., 2019). The nine symptoms are Depressed Mood (S1),
Loss of Interest or Pleasure (S2), Sleep Disturbance (S3), Fatigue or Low
Energy (S4), Changes in Appetite (S5), Feelings of Guilt or Worthlessness
(S6), Difficulty Concentrating (S7), Psychomotor Agitation or Retardation
(S8), and Suicidal Thoughts (S9). We consider these symptoms as the base
concepts in our embedding space to simulate human reasoning processes and
enhance ProtoDep interpretability.

The second step is to initialize a set of prototypes for each concept.
Manually creating exemplary sets for each prototype proves resource-intensive
and needs iterative refinement. To address this challenge, we leverage the
generative capabilities of LLMs. Specifically, we employ GPT-4 to
automatically generate relevant examples, focusing on different aspects of a
given symptom. These examples serve as our initial set of prototypes for
subsequent training. We note that the number of prototypes is an important
hyperparameter, and generating different numbers of examples from GPT-4 for
each experiment is inconsistent and impractical. Therefore, we generate a
maximum number of examples once and use the mean of the embedded examples as a
base prototype for each symptom. Then, for each experiment, we sample around
each base prototype with a normal distribution. We define $p_{base}^{j}$ as
the base prototype and $P^{j}$ to be set of $m$ prototypes for the symptom $j$
by:

$P^{j}\sim\mathcal{N}(p_{base}^{j},\,\sigma^{2})\,.$ (2)

where $\sigma^{2}$ is variance. Therefore, $P$ will be set for all symptom
prototypes.

B. Symptom Space Optimization: Given the lack of tweet-level labels, we
propose a novel approach that leverages supervised-initialized prototypes.
Specifically, we formulate the optimization process as a multi-label
classification task, where each tweet is labeled with the nearest symptom
within the embedding space. By adopting this strategy, we effectively use
prior knowledge from the initial symptom space while accommodating the lack of
labeled tweets. As a result, we define the total symptom loss as the sum of
two terms:

$L_{symp}=\lambda_{1}*L_{sinkhorn}+\lambda_{2}*L_{mse}$ (3)

where $\lambda_{1}$ and $\lambda_{2}$ are hyperparameters, and $L_{sinkhorn}$
is Sinkhorn loss, a mathematical tool that computes optimal transport between
two probability distributions (Cuturi, 2013). This choice has several
advantages over conventional loss functions. It enhances the stability and
robustness of the training process, mitigates the impact of noise and
overfitting, and accelerates the convergence rate (Feydy et al., 2019). For
calculating $L_{sinkhorn}$, first, we calculate a cosine similarity between a
tweet embedding $e_{i}$ and a symptom $p_{k}^{j}$:

$\textnormal{sim}_{i,j,k}=\cos(e_{i},p_{k}^{j})$ (4)

where $i\in\\{1,...,n\\}$ and $j\in\\{1,...,9\\}$ and $k\in\\{1,...,m\\}$.
Then, we assign a label denoted as $s_{i}$ to each tweet based on its nearest
symptom.

$s_{i}=\arg\max_{j}\frac{1}{m}\sum_{k=1}^{m}\textnormal{sim}_{i,j,k}$ (5)

By defining all tweet embeddings with the same symptom as:

$E^{j}=\\{e_{i}|s_{i}=j\\}$ (6)

The $L_{sinkhorn}$ will be:

$L_{sinkhorn}=\frac{1}{9}\sum_{j=1}^{9}\textnormal{sinkhorn}(E^{j},P^{j})$ (7)

Finally, $L_{mse}$ is a mean squared error loss and a measure of the
difference between the input samples and their reconstructions using the
nearest prototype. It encourages the prototypes to be representative of the
input data. For this purpose, we find the index of the nearest prototype to
each tweet embedding $c_{i}$ as:

$c_{i}=\arg\max_{j,k}\textnormal{sims}_{i,j,k}$ (8)

Next, we define the nearest prototype to $e_{i}$ as:

$p_{c_{i}}=\\{p_{k}^{j}|(j,k)=c_{i}\\}$ (9)

Now the $L_{mse}$ will be calculated as:

$L_{mse}=\frac{1}{n}\sum_{i=1}^{n}(e_{i}-p_{c_{i}})^{2}$ (10)

###  3.4 Step 3: Encoding the User

Following the approach of (Han et al., 2022), we use a multi-layer attention
mechanism and a feed-forward neural network to encode the sequential user
behavior. The encoder model for the ProtoDep framework can vary depending on
the problem domain and the data modality, which we will elaborate on in
section 5. We also append the similarity scores between the tweets and the
symptom prototypes to improve the tweet representations. Our experiments show
this can help the model learn better user representations and perform more
accurately. Formally:

$\textnormal{SympSims}=\frac{1}{m}\sum_{k=1}^{m}\textnormal{sim}_{i,j,k}$ (11)

And user embedding $e_{u}$ will be:

$e_{u}=\textnormal{UserEncoder}(E\oplus\textnormal{SympSims})$ (12)

###  3.5 Step 4: Learning User Prototypes

This step provides transparent, case-based reasoning to evaluate the user’s
depressive behavior. It follows the same principle as the learning symptom
prototype step and consists of two sub-steps: A. User Space Creation and B.
User Space Optimization. Figure 2 $(b)$ illustrates this step.

A. User Space Creation: Social media datasets for depression detection often
exhibit an imbalance between the number of users or tweets in each class. This
may negatively impact the reasoning of deep learning models as they may
prioritize the majority class during training. Inspired by (Das et al., 2022),
we encourage the model to find the best examples for both classes to find a
more effective decision boundary between them. Unlike the symptom prototype
space, which relies on predefined prototypes, the user prototype space allows
the model to learn the prototypes from the data. Consequently, we randomly
initialize $k$ different vectors per class as initial prototypes.

B. User Space Optimization: In this step, we adopt the same optimization
strategy as in step 2, but with a crucial difference. We leverage the user-
level labels to learn the prototypes in a supervised fashion—this way, we do
not require the computation of $s_{i}$. We denote the total loss for this step
as $L_{user}$.

###  3.6 Step 5: Classification

The final step of our model is to classify the users based on their similarity
to symptoms and user prototypes. First, we calculate the average of all tweet-
symptom similarities, providing an overall measure of the similarity between a
user’s tweets and the symptom prototypes. Then, we concatenate these scores
with the user prototype similarities and feed this into a linear layer
followed by a Softmax function to obtain the final classification. We use
binary cross-entropy (BCE) loss for this step, and the total loss function for
our model will be:

$L=L_{symp}+L_{user}+L_{BCE}$ (13)

##  4 Experiment Results

Dataset. We employ an openly available Twitter dataset, MDL, specifically
designed for depression detection. For comparison purposes, we use the
specific version of the MDL dataset provided by (Han et al., 2022). In this
dataset, individuals who posted tweets containing predefined phrases
indicative of depression, such as "I’m," "I was," "I am," or "I’ve been
diagnosed with depression," were labeled as depressive. Conversely, those who
never posted any tweet containing the term "depress" were labeled as non-
depressive. (Han et al., 2022) employed a random selection of users to create
five distinct datasets. The train, validation, and test sets utilize 60%, 20%,
and 20% of the entire dataset. This resulted in 2,524 users in the train set
and 842 users in each validation and test set. More details regarding the
dataset can be found in (Han et al., 2022).

Baselines. To evaluate the ProtoDep framework, we compare its performance to
four established depression detection baselines. Given its similarity in
approach, we consider (Han et al., 2022) the most relevant and significant
baseline for our model. Additionally, we compare our results to (Gui et al.,
2019; Lin et al., 2020; Zhang et al., 2021b), as these studies appeared to be
most pertinent to our work.

Setup. We trained all models using a GeForce RTX 3090 with 64GB of RAM and
Pytorch 2.0.1. We tuned the hyperparameters on the validation data and
optimized all neural models with the AdamW algorithm. The learning rate and
the batch size were 1e-3 and 64, respectively. We applied early stopping based
on the F1 score on the validation data. The maximum number of tweets per input
was 200. For symptom and user prototypes, we set the number of prototypes per
class to $m=7$ and $k=3$, respectively. We sampled normally around the base
prototypes with $\sigma=0.025$. We also used two-layer attention for Step 3.
We chose "all-mpnet-base-v2" (Reimers and Gurevych, 2019) as our Tweet Encoder
Model. We discuss more on these in section 5.

Result (1): Classification Performance. We assess ProtoDep using five
benchmark datasets and compare it against state-of-the-art methods, as
demonstrated in Table 1. ProtoDep achieves a competitive 94.4% average F1
score, providing more intuitive and interpretable prototypes for
classification decisions and maintaining consistent performance across
different randomly sampled datasets (D1-D5). We also explore another variant
of ProtoDep, ProtoDep-Acc, to demonstrate its power to achieve state-of-the-
art performance. ProtoDep-Acc leverages a different loss function for learning
prototypes. We discuss more about ProtoDep-Acc in section 5.

Model | P | R | F1
---|---|---|---
(Gui et al., 2019) | 0.900 | 0.901 | 0.900
(Lin et al., 2020) | 0.903 | 0.870 | 0.886
(Zhang et al., 2021b) | 0.909 | 0.904 | 0.912
(Han et al., 2022) | 0.975 | 0.969 | 0.972
ProtoDep-Acc | 0.985 | 0.995 | 0.990
ProtoDep (avg) | 0.934 | 0.954 | 0.944
D1 | 0.964 | 0.953 | 0.959
D2 | 0.898 | 0.951 | 0.924
D3 | 0.984 | 0.991 | 0.987
D4 | 0.931 | 0.931 | 0.931
D5 | 0.893 | 0.946 | 0.919
Table 1: Depression Detection Results. ProtoDep-Acc and ProtoDep (avg) are the
average values across D1-D5.

Result (2): Explainable Prototypes. We evaluate the quality of ProtoDep’s
learned prototypes using two distinct methods. First, we compare these
prototypes with manually labeled ground truth prototypes created by domain
experts. To achieve this, we leverage a specialized dictionary developed by
(Yazdavar et al., 2017). This lexicon contains an extensive collection of
depression-related terms specifically associated with the nine symptom
categories of the PHQ-9. These terms have undergone meticulous curation to
capture the subtle nuances inherent in depression symptoms.

By aligning our learned prototypes with this established lexicon, we can
assess their relevance and meaningfulness within the context of depression. To
quantify this alignment, we compute the mean representation for each symptom
prototype and subsequently measure its cosine similarity with the embedded
ground truth lexicon. Figure 3 visually illustrates this similarity, revealing
a robust alignment between the learned symptom prototypes and the clinically
relevant terms associated with depression. Notably, we focus on the diagonal
elements of the similarity matrix, which correspond to the self-similarity
scores. These high values indicate that our generated prototypes align with
the clinically relevant terminology. Another observation drawn from the figure
indicates a heightened similarity between the first two symptoms, specifically
‘Lack of Interest’ and ‘Feeling Down,’ and other symptoms. One plausible
explanation for this phenomenon is the initialization bias as the initial GPT
symptoms exhibited a similar pattern.

Figure 3: Similarity between learned symptom prototypes and ground truth
lexicon.

Second, we assessed the prototypes’ discriminative power by using the PRIDE
score (Ni et al., 2022), inspired by (Zhang et al., 2021a). This method
defines a "real" prototype for each category by averaging its data points and
then measures the similarity between these real prototypes and the learned
ones. A high PRIDE score indicates a learned prototype’s effectiveness in
capturing its designated category while differentiating itself from others. We
assume the nearest tweet to each symptom in the ground truth lexicon is the
real prototype in the dataset. Figure 4 demonstrates that ProtoDep achieves
positive PRIDE scores for all symptoms’ prototypes, implying effective
learning of distinct and representative symptom prototypes.

Figure 4: Visualization of the PRIDE score for learned prototypes for
ProtoDep, ProtoDep-Acc, and ProtoDep (Sinkhorn) Models.

We assess the efficacy of learned user prototypes by reporting the PRIDE
score. Specifically, we identify a representative ‘real’ user for each class
by computing the mean of all users within that class and subsequently
calculating the PRIDE score. Notably, both depressed and non-depressed classes
exhibit positive PRIDE scores (0.27 and 0.33, respectively), affirming that
ProtoDep effectively captures a meaningful prototype space for users.

Result (3): Transparent Reasoning. Beyond accurate depression detection,
ProtoDep offers valuable insights into its decision-making process through
several avenues. Examining the weights assigned to various symptoms within its
final layer unveils their relative importance in user classification. As
illustrated in Figure 5, ProtoDep across diverse datasets prioritizes symptoms
like "Fatigue or low energy" and "Lack of Interest," mirroring human expert
judgment reported in (Yazdavar et al., 2017). Interestingly, it assigns less
weight to "Sleep Disorder" and "Concentration problems," potentially due to
the ambiguity of these symptoms in textual data. For example, the tweet "lost
in my own mind" might not explicitly mention keywords indicating
"Concentration problems," making accurate classification challenging. This
finding highlights the inherent difficulty in capturing nuanced depressive
symptoms, even for human experts.

Furthermore, ProtoDep’s user embedding layer with stacked attention layers
holds promise for interpreting user classifications, similar to Han et al.
(2022). We analyzed attention scores to identify tweets that significantly
influence user classification. However, echoing prior research (Bibal et al.,
2022; Wen et al., 2022; Pruthi et al., 2020), our extensive evaluation across
both methods revealed no statistically significant association between
attended tweets and those crucial for accurate classification. This suggests
that while attention weights offer glimpses into model behavior, they might
not directly explain specific classification outcomes in this context.

Figure 5: Classification weights (absolute values) for different symptoms over
all datasets.

##  5 Ablation Study

We consider four different settings to validate the impact of different
hyperparameters.

Symptom Prototype Initialization. In this study, we explore alternative
methods for initializing symptom prototypes. We compare two novel
initialization approaches with the baseline method that utilizes a pre-trained
language model (LLM) for symptom initialization. In the first setting, we
leverage the ground truth lexicon as the foundation for symptom prototypes. We
extract ground truth embeddings from this lexicon and then sample additional
prototypes around them for each symptom class. A well-constructed lexicon
captures domain-specific nuances and expert knowledge, which can enhance the
quality of symptom prototypes. In the second setting, we depart from direct
lexicon embeddings. Instead, we identify the nearest tweet in our dataset to
each lexicon symptom and use its embedding as the basis for symptom
prototypes. We then continue sampling around these tweet-grounded prototypes.
By anchoring the initial prototypes to actual tweets, we aim to improve their
relevance and alignment with real-world symptom expressions. Our experimental
results in Table 2 demonstrate that both the lexicon-based and tweet-grounded
initialization outperform the LLM baseline. Notably, the curated lexicon’s
consideration of various combinations of the depression-indicative keywords
contributes to its effectiveness. However, the marginal performance difference
between these two settings and the baseline suggests that LLMs can achieve
competitive results even in scenarios lacking human annotation or domain-
specific knowledge.

Loss Function | GPT-4 | Lexicon | Lex. Tweets
---|---|---|---
Triplet+MSE.+Ent. | 0.990 | 0.990 | 0.989
Sinkhorn+MSE. | 0.947 | 0.969 | 0.954
Sinkhorn | 0.936 | 0.964 | 0.949
Avg. val. F1 | 0.958 | 0.975 | 0.964
Table 2: Comparison of average validation F1 scores for different
initialization methods compared to different loss functions. "MSE." refers to
Mean Squared Error loss, and "Ent." refers to Entropy loss.

Prototype Loss Function. To assess the influence of different loss functions
within the ProtoDep framework, we implemented two evaluation settings. The
first setting, ProtoDep (Sinkhorn), exclusively employed the Sinkhorn loss to
isolate the impact of the MSE loss in ProtoDep. The second setting, ProtoDep-
Acc, combined Triplet loss with MSE and Entropy losses, mimicking conventional
loss functions commonly used in prototype learning research.

As observed in Table 2, ProtoDep-Acc achieved strong performance,
demonstrating the capability of the ProtoDep framework. Notably, it surpassed
other settings in terms of F1 scores. However, as illustrated in Table 4,
while ProtoDep-Acc yielded higher F1 scores, its PRIDE scores indicated a
failure to learn meaningful symptom-level prototypes. Conversely, ProtoDep
(Sinkhorn) achieved better PRIDE scores, signifying successful prototype
learning, but yielded lower F1 scores than ProtoDep.

This evaluation highlights a trade-off between classification performance and
interpretability within the ProtoDep framework. While ProtoDep-Acc excelled in
F1 scores, its learned prototypes lacked interpretability. In contrast,
ProtoDep and ProtoDep (Sinkhorn) prioritized interpretability through
meaningful prototypes but compromised classification accuracy. These findings
suggest the need for careful consideration of loss function selection in
balancing interpretability and performance within the ProtoDep framework.

Attention Mechanism. Instead of employing the conventional user embedding
attention mechanism, we introduce a single-layer Multi-head attention
configuration. The outcomes are presented in Table 3.

Attention | Baseline | Multi-head
---|---|---
Avg. val. F1 | 0.9407 | 0.823
Table 3: Comparison of different attention mechanism.

Number of Prototypes. Figure 6 depicts the average F1 score across all five
datasets, varying the number of prototypes. While the overall trend suggests
that the model performs better with fewer prototypes, a nuanced examination
reveals that individual datasets often favor a larger number of prototypes.

Figure 6: Average F1 score over all test datasets for different numbers of
prototypes.

##  6 Conclusion

In this paper, we proposed ProtoDep, a novel framework that combines prototype
learning and Large Language Models (LLMs) to provide explainable depression
detection on Twitter. Unlike conventional "black-box" models, ProtoDep can
generate transparent and interpretable explanations at three levels: symptom-
level, case-based, and transparent decision-making weights. We evaluated
ProtoDep on five benchmark datasets and showed it achieves competitive
performance while learning meaningful and representative prototypes. We argue
that ProtoDep has the potential to improve the trustworthiness and
accountability of depression detection on social media, as well as to
facilitate the understanding and intervention of mental health professionals.
As a future work, we plan to investigate the applicability of ProtoDep to
other social media platforms and mental health domains and enhance the
explanation generation process with more clinical and contextual information.
To sum up, ProtoDep is a novel and promising framework for explainable
depression detection on social media, which can contribute to the well-being
of individuals and society.

##  7 Ethical Consideration

We used a publicly available dataset introduced by Han et al. (2022). Our
investigation focuses exclusively on textual content, deliberately excluding
user profile information. We emphasize the ethical implications of our model’s
application and strongly discourage misuse compromising data security or
privacy principles.

##  8 Limitations

While ProtoDep offers a promising approach to explainable depression detection
on Twitter, it is essential to acknowledge potential limitations associated
with this framework:

Data Representativeness: The performance and generalizability of ProtoDep
heavily rely on the quality and representativeness of the training data.
Biases or limitations within the training data can be reflected in the learned
prototypes, potentially leading to biased predictions for demographics or
groups underrepresented in the data.

Tweet Encoder Model: The effectiveness of the learned prototypes largely
relies on the selection of the tweet encoder model. If the encoder model
cannot accurately differentiate between various initial symptom prototypes, it
will fail to converge to a meaningful prototype space at the end of the
training process.

Privacy Concerns: Utilizing social media data for depression detection raises
inherent privacy concerns. It is crucial to ensure user privacy throughout the
data collection, processing, and explanation generation stages, adhering to
relevant ethical and legal guidelines.

Limited Scope: While ProtoDep focuses on Twitter data, it might not generalize
to other social media platforms with different content characteristics and
user behaviors. Further research is needed to explore the framework’s
applicability across diverse platforms.

Clinical Validation: Although ProtoDep aims to aid mental health professionals
by providing transparent case-based reasoning for each prediction, its
effectiveness in real-world clinical settings requires rigorous validation
through controlled studies with healthcare providers.

Hyperparameter Tuning: During our experiments with ProtoDep, we observed that
model performance and prototype quality significantly varied for different
hyperparameters. Careful hyperparameter optimization is necessary for
adaptation to different domains.

By acknowledging these limitations and actively working towards addressing
them, future research can refine ProtoDep to ensure its responsible, ethical,
and effective implementation in supporting mental health diagnosis and
intervention.

## Acknowledgements

This research was funded by the National Science Foundation under Grant
2033521 A1, titled “KnowWhereGraph: Enriching and Linking Cross-Domain
Knowledge Graphs using Spatially-Explicit AI Technologies.” The views,
findings, conclusions, or recommendations presented in this paper are those of
the authors and do not necessarily reflect the views of the National Science
Foundation.

## References

  * Ahmed et al. (2022) Arfan Ahmed, Sarah Aziz, Carla T Toro, Mahmood Alzubaidi, Sara Irshaidat, Hashem Abu Serhan, Alaa A Abd-Alrazaq, and Mowafa Househ. 2022.  Machine learning models to detect anxiety and depression through social media: A scoping review.  _Computer Methods and Programs in Biomedicine Update_ , page 100066.
  * Benton et al. (2017) Adrian Benton, Margaret Mitchell, and Dirk Hovy. 2017.  Multi-task learning for mental health using social media text.  _arXiv preprint arXiv:1712.03538_.
  * Bibal et al. (2022) Adrien Bibal, Rémi Cardon, David Alfter, Rodrigo Wilkens, Xiaoou Wang, Thomas François, and Patrick Watrin. 2022.  Is attention explanation? an introduction to the debate.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3889–3900.
  * Birnbaum et al. (2017) Michael L Birnbaum, Sindhu Kiranmai Ernala, Asra F Rizvi, Munmun De Choudhury, and John M Kane. 2017.  A collaborative approach to identifying social media markers of schizophrenia by employing machine learning and clinical appraisals.  _Journal of medical Internet research_ , 19(8):e7956.
  * Chancellor and De Choudhury (2020) Stevie Chancellor and Munmun De Choudhury. 2020.  Methods in predictive techniques for mental health status on social media: a critical review.  _NPJ digital medicine_ , 3(1):43.
  * Coppersmith et al. (2014) Glen Coppersmith, Craig Harman, and Mark Dredze. 2014.  Measuring post-traumatic stress disorder in twitter.  In _Proceedings of the international AAAI conference on web and social media_ , volume 8, pages 579–582.
  * Culotta (2014) Aron Culotta. 2014.  Estimating county health statistics with twitter.  In _Proceedings of the SIGCHI conference on human factors in computing systems_ , pages 1335–1344.
  * Cuturi (2013) Marco Cuturi. 2013.  Sinkhorn distances: Lightspeed computation of optimal transport.  _Advances in neural information processing systems_ , 26.
  * Das et al. (2022) Anubrata Das, Chitrank Gupta, Venelin Kovatchev, Matthew Lease, and Junyi Jessy Li. 2022.  ProtoTEx: Explaining model decisions with prototype tensors.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2986–2997, Dublin, Ireland. Association for Computational Linguistics.
  * Datta et al. (2016) Anupam Datta, Shayak Sen, and Yair Zick. 2016.  Algorithmic transparency via quantitative input influence: Theory and experiments with learning systems.  In _2016 IEEE symposium on security and privacy (SP)_ , pages 598–617. IEEE.
  * De Choudhury et al. (2013a) Munmun De Choudhury, Scott Counts, and Eric Horvitz. 2013a.  Social media as a measurement tool of depression in populations.  In _Proceedings of the 5th annual ACM web science conference_ , pages 47–56.
  * De Choudhury and De (2014) Munmun De Choudhury and Sushovan De. 2014.  Mental health discourse on reddit: Self-disclosure, social support, and anonymity.  In _Proceedings of the international AAAI conference on web and social media_ , volume 8, pages 71–80.
  * De Choudhury et al. (2013b) Munmun De Choudhury, Michael Gamon, Scott Counts, and Eric Horvitz. 2013b.  Predicting depression via social media.  In _Proceedings of the international AAAI conference on web and social media_ , volume 7, pages 128–137.
  * Feydy et al. (2019) Jean Feydy, Thibault Séjourné, François-Xavier Vialard, Shun-ichi Amari, Alain Trouvé, and Gabriel Peyré. 2019.  Interpolating between optimal transport and mmd using sinkhorn divergences.  In _The 22nd International Conference on Artificial Intelligence and Statistics_ , pages 2681–2690. PMLR.
  * Goodwin et al. (2022) Renee D. Goodwin, Lisa C. Dierker, Melody Wu, Sandro Galea, Christina W. Hoven, and Andrea H. Weinberger. 2022.  Trends in u.s. depression prevalence from 2015 to 2020: The widening treatment gap.  _American Journal of Preventive Medicine_ , 63(5):726–733.
  * Gui et al. (2019) Tao Gui, Liang Zhu, Qi Zhang, Minlong Peng, Xu Zhou, Keyu Ding, and Zhigang Chen. 2019.  Cooperative multimodal approach to depression detection in twitter.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 33, pages 110–117.
  * Guntuku et al. (2017) Sharath Chandra Guntuku, David B Yaden, Margaret L Kern, Lyle H Ungar, and Johannes C Eichstaedt. 2017.  Detecting depression and mental illness on social media: an integrative review.  _Current Opinion in Behavioral Sciences_ , 18:43–49.
  * Han et al. (2022) Sooji Han, Rui Mao, and Erik Cambria. 2022.  Hierarchical attention network for explainable depression detection on twitter aided by metaphor concept mappings.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 94–104.
  * Ji et al. (2018) Shaoxiong Ji, Celina Ping Yu, Sai-fu Fung, Shirui Pan, and Guodong Long. 2018.  Supervised learning for suicidal ideation detection in online user content.  _Complexity_ , 2018.
  * Ji et al. (2021) Shaoxiong Ji, Tianlin Zhang, Luna Ansari, Jie Fu, Prayag Tiwari, and Erik Cambria. 2021.  Mentalbert: Publicly available pretrained language models for mental healthcare.  _arXiv preprint arXiv:2110.15621_.
  * Jiang et al. (2020) Zheng Ping Jiang, Sarah Ita Levitan, Jonathan Zomick, and Julia Hirschberg. 2020.  Detection of mental health from reddit via deep contextualized representations.  In _Proceedings of the 11th international workshop on health text mining and information analysis_ , pages 147–156.
  * Kroenke et al. (2010) Kurt Kroenke, Robert L. Spitzer, Janet B.W. Williams, and Bernd Löwe. 2010.  The patient health questionnaire somatic, anxiety, and depressive symptom scales: a systematic review.  _General Hospital Psychiatry_ , 32(4):345–359.
  * Kruzan et al. (2022) Kaylee Payne Kruzan, Kofoworola DA Williams, Jonah Meyerhoff, Dong Whi Yoo, Linda C O’Dwyer, Munmun De Choudhury, and David C Mohr. 2022.  Social media-based interventions for adolescent and young adult mental health: A scoping review.  _Internet interventions_ , page 100578.
  * Levis et al. (2019) Brooke Levis, Andrea Benedetti, and Brett D Thombs. 2019.  Accuracy of patient health questionnaire-9 (phq-9) for screening to detect major depression: individual participant data meta-analysis.  _BMJ_ , 365.
  * Lin et al. (2020) Chenhao Lin, Pengwei Hu, Hui Su, Shaochun Li, Jing Mei, Jie Zhou, and Henry Leung. 2020.  Sensemood: depression detection on social media.  In _Proceedings of the 2020 international conference on Multimedia Retrieval_ , pages 407–411.
  * Livingston et al. (2014) James D Livingston, Michelle Cianfrone, Kimberley Korf-Uzan, and Connie Coniglio. 2014.  Another time point, a different story: one year effects of a social media intervention on the attitudes of young people towards mental health issues.  _Social psychiatry and psychiatric epidemiology_ , 49:985–990.
  * Lundberg and Lee (2017) Scott M Lundberg and Su-In Lee. 2017.  A unified approach to interpreting model predictions.  _Advances in neural information processing systems_ , 30.
  * Moreno et al. (2011) Megan A Moreno, Lauren A Jelenchick, Katie G Egan, Elizabeth Cox, Henry Young, Kerry E Gannon, and Tara Becker. 2011.  Feeling bad on facebook: Depression disclosures by college students on a social networking site.  _Depression and anxiety_ , 28(6):447–455.
  * Nguyen et al. (2021) Giang Nguyen, Daeyoung Kim, and Anh Nguyen. 2021.  The effectiveness of feature attribution methods and its correlation with automatic evaluation scores.  _Advances in Neural Information Processing Systems_ , 34:26422–26436.
  * Nguyen et al. (2014) Thin Nguyen, Dinh Phung, Bo Dao, Svetha Venkatesh, and Michael Berk. 2014.  Affective and content analysis of online depression communities.  _IEEE transactions on affective computing_ , 5(3):217–226.
  * Nguyen et al. (2022) Thong Nguyen, Andrew Yates, Ayah Zirikly, Bart Desmet, and Arman Cohan. 2022.  Improving the generalizability of depression detection by leveraging clinical questionnaires.  _arXiv preprint arXiv:2204.10432_.
  * Ni et al. (2022) Xinzhe Ni, Yong Liu, Hao Wen, Yatai Ji, Jing Xiao, and Yujiu Yang. 2022.  Multimodal prototype-enhanced network for few-shot action recognition.  _arXiv preprint arXiv:2212.04873_.
  * of Mental Health (2023) National Institute of Mental Health. 2023.  Mental illness.
  * Otsuka et al. (2023) Norio Otsuka, Yuu Kawanishi, Fumimaro Doi, Tsutomu Takeda, Kazuki Okumura, Takahira Yamauchi, Shuntaro Yada, Shoko Wakamiya, Eiji Aramaki, and Manabu Makinodan. 2023.  Diagnosing psychiatric disorders from history of present illness using a large-scale linguistic model.  _Psychiatry and Clinical Neurosciences_.
  * Park et al. (2013) Minsu Park, David McDonald, and Meeyoung Cha. 2013.  Perception differences between the depressed and non-depressed users in twitter.  In _Proceedings of the international AAAI conference on web and social media_ , volume 7, pages 476–485.
  * Paul and Dredze (2011) Michael Paul and Mark Dredze. 2011.  You are what you tweet: Analyzing twitter for public health.  In _Proceedings of the international AAAI conference on web and social media_ , volume 5, pages 265–272.
  * Pruthi et al. (2020) Danish Pruthi, Mansi Gupta, Bhuwan Dhingra, Graham Neubig, and Zachary C Lipton. 2020.  Learning to deceive with attention-based explanations.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4782–4793.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
  * Ribeiro et al. (2016) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016.  " why should i trust you?" explaining the predictions of any classifier.  In _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_ , pages 1135–1144.
  * Ridout and Campbell (2018) Brad Ridout and Andrew Campbell. 2018.  The use of social networking sites in mental health interventions for young people: systematic review.  _Journal of medical Internet research_ , 20(12):e12244.
  * Rumshisky et al. (2016) Anna Rumshisky, Marzyeh Ghassemi, Tristan Naumann, Peter Szolovits, VM Castro, TH McCoy, and RH Perlis. 2016.  Predicting early psychiatric readmission with natural language processing of narrative discharge summaries.  _Translational psychiatry_ , 6(10):e921–e921.
  * Saifullah et al. (2021) Shoffan Saifullah, Yuli Fauziah, and Agus Sasmito Aribowo. 2021.  Comparison of machine learning for sentiment analysis in detecting anxiety based on social media data.  _arXiv preprint arXiv:2101.06353_.
  * Sarkar et al. (2022) Shailik Sarkar, Abdulaziz Alhamadani, Lulwah Alkulaib, and Chang-Tien Lu. 2022.  Predicting depression and anxiety on reddit: a multi-task learning approach.  In _2022 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM)_ , pages 427–435. IEEE.
  * Sawhney et al. (2018) Ramit Sawhney, Prachi Manchanda, Puneet Mathur, Rajiv Shah, and Raj Singh. 2018.  Exploring and learning suicidal ideation connotations on social media with deep learning.  In _Proceedings of the 9th workshop on computational approaches to subjectivity, sentiment and social media analysis_ , pages 167–175.
  * Shapley et al. (1953) Lloyd S Shapley et al. 1953.  A value for n-person games.
  * Shen and Rudzicz (2017) Judy Hanwen Shen and Frank Rudzicz. 2017.  Detecting anxiety through reddit.  In _Proceedings of the Fourth Workshop on Computational Linguistics and Clinical Psychology—From Linguistic Signal to Clinical Reality_ , pages 58–65.
  * Tadesse et al. (2019) Michael Mesfin Tadesse, Hongfei Lin, Bo Xu, and Liang Yang. 2019.  Detection of suicide ideation in social media forums using deep learning.  _Algorithms_ , 13(1):7.
  * Tsugawa et al. (2015) Sho Tsugawa, Yusuke Kikuchi, Fumio Kishino, Kosuke Nakajima, Yuichi Itoh, and Hiroyuki Ohsaki. 2015.  Recognizing depression from twitter activity.  In _Proceedings of the 33rd annual ACM conference on human factors in computing systems_ , pages 3187–3196.
  * Wen et al. (2022) Bingyang Wen, Koduvayur Subbalakshmi, and Fan Yang. 2022.  Revisiting attention weights as explanations from an information theoretic perspective.  In _NeurIPS’22 Workshop on All Things Attention: Bridging Different Perspectives on Attention_.
  * Xu et al. (2021) Xuhai Xu, Prerna Chikersal, Janine M Dutcher, Yasaman S Sefidgar, Woosuk Seo, Michael J Tumminia, Daniella K Villalba, Sheldon Cohen, Kasey G Creswell, J David Creswell, et al. 2021.  Leveraging collaborative-filtering for personalized behavior modeling: a case study of depression detection among college students.  _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ , 5(1):1–27.
  * Yazdavar et al. (2017) Amir Hossein Yazdavar, Hussein S Al-Olimat, Monireh Ebrahimi, Goonmeet Bajaj, Tanvi Banerjee, Krishnaprasad Thirunarayan, Jyotishman Pathak, and Amit Sheth. 2017.  Semi-supervised approach to monitoring clinical depressive symptoms in social media.  In _Proceedings of the 2017 IEEE/ACM International Conference on advances in Social Networks Analysis and Mining 2017_ , pages 1191–1198.
  * Zhang et al. (2021a) Baoquan Zhang, Xutao Li, Yunming Ye, Zhichao Huang, and Lisai Zhang. 2021a.  Prototype completion with primitive knowledge for few-shot learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3754–3762.
  * Zhang et al. (2021b) Dongyu Zhang, Nan Shi, Ciyuan Peng, Abdul Aziz, Wenhong Zhao, and Feng Xia. 2021b.  Mam: A metaphor-based approach for mental illness detection.  In _International Conference on Computational Science_ , pages 570–583. Springer.