# Mental-LLM: Leveraging Large Language Models for Mental Health Prediction
via Online Text Data

Xuhai Xu<EMAIL_ADDRESS>0000-0001-5930-3899 Massachusetts Institute of
Technology & University of WashingtonUSA ,  Bingsheng Yao  0009-0004-8329-4610
Rensselaer Polytechnic InstituteUSA ,  Yuanzhe Dong  0009-0006-2013-1157
Stanford UniversityUSA ,  Saadia Gabriel  0009-0001-9353-951X Massachusetts
Institute of TechnologyUSA ,  Hong Yu  0000-0001-9263-5035 University of
Massachusetts LowellUSA ,  James Hendler  0000-0003-3056-1960 Rensselaer
Polytechnic InstituteUSA ,  Marzyeh Ghassemi  0000-0001-6349-7251
Massachusetts Institute of TechnologyUSA ,  Anind K. Dey  0000-0002-3004-0770
University of WashingtonUSA and  Dakuo Wang  0000-0001-9371-9441 Northeastern
UniversityUSA

###### Abstract.

Advances in large language models (LLMs) have empowered a variety of
applications. However, there is still a significant gap in research when it
comes to understanding and enhancing the capabilities of LLMs in the field of
mental health. In this work, we present a comprehensive evaluation of multiple
LLMs on various mental health prediction tasks via online text data, including
Alpaca, Alpaca-LoRA, FLAN-T5, GPT-3.5, and GPT-4. We conduct a broad range of
experiments, covering zero-shot prompting, few-shot prompting, and instruction
fine-tuning. The results indicate a promising yet limited performance of LLMs
with zero-shot and few-shot prompt designs for mental health tasks. More
importantly, our experiments show that instruction finetuning can
significantly boost the performance of LLMs for all tasks simultaneously. Our
best-finetuned models, Mental-Alpaca and Mental-FLAN-T5, outperform the best
prompt design of GPT-3.5 (25 and 15 times bigger) by 10.9% on balanced
accuracy and the best of GPT-4 (250 and 150 times bigger) by 4.8%. They
further perform on par with the state-of-the-art task-specific language model.
We also conduct an exploratory case study on LLMs’ capability on mental health
reasoning tasks, illustrating the promising capability of certain models such
as GPT-4. We summarize our findings into a set of action guidelines for
potential methods to enhance LLMs’ capability for mental health tasks.
Meanwhile, we also emphasize the important limitations before achieving
deployability in real-world mental health settings, such as known racial and
gender bias. We highlight the important ethical risks accompanying this line
of research.

Mental Health, Large Language Model, Instruction Finetuning

††copyright: acmlicensed††journal: IMWUT††journalyear: 2024††journalvolume:
8††journalnumber: 1††article: 32††publicationmonth: 3††doi:
10.1145/3643540††ccs: Human-centered computing Ubiquitous and mobile
computing††ccs: Applied computing Life and medical sciences

##  1\. Introduction

The recent surge of Large Language Models (LLMs), such as GPT-4 (Bubeck et
al., 2023), PaLM (Chowdhery et al., 2022), FLAN-T5 (Chung et al., 2022), and
Alpaca (Taori et al., 2023), demonstrates the promising capability of large
pre-trained models to solve various tasks in zero-shot settings (i.e., tasks
not encountered during training). Example tasks include question answering
(Omar et al., 2023; Robinson and Wingate, 2023), logic reasoning (Wei et al.,
2023; Zhou et al., 2023), machine translation (Brants et al., 2007; Gulcehre
et al., 2017), etc. A number of experiments have revealed that, built on
hundreds of billions of parameters, these LLMs have started to show the
capability to understand the human common sense beneath the natural language
and do proper reasoning and inference accordingly (Bubeck et al., 2023; Nori
et al., 2023).

Among different applications, one particular question yet to be answered is
how well LLMs can understand human mental health states through natural
language. Mental health problems represent a significant burden for
individuals and societies worldwide. A recent report suggested that more than
20% of adults in the U.S. experience at least one mental disorder in their
lifetime (America, 2022) and 5.6% have suffered from a serious psychotic
disorder that significantly impairs functioning (men, 2023b). The global
economy loses around $1 trillion annually in productivity due to depression
and anxiety alone (men, 2023a).

In the past decade, there has been a plethora of research in natural language
processing (NLP) and computational social science on detecting mental health
issues via online text data such as social media content (e.g., (Guntuku et
al., 2017; Eichstaedt et al., 2018; Coppersmith et al., 2015b; De Choudhury et
al., 2013a; De Choudhury and De, 2014)). However, most of these studies have
focused on building domain-specific machine learning (ML) models (i.e., one
model for one particular task, such as stress detection (Nijhawan et al.,
2022; Guntuku et al., 2019), depression prediction (Eichstaedt et al., 2018;
Tadesse et al., 2019a; Xu et al., 2019, 2021a), or suicide risk assessment (De
Choudhury et al., 2016; Coppersmith et al., 2018a)). Even for traditional pre-
trained language models such as BERT, they need to be finetuned for specific
downstream tasks (Devlin et al., 2019; Liu et al., 2019). Some studies have
also explored the multi-task setup (Benton et al., 2017), such as predicting
depression and anxiety at the same time (Sarkar et al., 2022). However, these
models are constrained to predetermined task sets, offering limited
flexibility. From a different aspect, another line of research has been
exploring the application of chatbots for mental health services (Cameron et
al., 2019, 2017; Lee et al., 2020). Most chatbots are simply rule-based and
can benefit from more advanced models that empower the chatbots (Abd-alrazaq
et al., 2019; Lee et al., 2020). Despite the growing research efforts of
empowering AI for mental health, it’s important to note that existing
techniques can sometimes introduce bias and even provide harmful advice to
users (Irene Y. Chen et al., 2019; Timmons et al., 2022; Lovejoy, 2019).

Since natural language is a major component of mental health assessment and
treatment (Sharma and De Choudhury, 2018; Gkotsis et al., 2016), LLMs could be
a powerful tool for understanding end-users’ mental states through their
written language. These instruction-finetuned and general-purpose models can
understand a variety of inputs and obviate the need to train multiple models
for different tasks. Thus, we envision employing a single LLM for a variety of
mental health-related tasks, such as multiple question-answering, reasoning,
and inference. This vision opens up a wide range of opportunities for UbiComp,
Human-Computer Interaction (HCI), and mental health communities, such as
online public health monitoring systems (Patel et al., 2018; Graham et al.,
2019), mental-health-aware personal chatbots (Abd-Alrazaq et al., 2021;
Denecke et al., 2020; Kernan Freire et al., 2023), intelligent assistants for
mental health therapists (Sharma et al., 2021), online moderation tools
(Franco et al., 2023), daily mental health counselors and supporters (Sharma
et al., 2023), etc. However, there is a lack of investigation into
understanding, evaluating, and improving the capability of LLMs for mental-
health-related tasks.

There are few recent studies on the evaluation of LLMs (e.g., ChatGPT) on
mental-health-related tasks, most of which are in zero-shot settings with
simple prompt engineering (Yang et al., 2023a; Amin et al., 2023; Lamichhane,
2023). Researchers have shown preliminary results that LLMs have the initial
capability of predicting mental health disorders using natural language, with
promising but still limited performance compared to state-of-the-art domain-
specific NLP models (Yang et al., 2023a; Lamichhane, 2023). This remaining gap
is expected since existing general-purpose LLMs are not specifically trained
on mental health tasks. However, to achieve our vision of leveraging LLMs for
mental health support and assistance, we need to address the research
question: How to improve LLMs’ capability of mental health tasks?

We conducted a series of experiments with six LLMs, including Alpaca (Taori et
al., 2023) and Alpaca-LoRA (LoRA-finetuned LLaMA on Alpaca dataset) (Hu et
al., 2021), which are representative open-source models focused on dialogue
and other tasks; FLAN-T5 (Chung et al., 2022), a representative open-source
model focused on task-solving; LLaMA2 (Touvron et al., 2023b), one of the most
advanced open-source model released by Meta; GPT-3.5 (noa, 2022) and GPT-4
(Bubeck et al., 2023), representative close-sourced LLMs over 100 billion
parameters. Considering the data availability, we leveraged online social
media datasets with high-quality human-generated mental health labels. Due to
the ethical concerns of existing AI research for mental health, we aim to
benchmark LLMs’ performance as an initial step before moving toward real-life
deployment. Our experiments contained three stages: (1) zero-shot prompting,
where we experimented with various prompts related to mental health, (2) few-
shot prompting, where we inserted examples into prompt inputs, and (3)
instruction-finetuning, where we finetuned LLMs on multiple mental-health
datasets with various tasks.

Our results show that the zero-shot approach yields promising yet limited
performance on various mental health prediction tasks across all models.
Notably, FLAN-T5 and GPT-4 show encouraging performance, approaching the
state-of-the-art task-specific model. Meanwhile, providing a few shots in the
prompt can improve the model performance to some extent ($\overline{\Delta}$ =
4.1%), but the advantage is limited. Finally and most importantly, we found
that instruction-finetuning significantly enhances the model performance
across multiple mental-health-related tasks and various datasets
simultaneously. Our finetuned Alpaca and FLAN-T5, namely Mental-Alpaca and
Mental-FLAN-T5, significantly outperform the best of GPT-3.5 across zero-shot
and few-shot settings ($\times$25 and 15 bigger than Alpaca and FLAN-T5) by an
average of 10.9% on balance accuracy, as well as the best of GPT-4 by 4.8%
($\times$250 and 150 bigger than Alpaca and FLAN-T5). Meanwhile, Mental-Alpaca
and Mental-FLAN-T5 can further perform on par with the task-specific state-of-
the-art Mental-RoBERTa (Ji et al., 2021). We further conduct an exploratory
case study on LLM’s capability of mental health reasoning (i.e., explaining
the rationale behind their predictions). Our results illustrate the promising
future of certain LLMs like GPT-4, while also suggesting critical failure
cases that need future research attention. We open-source our code and model
at https://github.com/neuhai/Mental-LLM.

Our experiments present a comprehensive evaluation of various techniques to
enhance LLMs’ capability in the mental health domain. However, we also note
that our technical results do not imply deployability. There are many
important limitations of leveraging LLMs in mental health settings, especially
along known racial and gender gaps (Abid et al., 2021; Ghosh and Caliskan,
2023). We discuss the important ethical risks to be addressed before achieving
real-world deployment.

The contribution of our paper can be summarized as follows:

  1. (1)

We present a comprehensive evaluation of prompt engineering, few-shot, and
finetuning techniques on multiple LLMs in the mental health domain.

  2. (2)

With online social media data, our results reveal that finetuning on a variety
of datasets can significantly improve LLM’s capability on multiple mental-
health-specific tasks across different datasets simultaneously.

  3. (3)

We release our model Mental-Alpaca and Mental-FLAN-T5 as open-source LLMs
targeted at multiple mental health prediction tasks.

  4. (4)

We provide a few technical guidelines for future researchers and developers on
turning LLMs into experts in specific domains. We also highlight the important
ethical concerns regarding leveraging LLMs for health-related tasks.

##  2\. Background

We briefly summarize the related work in leveraging online text data for
mental health prediction (Sec. 2.1). We also provide an overview of the
ongoing research in LLMs and their application in the health domain (Sec.
2.2).

###  2.1. Online Text Data and Mental Health

Online platforms, especially social media platforms, have been acknowledged as
a promising lens that is capable of revealing insights into the psychological
states, health, and well-being of both individuals and populations (Paul and
Dredze, 2011; Culotta, 2014; Chancellor and De Choudhury, 2020; Guntuku et
al., 2017; De Choudhury and De, 2014). In the past decade, there has been
extensive research about leveraging content analysis and social interaction
patterns to identify and predict risks associated with mental health issues,
such as anxiety (Ahmed et al., 2022; Saifullah et al., 2021; Shen and Rudzicz,
2017), major depressive disorder (Park et al., 2021; De Choudhury et al.,
2013b; Tsugawa et al., 2015; De Choudhury et al., 2013a; Xu et al., 2021a,
2023), suicide ideation (De Choudhury et al., 2016; Ruder et al., 2011; Burnap
et al., 2015; Tadesse et al., 2019b; Coppersmith et al., 2018b), and others
(Coppersmith et al., 2015a; Mitchell et al., 2015; Coppersmith et al., 2014;
Saha et al., 2017). The real-time nature of social media, along with its
archival capabilities, often helps in mitigating retrospective bias. The rich
amount of social media data also facilitates the identification, monitoring,
and potential prediction of risk factors over time. In addition to observation
and detection, social media platforms could further serve as effective
channels to offer in-time assistance to communities at risk (Livingston et
al., 2014; Ridout and Campbell, 2018; Kruzan et al., 2022).

From the computational technology perspective, early research started with
basic methods (Coppersmith et al., 2014; Mitchell et al., 2015; De Choudhury
et al., 2013a). For example, pioneering work by Coppersmith et al.
(Coppersmith et al., 2014) employed correlation analysis to reveal the
relationship between social media language data and mental health conditions.
Since then, researchers have proposed a wide range of feature engineering
methods and built machine-learning models for the prediction (Moreno et al.,
2011; Nguyen et al., 2014; Birnbaum et al., 2017; Tsugawa et al., 2015;
Rumshisky et al., 2016). For example, De Choudhury et al. (De Choudhury et
al., 2013b) extracted a number of linguistic styles and other features to
build an SVM model to perform depression prediction. Researchers have also
explored deep-learning-based models for mental health prediction to obviate
the need for hand-crafted features (Sawhney et al., 2018; Ji et al., 2018).
For instance, Tadesse et al. (Tadesse et al., 2019b) employed an LSTM-CNN
model and took word embeddings as the input to detect suicide ideation on
Reddit. More recently, pre-trained language models have become a popular
method for NLP tasks, including mental health prediction tasks (Ji et al.,
2021; Nguyen et al., 2022; Han et al., 2022). For example, Jiang et al. (Jiang
et al., 2020) used the contextual representations from BERT as input features
for mental health issue detection. Otsuka et al. (Otsuka et al., [n. d.])
evaluated the performance of BERT-based pre-trained models in clinical
settings. Meanwhile, researchers have also explored the multi-task setup
(Benton et al., 2017) that aims to predict multiple labels. For example,
Sarkar et al. (Sarkar et al., 2022) trained a multi-task model to predict
depression and anxiety at the same time. However, these multi-task models are
constrained to a predetermined task set and thus have limited flexibility. Our
work joins the same goal and aims to achieve a more flexible multi-task
capability. We focus on the next-generation technology of instruction-
finetuned LLMs, leverage their power in natural language understanding, and
explore their capability on mental health tasks with social media data.

###  2.2. LLM and Health Applications

After the great success of Transformer-based language models such as BERT
(Devlin et al., 2019) and GPT (Radford et al., 2018), researchers and
practitioners have advanced towards larger and more powerful language models
(e.g., GPT-3 (Brown et al., 2020) and T5 (Raffel et al., 2020)). Meanwhile,
researchers have proposed instruction finetuning, a method that utilizes
varied prompts across multiple datasets and tasks. This technique guides a
model during training and generation phases to perform diverse tasks within a
single unified framework (Wei et al., 2022). These instruction-finetuned LLMs,
such as GPT-4 (Bubeck et al., 2023), PaLM (Chowdhery et al., 2022), FLAN-T5
(Chung et al., 2022), LLaMA (Touvron et al., 2023a), Alpaca (Taori et al.,
2023), contain tens to hundreds of billions of parameters, demonstrating
promising performance across a variety of tasks, such as question answering
(Omar et al., 2023; Robinson and Wingate, 2023), logic reasoning (Wei et al.,
2023; Zhou et al., 2023), machine translation (Brants et al., 2007; Gulcehre
et al., 2017), etc.

Researchers have explored the capability of these LLMs in health fields (Jiang
et al., 2023; Singhal et al., 2023; Li et al., 2023b; Liu et al., 2023; Wu et
al., 2023; Nori et al., 2023; Lupetti et al., 2023). For example, Singhal et
al. (Singhal et al., 2023) finetuned PaLM-2 on medical domains and achieved
86.5% on MedQA dataset. Similarly, Wu et al. (Wu et al., 2023) finetuned LLaMA
on medical papers and showed promising results on multiple biomedical QA
datasets. Jo et al. (Jo et al., 2023) explored the deployment of LLMs for
public health scenarios. Jiang et al. (Jiang et al., 2023) trained a medical
language model on unstructured clinical notes from the electronic health
record and fine-tuned it across a wide range of clinical and operational
predictive tasks. Their evaluation indicates that such a model can be used for
various clinical tasks.

Table 1. A Summary of LLM-related Research for Mental Health Applications.

| LLMs | Methods | Tasks
---|---|---|---
Lamichhane (Lamichhane, 2023) | GPT-3.5 | Zero-shot | Classification
Amin et al. (Amin et al., 2023) | GPT-3.5 | Zero-shot | Classification
Yang et al. (Yang et al., 2023a) | GPT-3.5 | Zero-shot | Classification, Reasoning
Mental-LLaMA (Yang et al., 2023b) | LLaMA2, Vicuna (LLaMA-based) |  Zero-shot, Few-shot, Instruction Finetuning | Classification, Reasoning
Mental-LLM (Our Work) |  Alpaca, Alapca-LoRA, FLAN-T5, LLaMA2, GPT-3.5, GPT-4 |  Zero-shot, Few-shot, Instruction Finetuning | Classification, Reasoning
|  |  |

There is relatively less work in the mental health domain. Some work explored
the capability of LLMs for sentiment analysis and emotion reasoning (Qin et
al., 2023; Zhong et al., 2023; Kocoń et al., 2023).

Closer to our study, Lamichhane (Lamichhane, 2023) and Amin et al. (Amin et
al., 2023) tested the performance of ChatGPT (GPT-3.5) on multiple
classification tasks (e.g., stress, depression, and suicide detection). The
results showed that ChatGPT shows the initial potential for mental health
applications, but it still has a great room for improvement, with at least
5-10% performance gaps on accuracy and F1-score. Yang et al. (Yang et al.,
2023a) further evaluated the potential reasoning capability of GPT-3.5 for
reasoning tasks (e.g., potential stressors). However, most previous studies
focused solely on zero-shot prompting and did not explore other methods to
improve the performance of LLMs. Very recently, Yang et al. (Yang et al.,
2023b) released Mental-LLaMA, a set of LLaMA-based models finetuned on mental
health datasets for a set of mental health tasks.  Table 1 summarizes the
recent related work exploring LLMs’ capabilities on mental-health-related
tasks. None of the existing work explores the capability other than LLaMA or
GPT-3.5. In this work, we present a comprehensive and systematic exploration
of multiple LLMs’ performance on mental health tasks, as well as multiple
methods to improve their capabilities.

##  3\. Methods

We introduce our experiment design with LMMs on multiple mental health
prediction task setups, including zero-shot prompting (Sec. 3.1), few-shot
prompting (Sec. 3.2), and instruction finetuning (Sec. 3.3). These setups are
model-agnostic, and we will present the details of language models and
datasets employed for our experiment in the next section.

###  3.1. Zero-shot Prompting

The language understanding and reasoning capability of LLMs have enabled a
wide range of applications without the need for any domain-specific data, but
only providing appropriate prompts (Kojima et al., 2022; Wei et al., 2021).
Therefore, we start with prompt design for mental health tasks in a zero-shot
setting.

The goal of prompt design is to empower a pre-trained general-purpose LLM to
achieve good performance on tasks in the mental health domain. We propose a
general zero-shot prompt template ($\textit{Prompt}_{ZS}$) that consists of
four parts:

(1)
$\textit{Prompt}_{ZS}=\textit{TextData}+\textit{Prompt}_{\textit{Part1-S}}+\textit{Prompt}_{\textit{Part2-Q}}+\textit{OutputConstraint}$

where TextData is the online text data generated by end-users.
Prompt${}_{\textit{Part1-S}}$ provides specifications for a mental health
prediction target. Prompt${}_{\textit{Part2-Q}}$ poses the question for LLMs
to answer. And OutputConstraint controls the output of models (e.g., “Only
return yes or no” for a binary classification task).

We propose several design strategies for Prompt${}_{\textit{Part1-S}}$, as
shown in the top part of Table 2: (1) Basic, which leaves it as blank; (2)
Context Enhancement, which provides more social media context about the
TextData; (3) Mental Health Enhancement, which inserts mental health concept
by asking the model to act as an expert. (4) Context & Mental Health
Enhancement, which combines both enhancement strategies by asking the model to
act as a mental health expert under the social media context.

As for Prompt${}_{\textit{Part2-Q}}$, we mainly focus on two categories of
mental health prediction targets: (1) predicting critical mental states, such
as stress or depression, and (2) predicting high-stake risk actions, such as
suicide. We tailor the question description for each category. Moreover, for
both categories, we explore binary and multi-class classification tasks 111We
also conduct an exploratory case study on mental health reasoning tasks.
Please see more details in Sec. 5.4.. Thus, we also make small modifications
based on specific tasks to ensure appropriate questions (see Sec. 4 for our
mental health tasks). The bottom part of Table 2 summarizes the mapping.

Table 2. Prompt Design for Mental Health Prediction Tasks.
Prompt${}_{\textit{Part1-S}}$ aims to provide a better specification for LLMs
and Prompt${}_{\textit{Part2-Q}}$ poses the questions for LLMs to answer. For
Part 1, we propose three strategies: context enhancement, mental health
enhancement, and the combination of both. As for Part 2, we design different
content for multiple mental health problem categories and prediction tasks.
For each part, we propose two to three versions to improve its variation.

Strategy | Prompt${}_{\textit{Part1-S}}$
---|---
Basic |  $\bullet$ { blank }
Context Enhancement |  $\bullet$ This person wrote this paragraph on social media. $\bullet$ Consider this post on social media to answer the question.
Mental Health Enhancement |  $\bullet$ As a psychologist, read the post on social media and answer the question. $\bullet$ If you are a psychologist, read the post on social media and answer the question.
Context & Mental Health Enhancement |  $\bullet$ This person wrote this paragraph on social media. As a psychologist, read the post on social media and answer the question. $\bullet$ This person wrote this paragraph on social media. As a psychologist, consider the mental well-being condition expressed in this post, read the post on social media, and answer the question.
Category | Task | Prompt${}_{\textit{Part2-Q}}$
Mental state (e.g., stressed, depressed) |  Binary classification (e.g., yes or no) |  $\bullet$ Is the poster [stressed]? $\bullet$ Is the poster of this post [stressed]? $\bullet$ Determine if the poster of this post is [stressed].
Multi-class classification (e.g., multiple levels) |  $\bullet$ Which level is the person [stressed]? $\bullet$ How [stressed] is the person? $\bullet$ Determine how [stressed] the person is.
Critical risk action (e.g., suicide) |  Binary classification (e.g., yes or no) |  $\bullet$ Does the poster want to [suicide]? $\bullet$ Is the poster likely to [suicide]? $\bullet$ Determine if the poster of this post want to [suicide].
Multi-class classification (e.g., multiple levels) |  $\bullet$ Which level of [suicide] risk does the person have? $\bullet$ How [suicidal] is the person? $\bullet$ Determine which level of [suicide] risk does the person have.
|  |

For both Prompt${}_{\textit{Part1-S}}$ and Prompt${}_{\textit{Part2-Q}}$, we
propose several versions to improve its variability. We then evaluate these
prompts on multiple LLMs on different datasets and compare their performance.

###  3.2. Few-shot Prompting

In order to provide more domain-specific information, researchers have also
explored few-shot prompting with LLMs by providing few-shot demonstrations to
support in-context learning (e.g., (Agrawal et al., 2022; Dang et al., 2022)).
Note that these few examples are used solely in prompts, and the model
parameters remain unchanged. The intuition is to present a few “examples” for
the model to learn domain-specific knowledge in situ. In our setting, we also
test this strategy by adding additional randomly sampled
[$\textit{Prompt}_{ZS}-\textit{label}$] pairs. The design of the few-shot
prompt ($\textit{Prompt}_{FS}$) is straightforward:

(2) $\textit{Prompt}_{FS}=[\textit{Sample
Prompt}_{ZS}-\textit{label}]_{M}+\textit{Prompt}_{ZS}$

where $M$ is the number of prompt-label pairs and is capped by the input
length limit of a model. Note that both the $\textit{Sample Prompt}_{ZS}$ and
$\textit{Prompt}_{ZS}$ follow Eq. 1 and employ the same design of
Prompt${}_{\textit{Part1-S}}$ and Prompt${}_{\textit{Part2-Q}}$ to ensure
consistency.

###  3.3. Instruction Finetuning

In contrast to the few-shot prompting strategy in Sec. 3.2, the goal of this
strategy is closer to the traditional few-shot transfer learning, where we
further train the model with a small amount of domain-specific data (e.g.,
(Huang et al., 2022; Xu et al., 2021b; Liu et al., 2023)). We experiment with
multiple finetuning strategies.

####  3.3.1. Single-dataset Finetuning

Following most of the previous work in the mental health field (Yang et al.,
2023a; Coppersmith et al., 2015b; De Choudhury et al., 2016), we first conduct
basic finetuning on a single dataset (the training set). This finetuned model
can be tested on the same dataset (the test set) to evaluate its performance
and different datasets to evaluate its generalizability.

####  3.3.2. Multi-dataset Finetuning

From Sec. 3.1 to Sec. 3.3.1, we have been focusing on one single mental health
dataset $D$. More interestingly, we further experiment with finetuning across
multiple datasets simultaneously. Specifically, we leverage instruction
finetuning to enable LLMs to handle multiple tasks in different datasets
(Brown et al., 2020).

It is noteworthy that such an instruction finetuning setup differs from the
state-of-the-art mental-health-specific models (e.g., Mental-RoBERTa (Ji et
al., 2021)). The previous models are finetuned for a specific task, such as
depression prediction or suicidal ideation prediction. Once trained on task A,
the model becomes specific to task A and is only suitable for solving that
particular task. In contrast, we finetune LLMs on several mental health
datasets, employing diverse instructions for different tasks across these
datasets in a single iteration. This enables them to handle multiple tasks
without additional task-specific finetuning.

For both single- and multi-dataset finetuning, we follow the same two steps:

(3) $\displaystyle\begin{split}\text{Step 1:}&\text{ Finetune with
}[\textit{Prompt}_{ZS}-\textit{label}]_{\sum\limits^{I}N_{D_{i-train}}}\\\
\text{Step 2:}&\text{ Test with
}[\textit{Prompt}_{ZS}]_{\sum\limits^{I}N_{D_{i-test}}}\end{split}$

where $N_{D_{i-train}}$/$N_{D_{i-test}}$ is the total size of the
training/test dataset $D_{i}$, $I$ represents the set of datasets used for
finetuning, and $i$ indicates the specific dataset index ($i\in I,|I|\geq 1$).
Both $\textit{Prompt}_{ZS\textit{-train}}$ and
$\textit{Prompt}_{ZS\textit{-test}}$ follow Eq. 1. Similar to the few-shot
setup in Eq. 2, they employ the same design of Prompt${}_{\textit{Part1-S}}$
and Prompt${}_{\textit{Part2-Q}}$.

##  4\. Implementation

Our method design is agnostic to specific datasets or models. In this section,
we introduce the specific datasets (Sec. 4.1) and models (Sec. 4.2) involved
in our experiments. In particular, we highlight our instructional-finetuned
open-source models Mental-Alpaca and Mental-FLAN-T5 (Sec. 4.2.1).  We also
provide an overview of our experiment setup and evaluation metrics (Sec. 4.3).

###  4.1. Datasets and Tasks

Our experiment is based on four well-established datasets that are commonly
employed for mental health analysis. These datasets were collected from Reddit
due to their high-quality and availability. It is noteworthy that we
intentionally avoid using datasets with weak labels based on specific
linguistic patterns (e.g., whether a user ever stated “I was diagnosed with
X”). Instead, we used ones with human expert annotations or supervision. We
define six diverse mental health prediction tasks based on these datasets.

  * •

Dreaddit (Turcan and McKeown, 2019): This dataset collected posts via Reddit
PRAW API (Praw-Dev, [n. d.]) from Jan 1, 2017 to Nov 19, 2018, which contains
ten subreddits in the five domains (abuse, social, anxiety, PTSD, and
financial) and includes 2929 users’ posts. Multiple human annotators rated
whether sentence segments showed the stress of the poster, and the annotations
were aggregated to generate final labels. We used this dataset for a post-
level binary stress prediction (Task 1).

  * •

DepSeverity (Naseem et al., 2022): This dataset leveraged the same posts
collected in (Turcan and McKeown, 2019), but with a different focus on
depression. Two human annotators followed DSM-5 (Regier et al., 2013) and
categorized posts into four levels of depression: minimal, mild, moderate, and
severe. We employed this dataset for two post-level tasks: binary depression
prediction (i.e., whether a post showed at least mild depression, Task 2), and
four-level depression prediction (Task 3).

  * •

SDCNL (Haque et al., 2021): This dataset also collected posts from Python
Reddit API, including r/SuicideWatch and r/Depressionfrom 1723 users. Through
manual annotation, they labeled whether each post showed suicidal thoughts. We
employed this dataset for the post-level binary suicide ideation prediction
(Task 4).

  * •

CSSRS-Suicide (Gaur et al., 2019): This dataset contains posts from 15 mental
health-related subreddits from 2181 users between 2005 and 2016. Four
practicing psychiatrists followed Columbia Suicide Severity Rating Scale
(C-SSRS) guidelines (Posner et al., 2008) to manually annotate 500 users on
suicide risks in five levels: supportive, indicator, ideation, behavior, and
attempt. We leveraged this dataset for two user-level tasks: binary suicide
risk prediction (i.e., whether a user showed at least suicide indicator, Task
5), and five-level suicide risk prediction (Task 6).

In order to test the generalizability of our methods, we also leveraged three
other datasets from various platforms. Similarly, all datasets contain human
annotations as labels.

  * •

Red-Sam (Sampath and Durairaj, 2022; Kayalvizhi et al., 2022): This dataset
also collected posts with PRAW API (Praw-Dev, [n. d.]) , involving five
subreddits (Mental Health, depression, loneliness, stress, anxiety). Two
domain experts’ annotations were aggregated to generate depression labels. We
used this dataset as an external evaluation dataset on binary depression
detection (Task 2). Although also from Reddit, this dataset was not involved
in few-shot learning or instruction finetuning. We cross-checked datasets to
ensure there were no overlapping posts.

  * •

Twt-60Users (Jamil et al., 2017): This dataset collected twitters from 60
users during 2015 with Twitter API. Two human annotators labeled every tweet
with depression labels. We used this non-Reddit dataset as an external
evaluation dataset on depression detection (Task 2). Note that this dataset
has imbalanced labels (90.7% False), as most tweets did not indicate mental
distress.

  * •

SAD (Mauriello et al., 2021): This dataset contains SMS-like text messages
with nine types of daily stressor categories (work, school, financial problem,
emotional turmoil, social relationships, family issues, health, everyday
decision-making, and other). These messages were written by 3578 humans. We
used this non-Reddit dataset as an external evaluation dataset on binary
stress detection (Task 1). Note that human crowd-workers write the messages
under certain stressor-triggered instructions. Therefore, this dataset has
imbalanced labels on the other side (94.0% True).

Table 3 summarizes the information of the seven datasets and six mental health
prediction tasks. For each dataset, we conducted an 80%/20% train-test split.
Notably, to avoid data leakage, each user’s data were placed exclusively in
either the training or test set.

Table 3. Summary of Seven Mental Health Datasets Employed for Our Experiment.
The top four datasets are used for both training and testing, while the bottom
three datasets are used for external evaluation. We define six diverse mental
health prediction tasks on these datasets.

Dataset |  Task | Dataset Size | Text Length (Token)
---|---|---|---
Dreaddit (Turcan and McKeown, 2019) Source: Reddit |  #1: Binary Stress Prediction post-level |  Train: 2838 (47.6% False, 52.4% True) Test: 715 (48.4% False, 51.6% True) |  Train: 114 $\pm$ 41 Test: 113 $\pm$ 39
DepSeverity (Naseem et al., 2022) Source: Reddit |  #2: Binary Depression Prediction post-level |  Train: 2842 (72.9% False, 17.1% True) Test: 711 (72.3% False, 17.7% True) |  Train: 114 $\pm$ 41 Test: 113 $\pm$ 37
#3: Four-level Depression Prediction post-level |  Train: 2842 (72.9% Minimum, 8.4% Mild, 11.2% Moderate, 7.4% Severe) Test: 711 (72.3% Minimum, 7.2% Mild, 11.5% Moderate, 10.0% Severe) |  Train: 114 $\pm$ 41 Test: 113 $\pm$ 37
SDCNL (Haque et al., 2021) Source: Reddit |  #4: Binary Suicide Ideation Prediction post-level |  Train: 1516 (48.1% False, 51.9% True) Test: 379 (49.1% False, 50.9% True) |  Train: 101 $\pm$ 161 Test: 92 $\pm$ 119
CSSRS-Suicide (Gaur et al., 2019) Source: Reddit |  #5: Binary Suicide Risk Prediction user-level |  Train: 400 (20.8% False, 79.2% True) Test: 100 (25.0% False, 75.0% True) |  Train: 1751 $\pm$ 2108 Test: 1909 $\pm$ 2463
#6: Five-level Suicide Risk Prediction user-level |  Train: 400 (20.8% Supportive, 20.8% Indicator, 34.0% Ideation, 14.8% Behavior, 9.8% Attempt) Test: 100 (25.0% Supportive, 16.0% Indicator, 35.0% Ideation, 18.0% Behavior, 6.0% Attempt) |  Train: 1751 $\pm$ 2108 Test: 1909 $\pm$ 2463
Red-Sam (Sampath and Durairaj, 2022) Source: Reddit |  #2: Binary Depression Prediction post-level |  External Evaluation: 3245 (26.1% False, 73.9% True) |  External Evaluation: 151 $\pm$ 139
Twt-60Users (Jamil et al., 2017) Source: Twitter |  #2: Binary Depression Prediction post-level |  External Evaluation: 8135 (90.7% False, 9.3% True) |  External Evaluation: 15 $\pm$ 7
SAD (Mauriello et al., 2021) Source: SMS-like |  #1: Binary Stress Prediction post-level |  External Evaluation: 6185 (6.0% False, 94.0% True) |  External Evaluation: 13 $\pm$ 6
|  |  |

###  4.2. Models

We experimented with multiple LLMs with different sizes, pre-training targets,
and availability.

  * •

Alpaca (7B) (Taori et al., 2023): An open-source large model finetuned from
another open-sourced LLaMA 7B model (Touvron et al., 2023a) on instruction
following demonstrations. Experiments have shown that Alpaca behaves
qualitatively similarly to OpenAI’s text-davinci-003 on certain task metrics.
We choose the relatively small 7B version to facilitate running and finetuning
on consumer hardware.

  * •

Alpaca-LoRA (7B) (Hu et al., 2021): Another open-source large model finetuned
from LLaMA 7B model using the same dataset as Alpaca (Taori et al., 2023).
This model leverages a different finetuning technique called low-rank
adaptation (LoRA) (Hu et al., 2021), with the goal of reducing finetuning cost
by freezing the model weights and injecting trainable rank decomposition
matrices into each layer of the Transformer architecture. Despite the
similarity in names, it is important to note that Alpaca-LoRA is entirely
distinct from Alpaca. They are trained on the same dataset but with different
methods.

  * •

FLAN-T5 (11B) (Chung et al., 2022): An open-source large model T5 (Raffel et
al., 2020) finetuned with a variety of task-based datasets on instructions.
Compared to other LLMs, FLAN-T5 focuses more on task solving and is less
optimized for natural language or dialogue generation. We picked the largest
version of FLAN-T5 (i.e., FLAN-T5-XXL), which has a comparable size of Alpaca.

  * •

LLaMA2 (70B) (Touvron et al., 2023b): A recent open-source large model
released by Meta. We picked the largest version of LLaMA2, whose size is
between FLAN-T5 and GPT-3.5.

  * •

GPT-3.5 (175B) (noa, 2022): This large model is closed-source and available
through API provided by OpenAI. We picked the gpt-3.5-turbo, one of the most
capable and cost-effective models in the GPT-3.5 family.

  * •

GPT-4 (1700B) (Bubeck et al., 2023): This is the largest closed-source model
available through OpenAI API. We picked the gpt-4-0613. Due to the limited
availability of API, the cost of finetuning GPT-3.5 or GPT-4 is prohibitive.

It is worth noting that Alpaca, Alpaca-LoRA, GPT-3.5, LLaMA2 and GPT-4 are all
finetuned with natural dialogue as one of the optimization goals. In contrast,
FLAN-T5 is more focused on task-solving. In our case, the user-written input
posts resemble natural dialogue, whereas the mental health prediction tasks
are defined as specific classification tasks. It is unclear and thus
interesting to explore which LLM fits better with our goal.

####  4.2.1. Mental-Alpaca & Mental-FLAN-T5

Our methods of zero-shot prompting (Sec. 3.1) and few-shot prompting (Sec.
3.2) do not update model parameters during the experiment. In contrast,
instruction finetuning (Sec. 3.3) will update model parameters and generate
new models. To enhance their capability in the mental health domains, we
update Alpaca and FLAN-T5 on six tasks across the four datasets in Sec. 4.1
using the multi-dataset instruction finetuning method (Sec. 3.3.2), which
leads to our new model Mental-Alpaca and Mental-FLAN-T5.

###  4.3. Experiment Setup and Metrics

For zero-shot and few-shot prompting methods, we load open-source models
(Alpaca, Alpaca-LoRA, FLAN-T5, LLaMA2) with one to eight Nvidia A100 GPUs to
do the tasks, depending on the size of the model. For closed-source models
(GPT-3.5, and GPT-4), we use OpenAI API to conduct chat completion tasks.

As for finetuning Mental-Alpaca and Mental-FLAN-T5, we merge the four datasets
together and provide instructions for all six tasks (in the training set). We
use eight Nvidia A100 GPUs for instruction finetuning. With cross entropy as
the loss function, we backpropagate and update model parameters in 3 epochs,
with Adam optimizer and a learning rate as 2$e^{-5}$ (cosine scheduler, warmup
ratio 0.03).

We focus on balanced accuracy as the main evaluation metric, i.e., the mean of
sensitivity (true positive rate) and specificity (true negative rate). We
picked this metric since it is more robust to class imbalance compared to the
accuracy or F1 score (Brodersen et al., 2010; Xu et al., 2023). It is
noteworthy that the sizes of LLMs we compare are vastly different, with the
number of parameters ranging from 7B to 1700B. A larger model is usually
expected to have a better overall performance than a smaller model. We inspect
whether this expectation holds in our experiments.

##  5\. Results

We summarize our experiment results with zero-shot prompting (Sec. 5.1), few-
shot prompting (Sec. 5.2), and instruction finetuning (Sec. 5.3). Moreover,
although we mainly focus on prediction tasks in this research, we also present
the initial results of our exploratory case study on mental health reasoning
tasks in Sec. 5.4.

Overall, our results show that zero-shot and few-shot settings show promising
performance of LLMs for mental health tasks, although their performance is
still limited. Instruction-finetuning on multiple datasets (Mental-Alpaca and
Mental-FLAN-T5) can significantly boost models’ performance on all tasks
simultaneously. Our case study also reveals the strong reasoning capability of
certain LLMs, especially GPT-4. However, we note that these results do not
indicate the deployability. We highlight important ethical concerns and gaps
in Sec. 6.

###  5.1. Zero-shot Prompting Shows Promising yet Limited Performance

We start with the most basic zero-shot prompting with Alpaca, Alpaca-LoRA,
FLAN-T5, LLaMA2, GPT-3.5, and GPT-4. The balanced accuracy results are
summarized in the first sections of Table 4. AlpacaZS and Alpaca-LoRAZS
achieve better overall performance than the naive majority baseline
($\overline{\Delta}_{\text{Alpaca}}=5.5\%$, $\overline{\Delta}_{\text{Alpaca-
LoRA}}=5.6\%$), but they are far from the task-specific baseline models BERT
and Mental-RoBERTa (which have 20%-25% advantages). With much larger models
GPT-3.5ZS, the performance gets more promising
($\overline{\Delta}_{\text{GPT-3.5}}=12.4\%$ over baseline), which is inline
with previous work (Yang et al., 2023a). GPT-3.5’s advantage over Alpaca and
Alpaca-LoRA is expected due to its larger size (25$\times$).

Surprisingly, FLAN-T5ZS achieves much better overall results compared to
AlpacaZS ($\overline{\Delta}_{\text{FLAN-T5}\\_vs\\_\text{Alpaca}}=10.9\%$)
and Alpaca-LoRAZS ($\overline{\Delta}_{\text{FLAN-T5}\\_vs\\_\text{Alpaca-
LoRA}}=11.0\%$), and even LLaMA2
($\overline{\Delta}_{\text{FLAN-T5}\\_vs\\_\text{LLaMA2}}=1.0\%$) and GPT-3.5
($\overline{\Delta}_{\text{FLAN-T5}\\_vs\\_\text{GPT-3.5}}=4.2\%$). Note that
LLaMA2 is 6 times bigger than FLAN-T5 and GPT-3.5 is 15 times bigger. On Task
#6 (Five-level Suicide Risk Prediction), FLAN-T5ZS even outperforms the state-
of-the-art Mental-RoBERTa by 4.5%. Comparing these results, the task-solving-
focused model FLAN-T5 appears to be better at the mental health prediction
tasks in a zero-shot setting. We will introduce more interesting findings
after finetuning (see Sec. 5.3.1).

In contrast, the advantage of GPT-4 becomes relatively less remarkable
considering its gigantic size. GPT-4ZS’s average performance outperforms
FLAN-T5ZS (150$\times$ size), LLaMA2ZS (25$\times$ size), and GPT-3.5ZS
(10$\times$ size) by 6.4%, 7.5%, and 10.6%, respectively. Yet it is still very
encouraging to observe that GPT-4 is approaching the state-of-the-art on these
tasks ($\overline{\Delta}_{\text{GPT-4}\\_vs\\_\text{Mental-RoBERTa}}$
$=-7.9\%$), and it also outperforms Mental-RoBERTa on Task #6 by 4.5%. In
general, these results indicate the promising capability of LLMs on mental
health prediction tasks compared to task-specific models, even without any
domain-specific information.

Table 4.  Balanced Accuracy Performance Summary of Zero-shot, Few-shot and
Instruction Finetuning on LLMs. $ZS_{best}$ highlights the best performance
among zero-shot prompt designs, including context enhancement, mental health
enhancement, and their combination (see Table. 2). Detailed results can be
found in Table 10 in Appendix. Small numbers represent standard deviation
across different designs of Prompt${}_{\textit{Part1-S}}$ and
Prompt${}_{\textit{Part2-Q}}$. The baselines at the bottom rows do not have
standard deviation as the task-specific output is static, and prompt designs
do not apply. Due to the maximum token size limit, we only conduct few-shot
prompting on a subset of datasets and mark other infeasible datasets as “–”.
For each column, the best result is bolded, and the second best is underlined.

|  Dataset | Dreaddit | DepSeverity | SDCNL | CSSRS-Suicide
---|---|---|---|---|---
Category | Model | Task #1 | Task #2 | Task #3 | Task #4 | Task #5 | Task #6
Zero-shot Prompting | AlpacaZS | 0.593±0.039 | 0.522±0.022 | 0.431±0.050 | 0.493±0.007 | 0.518±0.037 | 0.232±0.076
AlpacaZS_best | 0.612±0.065 | 0.577±0.028 | 0.454±0.143 | 0.532±0.005 | 0.532±0.033 | 0.250±0.060
2-8 | Alpaca-LoRAZS | 0.571±0.043 | 0.548±0.027 | 0.437±0.044 | 0.502±0.011 | 0.540±0.012 | 0.187±0.053
| Alpaca-LoRAZS_best | 0.571±0.043 | 0.548±0.027 | 0.437±0.044 | 0.502±0.011 | 0.567±0.038 | 0.224±0.049
2-8 | FLAN-T5ZS | 0.659±0.086 | 0.664±0.011 | 0.396±0.006 | 0.643±0.021 | 0.667±0.023 | 0.418±0.012
| FLAN-T5ZS_best | 0.663±0.079 | 0.674±0.014 | 0.396±0.006 | 0.653±0.011 | 0.667±0.023 | 0.418±0.012
2-8 | LLaMA2ZS | 0.720±0.012 | 0.693±0.034 | 0.429±0.013 | 0.589±0.010 | 0.691±0.014 | 0.261±0.018
| LLaMA2ZS_best | 0.720±0.012 | 0.711±0.033 | 0.444±0.021 | 0.643±0.014 | 0.722±0.039 | 0.367±0.043
2-8 | GPT-3.5ZS | 0.685±0.024 | 0.642±0.017 | 0.603±0.017 | 0.460±0.163 | 0.570±0.118 | 0.233±0.009
| GPT-3.5ZS_best | 0.688±0.045 | 0.653±0.020 | 0.642±0.034 | 0.632±0.020 | 0.617±0.033 | 0.310±0.015
2-8 | GPT-4ZS | 0.700±0.001 | 0.719±0.013 | 0.588±0.010 | 0.644±0.007 | 0.760±0.009 | 0.418±0.009
| GPT-4ZS_best | 0.725±0.009 | 0.719±0.013 | 0.656±0.001 | 0.647±0.014 | 0.760±0.009 |  0.441±0.057
Few-shot Prompting | AlpacaFS | 0.632±0.030 | 0.529±0.017 | 0.628±0.005 | — | — | —
FLAN-T5FS | 0.786±0.006 | 0.678±0.009 | 0.432±0.009 | — | — | —
GPT-3.5FS | 0.721±0.010 | 0.665±0.015 | 0.580±0.002 | — | — | —
GPT-4FS | 0.698±0.009 | 0.724±0.005 | 0.613±0.001 | — | — | —
Instructional Finetuning | Mental-Alpaca |  0.816±0.006 |  0.775±0.006 |  0.746±0.005 | 0.724±0.004 | 0.730±0.048 | 0.403±0.029
Mental-FLAN-T5 | 0.802±0.002 | 0.759±0.003 | 0.756±0.001 | 0.677±0.005 | 0.868±0.006 | 0.481±0.006
Baseline | Majority | 0.500±— | 0.500±— | 0.250±— | 0.500±— | 0.500±— | 0.200±—
BERT | 0.783±— | 0.763±— | 0.690±— | 0.678±— | 0.500±— | 0.332±—
Mental-RoBERTa | 0.831±— | 0.790±— | 0.736±— |  0.723±— |  0.853±— | 0.373±—
|  |  |  |  |  |  |

####  5.1.1. The Effectiveness of Enhancement Strategies.

In Sec. 3.1, we propose context enhancement, mental health enhancement, and
their combination strategies for zero-shot prompt design to provide more
information about the domain. Interestingly, our results suggest varied
effectiveness on different LLMs and datasets.

Table 5.  Balanced Accuracy Performance Change using Enhancement Strategies.
The green/red color indicates increased/decreased accuracy. This table zooms
in on the zero-shot section of Table 4. $\uparrow$/$\downarrow$ marks the ones
with better/worse performance in comparison.

Dataset | Dreaddit | DepSeverity | SDCNL | CSSRS-Suicide |
---|---|---|---|---|---
Model | Task #1 | Task #2 | Task #3 | Task #4 | Task #5 | Task #6 |  $\overline{\Delta}-$All Six Tasks
$\Delta-$AlpacaZS_context | $\uparrow+0.019$ | $\uparrow+0.045$ | $\uparrow+0.023$ | $\uparrow+0.004$ | $\uparrow+0.014$ | $\uparrow+0.018$ | $\uparrow+0.021$
$\Delta-$AlpacaZS_mh | $\uparrow+0.000$ | $\uparrow+0.055$ | $\uparrow+0.013$ | $\downarrow-0.011$ | $\uparrow+0.006$ | $\uparrow+0.004$ | $\uparrow+0.011$
$\Delta-$AlpacaZS_both | $\downarrow-0.053$ | $\uparrow+0.037$ | $\downarrow-0.010$ | $\uparrow+0.039$ | $\downarrow-0.007$ | $\downarrow-0.010$ | $\downarrow-0.001$
$\Delta-$Alpaca-LoRAZS_context | $\downarrow-0.035$ | $\downarrow-0.047$ | $\downarrow-0.094$ | $\downarrow-0.030$ | $\uparrow+0.027$ | $\uparrow+0.027$ | $\downarrow-0.025$
$\Delta-$Alpaca-LoRAZS_mh | $\downarrow-0.071$ | $\downarrow-0.047$ | $\downarrow-0.105$ | $\downarrow-0.005$ | $\uparrow+0.017$ | $\uparrow+0.029$ | $\downarrow-0.031$
$\Delta-$Alpaca-LoRAZS_both | $\downarrow-0.071$ | $\downarrow-0.048$ | $\downarrow-0.051$ | $\downarrow-0.003$ | $\downarrow-0.023$ | $\uparrow+0.037$ | $\downarrow-0.027$
$\Delta-$FLAN-T5ZS_context |  $\uparrow$ +0.004 | $\uparrow+0.011$ | $\downarrow-0.018$ | $\uparrow+0.010$ | $\downarrow-0.018$ | $\downarrow-0.040$ | $\downarrow-0.009$
$\Delta-$FLAN-T5ZS_mh | $\downarrow-0.043$ | $\uparrow+0.003$ | $\downarrow-0.030$ | $\uparrow+0.005$ | $\downarrow-0.013$ | $\downarrow-0.046$ | $\downarrow-0.021$
$\Delta-$FLAN-T5ZS_both | $\downarrow-0.055$ | $\downarrow-0.003$ | $\downarrow-0.007$ | $\uparrow+0.002$ | $\downarrow-0.010$ | $\downarrow-0.036$ | $\downarrow-0.018$
$\Delta-$LLaMA2ZS_context | $\downarrow-0.062$ | $\uparrow+0.014$ | $\downarrow-0.019$ | $\uparrow+0.000$ | $\uparrow+0.031$ | $\uparrow+0.106$ | $\uparrow+0.012$
$\Delta-$LLaMA2ZS_mh | $\downarrow-0.102$ | $\uparrow+0.018$ | $\downarrow-0.033$ | $\uparrow+0.053$ | $\uparrow+0.004$ | $\uparrow+0.031$ | $\downarrow-0.005$
$\Delta-$LLaMA2ZS_both | $\downarrow-0.136$ | $\uparrow+0.011$ | $\uparrow+0.016$ | $\uparrow+0.054$ | $\downarrow-0.002$ | $\uparrow+0.067$ | $\uparrow+0.002$
$\Delta-$GPT-3.5ZS_context | $\uparrow+0.003$ | $\uparrow+0.011$ | $\downarrow-0.060$ | $\uparrow+0.157$ | $\uparrow+0.007$ | $\uparrow+0.031$ | $\uparrow+0.025$
$\Delta-$GPT-3.5ZS_mh | $\downarrow-0.006$ | $\downarrow-0.006$ | $\uparrow+0.039$ | $\uparrow+0.116$ | $\downarrow-0.093$ | $\uparrow+0.077$ | $\uparrow+0.021$
$\Delta-$GPT-3.5ZS_both | $\downarrow-0.005$ | $\downarrow-0.015$ | $\uparrow+0.014$ | $\uparrow+0.172$ | $\uparrow+0.047$ | $\uparrow+0.020$ | $\uparrow+0.039$
$\Delta-$GPT-4ZS_context | $\uparrow+0.006$ | $\uparrow+0.000$ | $\uparrow+0.001$ | $\uparrow+0.000$ | $\downarrow-0.007$ | $\uparrow+0.023$ | $\uparrow+0.004$
$\Delta-$GPT-4ZS_mh | $\uparrow+0.025$ | $\downarrow-0.035$ | $\uparrow+0.067$ | $\uparrow+0.002$ | $\downarrow-0.023$ | $\downarrow-0.022$ | $\uparrow+0.002$
$\Delta-$GPT-4ZS_both | $\uparrow+0.018$ | $\downarrow-0.031$ | $\uparrow+0.061$ | $\uparrow+0.003$ | $\downarrow-0.063$ | $\downarrow-0.006$ | $\downarrow-0.003$
|  |  |  |  |  |  |
$\overline{\Delta}-$All Six Models | $\downarrow-0.031$ | $\uparrow+0.000$ | $\downarrow-0.011$ | $\uparrow+0.032$ | $\downarrow-0.006$ | $\uparrow+0.017$ | $\uparrow+0.000$
$\overline{\Delta}-$Alpaca, GPT-3.5, GPT-4 | $\uparrow+0.001$ | $\uparrow+0.007$ | $\uparrow+0.017$ | $\uparrow+0.053$ | $\downarrow-0.013$ | $\uparrow+0.015$ | $\uparrow+0.013$
|  |  |  |  |  |  |

Table 5 provides a zoom-in summary of the zero-shot part in Table 4. For
Alpaca, LLaMA2, GPT-3.5, and GPT-4, the three strategies improved the
performance in general ($\overline{\Delta}_{\text{Alpaca}}=1.0\%$, 13 out of
18 tasks show positive changes; $\overline{\Delta}_{\text{LLaMA2}}=0.3\%$,
12/18 tasks positive; $\overline{\Delta}_{\text{GPT-3.5}}=2.8\%$, 12/18 tasks
positive; $\overline{\Delta}_{\text{GPT-4}}=0.2\%$, 11/18 tasks positive).
However, for Alpaca-LoRA and FLAN-T5, adding more context or mental health
domain information would reduce the model performance
($\overline{\Delta}_{\text{Alpaca-LoRA}}=-2.7\%$,
$\overline{\Delta}_{\text{FLAN-T5}}=-1.6\%$). For Alpaca-LoRA, this limitation
may stem from being trained with fewer parameters, potentially constraining
its ability to understand context or domain specifics. For FLAN-T5, this
reduced performance might be attributed to its limited capability in
processing additional information, as it is primarily tuned for task-solving.

The effectiveness of strategies on different datasets/tasks also varies. We
observe that Task#4 from the SDCNL dataset and Task#6 from the CSSRS-Suicide
dataset benefit the most from the enhancement. In particular, GPT-3.5 benefits
very significantly from enhancement on Task #4
($\overline{\Delta}_{\text{GPT-3.5}-\text{Task\\#4}}=14.8\%$). And LLaMA2
benefits significantly on Task #6
($\overline{\Delta}_{\text{GPT-3.5}-\text{Task\\#6}}=6.8\%$). These could be
caused by the different nature of datasets. Our results suggest that these
enhancement strategies are generally more effective for critical action
prediction (e.g., suicide, 2/3 tasks positive) than mental state prediction
(e.g., stress and depression, 1/3 task positive).

We also compare the effectiveness of different strategies on the four models
with positive effects: Alpaca, LLaMA2, GPT-3.5, and GPT-4. The context
enhancement strategy has the most stable improvement across all mental health
prediction tasks ($\overline{\Delta}_{\text{Alpaca}-context}=2.1\%$, 6/6 tasks
positive; $\overline{\Delta}_{\text{LLaMA2}-context}=1.2\%$, 4/6 tasks
positive; $\overline{\Delta}_{\text{GPT-3.5}-context}=2.5\%$, 5/6 tasks
positive; $\overline{\Delta}_{\text{GPT-4}-context}=0.4\%$, 5/6 tasks
positive). Comparatively, the mental health enhancement strategy is less
effective ($\overline{\Delta}_{\text{Alpaca}-mh}=1.1\%$, 5/6 tasks positive;
$\overline{\Delta}_{\text{LLaMA2}-mh}=-0.5\%$, 4/6 tasks positive;
$\overline{\Delta}_{\text{GPT-3.5}-mh}=2.1\%$, 3/6 tasks positive;
$\overline{\Delta}_{\text{GPT-4}-mh}=0.2\%$, 3/6 tasks positive). The
combination of the two strategies yields diverse results. It has the most
significant improvement on GPT-3.5’s performance, but not on all tasks
($\overline{\Delta}_{\text{GPT-3.5}-both}=3.9\%$, 4/6 tasks positive),
followed by LLaMA2 ($\overline{\Delta}_{\text{LLaMA2}-both}=0.2\%$, 4/6 tasks
positive). However, it has slightly negative impact on the average performance
of Alpaca ($\overline{\Delta}_{\text{Alpaca}-both}=-0.1\%$, 2/6 tasks
positive) or GPT-4 ($\overline{\Delta}_{\text{GPT-4}-both}=-0.3\%$, 3/6 tasks
positive). This indicates that larger language models (LLaMA2, GPT-3.5 vs.
Alpaca) have a strong capability to leverage the information embedded in the
prompts. But for the huge GPT-4, adding prompts seems less effective, probably
because it already contains similar basic information in its knowledge space.

We summarize our key takeaways from this section:

  * •

Both small-scale and large-scale LLMs show promising performance on mental
health tasks. FLAN-T5 and GPT-4’s performance is approaching task-specific NLP
models.

  * •

The prompt design enhancement strategies are generally effective for dialogue-
focused models, but not for task-solving-focused models. These strategies work
better for critical action prediction tasks such as suicide prediction.

  * •

Providing more contextual information about the task & input can consistently
improve performance in most cases.

  * •

Dialogue-focused models with larger trainable parameters (Alpaca vs. Alpaca-
LoRA, as well as LLaMA2/GPT-3.5 vs. Alpaca) can better leverage the contextual
or domain information in the prompts, yet GPT-4 shows less effect in response
to different prompts.

###  5.2. Few-shot Prompting Improves Performance to Some Extent

We then investigate the effectiveness of few-shot prompting. Note that since
we observe diverse effects of prompt design strategies in Table 5, in this
section, we only experiment with the prompts with the best performance in the
zero-shot setting. Moreover, we exclude Alpaca-LoRA due to its less promising
results and LLaMA2 due to its high computation cost.

Table 6.  Balanced Accuracy Performance Change with Few-shot Prompting. This
table is calculated between the zero-shot and the few-shot sections of Table
4.

Dataset | Dreaddit | DepSeverity |
---|---|---|---
Model | Task #1 | Task #2 | Task #3 |  $\overline{\Delta}-$All Three Tasks
$\Delta-$AlpacaFS_vs_ZS | $\uparrow+0.039$ | $\uparrow+0.007$ | $\uparrow+0.197$ | $\uparrow+0.081$
$\Delta-$FLAN-T5FS_vs_ZS |  $\uparrow$ +0.127 | $\uparrow+0.014$ | $\uparrow+0.036$ | $\uparrow+0.059$
$\Delta-$GPT-3.5FS_vs_ZS | $\uparrow+0.036$ | $\uparrow+0.023$ | $\downarrow-0.023$ | $\uparrow+0.012$
$\Delta-$GPT-4FS_vs_ZS | $\downarrow-0.002$ | $\uparrow+0.005$ | $\uparrow+0.025$ | $\uparrow+0.009$
$\overline{\Delta}-$All Four Models | $\uparrow+0.051$ | $\uparrow+0.012$ | $\uparrow+0.059$ | $\uparrow+0.041$
|  |  |  |

Due to the maximum input token length of models (2048), we focus on Dreaddit
and DepSeverity datasets that have a shorter input and experiment with $M=2$
in Eq. 2 for binary classification and $M=4/5$ for multi-class classification,
i.e., one sample per class. We repeat the experiment on each task three times
and randomize the few shot samples for each run.

We summarize the overall results in the second section of Table 4 and the
zoom-in comparison results in Table 6. Although language models with few-shot
prompting still underperform task-specific models, providing examples of the
task can improve model performance on most tasks compared to zero-shot
prompting ($\overline{\Delta}_{FS\text{\\_}vs\text{\\_}ZS}=4.1\%$).
Interestingly, few-shot prompting is more effective for AlpacaFS and FLAN-T5FS
($\overline{\Delta}_{\text{Alpaca}}=8.1\%$, 3/3 tasks positive;
$\overline{\Delta}_{\text{FLAN-T5}}=5.9\%$, 3/3 tasks positive) than GPT-3.5FS
and GPT-4FS ($\overline{\Delta}_{\text{GPT-3.5}}=1.2\%$, 2/3 tasks positive;
$\overline{\Delta}_{\text{GPT-4}}=0.9\%$, 2/3 tasks positive). Especially for
Task #3, we observe an improved balanced accuracy of 19.7% for Alpaca but a
decline of 2.3% for GPT-3.5, so that Alpaca outperforms GPT-3.5 on this task.
A similar situation is observed for FLAN-T5 (improved by 12.7%) and GPT-4
(declined by 0.2%) on Task #1. This may be attributed to the fact that smaller
models such as Alpaca and FLAN-T5 can quickly adapt to complex tasks with only
a few examples. In contrast, larger models like GPT-3.5 and GPT-4, with their
extensive “in memory” data, find it more challenging to rapidly learn from new
examples.

This leads to the key message from this experiment: Few-shot prompting can
improve the performance of LLMs on mental health prediction tasks to some
extent, especially for small models.

###  5.3. Instruction Finetuning Boost Performance for Multiple Tasks
Simultaneously

Our experiments so far have shown that zero-shot and few-shot prompting can
improve LLMs on mental health tasks to some extent, but their overall
performance is still below state-of-the-art task-specific models. In this
section, we explore the effectiveness of instruction finetuning.

Table 7.  Balanced Accuracy Performance Change with Instruction Finetuning.
This table is calculated between the finetuning and zero-shot section, as well
as the finetuning and few-shot section of Table 4. GPT-3.5Best and GPT-4Best
are the best results among zero-shot and few-shot settings.

Dataset | Dreaddit | DepSeverity | SDCNL | CSSRS-Suicide
---|---|---|---|---
Model | Task #1 | Task #2 | Task #3 | Task #4 | Task #5 | Task #6
$\Delta-$AlpacaFT_vs_ZS | $\uparrow+0.223$ | $\uparrow+0.253$ | $\uparrow+0.315$ | $\uparrow+0.231$ | $\uparrow+0.212$ | $\uparrow+0.171$
$\Delta-$AlpacaFT_vs_FS | $\uparrow+0.184$ | $\uparrow+0.246$ | $\uparrow+0.118$ | — | — | —
$\Delta-$FLAN-T5FT_vs_ZS | $\uparrow+0.143$ | $\uparrow+0.095$ | $\uparrow+0.360$ | $\uparrow+0.047$ | $\uparrow+0.201$ | $\uparrow+0.034$
$\Delta-$FLAN-T5FT_vs_FS | $\uparrow+0.016$ | $\uparrow+0.081$ | $\uparrow+0.324$ | — | — | —
Results comparison: |  |  |  |  |  |
Mental-RoBERTa | 0.831 | 0.790 | 0.736 | 0.723 | 0.853 | 0.373
GPT-3.5Best | 0.721 | 0.665 | 0.642 | 0.632 | 0.617 | 0.310
GPT-4Best | 0.725 | 0.724 | 0.656 | 0.647 | 0.760 | 0.441
Mental-Alpaca | 0.816 | 0.775 | 0.746 | 0.724 | 0.730 | 0.403
Mental-FLAN-T5 | 0.802 | 0.759 | 0.756 | 0.677 | 0.868 | 0.481
|  |  |  |  |  |

Due to the prohibitive cost and lack of transparency of GPT-3.5 and GPT-4
finetuning, we only experiment with Alpaca and FLAN-T5 that we have full
control of. We picked the most informative prompt to provide more embedded
knowledge during the finetuning. As introduced in Sec. 3.3.2 and Sec. 4.2.1,
we build Mental-Alpaca and Mental-FLAN-T5 by finetuning Alpaca and FLAN-T5 on
all six tasks across four datasets at the same time.

The third section of Table 4 summarizes the overall results, and Table 7
highlights the key comparisons. We observe that both Mental-Alpaca and Mental-
FLAN-T5 achieve significantly better performance compared to the unfinetuned
versions ($\overline{\Delta}_{\text{Alpaca}-FT\\_vs\\_ZS}$ = 23.4%,
$\overline{\Delta}_{\text{Alpaca}-FT\\_vs\\_FS}$ = 18.3%;
$\overline{\Delta}_{\text{FLAN-T5}-FT\\_vs\\_ZS}$ = 14.7%,
$\overline{\Delta}_{\text{FLAN-T5}-FT\\_vs\\_FS}$ = 14.0%). Both finetuned
models surpass GPT-3.5’s best performance among zero-shot and few-shot
settings across all six tasks ($\overline{\Delta}_{\text{Mental-
Alpaca}\\_vs\\_\text{GPT-3.5}}$ = 10.1%; $\overline{\Delta}_{\text{Mental-
FLAN-T5}\\_vs\\_\text{GPT-3.5}}$ = 11.6%) and outperform GPT-4’s best version
in most tasks ($\overline{\Delta}_{\text{Mental-Alpaca}\\_vs\\_\text{GPT-4}}$
= 4.0%, 4/6 tasks positive; $\overline{\Delta}_{\text{Mental-
FLAN-T5}\\_vs\\_\text{GPT-4}}$ = 5.5%, 5/6 tasks positive). Recall that
GPT-3.5/GPT-4 are 25/250 times bigger than Mental-Alpaca and 15/150 times
bigger than Mental-FLAN-T5.

More importantly, Mental-Alpaca and Mental-FLAN-T5 perform on par with the
state-of-the-art Mental-RoBERTa. Mental-Alpaca has the best performance on one
task and the second best on three tasks, while Mental-FLAN-T5 has the best
performance on three tasks. It is noteworthy that Mental-RoBERTa is a task-
specific model, which means it is specialized on one task after being trained
on it. In contrast, Mental-Alpaca and Mental-FLAN-T5 can simultaneously work
across all tasks with a single-round finetuning. These results show the strong
effectiveness of instruction finetuning: By finetuning LLMs on multiple mental
health datasets with instructions, the models can obtain better capability to
solve a variety of mental health prediction tasks.

####  5.3.1. Dialogue-Focused vs. Task-Solving-Focused LLMs

We further compare Mental-Alpaca and Mental-FLAN-T5. Overall, their
performance is quite close
($\overline{\Delta}_{\text{FLAN-T5}\\_vs\\_\text{Alpaca}}=1.4\%$), with
Mental-Alapca better at Task #4 on SDCNL and Mental-FLAN-T5 better at Task #5
and #6 on CSSRS-Suicide. In Sec. 5.1, we observe that FLAN-T5ZS has a much
better performance than AlpacaZS
($\overline{\Delta}_{\text{FLAN-T5}\\_vs\\_\text{Alpaca}}$ = 10.9%, 5/6 tasks
positive) in the zero-shot setting. However, after finetuning, FLAN-T5’s
advantage disappears.

Our comparison result indicates that Alpaca, as a dialogue-focused model, is
better at learning from human natural language data compared to FLAN-T5.
Although FLAN-T5 is good at task solving and thus has a better performance in
the zero-shot setting, its performance improvement after instruction
finetuning is relatively smaller than that of Alpaca. This observation has
implications for future stakeholders. If the data and computing resources for
finetuning are not available, using task-solving-focused LLMs could lead to
better results. When there are enough data and computing resources, finetuning
dialogue-based models can be a better choice. Furthermore, models like Alpaca,
with their dialogue conversation capabilities, may be more suitable for
downstream applications, such as mental well-being assistants for end-users.

####  5.3.2. Does Finetuning Generalize across Datasets?

We further measure the generalizability of LLMs after finetuning. To do this,
we instruction-finetune the model on one dataset and evaluate it on all
datasets (as introduced in Sec. 3.3.1). As the main purpose of this part is
not to compare different models but evaluate the finetuning method, we only
focus on Alpaca. Table 8 summarizes the results.

Table 8.  Balanced Accuracy Cross-Dataset Performance Summary of Mental-Alpaca
Finetuning on Single Dataset. Numbers indicate the results of the model
finetuned and tested on the same dataset. The bottom few rows are related
Alpaca versions for reference. $\uparrow$/$\downarrow$ marks the ones with
better/worse cross-dataset performance compared to the zero-shot version
AlpacaZS.

Test Dataset | Dreaddit | DepSeverity | SDCNL | CSSRS-Suicide
---|---|---|---|---
Finetune Dataset | Task #1 | Task #2 | Task #3 | Task #4 | Task #5 | Task #6
Dreaddit | 0.823 |  $\uparrow$ 0.720 |  $\uparrow$ 0.623 |  $\downarrow$ 0.474 |  $\uparrow$ 0.720 |  $\downarrow$ 0.156
DepSeverity |  $\uparrow$ 0.618 | 0.733 | 0.769 |  $|$ 0.493 |  $\uparrow$ 0.753 |  $\downarrow$ 0.156
SDCNL |  $\downarrow$ 0.468 |  $\downarrow$ 0.461 |  $\uparrow$ 0.623 | 0.730 |  $\uparrow$ 0.573 |  $\downarrow$ 0.156
CSSRS-Suicide |  $\downarrow$ 0.500 |  $\downarrow$ 0.500 |  $\uparrow$ 0.622 |  $\uparrow$ 0.500 | 0.753 | 0.578
Reference: |  |  |  |  |  |
AlpacaZS | 0.593 | 0.522 | 0.431 | 0.493 | 0.518 | 0.232
Mental-Alpaca | 0.816 | 0.775 | 0.746 | 0.724 | 0.730 | 0.403
|  |  |  |  |  |

We first find that finetuning and testing on the same dataset lead to good
performance, as indicated by the boxed entries on the diagonal in Table 8.
Some results are even better than Mental-Alpaca (5 out of 6 tasks) or Mental-
RoBERTa (3 out of 6 tasks), which is not surprising. More interestingly, we
investigate cross-dataset generalization performance (i.e., the ones off the
diagonal). Overall, finetuning on a single dataset achieves better performance
compared to the zero-shot setting
($\overline{\Delta}_{FT\text{-Single}\\_vs\\_ZS}=4.2\%$). However, the
performance changes vary across tasks. For example, finetuning on any dataset
is beneficial for Task #3 ($\overline{\Delta}=19.2\%$) and #5
($\overline{\Delta}=16.4\%$), but detrimental for Task #6
($\overline{\Delta}=-7.6\%$) and almost futile for Task #4
($\overline{\Delta}=-0.4\%$). Generalizing across Dreaddit and DepSeverity
shows good performance, but this is mainly because they share the language
corpus. These results indicate that finetuning on a single dataset can provide
mental health knowledge with a certain level and thus improve the overall
generalization results, but such improvement is not stable across tasks.

Table 9.  Balanced Accuracy Performance Summary on Three External Datasets.
These datasets come from diverse social media platforms. For each column, the
best result is bolded, and the second best is underlined.

|  Dataset | Red-Sam | Twt-60Users | SAD
---|---|---|---|---
Category | Model | Task #2 | Task #2 | Task #1
Zero-shot Prompting | AlpacaZS_best | 0.527±0.006 | 0.569±0.017 | 0.557±0.041
Alpaca-LoRAZS_best | 0.577±0.004 | 0.649±0.021 | 0.477±0.016
FLAN-T5ZS_best | 0.563±0.029 | 0.613±0.046 | 0.767±0.050
LLaMA2ZS_best | 0.574±0.008 |  0.736±0.019 | 0.704±0.026
GPT-3.5ZS_best | 0.506±0.004 | 0.571±0.000 | 0.750±0.027
GPT-4ZS_best | 0.511±0.000 | 0.566±0.017 | 0.854±0.006
Instructional Finetuning | Mental-Alpaca | 0.604±0.012 | 0.718±0.011 |  0.819±0.006
$\Delta-$AlpacaFT_vs_ZS | $\uparrow+0.077$ | $\uparrow+0.149$ | $\uparrow+0.262$
Mental-FLAN-T5 |  0.582±0.002 | 0.736±0.003 | 0.779±0.002
$\Delta-$FLAN-T5FT_vs_ZS | $\uparrow+0.019$ | $\uparrow+0.123$ | $\uparrow+0.012$
|  |  |  |

Moreover, we further evaluate the generalizability of our best models
instructional-finetuned on multiple datasets, i.e., Mental-Alpaca and Mental-
FLAN-T5. We leverage external datasets that are not included in the
finetuning. Table 9 highlights the key results. More detailed results can be
found in Table 10.

Consistent with the results in Table 7, the instruction finetuning enhances
the model performance on external datasets
($\overline{\Delta}_{Alpaca}=16.3\%$, $\overline{\Delta}_{FLAN-T5}=5.1\%$).
Both Mental-Alpaca and Mental-FLAN-T5 ranked top 1 or 2 in 2/3 external tasks.
It is noteworthy that Twt-60Users and SAD datasets are collected outside
Reddit, and their data is different from the source of finetuning datasets.
These results demonstrate strong evidence that instruction finetuning with
diverse tasks, even with data collected from a single social media platform,
can significantly enhance LLMs’ generalizability across multiple scenarios.

####  5.3.3. How Much Data Is Needed?

Additionally, we are interested in exploring how the size of the dataset
impacts the results of instruction finetuning. To answer this question, we
downsample the training set to 50%, 20%, 10%, 5%, and 1% of the original size
and repeat each one three times. We increase the training epoch accordingly to
make sure that the model is exposed to a similar amount of data. Similarly, we
also focus on Alpaca only. Figure 1 visualizes the results. With only 1% of
the data, the finetuned model is able to outperform the zero-shot model on
most tasks (5 out of 6). With 5% of the data, the finetuned model has a better
performance on all tasks. As expected, the model performance has an increasing
trend with more training data. For many tasks, the trend approaches a plateau
after 10%. The difference between 10% training data (less than 300 samples per
dataset) and 100% training data is not huge ($\overline{\Delta}$ = 5.9%).

Figure 1. Balanced Accuracy Performance Summary of Mental-Alpaca Finetuning
with Different Sizes of Training Set. The finetuning is conducted across four
datasets and six tasks. Each solid line represents the performance of the
finetuned model on each task. The dashed line indicates the AlpacaZS
performance baseline. Note that the x-axis is in the log scale.

####  5.3.4. More Data in One Dataset vs. Fewer Data across Multiple Datasets

In Sec. 5.3.2, the finetuning on a single dataset can be viewed as training on
a smaller set (around 5-25% of the original size) with less variation (i.e.,
no finetuning across datasets). Thus, the results in Sec. 5.3.2 are comparable
to those in Sec. 5.3.3. We found that the model’s overall performance is
better when the model is finetuned across multiple datasets when overall
training data sizes are similar
($\overline{\Delta}_{FT\text{-}5\%\\_vs\\_FT\text{-Single}}$ = 3.8%,
$\overline{\Delta}_{FT\text{-}10\%\\_vs\\_FT\text{-Single}}$ = 8.1%,
$\overline{\Delta}_{FT\text{-}20\%\\_vs\\_FT\text{-Single}}$ = 12.4%). This
suggests that increasing data variation can more effectively benefit
finetuning outcomes when the training data size is fixed.

These results can guide future developers and practitioners in collecting the
appropriate data size and sources to finetune LLMs for the mental health
domain efficiently. We have more discussion in the next section. In summary,
we highlight the key takeaways of our finetuning experiments as follows:

  * •

Instruction finetuning on multiple mental health datasets can significantly
boost the performance of LLMs on various mental health prediction tasks.
Mental-Alpaca and Mental-FLAN-T5 outperform GPT-3.5 and GPT-4, and perform on
par with the state-of-the-art task-specific model.

  * •

Although task-solving-focused LLMs may have better performance in the zero-
shot setting for mental health prediction tasks, dialogue-focused LLMs have a
stronger capability of learning from human natural language and can improve
more significantly after finetuning.

  * •

Finetuning LLMs on a small number of datasets and tasks may improve model
generalizable knowledge in mental health, but its effect is not robust.
Comparatively, finetuning on diverse tasks can robustly enhance
generalizability across multiple social media platforms.

  * •

Finetuning LLMs on a small number of samples (a few hundred) across multiple
datasets can already achieve favorable performance.

  * •

When the data size is the same, finetuning LLMs on data with larger variation
(i.e., more datasets and tasks) can achieve better performance.

###  5.4. Case Study of LLMs’ Capability on Mental Health Reasoning

Figure 2. A Case Study of Correct Reasoning Examples on Task #1 Binary Stress
Prediction on Dreaddit Dataset. Bolded texts highlight the mental-health-
related content in the input section, and the answers of LLMs. Underlined
texts highlight the reasoning content generated by LLMs, and italicized &
underlined texts indicate the wrong or unrelated content.

In addition to evaluating LLMs’ performance on classification tasks, we also
take an initial step to explore LLMs’ capability on mental health reasoning.
This is another strong advantage of LLMs since they can generate human-like
natural language based on embedded knowledge. Due to the high cost of a
systematic evaluation of reasoning outcomes, here we present a few examples as
a case study across different LLMs. It is noteworthy that we do not aim to
claim that certain LLMs have better/worse reasoning capabilities. Instead,
this section aims to provide a general sense of LLMs’ performance on mental
health reasoning tasks.

Specifically, we modify the prompt design by inserting a Chain-of-Thought
(CoT) prompt (Kojima et al., 2022) at the end of OutputConstraint in Eq. 1:
“Return [set of classification labels]. Provide reasons step by step”. We
compare Alpaca, FLAN-T5, GPT-3.5, and GPT-4. Our results indicate the
promising reasoning capability of these models, especially GPT-3.5 and GPRT-4.
We also experimented with the finetuned Mental-Alpaca and Mental-FLAN-T5.
Unfortunately, our results show that after finetuning solely on classification
tasks, these two models are no longer able to generate reasoning sentences
even with the CoT prompt. This suggests a limitation of the current finetuned
model.

####  5.4.1. Diverse Reasoning Capabilities across LLMs.

We present several examples as our case study to illustrate the reasoning
capability of these LLMs.  The first example comes from the binary stress
prediction task (Task #1) on the Dreaddit dataset (see Figure 2). All models
give the right classification, but with significantly different reasoning
capabilities. First, FLAN-T5 generates the shortest reason. Although it is
reasonable, it is superficial and does not provide enough insights. This is
understandable because FLAN-T5 is targeted at task-solving instead of
reasoning. Compared to FLAN-T5, Alpaca generates better reason. Among the five
reasons, two of them accurately analyze the user’s mental state given the
stressful situations. Meanwhile, GPT-3.5 and GPT-4 generate expert-level high-
quality reasons. The inference from the user’s statement is accurate and deep,
indicating their powerful capability of understanding human emotion and mental
health. Comparing the two models, GPT-3.5’s reason is simpler, following the
user’s statement point by point and adding basic comments, while GPT-4’s
output is more organic and insightful, yet more concise.

Figure 3. A Case Study of Mixed Reasoning Examples on Task #3 Four-level
Depression Prediction on DepSeverity Dataset. Alpaca wrongly predicted the
label, and GPT-3.5 provided a wrong inference on the meaning of “relief”.

We also have a similar observation in the second example from the four-level
depression prediction task (Task #3) on the DepSeverity dataset (see Figure
3). In this example, although FLAN-T5’s prediction is correct, it simply
repeats the fact stated by the user, without providing further insights.
Alpaca makes the wrong prediction, but it provides one sentence of accurate
reasoning (although relatively shallow). GPT-3.5 makes an ambiguous prediction
that includes the correct answer. In contrast, GPT-4 generates the highest
quality reasoning with the right prediction. With its correct understanding of
depressive symptoms, GPT-4 can accurately infer from the user’s situation,
link it to symptoms, and provides insightful analysis.

####  5.4.2. Wrong and Dangerous Reasoning from LLMs.

However, we also want to emphasize the incorrect reasoning content, which may
lead to negative consequences and risks.  In the first example, Alpaca
generated two wrong reasons for the hallucinated “reliance on others” and
“safety concerns”, along with an unrelated reason for readers instead of the
poster. In the second example, GPT-3.5 misunderstood what the user meant by
“relief”. To better illustrate this, we further present another example where
all four LLMs make problematic reasoning (see Figure 4). In this example, the
user was asking for others’ opinions on social anxiety, with their own job
interview experience as an example. Although the user mentioned situations
where they were anxious and stressed, it’s clear that they were calm when
writing this post and described their experience in an objective way. However,
FLAN-T5, GPT-3.5, and GPT-4 all mistakenly take the description of the anxious
interview experience as evidence to support their wrong prediction. Although
Alpaca makes the right prediction, it does not understand the main theme of
the post. The false positives reveal that LLMs may overly generalize in a
wrong way: Being stressed in one situation does not indicate that a person is
stressed all the time. However, the reasoning content alone reads smoothly and
logically. If the original post was not provided, the content could be very
misleading, resulting in a wrong prediction with reasons that “appears to be
solid”. These examples clearly illustrate the limitations of the current LLMs
for mental health reasons, as well as their risks of introducing dangerous
bias and negative consequences to users.

Figure 4. A Case Study of Incorrect Reasoning Examples on Task #1 Binary
Stress Prediction on Dreaddit Dataset. FLAN-T5, GPT-3.5, and GPT-4 all make
false positive predictions. All four LLMs provide problematic reasons.

The case study suggests that GPT-4 enjoys impressive reasoning capability,
followed by GPT-3.5 and Alpaca. Although FLAN-T5 shows a promising zero-shot
performance, it is not good at reasoning. Our results reveal the encouraging
capability of LLMs to understand human mental health and generate meaningful
analysis. However, we also present examples where LLMs can make mistakes and
offer explanations that appear reasonable but are actually flawed. This
further suggests the importance of more future research on LLMs’ ethical
concerns and safety issues before real-world deployment.

##  6\. Discussion

Our experiment results reveal a number of interesting findings. In this
section, we discuss potential guidelines for enabling LLMs for mental -health-
related tasks (Sec. 6.1). We envision promising future directions (Sec. 6.2),
while highlighting important ethical concerns and limitations with LLMs for
mental health (Sec. 6.3). We also summarize the limitations of the current
work (Sec. 6.4).

###  6.1. Guidelines for Empowering LLMs for Mental Health Prediction Tasks

We extract and summarize the takeaways from Sec. 5 into a set of guidelines
for future researchers and practitioners on how to empower LLMs to be better
at various mental health prediction tasks.

When computing resources are limited, combine prompt design & few-shot
prompting, and pick prompts carefully. As the size of large models continues
to grow, the requirement for hardware (mainly GPU) has also been increasing,
especially when finetuning an LLM. For example, in our experiment, Alpaca was
trained on eight 80GB A100s for three hours (Taori et al., 2023). With limited
computing resources, only running inferences or resorting to APIs is feasible.
In these cases, zero-shot and few-shot prompt engineering strategies are
viable options. Our results indicate that providing few-shot mental health
examples with appropriate enhancement strategies can effectively improve
prediction performance. Specifically, adding contextual information about the
online text data is always helpful. If the available model is large and
contains rich knowledge (at least 7B trainable parameters), adding mental
health domain information can also be beneficial.

With enough computing resources, instruction finetune models on various mental
health datasets. When there are enough computing resources and model
training/finetuning is possible, there are more options to enhance LLMs for
mental health prediction tasks. Our experiments clearly show that instruction
finetuning can significantly boost the performance of models, especially for
dialogue-based models since they can better understand and learn from human
natural language. When there are multiple datasets available, merging multiple
datasets and tasks altogether and finetuning the model in a single round is
the most effective approach to enhance its generalizability.

Implement efficient finetuning with hundreds of examples and prioritize data
variation when data resource is limited. Figure 1 shows that finetuning does
not require large datasets. If there is no immediately available dataset,
collecting small datasets with a few hundred samples is often good enough.
Moreover, when the overall amount of data is limited (e.g., due to resource
constraints), it is more advantageous to collect data from a variety of
sources, each with a smaller size, than to collect a single larger dataset.
Because instruction finetuning generalizes better when data and tasks have a
larger variation.

More curated finetuning datasets are needed for mental health reasoning. Our
case study suggests that Mental-Alpaca and Mental-FLAN-T5 can only generate
classification labels after being finetuned solely on classification tasks,
losing their reasoning capability. This is a major limitation of the current
models. A potential solution involves incorporating more datasets focused on
reasoning or causality into the instruction finetuning process, so that models
can also learn the relationship between mental health outcomes and causal
factors.

Limited Prediction and Reasoning Performance for Complex Contexts. LLMs tend
to make more mistakes when the conversation contexts are more complex
(Bhattacharya et al., 2023; Li et al., 2023a). Our results contextualize this
in the mental health domain. Section 5.4.2 shows an example case where most
LLMs not only predict incorrectly but also provide flawed reasoning processes.
Further analysis of mispredicted instances indicates a recurring difficulty:
LLMs often err when there’s a disconnect between the literal context and the
underlying real-life scenarios. he example in Figure 4 is a case where LLMs
are confused by the hypothetical stressful case described by the person. In
another example, all LLMs incorrectly assess a person with severe depression
(false negative): “I’m just blown away by this doctor’s willingness to help. I
feel so validated every time I leave his office, like someone actually
understands what I’m struggling with, and I don’t have to convince them of my
mental illness. Bottom line? Research docs if you can online, read their
reviews and don’t give up until you find someone who treats you the way you
deserve. If I can do this, I promise you can!” Here, LLMs are misled by the
outwardly positive sentiment, overlooking the significant cues such as regular
doctor visits and explicit mentions of mental illness. These observations
underscore a critical shortfall of LLMs: they cannot handle complex mental
health-related tasks, particularly those concerning chronic conditions like
depression. The variability of human expressions over time and the models’
susceptibility to being swayed by superficial text rather than underlying
scenarios present significant challenges.

Despite the promising capability of LLMs for mental health tasks, they are
still far from being deployable in the real world. Our experiments reveal the
encouraging performance of LLMs on mental health prediction and reasoning
tasks. However, as we note in Sec. 6.3, our current results do not indicate
LLMs’ deployability in real-life mental health settings. There are many
important ethical and safety concerns and gaps before deployment to be
addressed before achieving robustness and deployability.

###  6.2. Beyond Mental Health Prediction Task and Online Text Data

Our current experiments mainly involve mental health prediction tasks, which
are essentially classification problems. There are more types of tasks that
our experiments don’t cover, such as regression (e.g., predicting a score on a
mental health scale). In particular, reasoning is an attractive task as it can
fully leverage the capability of LLMs on language generation (Nori et al.,
2023; Bubeck et al., 2023). Our initial case study on reasoning is limited but
reveals promising performance, especially for large models such as GPT-4. We
plan to conduct more experiments on tasks that go beyond classification.

In addition, there is another potential extension direction. In this paper, we
mainly focus on online text data, which is one of the important data sources
of the ubiquitous computing ecosystem. However, there are more available data
streams that contain rich information, such as the multimodal sensor data from
mobile phones and wearables (e.g., (Xu et al., 2022; Nepal et al., 2022; Xu et
al., 2021c; Morais et al., 2023; J. Abrantes, 2023)). This leads to another
open question on how to leverage LLMs for time-series sensor data. More
research is needed to explore potential methods to merge the online text
information with sensor streams. These are another set of exciting research
questions to explore in future work.

###  6.3. Ethics in LLMs and Deployability Gaps for Mental Health

Although our experiments on LLMs have shown promising capability for mental-
health-related tasks, it still has a long way to go before being deployed in
real-life systems. Recent research has revealed the potential bias or even
harmful advice introduced by LLMs (Hoover, 2023), especially with the gender
(Ghosh and Caliskan, 2023) and racial (Abid et al., 2021) gaps. In mental
health, these gaps and disparities between population groups have been long-
standing (Irene Y. Chen et al., 2019). Meanwhile, our case study of incorrect
prediction, over-generalization, and “falsely reasonable” explanations further
highlight the risk of current LLMs. Recent studies are calling for more
research emphasis and efforts in assessing and mitigating these biases for
mental health (Irene Y. Chen et al., 2019; Timmons et al., 2022).

Although with a much stronger capability of understanding natural language
(and early signs of mental health domain knowledge in our case), LLMs are no
different from other modern AI models that are trained on a large amount of
human-generated content, which exhibit all the biases that humans do (Irene Y.
Chen et al., 2021; Ntoutsi et al., 2020; Wang et al., 2020). Meanwhile,
although we carefully picked datasets with human expert annotations, there
exist potential biases in the labels, such as stereotypes (Pessach and
Shmueli, 2022), confirmation bias (Gemalmaz and Yin, 2021), normative vs.
descriptive labels (Balagopalan et al., 2023). Besides, privacy is another
important concern. Although our datasets are based on public social media
platforms, it is necessary to carefully handle mental-health-related data and
guarantee anonymity in any future efforts. These ethical concerns need to
receive attention not only at the monitoring and prediction stage, but also in
the downstream applications, ranging from assistants for mental health experts
to chatbots for end-users. Careful efforts into safe development, auditing,
and regulation are very much needed to address these ethical risks.

###  6.4. Limitations

Our paper has a few limitations. First, although we carefully inspect the
quality of our dataset and cover different categories of LLM, the the range of
datasets and the types of LLMs included are still limited. Our findings are
based on the observations of these datasets and models, which may not
generalize to other cases. Related, our exploration of zero-shot few-shot
prompt design is not comprehensive. The limited input window of some models
also limits our exploration of more samples for few-shot prompting.
Furthermore, we have not conducted a systematic evaluation of these models’
performance in mental health reasoning. Future work can design larger-scale
experiments to include more datasets, models, prompt designs, and better
evaluation.

Second, our datasets were mainly from Reddit, which could be limited. Although
our analysis in Section 5.3.2 shows that finetuned models have cross-platform
generalizability, the finetuning was only based on Reddit and can introduce
bias. Meanwhile, although the labels are not directly accessible on the
platforms, it is possible that these text data have been included in the
initial training of these large models. We still argue that there is little
information leakage as long as the models haven’t seen the labels for our
tasks, but it is hard to measure how the initial training process may affect
the outcomes in our evaluation.

Third, another important limitation of the current work is the lack of
evaluation of model fairness. Our anonymous datasets do not include
comprehensive demographic information, making it hard to compare the
performance across different population groups. As we discussed in the
previous section, lots of future work on ethics and fairness is necessary
before deploying such systems in real life.

##  7\. Conclusion

In this paper, we present the first comprehensive evaluation of multiple LLMs
(Alpaca, Alpaca-LoRA, FLAN-T5, LLaMA2, GPT-3.5, and GPT-4) on mental health
prediction tasks (binary and multi-class classification) via online text data.
Our experiments cover zero-shot prompting, few-shot prompting, and instruction
finetuning. The results reveal a number of interesting findings. Our context
enhancement strategy can robustly improve performance for all LLMs, and our
mental health enhancement strategy can enhance models with a large number of
trainable parameters. Meanwhile, few-shot prompting can also robustly improve
model performance even by providing just one example per class. Most
importantly, our experiments show that instruction finetuning across multiple
datasets can significantly boost model performance on various mental health
prediction tasks at the same time, generalizing across external data sources
and platforms.. Our best finetuned models, Mental-Alpaca and Mental-FLAN-T5,
outperform much larger LLaMA2, GPT-3.5 and GPT-4, and perform on par with the
state-of-the-art task-specific model Mental-RoBERTa. We also conduct an
exploratory case study on these models’ reasoning capability, which further
suggests both the promising future and the important limitations of LLMs. We
summarize our findings as a set of guidelines for future researchers,
developers, and practitioners who want to empower LLMs with better knowledge
of mental health for downstream tasks. Meanwhile, we emphasize that our
current efforts of LLMs in mental health are still far from deployability. We
highlight the important ethical concerns accompanying this line of research.

## ACKNOWLEDGMENTS

This work is supported by VW Foundation, Quanta Computing, and the National
Institutes of Health (NIH) under Grant No. 1R01MD018424-01.

## References

  * (1)
  * noa (2022) 2022\.  Introducing ChatGPT.  https://openai.com/blog/chatgpt
  * men (2023a) 2023a.  Mental Health By the Numbers.  https://nami.org/mhstats
  * men (2023b) 2023b.  Mental Illness.  https://www.nimh.nih.gov/health/statistics/mental-illness
  * Abd-alrazaq et al. (2019) Alaa A. Abd-alrazaq, Mohannad Alajlani, Ali Abdallah Alalwan, Bridgette M. Bewick, Peter Gardner, and Mowafa Househ. 2019.  An overview of the features of chatbots in mental health: A scoping review.  _International Journal of Medical Informatics_ 132 (Dec. 2019), 103978.  https://doi.org/10.1016/j.ijmedinf.2019.103978
  * Abd-Alrazaq et al. (2021) Alaa A Abd-Alrazaq, Mohannad Alajlani, Nashva Ali, Kerstin Denecke, Bridgette M Bewick, and Mowafa Househ. 2021.  Perceptions and opinions of patients about mental health chatbots: scoping review.  _Journal of medical Internet research_ 23, 1 (2021), e17828.
  * Abid et al. (2021) Abubakar Abid, Maheen Farooqi, and James Zou. 2021\.  Persistent anti-muslim bias in large language models. In _Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society_. 298–306.
  * Agrawal et al. (2022) Monica Agrawal, Stefan Hegselmann, Hunter Lang, Yoon Kim, and David Sontag. 2022.  Large language models are few-shot clinical information extractors. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_. 1998–2022.
  * Ahmed et al. (2022) Arfan Ahmed, Sarah Aziz, Carla T Toro, Mahmood Alzubaidi, Sara Irshaidat, Hashem Abu Serhan, Alaa A Abd-Alrazaq, and Mowafa Househ. 2022\.  Machine learning models to detect anxiety and depression through social media: A scoping review.  _Computer Methods and Programs in Biomedicine Update_ (2022), 100066.
  * America (2022) Mental Health America. 2022\.  The state of mental health in America.
  * Amin et al. (2023) Mostafa M. Amin, Erik Cambria, and Björn W. Schuller. 2023\.  Will Affective Computing Emerge from Foundation Models and General AI? A First Evaluation on ChatGPT.  http://arxiv.org/abs/2303.03186
  * Balagopalan et al. (2023) Aparna Balagopalan, David Madras, David H. Yang, Dylan Hadfield-Menell, Gillian K. Hadfield, and Marzyeh Ghassemi. 2023.  Judging facts, judging norms: Training machine learning models to judge humans requires a modified approach to labeling data.  _Science Advances_ 9, 19 (May 2023), eabq0701.  https://doi.org/10.1126/sciadv.abq0701
  * Benton et al. (2017) Adrian Benton, Margaret Mitchell, and Dirk Hovy. 2017\.  Multi-task learning for mental health using social media text.  _arXiv preprint arXiv:1712.03538_ (2017).
  * Bhattacharya et al. (2023) Sourangshu Bhattacharya, Avishek Anand, et al. 2023\.  In-Context Ability Transfer for Question Decomposition in Complex QA.  _arXiv preprint arXiv:2310.18371_ (2023).
  * Birnbaum et al. (2017) Michael L Birnbaum, Sindhu Kiranmai Ernala, Asra F Rizvi, Munmun De Choudhury, and John M Kane. 2017\.  A collaborative approach to identifying social media markers of schizophrenia by employing machine learning and clinical appraisals.  _Journal of medical Internet research_ 19, 8 (2017), e7956.
  * Brants et al. (2007) Thorsten Brants, Ashok C Popat, Peng Xu, Franz J Och, and Jeffrey Dean. 2007.  Large language models in machine translation.  (2007).
  * Brodersen et al. (2010) Kay Henning Brodersen, Cheng Soon Ong, Klaas Enno Stephan, and Joachim M Buhmann. 2010.  The balanced accuracy and its posterior distribution. In _2010 20th international conference on pattern recognition_. IEEE, 3121–3124.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language Models are Few-Shot Learners. In _Advances in Neural Information Processing Systems_ , Vol. 33. Curran Associates, Inc., 1877–1901.  https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html
  * Bubeck et al. (2023) Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023\.  Sparks of Artificial General Intelligence: Early experiments with GPT-4.  http://arxiv.org/abs/2303.12712
  * Burnap et al. (2015) Pete Burnap, Walter Colombo, and Jonathan Scourfield. 2015\.  Machine Classification and Analysis of Suicide-Related Communication on Twitter. In _Proceedings of the 26th ACM Conference on Hypertext & Social Media - HT ’15_. ACM Press, Guzelyurt, Northern Cyprus, 75–84.  https://doi.org/10.1145/2700171.2791023
  * Cameron et al. (2017) Gillian Cameron, David Cameron, Gavin Megaw, Raymond Bond, Maurice Mulvenna, Siobhan O’Neill, Cherie Armour, and Michael McTear. 2017\.  Towards a chatbot for digital counselling.  https://doi.org/10.14236/ewic/HCI2017.24
  * Cameron et al. (2019) Gillian Cameron, David Cameron, Gavin Megaw, Raymond Bond, Maurice Mulvenna, Siobhan O’Neill, Cherie Armour, and Michael McTear. 2019\.  Assessing the Usability of a Chatbot for Mental Health Care.  In _Internet Science_ , Svetlana S. Bodrunova, Olessia Koltsova, Asbjørn Følstad, Harry Halpin, Polina Kolozaridi, Leonid Yuldashev, Anna Smoliarova, and Heiko Niedermayer (Eds.). Vol. 11551. Springer International Publishing, Cham, 121–132.  https://doi.org/10.1007/978-3-030-17705-8_11 Series Title: Lecture Notes in Computer Science.
  * Chancellor and De Choudhury (2020) Stevie Chancellor and Munmun De Choudhury. 2020.  Methods in predictive techniques for mental health status on social media: a critical review.  _npj Digital Medicine_ 3, 1 (March 2020), 43\.  https://doi.org/10.1038/s41746-020-0233-7
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022.  PaLM: Scaling Language Modeling with Pathways.  http://arxiv.org/abs/2204.02311 arXiv:2204.02311 [cs].
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022.  Scaling Instruction-Finetuned Language Models.  http://arxiv.org/abs/2210.11416 arXiv:2210.11416 [cs].
  * Coppersmith et al. (2015a) Glen Coppersmith, Mark Dredze, Craig Harman, and Kristy Hollingshead. 2015a.  From ADHD to SAD: Analyzing the Language of Mental Health on Twitter through Self-Reported Diagnoses. In _Proceedings of the 2nd Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality_. Association for Computational Linguistics, Denver, Colorado, 1–10.  https://doi.org/10.3115/v1/W15-1201
  * Coppersmith et al. (2015b) Glen Coppersmith, Mark Dredze, Craig Harman, Kristy Hollingshead, and Margaret Mitchell. 2015b.  CLPsych 2015 Shared Task: Depression and PTSD on Twitter. In _Proceedings of the 2nd Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality_. Association for Computational Linguistics, Denver, Colorado, 31–39.  https://doi.org/10.3115/v1/W15-1204
  * Coppersmith et al. (2014) Glen Coppersmith, Craig Harman, and Mark Dredze. 2014\.  Measuring Post Traumatic Stress Disorder in Twitter.  _Proceedings of the International AAAI Conference on Web and Social Media_ 8, 1 (May 2014), 579–582.  https://doi.org/10.1609/icwsm.v8i1.14574
  * Coppersmith et al. (2018a) Glen Coppersmith, Ryan Leary, Patrick Crutchley, and Alex Fine. 2018a.  Natural language processing of social media as screening for suicide risk.  _Biomedical informatics insights_ 10 (2018), 1178222618792860\.
  * Coppersmith et al. (2018b) Glen Coppersmith, Ryan Leary, Patrick Crutchley, and Alex Fine. 2018b.  Natural Language Processing of Social Media as Screening for Suicide Risk.  _Biomedical Informatics Insights_ 10 (Jan. 2018), 117822261879286\.  https://doi.org/10.1177/1178222618792860
  * Culotta (2014) Aron Culotta. 2014\.  Estimating county health statistics with twitter. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_. ACM, Toronto Ontario Canada, 1335–1344.  https://doi.org/10.1145/2556288.2557139
  * Dang et al. (2022) Hai Dang, Lukas Mecke, Florian Lehmann, Sven Goller, and Daniel Buschek. 2022.  How to prompt? Opportunities and challenges of zero-and few-shot learning for human-AI interaction in creative applications of generative models.  _arXiv preprint arXiv:2209.01390_ (2022).
  * De Choudhury et al. (2013a) Munmun De Choudhury, Scott Counts, and Eric Horvitz. 2013a.  Social media as a measurement tool of depression in populations. In _Proceedings of the 5th Annual ACM Web Science Conference_. ACM, Paris France, 47–56.  https://doi.org/10.1145/2464464.2464480
  * De Choudhury and De (2014) Munmun De Choudhury and Sushovan De. 2014.  Mental Health Discourse on reddit: Self-Disclosure, Social Support, and Anonymity.  _Proceedings of the International AAAI Conference on Web and Social Media_ 8, 1 (May 2014), 71–80.  https://doi.org/10.1609/icwsm.v8i1.14526
  * De Choudhury et al. (2013b) Munmun De Choudhury, Michael Gamon, Scott Counts, and Eric Horvitz. 2013b.  Predicting Depression via Social Media.  _Proceedings of the International AAAI Conference on Web and Social Media_ 7, 1 (2013), 128–137.  https://doi.org/10.1609/icwsm.v7i1.14432
  * De Choudhury et al. (2016) Munmun De Choudhury, Emre Kiciman, Mark Dredze, Glen Coppersmith, and Mrinal Kumar. 2016.  Discovering Shifts to Suicidal Ideation from Mental Health Content in Social Media. In _Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems_. ACM, San Jose California USA, 2098–2110.  https://doi.org/10.1145/2858036.2858207
  * Denecke et al. (2020) Kerstin Denecke, Sayan Vaaheesan, and Aaganya Arulnathan. 2020\.  A mental health chatbot for regulating emotions (SERMO)-concept and usability test.  _IEEE Transactions on Emerging Topics in Computing_ 9, 3 (2020), 1170–1182.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  _arXiv:1810.04805 [cs]_ (May 2019).  http://arxiv.org/abs/1810.04805
  * Eichstaedt et al. (2018) Johannes C Eichstaedt, Robert J Smith, Raina M Merchant, Lyle H Ungar, Patrick Crutchley, Daniel Preoţiuc-Pietro, David A Asch, and H Andrew Schwartz. 2018\.  Facebook language predicts depression in medical records.  _Proceedings of the National Academy of Sciences_ 115, 44 (2018), 11203–11208.
  * Franco et al. (2023) Mirko Franco, Ombretta Gaggi, and Claudio E Palazzi. 2023\.  Analyzing the use of large language models for content moderation with chatgpt examples. In _Proceedings of the 3rd International Workshop on Open Challenges in Online Social Networks_. 1–8.
  * Gaur et al. (2019) Manas Gaur, Amanuel Alambo, Joy Prakash Sain, Ugur Kursuncu, Krishnaprasad Thirunarayan, Ramakanth Kavuluru, Amit Sheth, Randy Welton, and Jyotishman Pathak. 2019.  Knowledge-aware Assessment of Severity of Suicide Risk for Early Intervention. In _The World Wide Web Conference_. ACM, San Francisco CA USA, 514–525.  https://doi.org/10.1145/3308558.3313698
  * Gemalmaz and Yin (2021) Meric Altug Gemalmaz and Ming Yin. 2021.  Accounting for Confirmation Bias in Crowdsourced Label Aggregation.. In _IJCAI_. 1729–1735.
  * Ghosh and Caliskan (2023) Sourojit Ghosh and Aylin Caliskan. 2023.  ChatGPT Perpetuates Gender Bias in Machine Translation and Ignores Non-Gendered Pronouns: Findings across Bengali and Five other Low-Resource Languages.  _arXiv preprint arXiv:2305.10510_ (2023).
  * Gkotsis et al. (2016) George Gkotsis, Anika Oellrich, Tim Hubbard, Richard Dobson, Maria Liakata, Sumithra Velupillai, and Rina Dutta. 2016.  The language of mental health problems in social media. In _Proceedings of the third workshop on computational linguistics and clinical psychology_. 63–73.
  * Graham et al. (2019) Sarah Graham, Colin Depp, Ellen E Lee, Camille Nebeker, Xin Tu, Ho-Cheol Kim, and Dilip V Jeste. 2019.  Artificial intelligence for mental health and mental illnesses: an overview.  _Current psychiatry reports_ 21 (2019), 1–18.
  * Gulcehre et al. (2017) Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, and Yoshua Bengio. 2017.  On integrating a language model into neural machine translation.  _Computer Speech & Language_ 45 (2017), 137–148.
  * Guntuku et al. (2019) Sharath Chandra Guntuku, Anneke Buffone, Kokil Jaidka, Johannes C Eichstaedt, and Lyle H Ungar. 2019.  Understanding and measuring psychological stress using social media. In _Proceedings of the international AAAI conference on web and social media_ , Vol. 13. 214–225.
  * Guntuku et al. (2017) Sharath Chandra Guntuku, David B Yaden, Margaret L Kern, Lyle H Ungar, and Johannes C Eichstaedt. 2017.  Detecting depression and mental illness on social media: an integrative review.  _Current Opinion in Behavioral Sciences_ 18 (Dec. 2017), 43–49.  https://doi.org/10.1016/j.cobeha.2017.07.005
  * Han et al. (2022) Sooji Han, Rui Mao, and Erik Cambria. 2022.  Hierarchical attention network for explainable depression detection on Twitter aided by metaphor concept mappings.  _arXiv preprint arXiv:2209.07494_ (2022).
  * Haque et al. (2021) Ayaan Haque, Viraaj Reddi, and Tyler Giallanza. 2021\.  Deep Learning for Suicide and Depression Identification with Unsupervised Label Correction.  http://arxiv.org/abs/2102.09427 arXiv:2102.09427 [cs].
  * Hoover (2023) Amanda Hoover. 2023\.  An eating disorder chatbot is suspended for giving harmful advice.  https://www.wired.com/story/tessa-chatbot-suspended/
  * Hu et al. (2021) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021\.  LoRA: Low-Rank Adaptation of Large Language Models.  http://arxiv.org/abs/2106.09685 arXiv:2106.09685 [cs].
  * Huang et al. (2022) Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2022\.  Large language models can self-improve.  _arXiv preprint arXiv:2210.11610_ (2022).
  * Irene Y. Chen et al. (2021) Irene Y. Chen, Emma Pierson, Sherri Rose, Shalmali Joshi, Kadija Ferryman, and Marzyeh Ghassemi. 2021.  Ethical Machine Learning in Healthcare.  _Annual Review of Biomedical Data Science_ 4, 1 (2021), 123–144.  https://doi.org/10.1146/annurev-biodatasci-092820-114757 _eprint: https://doi.org/10.1146/annurev-biodatasci-092820-114757.
  * Irene Y. Chen et al. (2019) Irene Y. Chen, Peter Szolovits, and Marzyeh Ghassemi. 2019\.  Can AI Help Reduce Disparities in General Medical and Mental Health Care?  _AMA Journal of Ethics_ 21, 2 (Feb. 2019), E167–179.  https://doi.org/10.1001/amajethics.2019.167
  * J. Abrantes (2023) M. J. N. Bento e Silva J. Abrantes. 2023\.  External validation of a deep learning model for breast density classification. In _ECR 2023 EPOS_.  https://epos.myesr.org/poster/esr/ecr2023/C-16014
  * Jamil et al. (2017) Zunaira Jamil, Diana Inkpen, Prasadith Buddhitha, and Kenton White. 2017.  Monitoring Tweets for Depression to Detect At-risk Users. In _Proceedings of the Fourth Workshop on Computational Linguistics and Clinical Psychology — From Linguistic Signal to Clinical Reality_ , Kristy Hollingshead, Molly E. Ireland, and Kate Loveys (Eds.). Association for Computational Linguistics, Vancouver, BC, 32–40.  https://doi.org/10.18653/v1/W17-3104
  * Ji et al. (2018) Shaoxiong Ji, Celina Ping Yu, Sai-fu Fung, Shirui Pan, and Guodong Long. 2018.  Supervised learning for suicidal ideation detection in online user content.  _Complexity_ 2018 (2018).
  * Ji et al. (2021) Shaoxiong Ji, Tianlin Zhang, Luna Ansari, Jie Fu, Prayag Tiwari, and Erik Cambria. 2021\.  MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcare.  http://arxiv.org/abs/2110.15621
  * Jiang et al. (2023) Lavender Yao Jiang, Xujin Chris Liu, Nima Pour Nejatian, Mustafa Nasir-Moin, Duo Wang, Anas Abidin, Kevin Eaton, Howard Antony Riina, Ilya Laufer, Paawan Punjabi, Madeline Miceli, Nora C. Kim, Cordelia Orillac, Zane Schnurman, Christopher Livia, Hannah Weiss, David Kurland, Sean Neifert, Yosef Dastagirzada, Douglas Kondziolka, Alexander T. M. Cheung, Grace Yang, Ming Cao, Mona Flores, Anthony B. Costa, Yindalon Aphinyanaphongs, Kyunghyun Cho, and Eric Karl Oermann. 2023.  Health system-scale language models are all-purpose prediction engines.  _Nature_ (June 2023).  https://doi.org/10.1038/s41586-023-06160-y
  * Jiang et al. (2020) Zheng Ping Jiang, Sarah Ita Levitan, Jonathan Zomick, and Julia Hirschberg. 2020.  Detection of mental health from reddit via deep contextualized representations. In _Proceedings of the 11th international workshop on health text mining and information analysis_. 147–156.
  * Jo et al. (2023) Eunkyung Jo, Daniel A Epstein, Hyunhoon Jung, and Young-Ho Kim. 2023.  Understanding the benefits and challenges of deploying conversational AI leveraging large language models for public health intervention. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_. 1–16.
  * Kayalvizhi et al. (2022) S Kayalvizhi, Thenmozhi Durairaj, Bharathi Raja Chakravarthi, et al. 2022\.  Findings of the shared task on detecting signs of depression from social media. In _Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion_. 331–338.
  * Kernan Freire et al. (2023) Samuel Kernan Freire, Mina Foosherian, Chaofan Wang, and Evangelos Niforatos. 2023.  Harnessing Large Language Models for Cognitive Assistants in Factories. In _Proceedings of the 5th International Conference on Conversational User Interfaces_. 1–6.
  * Kocoń et al. (2023) Jan Kocoń, Igor Cichecki, Oliwier Kaszyca, Mateusz Kochanek, Dominika Szydło, Joanna Baran, Julita Bielaniewicz, Marcin Gruza, Arkadiusz Janz, Kamil Kanclerz, et al. 2023\.  ChatGPT: Jack of all trades, master of none.  _Information Fusion_ (2023), 101861\.
  * Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022\.  Large Language Models are Zero-Shot Reasoners. In _36th Conference on Neural Information Processing Systems_.
  * Kruzan et al. (2022) Kaylee Payne Kruzan, Kofoworola D.A. Williams, Jonah Meyerhoff, Dong Whi Yoo, Linda C. O’Dwyer, Munmun De Choudhury, and David C. Mohr. 2022.  Social media-based interventions for adolescent and young adult mental health: A scoping review.  _Internet Interventions_ 30 (Dec. 2022), 100578\.  https://doi.org/10.1016/j.invent.2022.100578
  * Lamichhane (2023) Bishal Lamichhane. 2023\.  Evaluation of ChatGPT for NLP-based Mental Health Applications.  http://arxiv.org/abs/2303.15727
  * Lee et al. (2020) Yi-Chieh Lee, Naomi Yamashita, and Yun Huang. 2020\.  Designing a Chatbot as a Mediator for Promoting Deep Self-Disclosure to a Real Mental Health Professional.  _Proceedings of the ACM on Human-Computer Interaction_ 4, CSCW1 (May 2020), 1–27.  https://doi.org/10.1145/3392836
  * Li et al. (2023a) Yucheng Li, Bo Dong, Chenghua Lin, and Frank Guerin. 2023a.  Compressing Context to Enhance Inference Efficiency of Large Language Models.  _arXiv preprint arXiv:2310.06201_ (2023).
  * Li et al. (2023b) Yunxiang Li, Zihan Li, Kai Zhang, Ruilong Dan, Steve Jiang, and You Zhang. 2023b.  ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge.  http://arxiv.org/abs/2303.14070 arXiv:2303.14070 [cs].
  * Liu et al. (2023) Xin Liu, Daniel McDuff, Geza Kovacs, Isaac Galatzer-Levy, Jacob Sunshine, Jiening Zhan, Ming-Zher Poh, Shun Liao, Paolo Di Achille, and Shwetak Patel. 2023\.  Large Language Models are Few-Shot Health Learners. In _arXiv_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  http://arxiv.org/abs/1907.11692 arXiv:1907.11692 [cs].
  * Livingston et al. (2014) James D. Livingston, Michelle Cianfrone, Kimberley Korf-Uzan, and Connie Coniglio. 2014.  Another time point, a different story: one year effects of a social media intervention on the attitudes of young people towards mental health issues.  _Social Psychiatry and Psychiatric Epidemiology_ 49, 6 (June 2014), 985–990.  https://doi.org/10.1007/s00127-013-0815-7
  * Lovejoy (2019) Christopher A Lovejoy. 2019\.  Technology and mental health: the role of artificial intelligence.  _European Psychiatry_ 55 (2019), 1–3.
  * Lupetti et al. (2023) Maria Luce Lupetti, Emma Hagens, Willem Van Der Maden, Régine Steegers-Theunissen, and Melek Rousian. 2023\.  Trustworthy Embodied Conversational Agents for Healthcare: A Design Exploration of Embodied Conversational Agents for the periconception period at Erasmus MC. In _Proceedings of the 5th International Conference on Conversational User Interfaces_. 1–14.
  * Mauriello et al. (2021) Matthew Louis Mauriello, Thierry Lincoln, Grace Hon, Dorien Simon, Dan Jurafsky, and Pablo Paredes. 2021.  SAD: A Stress Annotated Dataset for Recognizing Everyday Stressors in SMS-like Conversational Systems. In _Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems_. ACM, Yokohama Japan, 1–7.  https://doi.org/10.1145/3411763.3451799
  * Mitchell et al. (2015) Margaret Mitchell, Kristy Hollingshead, and Glen Coppersmith. 2015\.  Quantifying the Language of Schizophrenia in Social Media. In _Proceedings of the 2nd Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality_. Association for Computational Linguistics, Denver, Colorado, 11–20.  https://doi.org/10.3115/v1/W15-1202
  * Morais et al. (2023) Margarida Morais, Francisco Maria Calisto, Carlos Santiago, Clara Aleluia, and Jacinto C Nascimento. 2023.  Classification of breast cancer in Mri with multimodal fusion. In _2023 IEEE 20th international symposium on biomedical imaging (ISBI)_. IEEE, 1–4.
  * Moreno et al. (2011) Megan A Moreno, Lauren A Jelenchick, Katie G Egan, Elizabeth Cox, Henry Young, Kerry E Gannon, and Tara Becker. 2011.  Feeling bad on Facebook: Depression disclosures by college students on a social networking site.  _Depression and anxiety_ 28, 6 (2011), 447–455.
  * Naseem et al. (2022) Usman Naseem, Adam G. Dunn, Jinman Kim, and Matloob Khushi. 2022\.  Early Identification of Depression Severity Levels on Reddit Using Ordinal Classification. In _Proceedings of the ACM Web Conference 2022_. ACM, Virtual Event, Lyon France, 2563–2572.  https://doi.org/10.1145/3485447.3512128
  * Nepal et al. (2022) Subigya Nepal, Gonzalo J. Martinez, Shayan Mirjafari, Koustuv Saha, Vedant Das Swain, Xuhai Xu, Pino G. Audia, Munmun De Choudhury, Anind K. Dey, Aaron Striegel, and Andrew T. Campbell. 2022.  A Survey of Passive Sensing in the Workplace.  arXiv:2201.03074 [cs.HC]
  * Nguyen et al. (2014) Thin Nguyen, Dinh Phung, Bo Dao, Svetha Venkatesh, and Michael Berk. 2014.  Affective and content analysis of online depression communities.  _IEEE transactions on affective computing_ 5, 3 (2014), 217–226.
  * Nguyen et al. (2022) Thong Nguyen, Andrew Yates, Ayah Zirikly, Bart Desmet, and Arman Cohan. 2022.  Improving the generalizability of depression detection by leveraging clinical questionnaires.  _arXiv preprint arXiv:2204.10432_ (2022).
  * Nijhawan et al. (2022) Tanya Nijhawan, Girija Attigeri, and T Ananthakrishna. 2022\.  Stress detection using natural language processing and machine learning over social interactions.  _Journal of Big Data_ 9, 1 (2022), 1–24.
  * Nori et al. (2023) Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. 2023\.  Capabilities of GPT-4 on Medical Challenge Problems.  http://arxiv.org/abs/2303.13375 arXiv:2303.13375 [cs].
  * Ntoutsi et al. (2020) Eirini Ntoutsi, Pavlos Fafalios, Ujwal Gadiraju, Vasileios Iosifidis, Wolfgang Nejdl, Maria-Esther Vidal, Salvatore Ruggieri, Franco Turini, Symeon Papadopoulos, Emmanouil Krasanakis, et al. 2020\.  Bias in data-driven artificial intelligence systems—An introductory survey.  _Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery_ 10, 3 (2020), e1356.
  * Omar et al. (2023) Reham Omar, Omij Mangukiya, Panos Kalnis, and Essam Mansour. 2023.  Chatgpt versus traditional question answering for knowledge graphs: Current status and future directions towards knowledge graph chatbots.  _arXiv preprint arXiv:2302.06466_ (2023).
  * Otsuka et al. ([n. d.]) Norio Otsuka, Yuu Kawanishi, Fumimaro Doi, Tsutomu Takeda, Kazuki Okumura, Takahira Yamauchi, Shuntaro Yada, Shoko Wakamiya, Eiji Aramaki, and Manabu Makinodan. [n. d.].  Diagnosing Psychiatric Disorders from History of Present Illness Using a Large-Scale Linguistic Model.  _Psychiatry and Clinical Neurosciences_ ([n. d.]).
  * Park et al. (2021) Minsu Park, David McDonald, and Meeyoung Cha. 2021\.  Perception Differences between the Depressed and Non-Depressed Users in Twitter.  _Proceedings of the International AAAI Conference on Web and Social Media_ 7, 1 (Aug. 2021), 476–485.  https://doi.org/10.1609/icwsm.v7i1.14425
  * Patel et al. (2018) Vivek Patel, Piyush Mishra, and JC Patni. 2018.  PsyHeal: An Approach to Remote Mental Health Monitoring System. In _2018 International Conference on Advances in Computing and Communication Engineering (ICACCE)_. IEEE, 384–393.
  * Paul and Dredze (2011) Michael Paul and Mark Dredze. 2011.  You Are What You Tweet: Analyzing Twitter for Public Health.  _Proceedings of the International AAAI Conference on Web and Social Media_ 5, 1 (2011), 265–272.  https://doi.org/10.1609/icwsm.v5i1.14137
  * Pessach and Shmueli (2022) Dana Pessach and Erez Shmueli. 2022.  A review on fairness in machine learning.  _ACM Computing Surveys (CSUR)_ 55, 3 (2022), 1–44.
  * Posner et al. (2008) K Posner, D Brent, C Lucas, M Gould, B Stanley, G Brown, P Fisher, J Zelazny, A Burke, MJNY Oquendo, et al. 2008\.  Columbia-suicide severity rating scale (C-SSRS).  _New York, NY: Columbia University Medical Center_ 10 (2008), 2008.
  * Praw-Dev ([n. d.]) Praw-Dev. [n. d.].  Praw-dev/PRAW: PRAW, an acronym for “Python reddit api wrapper”, is a python package that allows for simple access to Reddit’s API.  https://github.com/praw-dev/praw
  * Qin et al. (2023) Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023\.  Is ChatGPT a general-purpose natural language processing task solver?  _arXiv preprint arXiv:2302.06476_ (2023).
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018.  Improving Language Understanding by Generative Pre-Training.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the Limits of Transfer Learning with a Uniﬁed Text-to-Text Transformer.  _Journal of Machine Learning Research_ (2020).
  * Regier et al. (2013) Darrel A Regier, Emily A Kuhl, and David J Kupfer. 2013\.  The DSM-5: Classification and criteria changes.  _World psychiatry_ 12, 2 (2013), 92–98.
  * Ridout and Campbell (2018) Brad Ridout and Andrew Campbell. 2018.  The Use of Social Networking Sites in Mental Health Interventions for Young People: Systematic Review.  _Journal of Medical Internet Research_ 20, 12 (Dec. 2018), e12244.  https://doi.org/10.2196/12244
  * Robinson and Wingate (2023) Joshua Robinson and David Wingate. 2023.  Leveraging Large Language Models for Multiple Choice Question Answering. In _The Eleventh International Conference on Learning Representations_.  https://openreview.net/forum?id=yKbprarjc5B
  * Ruder et al. (2011) Thomas Ruder, Gary Hatch, Garyfalia Ampanozi, Michael Thali, and Nadja Fischer. 2011.  Suicide Announcement on Facebook.  _Crisis_ 32 (June 2011), 280–2.  https://doi.org/10.1027/0227-5910/a000086
  * Rumshisky et al. (2016) Anna Rumshisky, Marzyeh Ghassemi, Tristan Naumann, Peter Szolovits, VM Castro, TH McCoy, and RH Perlis. 2016.  Predicting early psychiatric readmission with natural language processing of narrative discharge summaries.  _Translational psychiatry_ 6, 10 (2016), e921–e921.
  * Saha et al. (2017) Koustuv Saha, Larry Chan, Kaya De Barbaro, Gregory D. Abowd, and Munmun De Choudhury. 2017.  Inferring Mood Instability on Social Media by Leveraging Ecological Momentary Assessments.  _Proc. ACM Interact. Mob. Wearable Ubiquitous Technol._ 1, 3, Article 95 (sep 2017), 27 pages.  https://doi.org/10.1145/3130960
  * Saifullah et al. (2021) Shoffan Saifullah, Yuli Fauziah, and Agus Sasmito Aribowo. 2021\.  Comparison of machine learning for sentiment analysis in detecting anxiety based on social media data.  _arXiv preprint arXiv:2101.06353_ (2021).
  * Sampath and Durairaj (2022) Kayalvizhi Sampath and Thenmozhi Durairaj. 2022.  Data Set Creation and Empirical Analysis for Detecting Signs of Depression from Social Media Postings.  In _Computational Intelligence in Data Science_ , Lekshmi Kalinathan, Priyadharsini R., Madheswari Kanmani, and Manisha S. (Eds.). Vol. 654. Springer International Publishing, Cham, 136–151.  https://doi.org/10.1007/978-3-031-16364-7_11 Series Title: IFIP Advances in Information and Communication Technology.
  * Sarkar et al. (2022) Shailik Sarkar, Abdulaziz Alhamadani, Lulwah Alkulaib, and Chang-Tien Lu. 2022.  Predicting depression and anxiety on reddit: a multi-task learning approach. In _2022 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM)_. IEEE, 427–435.
  * Sawhney et al. (2018) Ramit Sawhney, Prachi Manchanda, Puneet Mathur, Rajiv Shah, and Raj Singh. 2018.  Exploring and learning suicidal ideation connotations on social media with deep learning. In _Proceedings of the 9th workshop on computational approaches to subjectivity, sentiment and social media analysis_. 167–175.
  * Sharma et al. (2021) Ashish Sharma, Inna W. Lin, Adam S. Miner, David C. Atkins, and Tim Althoff. 2021.  Towards Facilitating Empathic Conversations in Online Mental Health Support: A Reinforcement Learning Approach. In _Proceedings of the Web Conference 2021_. ACM, Ljubljana Slovenia, 194–205.  https://doi.org/10.1145/3442381.3450097
  * Sharma et al. (2023) Ashish Sharma, Inna W. Lin, Adam S. Miner, David C. Atkins, and Tim Althoff. 2023.  Human–AI collaboration enables more empathic conversations in text-based peer-to-peer mental health support.  _Nature Machine Intelligence_ 5, 1 (Jan. 2023), 46–57.  https://doi.org/10.1038/s42256-022-00593-2
  * Sharma and De Choudhury (2018) Eva Sharma and Munmun De Choudhury. 2018.  Mental health support and its relationship to linguistic accommodation in online communities. In _Proceedings of the 2018 CHI conference on human factors in computing systems_. 1–13.
  * Shen and Rudzicz (2017) Judy Hanwen Shen and Frank Rudzicz. 2017.  Detecting Anxiety through Reddit. In _Proceedings of the Fourth Workshop on Computational Linguistics and Clinical Psychology — From Linguistic Signal to Clinical Reality_. Association for Computational Linguistics, Vancouver, BC, 58–65.  https://doi.org/10.18653/v1/W17-3107
  * Singhal et al. (2023) Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, Mike Schaekermann, Amy Wang, Mohamed Amin, Sami Lachgar, Philip Mansfield, Sushant Prakash, Bradley Green, Ewa Dominowska, Blaise Aguera y Arcas, Nenad Tomasev, Yun Liu, Renee Wong, Christopher Semturs, S. Sara Mahdavi, Joelle Barral, Dale Webster, Greg S. Corrado, Yossi Matias, Shekoofeh Azizi, Alan Karthikesalingam, and Vivek Natarajan. 2023.  Towards Expert-Level Medical Question Answering with Large Language Models.  http://arxiv.org/abs/2305.09617 arXiv:2305.09617 [cs].
  * Tadesse et al. (2019a) Michael M Tadesse, Hongfei Lin, Bo Xu, and Liang Yang. 2019a.  Detection of depression-related posts in reddit social media forum.  _IEEE Access_ 7 (2019), 44883–44893.
  * Tadesse et al. (2019b) Michael Mesfin Tadesse, Hongfei Lin, Bo Xu, and Liang Yang. 2019b.  Detection of Suicide Ideation in Social Media Forums Using Deep Learning.  _Algorithms_ 13, 1 (Dec. 2019), 7\.  https://doi.org/10.3390/a13010007
  * Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023.  Stanford alpaca: An instruction-following llama model.
  * Timmons et al. (2022) Adela C Timmons, Jacqueline B Duong, Natalia Simo Fiallo, Theodore Lee, Huong Phuc Quynh Vo, Matthew W Ahle, Jonathan S Comer, LaPrincess C Brewer, Stacy L Frazier, and Theodora Chaspari. 2022.  A Call to Action on Assessing and Mitigating Bias in Artificial Intelligence Applications for Mental Health.  _Perspectives on Psychological Science_ (2022), 17456916221134490.
  * Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a.  LLaMA: Open and Efficient Foundation Language Models.  http://arxiv.org/abs/2302.13971 arXiv:2302.13971 [cs].
  * Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023b.  Llama 2: Open Foundation and Fine-Tuned Chat Models.  http://arxiv.org/abs/2307.09288 arXiv:2307.09288 [cs].
  * Tsugawa et al. (2015) Sho Tsugawa, Yusuke Kikuchi, Fumio Kishino, Kosuke Nakajima, Yuichi Itoh, and Hiroyuki Ohsaki. 2015\.  Recognizing Depression from Twitter Activity. In _Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems_. ACM, Seoul Republic of Korea, 3187–3196.  https://doi.org/10.1145/2702123.2702280