Further author information: (Send correspondence to Guanghui Fu,
<EMAIL_ADDRESS>
# Evaluating the Efficacy of Supervised Learning vs. Large Language Models for
Identifying Cognitive Distortions and Suicidal Risks in Chinese Social Media

Hongzhi Qi  Faculty of Information Technology, Beijing University of
Technology, Beijing, 100124, China  Qing Zhao  Faculty of Information
Technology, Beijing University of Technology, Beijing, 100124, China  Changwei
Song  Faculty of Information Technology, Beijing University of Technology,
Beijing, 100124, China  Wei Zhai  Faculty of Information Technology, Beijing
University of Technology, Beijing, 100124, China  Dan Luo  Center for Wise
Information Technology of Mental Health Nursing Research, School of Nursing,
Wuhan University, Wuhan, China  Shuo Liu  Center for Wise Information
Technology of Mental Health Nursing Research, School of Nursing, Wuhan
University, Wuhan, China  Yi Jing Yu  Center for Wise Information Technology
of Mental Health Nursing Research, School of Nursing, Wuhan University, Wuhan,
China  Fan Wang  Center for Wise Information Technology of Mental Health
Nursing Research, School of Nursing, Wuhan University, Wuhan, China  Huijing
Zou  Center for Wise Information Technology of Mental Health Nursing Research,
School of Nursing, Wuhan University, Wuhan, China  Bing Xiang Yang  Center for
Wise Information Technology of Mental Health Nursing Research, School of
Nursing, Wuhan University, Wuhan, China  Jianqiang Li  Faculty of Information
Technology, Beijing University of Technology, Beijing, 100124, China  Guanghui
Fu* Sorbonne Université, Institut du Cerveau - Paris Brain Institute - ICM,
CNRS, Inria, Inserm, AP-HP, Hôpital de la Pitié Salpêtrière, Paris, France

###### Abstract

Large language models, particularly those akin to the rapidly progressing GPT
series, are gaining traction for their expansive influence. While there is
keen interest in their applicability within medical domains such as
psychology, tangible explorations on real-world data remain scant.
Concurrently, users on social media platforms are increasingly vocalizing
personal sentiments; under specific thematic umbrellas, these sentiments often
manifest as negative emotions, sometimes escalating to suicidal inclinations.
Timely discernment of such cognitive distortions and suicidal risks is crucial
to effectively intervene and potentially avert dire circumstances. Our study
ventured into this realm by experimenting on two pivotal tasks: suicidal risk
and cognitive distortion identification on Chinese social media platforms.
Using supervised learning as a baseline, we examined and contrasted the
efficacy of large language models via three distinct strategies: zero-shot,
few-shot, and fine-tuning. Our findings revealed a discernible performance gap
between the large language models and traditional supervised learning
approaches, primarily attributed to the models’ inability to fully grasp
subtle categories. Notably, while GPT-4 outperforms its counterparts in
multiple scenarios, GPT-3.5 shows significant enhancement in suicide risk
classification after fine-tuning. To our knowledge, this investigation stands
as the maiden attempt at gauging large language models on Chinese social media
tasks. This study underscores the forward-looking and transformative
implications of using large language models in the field of psychology. It
lays the groundwork for future applications in psychological research and
practice.

###### keywords:

Large language model, Deep learning, Mental health, Social media

##  1 Introduction

The omnipresent specter of mental illness, particularly depression, continues
to impose significant challenges globally [1]. According to the World Health
Organization (WHO), an estimated 3.8% of the global population experiences
depression [1]. Specifically in China, the prevalence of depression is notably
high, with estimates around 6.9% [2], underscoring the escalating mental
health concerns in the nation. Such severe depression can often precipitate
suicidal behaviors [3]. As digital avenues for communication flourish, social
media platforms like Twitter and Sina Weibo have evolved into reflective
mirrors, offering glimpses into the emotional landscapes of countless users
[4]. Within these platforms, a specific subset of topics recurrently surfaces,
with users frequently conveying deep-seated negative emotions and, alarmingly,
pronounced suicidal inclinations [5, 6].

Artificial intelligence (AI), especially the branch underscored by deep
learning and natural language processing technique, is an avenue that holds
promise in addressing this challenge [7]. Over recent years, AI research has
resulted in the formulation of a plethora of algorithms tailored for emotion
recognition within textual data [8]. However, these advancements are not
without obstacles [9]. Constructing a potent deep learning model often demands
considerable time and financial resources. The intricacies of data labeling,
predominantly the need to enlist domain experts and the model’s variance in
performance when shifted across different application areas, highlight
pressing challenges [10]. This underscores a compelling need for more agile
and adaptable algorithmic solutions especially in medical domain [11]. It is
in this context that the emergence and proliferation of large language models
are particularly noteworthy.

Large language models, characterized by their expansive parameters and the
depth of their training datasets, stand as vanguards in the realm of
computational linguistics [12]. Their prowess lies in their ability to
comprehend and emulate human-like text nuances. Despite their promising
potential, several studies have sought to validate their practical
implications. For instance, Xu et al. [13] examined four public datasets
related to online social media sentiment detection. However, their study
focused solely on English data, and the classification granularity was
relatively broad. To date, there is a notable gap in research concerning the
Chinese context, particularly in the area of fine-grained emotion recognition,
which is often of greater significance. The lack of comprehensive evaluations
and practical tests has inadvertently led to a cautious approach, especially
in sectors demanding high reliability, like medicine and healthcare [14].
Motivated by these challenges and opportunities, our research embarks on a
nuanced exploration of two psychology-related tasks using data from Chinese
social media platforms. We offer the following contributions:

  * •

To our best knowledge, our research presents the first time a detailed
comparison between supervised learning methodologies and large language models
using datasets from Chinese social media, with a specific focus on
psychological tasks. These tasks are not just academic exercises; they have
real-world implications, potentially shaping strategies for suicide prevention
and interventions for cognitive distortions.

  * •

We move beyond the foundational aspects and critically assess the performance
of large language models, paying particular attention to the role of prompt.
This endeavor aims to shed light on the optimal strategies for employing these
models in specific contexts, offering a roadmap for future researchers.

  * •

Lastly, our study pioneers the exploration of fine-tuning capabilities of
GPT-3.5, leveraging real-world data. This endeavor seeks to determine the
adaptability and specialized performance enhancements possible with the model,
a facet largely uncharted in current literature.

##  2 Related work

The intertwining of artificial intelligence (AI) with myriad fields has
spurred innovations and transformations at an unprecedented scale. A
quintessential example is the fusion of natural language processing (NLP)
tools, notably deep learning based model, with domains as critical as the
mental health field [15]. Additionally, as digital interactions burgeon,
especially on social media, the urgency to understand and analyze human
sentiments becomes paramount. In this section, we will delve into deep
learning techniques for sentiment analysis utilizing text data (Section 2.1).
Subsequently, we will discuss the evolution, potential, and current research
on large language models in this domain (Section 2.2).

###  2.1 Text sentiment analysis

In the swiftly evolving digital era, social networking platforms have emerged
as pivotal channels for expressing emotions globally. These platforms generate
vast amounts of unstructured data every second. Accurately and promptly
discerning the emotions embedded within this data presents a formidable
challenge to computational algorithms [8]. Fu et al. [16] presented a distant
supervision method designed to build systems that classify high and low
suicide risk levels using Chinese social media data. This approach minimizes
the need for human experts of varying expertise levels to perform annotations.
By integrating this model with crucial psychological features extracted from
user blogs, they attained an F1 score of 77.98%. Singh et al. [17] employed a
BERT-based model for sentiment analysis on tweets sourced globally and another
dataset specifically from India, both focusing on the topic of COVID-19. They
reported achieving an accuracy of 94%. Wan [18] introduced a method for
sentiment analysis of comments on Weibo platforms, leveraging deep neural
networks. The data undergoes feature extraction through multilevel pooling and
convolution layers. Comments are preprocessed and transformed into text
representations using the word2vec algorithm. Subsequently, key features are
extracted from the feature matrix using a CNN. For the final classification
and sentiment analysis, the softmax logistic regression method is employed.
Zhang et al. [19] explored the correlations among emotion labels, social
interactions, and temporal patterns within an annotated Twitter dataset. They
introduced a factor graph-based emotion recognition model that seamlessly
integrates these correlations into a unified framework. This model adeptly
identifies multiple emotions by applying a multi-label learning approach to
Twitter datasets. Wang et al. [20] introduced a topic modeling technique,
termed LDA, to examine the primary concerns expressed on Weibo during the
COVID-19 pandemic. They assessed the emotional inclinations of these topics,
determined their proportional distributions, and conducted user behavior
analysis based on metrics such as likes, comments, and retweets. Furthermore,
they explored shifts in user concerns and variations in engagement among
residents from different regions of mainland China. Such insights guide public
sentiment and actions during health emergencies, emphasizing the importance of
vigilant social media monitoring.

Although deep learning algorithms typically demonstrate impressive results,
they often require a significant volume of labeled data to perform optimally.
The distant supervision approach highlighted in Fu et al.’s research [16] aims
to reduce the need for labeling, but it still requires the involvement of
three different expert groups at various expertise levels to yield desired
results. Nonetheless, when applying these models to new datasets or tasks,
domain adaptation issues often arise. These trained models can see a decline
in their efficacy, making deep learning algorithms both costly and inflexible.
Given these hurdles, there’s a growing demand for efficient and user-centric
methods to assist individuals in emotion detection on social media platforms.
The recent advancements in large language models present a potential solution
to this challenge, but their precise impact still warrants examination from
multiple perspectives and specialists.

###  2.2 Large language model and its applications in medical domain

The advent of Large Language Models (LLMs), such as OpenAI’s ChatGPT [12], has
revolutionized the field of natural language processing [21]. These LLMs
demonstrate emergent abilities that significantly outperform those of their
smaller, pre-trained models [22]. Initially conceived for understanding and
generating human-like text, LLMs have found diverse applications ranging from
content generation [23], medical report assistant [24], coding assistance
[25], education [26], and answering medical related questions [27]. The sheer
scale of these models enables them to generate complex, contextually relevant
content. LLMs have garnered significant attention in medical domain [14]. For
instance, Jiang et al. [28] developed a clinical LLM named NYUTron to assist
physicians and healthcare administrators in making time-sensitive decisions.
This model can process on unstructured clinical notes from electronic health
record. And it can achieve good performance with AUC score ranging from
78.7–94.9%. The model has been successfully deployed in a prospective trial,
indicating its potential for real-world application in providing point-of-care
guidance to physicians.

Concurrently, research in psychology-related domains has also been conducted
by other researcher [29, 29]. Qin et al. [30] devised an interpretable and
interactive depression detection system employing large language models
(LLMs). This innovative approach allows for the detection of mental health
indicators through social media activity and encourages users to interact with
the system using natural language. While this facilitates a more personalized
understanding of an individual’s mental state, it also raises ethical
concerns. The absence of human oversight could lead to biased outcomes,
thereby posing potential risks to users. Additionally, if this system were to
become a foundational diagnostic tool for future psychological counseling,
issues related to user privacy could become a point of concern. Chen et al.
[31] developed a tool designed to improve the realism of psychiatrist-patient
simulations using ChatGPT-based chatbots. Their approach involved using
distinct prompts to enable large language models (LLMs) to emulate the roles
of both a depressed patient and a psychiatrist. The study confirmed the
feasibility of utilizing ChatGPT-driven chatbots in psychiatric contexts.
However, the research also acknowledged limitations: individual patients and
counselors have unique communication styles, and some patients may be
reluctant to engage in conversation. These nuances present a challenge for
achieving truly realistic simulations with ChatGPT. Addressing the simulation
of diverse personalities in a meaningful way remains a key area for further
investigation. Fu et al. [32] developed a counseling support system designed
to augment the capabilities of non-professional counselors. The system
provides multiple features, including mental health analysis, evaluation of
therapist responses, and suggested interventions. This application serves as a
valuable use case for language models in the mental health sector. Ten
professional psychologists assessed the system on five critical dimensions,
and the findings were favorable, with a 78% expert approval rate indicating
that the system can deliver effective treatment strategies. Ayers et al. [33]
developed a ChatGPT-based chatbot and compared its responses with those of
physicians to patient inquiries on a social media forum. Notably, 78.6% of the
evaluators preferred the chatbot’s responses, citing their speed and greater
empathetic tone. However, a key limitation of this study lies in its exclusive
focus on interactions within online forums. Such settings may not accurately
reflect the nuances of real-world patient-physician dialogues, as physicians
often tailor their responses based on pre-existing relationships and the
context of a clinical setting. In summary, there is active research into the
utilization of LLMs in the field of psychology, and these research demonstrate
considerable potential. However, delineating the limitations of LLMs remains a
crucial issue that warrants further investigation. Additional studies are
needed to comprehensively evaluate the capabilities and boundaries of LLMs in
psychological applications.

Xu et al. [13] present a pioneering evaluation of multiple Large Language
Models (LLMs) across various mental health prediction tasks using four
publicly available online text datasets. Their insights offer guidance to
practitioners on optimizing the use of LLMs for specific applications. While
their research stands as a monumental verification of LLMs’ potential in the
mental health domain, it is noteworthy that their datasets are exclusively in
English and do not address multi-label classification tasks. Yang et al. [34]
assessed ChatGPT’s capabilities in mental health analysis and emotional
reasoning by evaluating its performance on 11 datasets across five tasks. The
study also investigated the impact of different emotion-based prompting
strategies. Experimental results indicate that while ChatGPT surpasses
traditional neural network-based approaches, it still lags behind more
advanced, task-specific methods. Nevertheless, ChatGPT demonstrates
significant potential in the area of explainable mental health analysis. In
conclusion, while the integration of LLMs in medicine presents compelling
prospects, there’s an imperative to ensure privacy and uphold ethical
standards. Responses generated may not always be flawless [35]. Particularly
in mental health, relying solely on LLM-driven systems for diagnosis or
support introduces numerous unpredictable variables. It’s crucial to recognize
that LLMs warrant meticulous scrutiny and validation [36]. Evaluation should
be considered an essential discipline to facilitate the more effective
development of large language models (LLMs) [37].

##  3 Methods

We conducted experiments to classify suicide risk and cognitive distortions on
Chinese social media data using supervised learning methods and large language
models (LLMs). Within the realm of supervised learning, we explored two models
BERT [38] and LSAN [39] as baseline, detailed in Section 3.1. For the large
language models, we utilized zero-shot prompt, few-shot prompt, and fine-
tuning methods. Subsequent sections provide a comprehensive introduction of
these methods.

###  3.1 Baseline supervised learning model

We experimented with two representative models: LSAN [39] and BERT [38]. LSAN
is adept at uncovering the relationships between labels, making it
particularly suitable for our cognitive distortion recognition task. On the
other hand, BERT represents a groundbreaking pre-trained model architecture
that had achieved state-of-the-art (SOTA) on 11 distinct NLP tasks. We discuss
each in detail below:

  * •

LSAN: The LSAN model is engineered to utilize label semantics for identifying
the relationships between labels and documents, thereby creating a label-
specific document representation. The model also employs a self-attention
mechanism to focus on this representation, which is derived from the
document’s content. An adaptive fusion strategy integrates these components
effectively, facilitating the generation of a comprehensive document
representation suitable for multi-label text classification. The LSAN model
has proven effective, particularly in predicting low-frequency labels.

  * •

BERT: Bidirectional Encoder Representations from Transformers (BERT) has been
a pivotal development in natural language processing (NLP). Unlike traditional
NLP models that process text unidirectionally, BERT uses a bidirectional
approach, facilitated by the Transformer architecture, to understand the full
context of each word. It is pre-trained using a masked language model
objective, where random words are replaced with a ’[MASK]’ token and the model
predicts the original word. This design has enabled BERT to set new
performance standards in diverse NLP tasks, such as question-answering and
sentiment analysis, especially when fine-tuned on specific task data.

###  3.2 Large language models

Given that our data is in Chinese, we explored the open-source models
ChatGLM2-6B and GLM-130B [40], both of which support Chinese language
processing. The primary distinction between these two models lies in the
number of parameters they possess. GPT-3.5 [41] stands as a flagship large-
scale language model. We experimented with various prompt word constructions
and sought to integrate prior knowledge from the psychological domain, along
with the most recent public fine-tuning functionalities. GPT-4 [42], being the
latest iteration, was also included in our assessment. Detailed introduction
on these models are provided in the subsequent sections.

  * •

ChatGLM2-6B: ChatGLM2-6B is an open-source bilingual language model with 6.2
billion parameters, optimized for Chinese question-answering and dialogue. It
employs similar technology to ChatGPT and is trained on roughly 1 terabyte of
Chinese and English text data. The model can be fine-tuned through various
techniques like supervised learning and human feedback. It also features an
efficient tuning method based on P-Tuning v2, requiring at least 7GB of GPU
memory for customization. Due to quantization techniques, it can run on
consumer-grade graphics cards with only 6GB of memory.

    * –

Source code: https://github.com/thudm/chatglm2-6b

    * –

Unofficial demo: https://huggingface.co/spaces/mikeee/chatglm2-6b-4bit

  * •

GLM-130B: GLM-130B is a bilingual pre-trained language model optimized for
both English and Chinese, boasting a substantial 130 billion parameters. This
model aims to provide an open-source alternative of a scale comparable to
GPT-3, while shedding light on the complexities of training such large-scale
models. During its development, the team faced a host of technical and
engineering challenges, including issues related to loss spikes and model
convergence. Impressively, GLM-130B surpasses GPT-3 175B on multiple English
benchmarks and outperforms ERNIE TITAN 3.0 260B [43], the largest existing
Chinese language model, on relevant benchmarks. A distinctive feature of
GLM-130B is its capability for INT4 quantization without substantial
performance degradation, thus facilitating efficient inference on widely
available GPUs.

    * –

Source code: https://github.com/THUDM/GLM-130B

    * –

Official online demo: https://chatglm.cn/detail

  * •

GPT-3.5: GPT-3.5 is a cutting-edge language model developed by OpenAI,
designed to offer enhanced conversational capabilities. Building on the
foundation of its predecessor, GPT-3, this iteration introduces improvements
in both performance and cost-efficiency. OpenAI’s commitment to refining and
advancing the capabilities of their models is evident in GPT-3.5, which
provides users with a more coherent, context-aware, and responsive
conversational experience. As part of OpenAI’s mission to ensure that
artificial general intelligence benefits all of humanity, GPT-3.5 is a
testament to the organization’s dedication to innovation and excellence in the
realm of natural language processing.

    * –

Web application (also for GPT 4): https://chat.openai.com/

    * –

Fine-tuning details: https://openai.com/blog/gpt-3-5-turbo-fine-tuning-and-
api-updates

  * •

GPT 4: GPT-4 is a groundbreaking multimodal model capable of processing both
image and text inputs to generate text-based outputs. Marking a significant
advancement over its predecessors, GPT-4 exhibits human-level performance
across a range of professional and academic benchmarks, including a top 10%
score on a simulated bar exam. Built upon the Transformer architecture, the
model is initially trained to predict subsequent tokens in a given sequence
and later undergoes a post-training alignment process to improve its
factuality and behavior. A critical component of the project involved the
development of scalable infrastructure and optimization techniques that
function consistently across various sizes, allowing the team to extrapolate
GPT-4’s performance metrics based on smaller models. Despite its notable
capabilities, GPT-4 does inherit certain limitations from earlier versions,
such as occasional content ”hallucinations” and a constrained context window.

The large language model is widely recognized as being pre-trained on vast
amounts of text data. However, the manner in which prompt are inputted is
crucial, as it directly influences the LLM’s comprehension and output for a
given task. In light of this, we have formulated the following prompts.

#### LLM Zero-shot Prompting

We initiate our exploration with prompt design tailored for tasks within a
zero-shot paradigm. This process encompasses various strategies, including
direct task requests (acting as the basic), role-definition, scene-definition,
and hybrid approaches. For illustrative purposes, the cognitive distortion
classification task serves as the focal point. The design is elaborated as
follows:

  1. 1.

basic: A direct task directive devoid of specific contextual emphasis.

    1. (a)

English translation: ”Please conduct a multi-classification task to ascertain
if it encompasses any of the specified 12 cognitive distortions ([list of
cognitive distortions]). ”

    2. (b)

Formulaic representation: $M(T,12CD)$, where $M$ stands for multi-
classification, $T$ symbolizes the task, and $12CD$ represents the 12
cognitive distortions.

  2. 2.

Role-definition Prompting: The prompt delineates the role of the respondent
(in this case, a psychologist) and emphasizes reliance on psychological
insights.

    1. (a)

English translation: ”Assuming the role of a psychologist and leveraging
psychological insights, please conduct a multi-classification task to discern
if it integrates any of the 12 cognitive distortions ([list of cognitive
distortions]).”

    2. (b)

Formulaic representation: $R(M(T,12CD))$, where $R$ embodies the role-
definition of being a psychologist.

  3. 3.

Scene-definition Prompting: The context of a social media setting is
introduced, highlighting user identifiers to preclude ambiguity.

    1. (a)

English translation: ”Considering the provided user ID and the associated
posts on social media, please based on the post content, engage in a multi-
classification task to determine the presence of any of the 12 cognitive
distortions ([list of cognitive distortions]).”

    2. (b)

Formulaic representation: $S(M(T,12CD))$, with $S$ denoting the scene, which
in this scenario, pertains to the user’s ID and corresponding social media
posts.

  4. 4.

Hybrid Prompting: A synthesis of both role and scene definitions, offering an
integrative instruction.

    1. (a)

English translation: ”With the given user ID and their respective social media
posts, and adopting the role of a psychologist fortified with psychological
expertise, please execute a multi-classification task to verify the inclusion
of any of the 12 cognitive distortions ([list of cognitive distortions]).”

    2. (b)

Formulaic representation: $S+R(M(T,12CD))$, intertwining the scene context
($S$) with the role-definition ($R$).

#### LLM Few-shot Prompting

In this segment, few-shot prompting is construed as the provision of prior
knowledge or a batch of $n$ training instances to LLMs, thereby enabling them
to internalize this information and adeptly execute the stipulated task. This
methodology unfolds as:

  1. 1.

Background Knowledge: The model is furnished with psychological definitions
supplemented by emblematic cases, followed by one of the four prompting
strategies devised from zero-shot prompting. Prompts that integrate background
knowledge and employ the hybrid strategy from zero-shot prompting are detailed
as follows:

    1. (a)

English translation: ”Given the definitions of cognitive distortions denoted
by $D$ and the prototypical cases represented by $C$, and in light of the
supplied user ID and associated social media posts, you are assumed to be a
psychological expert well-versed in the aforementioned definitions and cases.
Drawing from this backdrop, please conduct a multi-classification task to
evaluate the correlation with any of the 12 cognitive distortions ([list of
cognitive distortions]).”

    2. (b)

Formulaic representation: $D+C+S+R(M(T,12CD))$, where $D$ encapsulates the
background definition, and $C$ signifies the prototypical instances from
academic literature, $S$ represents scene-definition and $R$ stands for role-
definition.

  2. 2.

Training with $n$ Samples per Category: In this approach, $n$ training
instances are randomly selected for each category to train the LLM, followed
by one of the four prompting strategies designed from zero-shot prompting.
These instances are represented as ${train}_{n}$ in the following tables.
Prompts that incorporate the training instances employ the hybrid strategy
from zero-shot prompting are detailed as follows:

    1. (a)

English translation: ”You are provided with learning samples denoted by $T$.In
light of the supplied user ID and associated social media posts, and assuming
your role as a psychologist with the relevant expertise.Drawing from this
backdrop, please conduct a multi-classification task to evaluate the
correlation with any of the 12 cognitive distortions ([list of cognitive
distortions]).”

    2. (b)

Formulaic representation: $T+S+R(M(T,12CD))$, integrating $T$ as the training
set with the scene-definition ($S$) and role-definition ($R$).

  3. 3.

Background knowledge and training with $n$ samples per category: This approach
investigates whether enhancing sample diversity in few-shot prompting augments
the LLM’s comprehension of psychological health tasks. It incorporates
psychological definitions, symbolic examples, and provides $n$ training
instances per category for LLM training. A command is subsequently issued
using a previously described few-shot prompting strategy. The following
example integrates background knowledge and training instances, and poses a
query using the hybrid strategy from zero-shot prompting:

    1. (a)

English translation: ”Given the definitions of cognitive distortions
represented by $D$ and the prototypical cases denoted by $C$, you are also
provided with learning samples represented by $T$. Assuming your expertise as
a psychological expert familiar with the aforementioned definitions and cases,
and in consideration of the supplied user ID and associated social media
posts, please conduct a multi-classification task to evaluate the correlation
with any of the 12 cognitive distortions ([list of cognitive distortions]).”

    2. (b)

Formulaic representation: $D+C+T+S+R(M(T,12CD))$, where $D$ encapsulates the
background definition, $C$ signifies the prototypical instances from academic
literature, $T$ is integrated as the training set, $S$ denotes the scene-
definition, and $R$ represents the role-definition.

#### LLM Fine-tuning

Fine-tuning represents a potent paradigm provided by OpenAI, enabling users to
optimize the aptitude of pre-trained models, such as GPT-3.5. While GPT-3.5 is
inherently trained on an expansive text corpus, the fine-tuning process
sharpens its proficiency for specialized tasks by exposing it to additional
task-specific instances. Details can be seen in the official document:
https://platform.openai.com/docs/guides/fine-tuning. Following the fine-
tuning, our evaluation paradigm retained the role, scene and hybrid
definitions from the zero-shot prompting for consistency and comparative
assessment:

  1. 1.

Role-definition Prompting: Post fine-tuning with relevant training samples, we
employed the prompt delineated in the role-definition section (refer to
Section 3.2).

  2. 2.

Scene-definition Prompting: Analogously, after the fine-tuning process, we
reverted to the prompt illustrated in the scene-definition segment of the
zero-shot prompting.

  3. 3.

Hybrid Prompting: Similarly, after the fine-tuning process, we adopted the
prompt presented in the hybrid strategy segment of the zero-shot prompting.

##  4 Experiments and results

###  4.1 Datasets and Evaluation Metrics

We undertook two psychology-related classification tasks: suicide risk and
cognitive distortion. The suicide risk task primarily differentiates between
high and low suicide risks, while the cognitive distortion task focuses on
classifications defined by Burns [44]. We sourced our data by crawling
comments from the “Zoufan”111https://www.weibo.com/xiaofan116?is_all=1 blog
within the Weibo social platform. Subsequently, a team of qualified
psychologists were enlisted to annotate the data. Given that this data is
publicly accessible, there are no concerns related to privacy breaches.

For the suicide detection data, there were 645 records with low suicide risk
and 601 records with high suicide risk. The dataset for cognitive distortion
consists of a total of 910 entries. The classification labels employed for
this data are as follows: all-or-nothing thinking, over-generalization, mental
filter, disqualifying the positive, mind reading, the fortune teller error,
magnification, emotional reasoning, should statements, labeling and
mislabeling, blaming oneself and blaming others. For both sets of data, the
training set and test set are divided according to the ratio of 4:1. We
utilize three evaluation metrics to measure the performance of different
algorithms for our two tasks: precision, recall, and $F_{1}$ score. Precision
is the ratio of correctly predicted positive observations to the total
predicted positives and recall (or Sensitivity) represents the ratio of
correctly predicted positive observations to all the actual positives. These
two metrics provide a comprehensive view of the algorithm’s performance in
terms of its positive predictions. The $F_{1}$ score offers a more holistic
view of the model’s performance, especially when the distribution of the true
positive and true negative rates is uneven.

###  4.2 Experiment design

Our experimental methodology is both hierarchical and greedy. Using cognitive
distortions as an example to show our points, our evaluations spanned several
dimensions:

  * •

Prompt Design Perspective: Initially, we assessed four prompting strategies
within the zero-shot learning framework. Subsequently, based on their
performance metrics, the top two strategies were selected for further
evaluation in the few-shot learning setting across various LLMs.

  * •

LLM Performance Perspective: Across all zero-shot prompts, ChatGLM2-6B’s
performance was found to be lacking, resulting in our decision to omit it from
subsequent few-shot prompting experiments. For GPT-3.5, its token limitation
prevented us from entering five samples for each category during few-shot
prompting. Consequently, we reserved the train5 approach exclusively for
GPT-4.

  * •

Fine-tuning Perspective: A discernible performance chasm exists between
GPT-3.5 and GPT-4. However, OpenAI’s recent introduction of fine-tuning
capabilities for GPT-3.5 and reports from official channels suggest that,
under specific conditions, GPT-3.5 might outperform GPT-4 post fine-tuning.
Consequently, our attention was centered on the fine-tuning of GPT-3.5.
Regrettably, the current iteration of GPT-4 lacks fine-tuning functionalities,
curtailing our capacity to assess its potential in this dimension.

The detailed experimental setup is as follows:

  * •

LSAN: We used word2vec to train 300-dimensional embeddings for both document
and randomly-initialized label texts. The attention mechanism helped us
compute word contributions to labels and create label-specific document
representations. Dot products between these document and label vectors refined
these relationships further. These two types of document representations were
then fused using weighted combinations. For predictions, we employed a fully
connected layer, followed by RELU and a sigmoid function. Losses were
calculated using a cross-entropy function during training.

  * •

BERT: We employ BERT to extract 768-dimensional vectors from Chinese
sentences. To mitigate overfitting, a dropout function is applied to these
sentence vectors. Subsequently, a fully connected layer is introduced to
independently classify suicide risk and cognitive distortions. The sigmoid
function serves as the activation function for the output layer. Both the BERT
layer and the fully connected layer are trained simultaneously.

  * •

LLM-zero shot: Both GPT-3.5 and GPT-4 are closed-source and available through
API provided by OpenAI. We picked the gpt-3.5-turbo, one of the most capable
and cost-effective models in the GPT-3.5 family, and the gpt-4, more capable
than any GPT-3.5 model, able to do more complex tasks, and optimized for chat.
As for the GLM models, we employed the smaller, open-source variant,
ChatGLM2-6B, suitable for deployment on consumer-grade hardware. Given the
extensive parameter count of GLM-130B, it posed deployment challenges due to
its elevated operational costs. Furthermore, its API lacked the capability to
handle cognitive distortion multi-label classification task, leading us to
conduct tests via its official website. Acknowledging the inherent variability
in LLM’s outputs, our experimental design involved averaging the outcomes over
five runs. For GPT-3.5, GPT-4, and ChatGLM2-6B, we adjusted the temperature to
values of 0.1, 0.3, 0.5, 0.7, and 0.9, conducting experiments at each setting.
Given the absence of a temperature setting for GLM-130B on its platform, we
simply executed five repeated runs and computed the mean performance. For
zero-shot evaluations, we initiated performance validation on the basic
strategy across the LLMs, subsequently examining the efficacy of role-
definition, scene-definition, and hybrid strategies, aiming to discern the
influence of domain-specific information on LLM’s performance.

  * •

LLM-few shot: We conducted a assessment using the top two performing prompt
strategies from the zero-shot tests, determined by their F1-scores.The impact
on performance was assessed when augmenting these strategies with
$background$, $train_{n}$ , and their combination ($background+train_{n}$).
Specifically, $background$ denotes the incorporation of prior knowledge,
$train_{n}$ represents the addition of training samples, where $n$ is the
number of positive samples chosen for each category. $background+train_{n}$
suggests simultaneous enrichment with prior knowledge and training samples.
Given the varying token input constraints among different models, the sample
size selected for each model differed. In addition, we also experimented with
the integration of basic, role, scene, and hybrid strategies in the zero-shot
prompting scenario.

  * •

LLM-fine-tunning: We fine-tuned the GPT-3.5 Turbo model for predicting suicide
risk and cognitive distortions using the API interface provided by OpenAI. We
utilized three types of prompts: role-based, scene-based, and hybrid
strategies.

##  5 Results

In our study, we focused on two specific tasks: suicide classification and
multi-label classification of cognitive distortions. And the results can be
seen in Table 1 and Table 2 respectively. Our analysis examined these two
tasks in Section 5.1 and Section 5.2 respectively from three distinct aspects:
training strategy, the construction of prompt, and a comparative evaluation
across various LLMs. Ultimately, we assessed and compared the model’s
performance on these two psychological tasks to draw conclusions in Section
5.3.

###  5.1 Suicide Risk

#### Training strategies

In our training strategy comparison, we observed varying degrees of
effectiveness across different models. The LSAN model showed a modest
improvement over the pre-trained BERT model, with a 1.59% F1-score increase in
performance; however, this difference was not statistically significant. In
contrast, fine-tuning GPT-3.5 led to a substantial performance gain, achieving
an F1-score of 78.44%. This represented a notable 11.42% improvement in
F1-score when compared to its base model (fine-tuning hybrid vs.zero-shot
hybrid), bringing its performance closer to that of supervised learning
models.

#### Design of prompts

Our investigation into prompt design for large language models revealed
nuanced outcomes across different strategies and models. In the realm of zero-
shot prompts, we found that while a mixed strategy yielded satisfactory
results, the performance differences among various types of prompts were not
statistically significant. For few-shot prompts, adding more data did not
consistently improve performance; this was evident in the ChatGLM2-6B model
where additional data sometimes reduced effectiveness. Conversely, GPT-4’s
performance remained stable irrespective of the data size. Notably, the
background+trainn+hybrid strategy emerged as the most effective across
multiple models.

We also studied the impact of extra training data in few-shot scenarios and
observed that using role-define and trainn+role-define prompts often led to
diminished performance. The role of background knowledge was model-dependent;
in smaller models like ChatGLM2-6B, incorporating background knowledge led to
a performance increase from 53.75% to 64.41%. However, this could not be
universally verified due to token limitations. Finally, our comparison between
few-shot and zero-shot prompts showed that few-shot prompts did not
significantly outperform their zero-shot counterparts.

#### Comparison of LLMs

In our comparative analysis of large language models, we observed several
trends that highlight the complexities of model performance. Generally, GPT-4
outperformed GPT-3.5, and GLM-130B excelled over ChatGLM2-6B, suggesting the
benefits of larger model architectures and more extensive training data.
However, this narrative was disrupted when GPT-3.5 was fine-tuned, as it
surpassed GPT-4 by a margin of 2.63%. Additionally, GLM-130B demonstrated a
performance comparable to GPT-4 and superior to GPT-3.5 for the specific task
under study. These findings indicate that while larger models typically offer
advantages, fine-tuning and task-specific capabilities can alter the
performance landscape significantly.

###  5.2 Cognitive Distortion

#### Training strategies

Our investigation into training strategies for large language models revealed
nuanced performance outcomes. Initially, the pre-trained BERT model
demonstrated a 2.83% performance advantage over LSAN trained from scratch.
However, this difference was not statistically significant, implying that the
observed discrepancy may not be meaningful. On the other hand, fine-tuning
GPT-3.5 surprisingly led to a decrease in performance rather than the
anticipated improvement. This underscores the complexity of model training and
the need for careful consideration when implementing fine-tuning strategies.

#### Design of prompts

In the design of prompts for large language models, our study examined the
performance of both zero-shot and few-shot prompts. For zero-shot prompts, we
found that a meticulous design focusing on scene and role settings is crucial;
otherwise, a basic task-oriented prompt is generally more effective. In the
realm of few-shot prompts, we observed that prompts providing specific data
points outperformed those that simply offered background knowledge.
Interestingly, increasing the training data in these prompts did not lead to
better performance. A comparative analysis revealed that although few-shot
prompts outperformed zero-shot prompts, they still fell short of fully meeting
the task requirements, as evidenced by GPT-4’s F1-score of approximately 30%.

#### Comparison of LLMs

Consistently, larger models like GPT-4 outperformed their smaller counterparts
such as GPT-3.5. When it came to the complex tasks in our study, The
performance of ChatGLM2-6B was insufficient for handling complex tasks, while
GLM-130B fared better but was still outdone by GPT-4. Given that our dataset
consists of comments from social networks, the text is generally concise. As a
result, token length did not substantially affect the performance of the
models in our tasks. Rather, the selection of representative data for prompt
construction emerged as a more crucial factor than merely increasing the
number of tokens.

###  5.3 Cross-Task Comparison

As task complexity increased from binary to multi-label classification, large
language models did not sustain their performance. In contrast, supervised
learning models maintained a relatively stable F1-score close to 80% across
both types of tasks. This highlights the limitations of large language models
in replacing supervised learning for specialized tasks. While fine-tuning may
benefit simpler tasks, it does not adequately address the challenges posed by
complex tasks, calling for further investigation into fine-tuning mechanisms
for large language models.

Table 1: Result for suicide binary classification task

Model category | Model name | Type | Sub-type | Train data | Test data | Precision | Recall | F1-score
---|---|---|---|---|---|---|---|---
Supervised learning | LSAN | train from scarch | - | 999 | 250 | 81.86% | 82.29% | 82.07%
BERT | fine-tuning | - | 999 | 80.16% | 80.80% | 80.48%
LLM | ChatGLM2-6B | zero-shot | basic | 0 | 69.07% | 37.10% | 48.07%
role-define | 0 | 65.77% | 35.81% | 46.15%
scene-define | 0 | 64.52% | 45.16% | 53.01%
hybrid | 0 | 65.68% | 46.13% | 53.74%
few-shot | background+scene-define | 0 | 58.56% | 47.26% | 51.45%
background+hybrid | 0 | 60.37% | 70.64% | 64.41%
train12+scene-define | 24 | 67.19% | 59.52% | 63.04%
train12+hybrid | 24 | 64.29% | 49.20% | 55.56%
background+train12+scene-define | 24 | 49.74% | 56.61% | 52.70%
background+train12+hybrid | 24 | 58.91% | 73.23% | 64.78%
train30+scene-define | 60 | 57.71% | 26.77% | 36.14%
train30+hybrid | 60 | 50.60% | 24.84% | 32.60%
background+train30+scene-define | 60 | 62.97% | 52.90% | 57.02%
background+train30+hybrid | 60 | 60.14% | 47.10% | 51.88%
GLM-130B | zero-shot | basic | 0 | 54.58% | 95.81% | 69.52%
role-define | 0 | 55.51% | 94.84% | 70.02%
scene-define | 0 | 55.05% | 93.87% | 69.40%
hybrid | 0 | 57.37% | 97.42% | 72.20%
few-shot | background+role-define | 0 | 56.55% | 90.32% | 69.55%
background+hybrid | 0 | 56.91% | 92.42% | 70.43%
train12+role-define | 24 | 53.18% | 83.23% | 64.89%
train12+hybrid | 24 | 55.30% | 88.39% | 68.02%
background+train12+role-define | 24 | 57.84% | 83.38% | 68.30%
background+train12+hybrid | 24 | 60.88% | 90.00% | 72.61%
GPT-3.5 | zero-shot | basic | 0 | 52.00% | 88.23% | 65.42%
role-define | 0 | 53.31% | 96.13% | 68.59%
scene-define | 0 | 52.16% | 89.03% | 65.76%
hybrid | 0 | 52.55% | 92.26% | 66.95%
few-shot | background+role-define | 0 | 54.90% | 88.55% | 67.76%
background+hybrid | 0 | 55.27% | 89.03% | 68.19%
train12+role-define | 0 | 56.34% | 83.39% | 67.22%
train12+hybrid | 0 | 57.19% | 84.68% | 68.27%
background+train12+role-define | 0 | 59.37% | 81.61% | 68.71%
background+train12+hybrid | 24 | 58.26% | 82.90% | 68.41%
fine-tuning | role-define | 999 | 84.76% | 71.77% | 77.29%
scene-define | 999 |  | 63.20% | 83.15% | 72.14%
hybrid | 999 |  | 84.25% | 73.38% | 78.44%
GPT-4 | zero-shot | basic | 0 |  | 57.43% | 95.48% | 71.72%
role-define | 0 |  | 57.29% | 97.26% | 72.10%
scene-define | 0 |  | 58.81% | 97.58% | 73.39%
hybrid | 0 |  | 57.47% | 97.42% | 72.30%
few-shot | background+scene-define | 0 |  | 64.91% | 73.55% | 68.86%
background+hybrid | 0 |  | 63.24% | 84.03% | 72.05%
train12+scene-define | 24 |  | 60.70% | 94.35% | 73.87%
train12+hybrid | 24 |  | 59.77% | 84.19% | 69.87%
background+train12+scene-define | 24 |  | 65.44% | 81.77% | 72.63%
background+train12+hybrid | 24 |  | 65.65% | 78.87% | 71.60%
train30+scene-define | 60 |  | 61.11% | 92.42% | 73.56%
train30+hybrid | 60 |  | 60.79% | 89.03% | 72.22%
background+train30+scene-define | 60 |  | 63.86% | 83.06% | 72.16%
background+train30+hybrid | 60 |  | 70.16% | 82.58% | 75.81%

Table 2: Result for cognitive distortion multi-label classification task.

Model category | Model name | Type | Sub-type | Train data | Test data | Precision | Recall | F1-score
---|---|---|---|---|---|---|---|---
Supervised learning | LSAN | train from scratch | - | 728 | 182 | 76.79% | 77.95% | 76.08%
BERT | fine-tuning | - | 728 | 79.85% | 80.49% | 78.91%
LLM | GLM-130B | zero-shot | basic | 0 | 9.56% | 57.39% | 16.39%
role-define | 0 | 9.00% | 65.31% | 15.78%
scene-define | 0 | 8.75% | 60.43% | 15.19%
hybrid | 0 | 9.93% | 67.83% | 17.31%
few-shot | train1+basic | 12 | 7.59% | 39.31% | 12.69%
train1+hybrid | 12 | 8.18% | 44.09% | 13.73%
GPT-3.5 | zero-shot | basic | 0 | 10.25% | 4.87% | 6.49%
role-define | 0 | 12.58% | 4.35% | 6.18%
scene-define | 0 | 9.2% | 3.91% | 5.34%
hybrid | 0 | 8.61% | 5.39% | 6.38%
few-shot | background+hybrid | 0 | 2.39% | 7.91% | 11.63%
background+basic | 0 | 24.21% | 10.09% | 14.06%
| train2+hybrid | 24 | 13.32% | 10.09% | 11.46%
| train2+basic | 24 | 12.51% | 10.00% | 11.10%
fine-tuning | role-define | 728 | 10.79% | 8.26% | 6.78%
| scene-define | 728 | 13.17% | 10.43% | 7.88%
| hybird | 728 | 10.79% | 8.26% | 6.27%
GPT-4 | zero-shot | basic | 0 | 16.46% | 46.09% | 24.18%
role-define | 0 | 16.69% | 42.09% | 23.86%
scene-define | 0 | 18.12% | 43.22% | 25.43%
hybrid | 0 | 16.26% | 38.87% | 22.84%
few-shot | background+basic | 0 | 21.96% | 31.04% | 25.54%
background+scene-define | 0 | 22.89% | 34.18% | 26.59%
train2+basic | 24 | 31.25% | 34.17% | 32.47%
train2+scene-define | 24 | 27.00% | 27.74% | 27.06%
|  | background+train2+basic | 24 | 24.35% | 32.00% | 27.63%
|  | background+train2+scene-define | 24 | 24.39% | 28.09% | 25.94%
|  | train5+basic | 60 |  | 25.62% | 35.65% | 29.57%
|  |  | train5+scene-define | 60 |  | 29.46% | 34.61% | 31.57%

##  6 Discussion

Our study systematically evaluated the effectiveness of large language models
(LLMs) across two mental health related tasks on Chinese social media: suicide
risk classification and cognitive distortion multi-label classification. Our
results also reveal the nuanced role of prompt design. While the ’hybrid’
prompt performed well in zero-shot settings, the benefits of increasing data
in few-shot prompts were not universally beneficial. For more straightforward
tasks, adding background knowledge appeared to help smaller models
(ChatGLM2-6B), but its utility diminished in more complex models or tasks.
This calls for a more customized approach to prompt engineering tailored to
the specific task and the size of the model being used. If high-quality data
is unavailable or prompt design proves challenging, allowing a LLM to directly
handle the task may still yield acceptable performance. Larger language models
like GPT-4 and GLM-130B generally outperform smaller variants such as GPT-3.5
and ChatGLM2-6B. However, it’s important to note that these large models are
not always competent at handling complex tasks and should not be seen as
replacements for supervised learning algorithms. For simpler tasks, such as
the suicide risk classification task examined in our study, the performance of
LLMs is satisfactory. Interestingly, after fine-tuning, GPT-3.5 even
outperforms GPT-4, achieving results that are nearly on par with those
obtained through supervised learning methods. While there is often a
preference for large input limits in large language models (LLMs), it’s
crucial to tailor these settings to the specific task at hand. For tasks
involving shorter text, such as our study on sentiment analysis of social
network data, the long input capability of an LLM may not be a primary
concern. Our experiments indicate that extending the input data to construct
few-shot prompts does not necessarily lead to improved performance. Therefore,
it is important to carefully consider the nature of the task when configuring
the input parameters of an LLM.

Our study does have some limitations. For instance, due to token constraints,
we were unable to conduct certain tests—particularly those involving smaller
models supplemented with background knowledge—across all tasks. Looking ahead,
we plan to conduct more comprehensive studies that encompass a wider variety
of tasks and models. This will allow us to draw more definitive conclusions
regarding the comparative effectiveness of large language models and
supervised learning algorithms. Additionally, the fine-tuning mechanisms of
LLMs warrant further exploration, particularly for more efficient handling of
complex tasks. The development of advanced prompt engineering techniques could
also help optimize the performance of LLMs across various tasks.

##  7 Conclusion

In this study, we evaluated the performance of multiple large language models
(LLMs) in two psychology-related tasks and compared their efficacy with that
of supervised learning algorithms. Although LLMs show promise in various
natural language processing applications, they are not yet a comprehensive
substitute for supervised learning, particularly in domain-specific tasks.
Fine-tuning LLMs can enhance performance on simpler tasks but is less
effective for more complex challenges. The success of different training
strategies and prompt designs is highly contingent on both the task and the
size of the model, underscoring the necessity for task-specific customization.
In summary, our research suggests that while LLMs offer considerable
potential, significant work remains to make them universally effective across
a broad array of complex tasks.

##  8 Acknowledgments

This work was supported by grants from the National Natural Science Foundation
of China (grant numbers:72174152, 72304212 and 82071546), Fundamental Research
Funds for the Central Universities (grant numbers: 2042022kf1218;
2042022kf1037), the Young Top-notch Talent Cultivation Program of Hubei
Province. Guanghui Fu is supported by a Chinese Government Scholarship
provided by the China Scholarship Council (CSC).

##  9 Competing interests

All authors declare no financial or non-financial competing interests.

## References

  * [1] World health organization, “Depressive disorder (depression),” (2023).
  * [2] Huang, Y., Wang, Y., Wang, H., Liu, Z., Yu, X., Yan, J., Yu, Y., Kou, C., Xu, X., Lu, J., et al., “Prevalence of mental disorders in China: a cross-sectional epidemiological study,” The Lancet Psychiatry 6(3), 211–224 (2019).
  * [3] World health organization, “Suicide,” (2023).
  * [4] Keles, B., McCrae, N., and Grealish, A., “A systematic review: the influence of social media on depression, anxiety and psychological distress in adolescents,” International journal of adolescence and youth 25(1), 79–93 (2020).
  * [5] Robinson, J., Cox, G., Bailey, E., Hetrick, S., Rodrigues, M., Fisher, S., and Herrman, H., “Social media and suicide prevention: a systematic review,” Early intervention in psychiatry 10(2), 103–121 (2016).
  * [6] Luxton, D. D., June, J. D., and Fairall, J. M., “Social media and suicide: a public health perspective,” American journal of public health 102(S2), S195–S200 (2012).
  * [7] Coppersmith, G., Leary, R., Crutchley, P., and Fine, A., “Natural language processing of social media as screening for suicide risk,” Biomedical informatics insights 10, 1178222618792860 (2018).
  * [8] Nandwani, P. and Verma, R., “A review on sentiment analysis and emotion detection from text,” Social Network Analysis and Mining 11(1), 81 (2021).
  * [9] Acheampong, F. A., Wenyu, C., and Nunoo-Mensah, H., “Text-based emotion detection: Advances, challenges, and opportunities,” Engineering Reports 2(7), e12189 (2020).
  * [10] Saunders, D., “Domain adaptation and multi-domain adaptation for neural machine translation: A survey,” Journal of Artificial Intelligence Research 75, 351–424 (2022).
  * [11] Laparra, E., Bethard, S., and Miller, T. A., “Rethinking domain adaptation for machine learning over clinical language,” JAMIA open 3(2), 146–150 (2020).
  * [12] Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al., “A survey of large language models,” arXiv preprint arXiv:2303.18223 (2023).
  * [13] Xu, X., Yao, B., Dong, Y., Yu, H., Hendler, J., Dey, A. K., and Wang, D., “Leveraging large language models for mental health prediction via online text data,” arXiv preprint arXiv:2307.14385 (2023).
  * [14] Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., and Ting, D. S. W., “Large language models in medicine,” Nature Medicine , 1–11 (2023).
  * [15] Le Glaz, A., Haralambous, Y., Kim-Dufor, D.-H., Lenca, P., Billot, R., Ryan, T. C., Marsh, J., Devylder, J., Walter, M., Berrouiguet, S., et al., “Machine learning and natural language processing in mental health: systematic review,” Journal of Medical Internet Research 23(5), e15708 (2021).
  * [16] Fu, G., Song, C., Li, J., Ma, Y., Chen, P., Wang, R., Yang, B. X., and Huang, Z., “Distant supervision for mental health management in social media: suicide risk classification system development study,” Journal of medical internet research 23(8), e26119 (2021).
  * [17] Singh, M., Jakhar, A. K., and Pandey, S., “Sentiment analysis on the impact of coronavirus in social life using the bert model,” Social Network Analysis and Mining 11(1), 33 (2021).
  * [18] Wan, F., “Sentiment analysis of weibo comments based on deep neural network,” in [2019 international conference on communications, information system and computer engineering (CISCE) ], 626–630, IEEE (2019).
  * [19] Zhang, X., Li, W., Ying, H., Li, F., Tang, S., and Lu, S., “Emotion detection in online social networks: a multilabel learning approach,” IEEE Internet of Things Journal 7(9), 8133–8143 (2020).
  * [20] Wang, J., Zhou, Y., Zhang, W., Evans, R., and Zhu, C., “Concerns expressed by chinese social media users during the COVID-19 pandemic: content analysis of sina weibo microblogging data,” Journal of medical Internet research 22(11), e22152 (2020).
  * [21] Kaddour, J., Harris, J., Mozes, M., Bradley, H., Raileanu, R., and McHardy, R., “Challenges and applications of large language models,” arXiv preprint arXiv:2307.10169 (2023).
  * [22] Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al., “Emergent abilities of large language models,” arXiv preprint arXiv:2206.07682 (2022).
  * [23] Liebrenz, M., Schleifer, R., Buadze, A., Bhugra, D., and Smith, A., “Generating scholarly content with ChatGPT: ethical challenges for medical publishing,” The Lancet Digital Health 5(3), e105–e106 (2023).
  * [24] Jeblick, K., Schachtner, B., Dexl, J., Mittermeier, A., Stüber, A. T., Topalis, J., Weber, T., Wesp, P., Sabel, B., Ricke, J., et al., “ChatGPT makes medicine easy to swallow: An exploratory case study on simplified radiology reports,” arXiv preprint arXiv:2212.14882 (2022).
  * [25] Surameery, N. M. S. and Shakor, M. Y., “Use ChatGPT to solve programming bugs,” International Journal of Information Technology & Computer Engineering (IJITC) ISSN: 2455-5290 3(01), 17–22 (2023).
  * [26] Kasneci, E., Seßler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al., “Chatgpt for good? on opportunities and challenges of large language models for education,” Learning and individual differences 103, 102274 (2023).
  * [27] Yeo, Y. H., Samaan, J. S., Ng, W. H., Ting, P.-S., Trivedi, H., Vipani, A., Ayoub, W., Yang, J. D., Liran, O., Spiegel, B., et al., “Assessing the performance of chatgpt in answering questions regarding cirrhosis and hepatocellular carcinoma,” medRxiv , 2023–02 (2023).
  * [28] Jiang, L. Y., Liu, X. C., Nejatian, N. P., Nasir-Moin, M., Wang, D., Abidin, A., Eaton, K., Riina, H. A., Laufer, I., Punjabi, P., et al., “Health system-scale language models are all-purpose prediction engines,” Nature , 1–6 (2023).
  * [29] Farhat, F., “ChatGPT as a complementary mental health resource: a boon or a bane,” Annals of Biomedical Engineering , 1–4 (2023).
  * [30] Qin, W., Chen, Z., Wang, L., Lan, Y., Ren, W., and Hong, R., “Read, diagnose and chat: Towards explainable and interactive LLMs-augmented depression detection in social media,” arXiv preprint arXiv:2305.05138 (2023).
  * [31] Chen, S., Wu, M., Zhu, K. Q., Lan, K., Zhang, Z., and Cui, L., “LLM-empowered chatbots for psychiatrist and patient simulation: Application and evaluation,” arXiv preprint arXiv:2305.13614 (2023).
  * [32] Fu, G., Zhao, Q., Li, J., Luo, D., Song, C., Zhai, W., Liu, S., Wang, F., Wang, Y., Cheng, L., et al., “Enhancing psychological counseling with large language model: A multifaceted decision-support system for non-professionals,” arXiv preprint arXiv:2308.15192 (2023).
  * [33] Ayers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al., “Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum,” JAMA internal medicine (2023).
  * [34] Yang, K., Ji, S., Zhang, T., Xie, Q., Kuang, Z., and Ananiadou, S., “Towards interpretable mental health analysis with ChatGPT,” (2023).
  * [35] Vaishya, R., Misra, A., and Vaish, A., “ChatGPT: Is this version good for healthcare and research?,” Diabetes & Metabolic Syndrome: Clinical Research & Reviews 17(4), 102744 (2023).
  * [36] Editorial, “Will ChatGPT transform healthcare?,” Nature Medicine , 505–506 (2023).
  * [37] Chang, Y., Wang, X., Wang, J., Wu, Y., Zhu, K., Chen, H., Yang, L., Yi, X., Wang, C., Wang, Y., et al., “A survey on evaluation of large language models,” arXiv preprint arXiv:2307.03109 (2023).
  * [38] Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K., “BERT: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805 (2018).
  * [39] Xiao, L., Huang, X., Chen, B., and Jing, L., “Label-specific document representation for multi-label text classification,” in [Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP) ], 466–475 (2019).
  * [40] Zeng, A., Liu, X., Du, Z., Wang, Z., Lai, H., Ding, M., Yang, Z., Xu, Y., Zheng, W., Xia, X., et al., “GLM-130b: An open bilingual pre-trained model,” arXiv preprint arXiv:2210.02414 (2022).
  * [41] OpenAI, “Introducing chatgpt,” (2023).
  * [42] OpenAI, “Gpt-4 technical report,” (2023).
  * [43] Wang, S., Sun, Y., Xiang, Y., Wu, Z., Ding, S., Gong, W., Feng, S., Shang, J., Zhao, Y., Pang, C., et al., “Ernie 3.0 titan: Exploring larger-scale knowledge enhanced pre-training for language understanding and generation,” arXiv preprint arXiv:2112.12731 (2021).
  * [44] Burns, D. D., [Feeling good ], Signet Book (1981).


Bayesian Screening: Multi-test Bayesian Optimization Applied to in silico
Material Screening James Hook, Calum Hand, Emma Whitfield

###### Abstract

We present new multi-test Bayesian optimization models and algorithms for use
in large scale material screening applications. Our screening problems are
designed around two tests, one expensive and one cheap. This paper differs
from other recent work on multi-test Bayesian optimization through use of a
flexible model that allows for complex, non-linear relationships between the
cheap and expensive test scores. This additional modeling flexibility is
essential in the material screening applications which we describe. We
demonstrate the power of our new algorithms on a family of synthetic toy
problems as well as on real data from two large scale screening studies.

##  1 Introduction

In a material (or chemical) screen, the aim is to select from a large number
of candidate materials the material or set of materials that perform best in
some particular test. This test could be a laboratory experiment or as in our
case a computer simulation. Bayesian optimization has previously been
suggested as a way to improve the efficiency of such screens, dramatically
reducing the number of tests necessary to find the best performing materials.
See for example [5, 7, 12]. Bayesian optimization is a family of methods for
optimizing the output of a black-box function, which work by modeling the
input-output relationship of the function as a random variable. Applied to a
materials screen, Bayesian optimization works by iteratively selecting new
materials for testing whilst simultaneously using the data generated by the
previous tests to model the relationship between known properties of the
candidate materials and their performance in the test. The model’s predictions
are then used to guide further experimentation.

There has been recent interest in developing Bayesian optimization methods
that can efficiently choose between a range of tests with different
cost/accuracy trade offs (in _in silico_ material screening the cost is
computional time). See [15] for an introduction. Application examples include
tuning the hyperparameters for deep learning models, where a cheap but less
accurate test can be made by training on a smaller data set or with fewer
iterations [18]. Another proposed application is testing a robot control
strategy either with a real life experiment or a computer simulation [9].

Chemical engineers conducting screening studies have access to a huge number
of possible tests. These range from calculating basic features of the
candidate materials, which can be done in factions of a second, to detailed
real life experiments, which might takes weeks or months to prepare. Screening
studies typically work by applying a cheaper test to all of the candidates and
then selecting some top fraction of performers to move onto a second round and
so on. Utilizing high performance computing, this approach allows for high-
throughput _in silico_ screening of huge databases of materials [2]. For
example the authors of [14] apply three rounds of increasingly accurate but
expensive computer simulated experiments to find the most promising candidate
materials in a database of 137,953 structures. This method is also widely used
in screens where all of the test are real life experiments, including in
industry [8]. In practise this approach has been shown to work well in many
applications but the choice of which tests to apply and how many candidates to
pass through each round tends to be carried out in an _ad hoc_ fashion without
the use of available statistical and AI techniques.

In this paper we present new algorithms for multi-test Bayesian optimization,
with a novel objective and statistical model for application in large scale
materials screening problems.

##  2 Models for screening and data

There are $n$ candidates which we index by $1,2,\dots,n$. Each candidate has a
vector of features $\bm{x}_{i}\in\mathbb{R}^{d}$, which is visible to the
decision making algorithm throughout the process. Additionally each candidate
has a cheap test score $y_{i}^{(C)}\in\mathbb{R}$ and an expensive test score
$y_{i}^{(E)}\in\mathbb{R}$, which are initially hidden and only become
available after they are tested for. Applying the cheap test costs $c_{C}$ and
applying the expensive test costs $c_{E}$. We consider two objectives for the
screening problem as follows.

Optimization: Our aim is to use a fixed testing budget $B$ to find the
candidate with the highest expensive test score that we can.

Mining: Our aim is to use a fixed testing budget $B$ to find as many of the
candidates with expensive test scores in the top $N$ as we can.

Previous work applying Bayesian optimization to material screening has focused
on the optimization objective. However, when searching very large computer
databases of candidate materials, the chemical engineers that we have spoken
to say that they really want to find all of the top performing materials. This
can be for many reasons including that some materials in the database will be
difficult or impossible to synthesize. Therefore the mining objective is a
better model for what the chemical engineers actually want and targeting this
objective may result in better performance in practice. We also consider two
different models for the hidden test scores as follows.

Multi-Fidelity Testing Model: To model the scores, we consider a Gaussian
Process $\\{f(\bm{x})\\}_{\bm{x}\in\mathbb{R}^{d}}$, then define the cheap and
expensive test scores $\bm{y}^{(C)},\bm{y}^{(E)}\in\mathbb{R}^{n}$, by

$y^{(C)}_{i}=f(\bm{x}_{i})+\epsilon_{i}^{(C)},\quad
y^{(E)}_{i}=f(\bm{x}_{i})+\epsilon_{i}^{(E)},$ (1)

for $i=1,\dots,n$, where $(\epsilon_{i}^{(C)})_{i=1}^{n}$ are i.i.d.
$\mathcal{N}(0,\sigma_{C}^{2})$ and $(\epsilon_{i}^{(E)})_{i=1}^{n}$ are
i.i.d. $\mathcal{N}(0,\sigma_{E}^{2})$.

Covariate Testing Model: To model the scores, we consider two Gaussian Process
$\\{f(\bm{x})\\}_{\bm{x}\in\mathbb{R}^{d}}$ and
$\\{g(\bm{x},y)\\}_{\bm{x}\in\mathbb{R}^{d},y\in\mathbb{R}}$, then define the
cheap and expensive test scores $\bm{y}^{(C)},\bm{y}^{(E)}\in\mathbb{R}^{n}$,
by

$y^{(C)}_{i}=f(\bm{x}_{i})+\epsilon_{i}^{(C)},\quad
y^{(E)}_{i}=g(\bm{x}_{i},y_{i}^{(C)})+\epsilon_{i}^{(E)},$ (2)

for $i=1,\dots,n$, where $(\epsilon_{i}^{(C)})_{i=1}^{n}$ are i.i.d.
$\mathcal{N}(0,\sigma_{C}^{2})$ and $(\epsilon_{i}^{(E)})_{i=1}^{n}$ are
i.i.d. $\mathcal{N}(0,\sigma_{E}^{2})$.

Previous work on multi-test Bayesian optimization has focused on the multi-
fidelity model or on multi-dimensional Gaussian process models that also
result in linear relationships between the cheap and expensive test scores.
These models results in Gaussian posterior distributions which are easy to
work with but are only appropriate in screening problems where the cheap test
gives an unbiased estimate of the expensive test or where the cheap test and
expensive test are related in a fixed linear way. We are interested in
screening problems where the cheap test scores provides useful information
about the expensive test scores but via some initially unknown non-linear
relationship. In such problems we need the extra flexibility of the covariate
testing model. However this flexibility comes at the price of tractability as
the resulting posterior distributions are non-Gaussian which makes inference
more complicated and expensive.

###  2.1 Markov Decision Process (MDP) formulation

Assuming any of the four possible objective/model combinations gives rise to a
MDP model for the screening problem. See [16] Chapter 3 for an introduction.
The state space of the MDP is given by
$S=\big{(}(\mathbb{R}\cup\\{?\\})^{2}\big{)}^{n}$, where real values represent
scores that have been tested for and question marks represent so far unknown
test scores. For example, a state $s\in S$, with $s_{i}=(a,?)$, where
$a\in\mathbb{R}$, indicates that candidate $i$ has been tested with the cheap
test and scored $y_{i}^{(C)}=a$ but that candidate $i$ has not been tested
with the expensive test. For a state $s\in S$, define

$\begin{array}[]{c}\begin{array}[]{cc}I_{\hbox{uu}}=\\{i~{}:~{}s_{i}=(?,?)\\},&I_{\hbox{tu}}=\\{i~{}:~{}s_{i}\in\mathbb{R}\times\\{?\\}\\},\\\
I_{\hbox{ut}}=\\{i~{}:~{}s_{i}\in\\{?\\}\times\mathbb{R}\\},&I_{\hbox{tt}}=\\{i~{}:~{}s_{i}\in\mathbb{R}\times\mathbb{R}\\},\end{array}\\\
\begin{array}[]{c}D=\\{\bm{y}^{(C)}_{I_{\hbox{tu}}\cup
I_{\hbox{tt}}},\bm{y}^{(E)}_{I_{\hbox{ut}}\cup I_{\hbox{tt}}}\\},\\\
b=B-c_{C}(|I_{\hbox{tu}}|+|I_{\hbox{tt}}|)-c_{E}(|I_{\hbox{ut}}|+|I_{\hbox{tt}}|),\\\
\mathcal{A}=\\{(i,C)~{}:~{}i\in I_{\hbox{uu}}\cup
I_{\hbox{ut}}\\}\cup\\{(i,E)~{}:~{}i\in I_{\hbox{uu}}\cup I_{\hbox{tu}}\\}\\\
y_{\max}^{(E)}=\max_{i\in I_{\hbox{ut}}\cup
I_{\hbox{tt}}}y_{i}^{(E)}.\end{array}\end{array}$ (3)

From state $s$ the available actions, denoted by the set $\mathcal{A}$, are to
apply the cheap test or expensive test to any candidate that has not taken
that test so far. If there is not sufficient budget for any further tests then
the MDP is terminated. When an action is taken the MDP transitions to a new
state $s\mapsto s^{\prime}$ by revealing the score of the chosen test.
Assuming either of the statistical models for the test scores, this gives rise
to random transitions. For example if we choose action $(i,E)$, i.e. to test
candidate $i$ with the expensive test, then
$(s^{\prime}_{i})_{2}\sim\rho(y_{i}^{(E)}|D)$ and all other entries of
$s^{\prime}$ are equal to those of $s$ and where conditioning on $D$ mean
conditioning on all of the previously tested values for $\bm{y}^{(C)}$ and
$\bm{y}^{(E)}$.

When the MDP transitions $s\mapsto s^{\prime}$ a reward is obtained. The
rewards for applying the cheap test are all equal to zero. For the expensive
test under the optimization objective the reward is given by

$R\big{(}(i,E),s\big{)}=\max\left\\{\left(y_{i}^{(E)}-y_{\max}^{(E)}\right),0\right\\},$
(4)

i.e. the improvement in the running maximum, and under the mining objective
the reward is given by

$R\big{(}(i,E),s\big{)}=\left\\{\begin{array}[]{cc}1&\hbox{if $i\in$
top${}_{N}$,}\\\ 0&\hbox{otherwise,}\end{array}\right.$ (5)

where topN is the set of the $N$ top expensive test scoring candidates. Note
that the rewards signal will be hidden to the MDP agent in the case of the
mining objecting as the agent cannot know for sure which candidates are in
topN until it has tested all of them.

A policy is a (possibly random) mapping $\pi:S\mapsto\mathcal{A}$ from states
to actions that defined a screening strategy. Once we fix a policy $\pi$ the
MDP becomes a Markov chain

$s_{0}\overset{\pi(s_{0})}{\longmapsto}s_{1}\overset{\pi(s_{1})}{\longmapsto}s_{2}\overset{\pi(s_{2})}{\longmapsto}\cdots\overset{\pi(s_{T-1})}{\longmapsto}s_{T},$
(6)

where $s_{0}=\underline{?}$ is the initial state where we have no test data,
$s_{t+1}$ is the state we transition to after taking action $\pi(s_{t})$ from
state $s_{t}$ for $t=0,1,\dots,T-1$ and where $s_{T}$ is the first state at
which we do not have sufficient budget to take any further action.

The screening problems can now be restated as MDP policy optimization problems
by

$\max_{\pi}\mathbb{E}\left[\sum_{t=0}^{T-1}R\big{(}\pi(s_{t}),s_{t}\big{)}\right].$
(7)

The focus of this paper is to develop algorithmic policies to work with the
mining objective and covariate testing model in large scale material screening
applications.

##  3 Single-Test Bayesian Optimization

Classical single-test Bayesian optimization works by updating an _acquisition
function_ , $\alpha(D)\in\mathbb{R}^{n}$, at each stage and then sampling the
candidate that maximizes it. See [4] for an introduction. Note that in the
case of single-test Bayesian optimization there are only two sets of
candidates of interest: the untested candidates $I_{\hbox{u}}$ and the tested
candidates $I_{\hbox{t}}$. See Algorithm 1.

Algorithm 1 Bayesian Optimization

1:while $b>0$ do

2: $i^{\ast}=\arg\max_{i\in I_{\hbox{u}}}\alpha_{i}(D)$

3: apply test to $i^{\ast}$

4:end while

##  4 Two-Test Sequential Bayesian Optimization

We will restrict ourselves to sequential methods in which candidates have to
be tested with the cheap test before they can be tested with the expensive
test. This restriction means that $I_{\hbox{ut}}$ will always be empty, which
avoids some major difficulties in making inferences with the covariate testing
model. In many cases this restriction will be forced on us by practical
considerations, for example if the cheap test is an intermediate result that
must be tested for as part of the expensive test. However there may also be
cases where the two tests are not related in this way and in those cases
sequential testing may not be optimal. For example if the cheap test score is
not useful or not useful enough to justify its cost then the optimal policy
would learn to skip this test, but that will not be possible for a sequential
method.

All of our proposed two-test sampling methods are implementations of the same
high level algorithm. At each stage Algorithm 2 either applies the cheap test
to advance a candidate from $I_{\hbox{uu}}$ to $I_{\hbox{tu}}$ or applies the
expensive test to advance a candidate from $I_{\hbox{tu}}$ to $I_{\hbox{tt}}$.

Exactly as in single-test Bayesian optimization an acquisition function is
updated at each stage to identify candidates for testing. However now a
controller decides whether to apply the cheap test to the best candidate from
$I_{\hbox{uu}}$ or to apply the expensive test to the best candidate from
$I_{\hbox{tu}}$. Note that unlike in single-test Bayesian optimization the
acquisition function depends on all of the available $\bm{y}^{(C)}$ and
$\bm{y}^{(E)}$ data and assigns values to candidates in both $I_{\hbox{uu}}$
and $I_{\hbox{tu}}$. Despite these differences we are able to adapt widely
used single-test acquisition functions to our setting using almost exactly the
same mathematical definitions, we just need to calculate them slightly
differently. We will use the following acquisition functions:

Two-Test Greedy Expected Improvement: This acquisition function returns the
expected reward for applying the expensive test to candidate $i$ under the
optimization objective.

$\alpha_{i}(D)=\mathbb{E}\big{[}\max(y_{i}^{(E)}-y_{\max}^{(E)},0)\big{|}D].$
(8)

Two-Test Greedy Mining: This acquisition function returns the expected reward
for applying the expensive test to candidate $i$ under the mining objective.

$\alpha_{i}(D)=\mathbb{P}[i\in\hbox{top}_{N}|D].$ (9)

Two-Test Greedy Threshold: This acquisition can be used to approximate (9).

$\alpha_{i}(D)=\mathbb{P}[y_{i}^{(E)}\geq\tau|D].$ (10)

Two-Test Thompson: This random acquisition function is obtained by sampling
from the posterior of the expensive test scores

$\alpha(D)\sim\rho(\bm{y}^{(E)}|D).$ (11)

Accurately estimating the greedy mining acquisition function requires a large
number of samples and this can be prohibitively expensive. We therefore
propose using the greedy threshold acquisition function as an approximation.
The threshold $\tau\in\mathbb{R}$ is chosen to be the posterior median of the
$y^{(E)}$ score of the $N$th highest scoring candidate. Although this value
also needs to be estimated through sampling, it will have a much lower
variance than the greedy acquisition function, so can be accurately estimated
from a modest number of samples, and does not need to be updated on every
iteration. Because the threshold score is based on the absolute value of each
candidates score, rather than their ranking, this approximation cuts out a lot
of complex dependencies.

If we fix the choice of acquisition function then the MDP screening problem
can be restated from the point of view of the controller as follows. From a
state $s\in S$, with sufficient budget, there are two actions available:

  1. 1.

Apply the cheap test test to $i_{uu}=\arg\max_{i\in
I_{\hbox{uu}}}\alpha_{i}(D)$.

  2. 2.

Apply the expensive test to $i_{tu}=\arg\max_{i\in
I_{\hbox{tu}}}\alpha_{i}(D)$.

Algorithm 2 Sequential Bayesian Optimization

1:while $b>0$ do

2: $i_{uu}=\arg\max_{i\in I_{\hbox{uu}}}\alpha_{i}(D)$

3: $i_{tu}=\arg\max_{i\in I_{\hbox{tu}}}\alpha_{i}(D)$

4: if controller chooses action 1 then

5: apply cheap test to $i_{uu}$

6: else

7: apply expensive test to $i_{tu}$

8: end if

9:end while

###  4.1 Sequential Greedy

The Sequential Greedy (SG) method combines a greedy acquisition function with
a greedy controller that chooses which test to apply from a state $s\in S$ by
comparing the expected reward to cost ratio of two different sequences of
actions.

  1. 1.

Apply the cheap test to $i_{uu}$ and then apply the expensive test to
whichever is the most promising of $i_{tu}$ and $i_{uu}$ given the new data
from the cheap test.

  2. 2.

Apply the expensive test to $i_{tu}$.

The greedy controller chooses action 1 whenever

$\displaystyle\frac{\mathbb{E}\left[\max\big{(}\alpha_{i(tu)}(D,y^{(C)}_{i(uu)}),\alpha_{i(uu)}(D,y^{(C)}_{i(uu)})\big{)}~{}|~{}D\right]}{c_{C}+c_{E}}>\frac{\alpha_{i(tu)}(D)}{c_{E}}$

where $\alpha$ is one of the greedy acquisition functions. See Algorithm 4.1
in the supplementary material.

###  4.2 Sequential Thompson with Random Controller

The Sequential Thompson with Random controller (STR) method combines the
Thompson sampling acquisition function with a random controller. The
controller chooses between action 1 and action 2 independently and randomly at
each stage according to a probability distribution which is chosen as a
parameter of the method. One possibility is to set the probability taking
action 1 by

$p_{1}=\frac{c_{E}+2c_{C}}{c_{E}+3c_{C}}$ (12)

which is chosen so that roughly half of the budget is spent on applying the
cheap test to candidates which are never tested with the expensive test. See
Algorithm 4.2 in the supplementary material.

(a) $\theta=0$ (b) $\theta=\pi/4$ (c) $\theta=\pi/2$

Figure 1: Example synthetic test data described in Section 6 with top 10
candidates highlighted.

##  5 Asynchronous Parallel Implementation

In large scale screening applications it is essential that the sampling method
can work efficiently with a large number of asynchronous parallel workers. In
[10] the authors show that single-test Thompson sampling is well suited to
asynchronous parallel implementation. In [11] the authors adapt single-test
expected improvement to this stetting by conditioning on and then
marginalizing out the scores of any candidates that are currently being
tested. The same adaptation could be applied to two-test expected improvement
but unfortunately their method would not make any difference to the two-test
greedy threshold acquisition function.

In all of the experiments in this paper we simulate the behaviour of
$w\in\mathbb{N}$ asynchronous parallel workers. The times taken to carry out
the cheap tests are i.i.d. samples from $U[c_{C}/2,3c_{C}/2]$ and the
expensive tests i.i.d. samples from $U[c_{E}/2,3c_{E}/2]$. Each worker uses
the sampling algorithm to choose an available action, carries out the
associated test and then as soon as it finishes immediately chooses a new
action to start.

##  6 Test on synthetic data

In this section we test our algorithms in simulated screens on synthetic data.
Each problem is generated as follows. We set $n=500$, then sample
$\bm{x}\in\mathbb{R}^{n}$ with i.i.d. uniform $[0,1]$ entries, then we sample
$\bm{y}^{(C)}\in\mathbb{R}^{n}$ from
$\mathcal{N}(\underline{0},\Sigma^{(C)})$, where

$\big{(}\Sigma^{(C)}\big{)}_{ij}=0.25^{2}\exp\left(\frac{-(x_{i}-x_{j})}{2\times
0.25^{2}}\right)+0.25^{2}\delta_{ij},$ (13)

for $i,j=1,\dots,n$, then we sample $\bm{y}^{(E)}\in\mathbb{R}^{n}$ from
$\mathcal{N}(\underline{0},\Sigma^{(E)})$, where

$\big{(}\Sigma^{(E)}\big{)}_{ij}=\exp\left(-\frac{(x_{i}-x_{j})}{2\times
0.25^{2}}\sin^{2}(\theta)-\frac{(y^{(C)}_{i}-y^{(C)}_{j})}{2\times
0.25^{2}}\cos^{2}(\theta)\right)+0.05^{2}\delta_{ij},$ (14)

for $i,j=1,\dots,n$, where $\theta\in[0,\pi/2]$.

Note that in these tests, the sampling algorithms will have full knowledge of
the generative model including all of the hyperparameter values. The
hyperparameter $\theta$ varies the expensive score’s length scales with
respect to the $x$ values, which are visible to the algorithm throughout with
no cost, and the $y^{(C)}$ values, which need to be tested for to be revealed.
The cheap test therefore provides more useful information for smaller values
of $\theta$. See Figure 5.

Figure 2: Comparison of algorithm performance in screens on synthetic data
described in Section 6.1. (a,b,c) Experiment 1, (d,e,f) Experiment 2 and
(g,h,i) Experiment 3. Bars indicate standard error of mean estimates.  Figure
3: Varying $p_{1}$ parameter in Sequential Thompson Random method as described
in Section 6.1. Bars indicate standard error of mean estimates.

###  6.1 Results

We fix $c_{E}=1$ and $B=50$ and will vary $\theta$, $c_{C}$ and $w$.

  1. 1.

We fix $c_{C}=0.2$ and $w=1$ and vary $0\leq\theta\leq\pi/2$.

  2. 2.

We fix $w=1$, $\theta=\pi/4$ and vary $0.1\leq c_{C}\leq 0.5$.

  3. 3.

We fix $\theta=\pi/4$, $c_{C}=1$ and vary $1\leq w\leq 16$.

For each set of experiment parameters we repeat 1000 independent trials and
record the average optimization regret, average mining regret and the average
number of expensive tests carried out. See Figure 2.

Note that Sequential Greedy Expected Improvement (SGEI), which targets the
optimization objective, achieves the lowest average optimization regret for
nearly all of the experiment parameter vales and likewise Sequential Greedy
Threshold (SGT), which targets the mining objective, achieves the lowest
average mining regret. Sequential Thompson Random (STR) is outperformed by
both Greedy methods in both metrics except for optimization regret when using
the largest number of parallel workers.

We also experimented with varying $p_{1}$ in the STR method using $c_{C}=0.2$,
$\theta=\pi/4$ and $w=1$. See Figure 3. Note that although the optimally tuned
method outperforms the method tuned using (12), it is still some way behind
SGT and that optimally tuning a parameter in this way would be impossible in a
practical problem.

##  7 Simulated screen on real data

In this section we test our algorithms in simulated screens using real data
from _in silico_ chemical engineering experiments. Metal Organic Frameworks
(MOFs) [19] and Covalent Organic Frameworks (COFs) [1] are families of porous
solids that can be grown from a variety of component molecules into a vast
array of different structures. Both MOFs and COFs have extremely high surface
area to volume ratios which mean they can interact with gasses in special ways
and have potential applications in a number of important industrial processes.

We use the covariate testing model and the mining objective with $N=100$ so
that the aim of the screen is to find as many of the top-100 materials as
possible. We will compare our two-stage sampling algorithms with the following
baseline methods:

Single-Test Poor. In this method we ignore the cheap test and apply standard
single-test Bayesian optimization using the expensive test with the feature
matrix $X$.

Single-Test Rich. In this method we apply the cheap test to all of the
candidates then apply standard single-test Bayesian optimization using the
expensive test with the expanded feature matrix $[X,\bm{y}^{(E)}]$.

In all of these experiments we fit the Gaussian process modelling
hyperparameters to minimize the NLL of the data available at each step. Our
code is based around the GPy package [6] but with some custom modifications to
to implement the covariate testing model and for sampling from large scale
posteriors. See supplementary material for details.

###  7.1 Methane deliverable capacity in COFs

The Hypothetical Covalent Organic Framework (HCOF) database contains 69,839
material structures [13]. Each structure is provided with a vector of features
that describe its composition and shape as well as its results in a simulated
methane adsorption experiment.

We choose methane deliverable capacity as the expensive test score which we
want to target in the screen. Deliverable capacity in methane ($CH_{4}$) is
defined by

$\hbox{DC}_{CH_{4}}=\hbox{N}_{CH_{4}}^{{65bar}}-\hbox{N}_{CH_{4}}^{{5.8bar}}$
(15)

where $\hbox{N}_{CH_{4}}^{{p}}$ denotes the volume of methane at STP adsorbed
by one unit volume of the material at pressure $p$. Roughly speaking this
measures how efficiently the COF can store and release methane. We use 7
chemical composition features along with density as the 8 basic features in
our screening problem. We use the void fraction for the cheap test score. This
measures what fraction of the structure’s volume is open to the gas molecules.
Note that void fraction is very important in determining deliverable capacity
but via a non-linear relationship. See Figure 5. We use indicative costs of
$c_{C}=0.1$, $c_{E}=1$ and allow a testing budget of $B=1000$.

Table 1: Results for COF screen. All values averaged over 10 independent trials with standard deviation in brackets. method | workers | cheap tests | expensive tests | total cost | total reward
---|---|---|---|---|---
GT - Rich | 1 | 69839 | 1000 | 70839 | 79.5 (2.6)
GT - Poor | 1 | 0 | 1000 | 1000 | 42.3 (4.3)
SGT | 1 | 2742.2 (247.7) | 725.1 (24.7) | 1000 | 63.9 (9.3)
T - Rich | 100 | 69839 | 1000 | 70839 | 76.7 (3.2)
T - Poor | 100 | 0 | 1000 | 1000 | 39.0 (5.6)
SRT | 100 | 4901.1 (105.7) | 509.2 (10.6) | 1000 | 56.9 (2.9)
Table 2: Results for MOF screen. All values averaged over 10 independent trials with standard deviation in brackets. method | workers | cheap tests | expensive tests | total cost | total reward
---|---|---|---|---|---
GT - Rich | 1 | 137953 | 1000 | 138953 | 98.0 (0.0)
GT - Poor | 1 | 0 | 1000 | 1000 | 70.1 (1.2)
SGT | 1 | 1613.4 (26.8) | 386.6 (26.8) | 1000 | 81.3 (2.2)
T - Rich | 100 | 137953 | 1000 | 138953 | 100.0 (0.0)
T - Poor | 100 | 0 | 1000 | 1000 | 44.5 (6.8)
SRT | 100 | 1473.1 (18.9) | 526.9 (18.9) | 1000 | 67.1 (7.7)

Figure 4: Test data used in screening problems. a) COF, b) MOFs.

Figure 5: Comparison of algorithm performance in screens on (a,b,c) COF and
(d,e,f) MOF data. (a,d) compare sequential methods over a range of worker
numbers. (b,e) compare greedy methods with a single worker. (c,f) compare
Thompson methods with 100 workers. Bars in (a,d) indicate standard deviation.
Bold curves in (b,c,e,f) indicate averaged performance.

###  7.2 Nitrogen Carbon dioxide separation in MOFs

The HMOF database contains 137,953 material structures [17]. Each structure is
provided with a vector of features that describe its composition and shape. We
use 12 chemical composition features along with 6 physical features as the 18
features in our screening problem. We also use data from [3] where each
structure was tested for its ability to separate nitrogen and carbon dioxide.
Rather than simulating a true mixture of the two gasses, which is far more
computationally expensive, the authors simulated each gas adsorption
separately and then combined the results of the two experiments to give an
Adsorption Performance Indicator (API).

$API=\frac{\hbox{DC}_{CO2}\times\hbox{N}_{CO2}^{{0.1bar}}}{\hbox{N}_{N2}^{{0.9bar}}+1}.$
(16)

In our screening problem we will use

$\bm{y}^{(C)}=\log\big{(}\hbox{DC}_{CO2}\times\hbox{N}_{CO2}^{{0.1bar}}\big{)},\quad\bm{y}^{(E)}=\log\big{(}API\big{)},$
(17)

so that the cheap test scores are the contribution to the API from the CO2
experiment and the expensive test scores are the final API. Note that we have
taken the log so that the data is better suited to our Gaussian Process model.
Also note that in this example the sequential testing restriction is naturally
implied by the fact that the cheap test forms part of the expensive test. See
Figure 5. We use indicative costs $c_{C}=0.5$, $c_{E}=0.5$ and allow a testing
budget of $B=1000$. When scoring single-test methods on this problem we use
$c_{E}=1$ for a fair comparison.

###  7.3 Results

We simulated the screening process using the Sequential Greedy Threshold (SGT)
and Sequential Thompson Random (STR) two-test methods and compared them to the
Greedy Threshold (GT) and Thompson (T) single-test methods with the Rich and
Poor datasets. Each experiment was repeated over ten independent trials.

Figure 5 (a,d) compares the performance of SGT and STR for the two databases.
Note that for a single worker the SGT method is most efficient but that for
fifty or more workers the STR method performs best. In the COF experiment we
actually observed improved performance with increased number of workers for
$1\leq w\leq 50$, which may be due to the increased number of (uniformly)
randomly chosen initial samples taken with more workers.

Figure 5 (b,c,e,f) compares the performance of our chosen two-test method to
the corresponding single-test method using SGT with one worker or STR with one
hundred workers. The results of these experiments are also summarised in
Tables 1 and 2. In both cases the two-test method is able to significantly
outperform the single-test method on the poor dataset (which is its only fair
comparison).

##  8 Conclusions

We presented new models and algorithms for multi-test Bayesian optimization
for application in large scale materials screening problems. We have
demonstrated the potential power of these techniques in two simulated screens
on real data from previous screening studies. Using the covariate testing
model allows our algorithms to learn complex non-linear patterns but makes
computation difficult.

Some possible direction for future algorithm development include the
following.

Non-sequential methods. As discussed the sequential condition to always apply
the cheap test before the expensive test could be inefficient in some
problems. However making inferences when $I_{\hbox{ut}}$ is non-empty is a
difficult missing data problem that will very challenging in large-scale
moderate-dimensional problems.

Entropy based methods. We do not currently have an efficient way to adapt the
entropy acquisition function to the multi-test setting. We did experiment with
an entropy based controller for use with Thompson sampling but this was too
slow to even carry out a large number of small scale experiments.

More than two tests. The covariate testing model could be adapted to support a
range of different cheap tests, possibly all related to the expensive test
scores by very different non-linear relationships. Any efficient sampling
method that worked with this model would need to work in a non-sequential
manner as described above.

##  9 Acknowledgements

This work was supported by the Engineering and Physical Sciences Research
Council EP/L016354/1. This project has received funding from the European
Research Council (ERC) under the European Union’s Horizon 2020 research and
innovation programme (grant agreement No 648283 GROWMOF) This research made
use of the Balena High Performance Computing (HPC) Service at the University
of Bath.

##  10 Supporting Information

The code used as part of this study can be found at:

        https://gitlab.com/AMInvestigator/ame/-/tree/multi_test_bayesian_optimisation

## References

  * [1] Shuai Cao, Bing Li, Rongmei Zhu, and Huan Pang.  Design and synthesis of covalent organic frameworks towards energy and environment fields.  Chemical Engineering Journal, 2019.
  * [2] Stefano Curtarolo, Gus Hart, Marco Buongiorno Nardelli, Natalio Mingo, Stefano Sanvito, and Ohad Levy.  The high-throughput highway to computational materials design.  Nature materials, 2013.
  * [3] Michael Fernandez, Tom K. Woo, Christopher E. Wilmer, and Randall Q. Snurr.  Large-Scale Quantitative Structure–Property Relationship (QSPR) Analysis of Methane Storage in Metal–Organic Frameworks.  The Journal of Physical Chemistry C, 2013.
  * [4] Peter I. Frazier.  A Tutorial on Bayesian Optimization.  arXiv, 2018.
  * [5] Peter I. Frazier and Jialei Wang.  Bayesian Optimization for Materials Design. In Information Science for Materials Discovery and Design.  2016\.
  * [6] GPy.  GPy: A gaussian process framework in python.  http://github.com/SheffieldML/GPy, since 2012.
  * [7] José Miguel Hernández-Lobato, James Requeima, Edward O. Pyzer-Knapp, and Alán Aspuru-Guzik.  Parallel and distributed thompson sampling for large-scale accelerated exploration of chemical space.  ICML2017, pages 1470–1479, 2017.
  * [8] James Hook, Kamil Kulesza, P. Morawiecki, and Eddied Wilson.  Understanding the screening process of new molecules.  In European Study Group With Industry 138, 2018.
  * [9] Kirthevasan Kandasamy, Gautam Dasarathy, Junier Oliva, Jeff Schneider, and Barnabás Póczos.  Gaussian process bandit optimisation with multi-fidelity evaluations.  NIPS2016, 2016.
  * [10] Kirthevasan Kandasamy, Akshay Krishnamurthy, Jeff Schneider, and Barnabas Poczos.  Parallelised bayesian optimisation via thompson sampling.  Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics, 2018.
  * [11] Benjamin Letham, Brian Karrer, Guilherme Ottoni, and Eytan Bakshy.  Constrained bayesian optimization with noisy experiments.  Bayesian Analysis, 2017.
  * [12] Turab Lookman, Prasanna Balachandran, Dezhen Xue, and Ruihao Yuan.  Active learning in materials science with emphasis on adaptive sampling using uncertainties for targeted design.  npj Computational Materials, 2019.
  * [13] Rocío Mercado, Rueih-Sheng Fu, Aliaksandr V Yakutovich, Leopold Talirz, Maciej Haranczyk, and Berend Smit.  In Silico Design of 2D and 3D Covalent Organic Frameworks for Methane Storage Applications.  Chemistry of Materials, 2018.
  * [14] Benjamin J. Sikora, Christopher E. Wilmer, Michael L. Greenfield, and Randall Q. Snurr.  Thermodynamic analysis of Xe/kr selectivity in over 137000 hypothetical metal organic frameworks.  Chem. Sci., 2012.
  * [15] Jialin Song, Yuxin Chen, and Yisong Yue.  A General Framework for Multi-fidelity Bayesian Optimization with Gaussian Processes.  Proceedings of Machine Learning Research, 2019.
  * [16] Richard S. Sutton and Andrew G. Barto.  Introduction to Reinforcement Learning.  MIT Press, 1998.
  * [17] Christopher E Wilmer, Omar K Farha, Youn-Sang Bae, Joseph T Hupp, and Randall Q Snurr.  Structure–property relationships of porous materials for carbon dioxide separation and capture.  Energy & Environmental Science, 2012.
  * [18] Jian Wu, Saul Toscano-Palmerin, Peter I. Frazier, and Andrew Gordon Wilson.  Practical multi-fidelity bayesian optimization for hyperparameter tuning.  arXiv, 2019.
  * [19] Shuai Yuan, Liang Feng, Kecheng Wang, Jiandong Pang, Matheiu Bosch, Christina Lollar, Yujia Sun, Junsheng Qin, Xinyu Yang, Peng Zhang, Qi Wang, Lanfang Zou, Yingmu Zhang, Liangliang Zhang, Yu Fang, Jialuo Li, and Hong-Cai Zhou.  Stable Metal–Organic Frameworks: Design, Synthesis, and Applications.  Advanced Materials, 2018.