# Data Behind the Walls – An Advanced Architecture for Data Privacy Management

Amen Faridoon  School of Computer Science
University College Dublin
Dublin, Ireland
<EMAIL_ADDRESS>M. Tahar Kechadi  School of Computer Science
University College Dublin
Dublin, Ireland
<EMAIL_ADDRESS>
###### Abstract

In today’s highly connected society, we are constantly asked to provide
personal information to retailers, voter surveys, medical professionals, and
other data collection efforts. The collected data is stored in large data
warehouses. Organisations and statistical agencies share and use this data to
facilitate research in public health, economics, sociology, etc. However, this
data contains sensitive information about individuals, which can result in
identit theft, financial loss, stress and depression, embarrassment, abuse,
etc. Therefore, one must ensure rigorous management of individuals’ privacy.
We propose, an advanced data privacy management architecture composed of three
layers. The data management layer consists of de-identification and
anonymisation, the access management layer for re-enforcing data access based
on the concepts of Role-Based Access Control and the Chinese Wall Security
Policy, and the roles layer for regulating different users. The proposed
system architecture is validated on healthcare datasets.

###### Index Terms:

Data privacy management; Data security; Access control; RBAC Model; CWSP
Model.

##  I INTRODUCTION

New technologies, such as artificial intelligence, cloud computing, sensors
and wireless networks, are constantly integrated into our daily lives at a
rapid pace. We live in a digital word, also called the fourth industrial
revolution. This digital world produces a considerable amount of data, which
is highly valuable and can be seen as the new gold or the new oil of this
century111https://www.cutimes.com/2020/07/17/big-data-the-new-oil-
fields/?slreturn=20220716063653. This data enables businesses or organisations
to enhance their productions and profits [1]. Many data organisations, such as
Facebook, Amazon, Google, …, have emerged as dominant in this field. They
invested a lot of resources to create, acquire, maintain, and store the data
to make constructive and timely decisions and improve customer services.
Therefore, with the rise of the data economy, companies find enormous value in
collecting, sharing, and using data.

Data breaches and data theft affect both individuals and organisations. Most
businesses have elaborated security mechanisms to protect individual private
information [2]. However, due to vulnerabilities in software, phishing, stolen
credentials, and malicious attacks, cybercriminals can infiltrate and steal
personal data, which results in identity theft, financial loss, stress and
depression, discrimination, embarrassment, abuse, etc. [3] [4]. Many
businesses and government agencies have suffered a massive data breach leading
to layoffs, lawsuits, and frauds. In 2016 IBM reported that around $60\%$ of
company data breaches were made by its employees, out of which $75\%$ were
deliberate222https://www.ciab.com/resources/ibm-60-percent-attacks-carried-
insiders/.

An insider threat is defined as any malicious activity executed by users with
legitimate access to the organisation’s network, applications, or data stores
[5]. One can distinguish three types of insider threats: 1) malicious insider
who deliberately looks to steal information, 2) careless insider who does not
follow proper IT procedures and accidentally exposes sensitive corporate data
at risk and 3) compromised insider who has compromised credentials by an
external attacker. Unlike threats from outside the organisation, insider
threats are much more difficult to detect and prevent. Insider threats’
behaviour is difficult to identify as suspicious because they have legitimate
credentials and access to the systems and data. Once inside the network, it is
difficult to pick apart the malicious behaviour from a large volume of
legitimate day-to-day activities. The data requires robust privacy management
systems to prevent identity theft and unauthorised access from external and
malicious insider attackers.

Therefore, the main challenge of this study is to design a data privacy
management architecture that can prevent breaches of an individual’s identity
and keep the data secure from unauthorised access. The main objectives of our
data privacy management architecture are twofold: 1) create security walls
between subjects and objects for access control and formulate authorisation
rules for legitimate users, and 2) secure the identity and sensitive
information of an individual from both the third parties and insider threats.

##  II Related Work

Numerous data privacy management models have been proposed and implemented by
addressing the problem of unauthorized access control whereas the most common
are; Mandatory Access Control (MAC), Discretionary Access Control (DAC),
Attribute Based Access Control (ABAC) etc. However, these models have
limitations. MAC is a very inflexible security model where each user is
assigned a clearance, and each object has a classification and compartment
stored in its security label. This model is mostly suitable for structured
environments where we have classified data. DAC is less central system
administration model because the control is not dictated by a company policy
but rather by the owner or creator of the information. Whereas, the drawbacks
of ABAC limits its implementation such as; time-consuming initial setup in
term of identifying right resources, objects and subjects attributes, policies
etc, cannot factor everything in environments like big organizations, and it
also not ideal for small businesses.
According to the organizational structure and needs the most considerable
models is Role Based Access Control (RBAC). Most of the studies [6] [7] [8]
[9] took an advantage of RBAC security model and their extended versions in
different environments (cloud, distributed data storage systems etc.) to
restrict the anuthorized access. Whereas, in the proposed systems the
administration of permission groups having the main responsibility for
managing the users, their roles and permissions. Therefore, by exploiting
their authority, the access permissions of the roles can easily be changed.
Moreover, researchers also adopt this model in healthcare domain. Authors in
[10] identified the need for roles and their acting procedures in an EHR
system. They highlighted that the processes for defining the roles in an EHR
system play a vital part in access privileges management. Others proposed
structured roles and grant access only to entities located in the
authorisation table [11]. The privacy and security maintenance committee is
the only authority that can act on the authorisation table. However,
structured roles are not enough, and the dynamic and structural aspects of
roles are necessary [12]. For instance, one should only provide data access to
physicians if they are engaged in their treatment. Therefore, one can deduce
roles have more contextual variables (e.g., place, time, association, etc.).
Hence, the first model, known as contextual role-based access control
authorisation, was proposed in [13]. The purpose of this model is to improve
privacy and confidentiality of patients’ records, as they were not robust
enough. The model considers a hierarchy of roles with authorisation
inheritance and a data model which covers the different data types (e.g.,
prescriptions, demographics). The model also includes a technique for
authorisation conflict management. However, most of these models consider
access to healthcare practitioners only, they do not have explicit policies on
the complete dataset. More data professionals should be considered and be able
to access the data with predefined usage and ethics.
The applications of Chinese Wall Security Policy (CWSP) in different
environments have attracted significant research interests in the last few
years. Studies deployed the CWSP model on the cloud at the IaaS level to
control the information flow between the conflicting parties [14][15].
Moreover, various scenarios have been modelled with CWSP, mainly when
conflict-of-interests are involved. For instance, studies [16][17] focused on
protecting subjects (companies) and objects (datasets) by creating subject
walls and object walls. Recently, a security model for cryptographic key
management in the cloud was proposed and offered security as a Service
(CKMS$-$SecaaS’s) [18].
These studies, while efficient, need further investigations because the
spectrum of their applications is limited to distributed or cloud
environments. Their main objective was to secure the datasets or assets of an
organisation from third-party (conflicting organisation) attacks.
Nevertheless, guaranteeing the privacy and security of personal and sensitive
datasets within an organisation from insider attacks is still challenging.

##  III Proposed Model Concepts

The proposed model consists of three modules: the roles (or users) module, the
data access management module, and the data management module. We combine the
concept of role-based access control with the Chinese wall security policy to
accomplish data security. The objective is to manage data storage and usage so
that individuals’ data cannot be exposed to malicious parties.

###  III-A Roles, Subjects, and Objects

The Role-Based Access Control model was proposed in 1996 in [19]. It is
considered one of the most accepted access control models. The model exploits
the “need-to-know” principle, widely adopted and utilised. The whole model
assumes that privacy is preserved as long as the data is only accessed when
necessary for the right motive and only minimum information details are
disclosed.

The core components shown in Figure 1 of the role-based access control model
are Users or Subjects, Roles, Operations, and Objects.

  * •

Users are the skilled individuals and they are grouped into roles.

  * •

Roles represent a set of responsibilities defined by the administration of an
organisation.

  * •

Operations (OPS) are functions performed by the subjects (users) on the
objects.

  * •

Objects (OBS) are the resources that are managed and used by the subjects. A
resource can be an application, database, file, etc.

Figure 1: Core Role-Based Access Control Model

###  III-B Access Management and Walls

The proposed data access management, based on the Chinese Wall security policy
(CWSP), is defined as a dynamic firewall mechanism shown in Figure 2. We
implemented the concept of CWSP for data access control between
competing/conflicting roles based on access control rules [16, 17, 20]. The
walls defined in this model prevent access between the roles and data
warehouses inside the same organisation. The adoption of the CWSP for the data
access management layer is summarised below:

  * •

Users/subjects cannot access the object that is in conflict with the object
they already possess. Therefore, each subject has a granted and a denied
objects, where each subject in the granted set has their conflicting subjects
in the denied set. The pairwise (Granted, Denied) of a subject is called the
subject wall. It is not possible to find two competing objects inside the same
wall.

  * •

The subjects can perform read and write operations via authorised access to
the objects. In this case, the wall, called as “object wall”, is created
around the objects based on the same rules as the subjects.

Figure 2: Data Access Control Based on CWSP

###  III-C Wall Placement

Chinese wall security policy is based on wall placements to prevent access to
data platforms from unauthorised subjects. Our model acts on the same entities
as previous ones: subjects, objects, and access rights. In this study, we have
data warehouses (DWs) as objects and a set of users of different
responsibilities on DWs. We achieve the privacy of individuals’ records in two
steps: de-identification and anonymisation of original data records.

The typical data platform architectures should deal with data collection,
integration, and analytics. We divide the collected data stores into three
main layers, depending on the level of privacy and security they may need to
handle. They are described in the following:

##### Original Data Warehouse (ODW)

The collected raw data via different devices and from various sources are
stored in data marts. These data marts are carefully integrated into a data
warehouse containing original data. This data warehouse maintains the quality,
accuracy, validity, timeliness, completeness, uniqueness, and consistency of
the original data records. We consider this data warehouse as an object (ODW)
that must be secured from unauthorised access and the limited number of users
who have access to these original records.

##### De-identified Data Warehouse (DDW)

All information about users’ identities is removed to prevent identity
disclosure. This de-identified data warehouse consists of individual
aggregated and derived statistical data that is computed from the original
data. We consider this de-identified data warehouse as a separate object (DDW)
that needs to prevent the identity of an entity during analysis. But, with the
presence of quasi-identifiers, this data needs further data protection.

##### Anonymised Data Warehouse (ADW)

After the de-identification, we further anonymise the data so that the
original data cannot be recovered from the knowledge extracted during the data
analysis. We consider this anonymised data warehouse as an object (ADW) that
must be secured from malicious third parties or the linkage and other attacks.

##  IV Model Formulation

In the previous section, we identified the main concepts of the model: roles,
objects, and walls. The roles are the subjects that own the set of objects.
The objects represent datasets, data records, or data warehouses (ODW, DDW and
ADW). Finally, the walls implement the conflicting relation on which the CWSP
is based. In the following, we give formal definitions of all these entities
and relations.

The overall system can be considered as $(R,O,CIR,CIC,W,A)$, where
$R=\\{r_{1},\cdots,r_{n}\\}$, $O=\\{o_{1}\cdots,o_{m}\\}$,
$CIR=\\{cr_{1}\cdots,cr_{k}\\}$, $CC=\\{cc_{1}\cdots,cc_{p}\\}$,
$W=\\{w_{1},\cdots,w_{l}\\}$, $A=\\{a_{1},\cdots,a_{h}\\}$ sets of Roles
(subjects), Objects, Conflict-relations, conflict-classes, Walls, and access
rights, respectively. They are defined as follows:

The proposed system has following characteristics:

  1. 1.

Some roles are in conflict:
Let $\\{R_{i}\\}$ and $\\{R_{j}\\}$ are in conflict: if u $\in\\{R_{i}\\}$,
then u $\notin\\{R_{j}\\}$ and u cannot be in $\\{R_{j}\\}$ at all. Therefore,
we need a wall between $\\{R_{i}\\}$ and $\\{R_{j}\\}$.

  2. 2.

Some roles are cooperative:
Let $\\{R_{x}\\}$ and $\\{R_{y}\\}$ are cooperative: if u $\in\\{R_{x}\\}$,
then u $\notin\\{R_{y}\\}$ but u can switch to $\\{R_{y}\\}$. So, there is no
need to have a wall between $\\{R_{x}\\}$ and $\\{R_{y}\\}$.

Suppose, $D$ is a set of domains, $D=\\{d_{1},\cdots,d_{n}\\}$, $\Phi$ denotes
Execution Denied, $\\{O_{d_{i}}\\}$ denotes Object of domain $\\{d_{n}\\}$,
and $\\{F_{d_{i}}()\\}$ is a function or an operation of the environment
$\\{d_{n}\\}$

###### Definition 1

A Role is a set of users group or subjects that can execute some given
operation F() within a given domain. Each users group can be scaled down to a
single user. The role $\\{R_{i}\\}$ within a system is defined as follows:

$\\{R_{i}\\}=\langle U_{i},F_{i}(),d_{i}\rangle$ Where,
$U_{i}=\\{\\{u_{j}\\}_{j}\\}\mid u_{j}$ can execute
$\\{F_{d_{i}}(O_{d_{i}})\\}$

###### Definition 2

Objects are the entities that can be manipulated by subjects. An object can be
a record, file, data warehouse, or any entity on which an operation can be
executed by a user or subject. The object $\\{O_{i}\\}$ within a system is
defined as follows:

$\\{O_{i}\\}=\langle E_{i},OP_{i}()\rangle$
$(e_{j}\in\\{E_{i}\\})\wedge(op_{j}\in OP_{i}())\Rightarrow
op_{j}(e_{j})\neq\Phi$

Whereas, $\\{E_{i}\\}$ is a set of entities (records, files, data warehouses)
and $\\{Op_{i}\\}$ is a set of operations (read, write, …)

###### Definition 3

A Conflict-relation is a relation that exist between the competing roles.
Roles have conflict with each other on the bases of access granted or denied
permissions on objects of different domains. Conflict of interest relation is
a binary relation, and within a system is defined as follows:

CIR $\subseteq\\{{R}^{2}\\}\Rightarrow\\{0,1\\}$ — if $r_{i}\in R=1,$
otherwise 0

Moreover, let CIR $\subseteq\\{RXR\\}$ must satisfies the following
properties:

  1. 1.

CIR-1: CIR is symmetric: If role $R_{i}$ is in conflict with the role $R_{j}$,
then $R_{j}$ is certainly in conflict with $R_{i}$.

$\forall{(r_{i},r_{j})}\in R^{2};RC_{i}(r_{i},r_{j})=RC_{i}(r_{j},r_{i})$

  2. 2.

CIR-2: CIR is anti-reflexive: The role cannot conflict to itself.

$\forall{(r_{i},r_{i})}$ in $R^{2};RC_{i}(r_{i},r_{i})=0$

  3. 3.

CIR-3: CIR is anti-transitive: If role $R_{i}$ is in conflict with the role
$R_{j}$, and $R_{j}$ is in conflict with the role $R_{k}$ then role $R_{i}$
may or may not in conflict with $R_{k}$.

$\forall{(r_{i},r_{j},r_{k})}\in
R;RC_{i}(r_{i},r_{j})=1,RC_{i}(r_{j},r_{k})=1$ Then
$RC_{i}(r_{i},r_{k})=[0,1]$

###### Definition 4

A conflict-class is defined on the notion of equivalence class, that is the
subset of roles which are equivalent to each other called as roles class (RC).
Whereas, each role within the class must satisfy the defined properties of
conflict-relation. So that we cannot find the conflicting roles within the
same class. Conflict-Class within a system is defined as follows:

$[RC_{i}]=\\{r_{i}\mid(r_{j},r_{i})\in CIR\\}$
$(r_{i}\in\\{RC_{i}\\})\wedge(o_{i}\in\\{RC_{i}\\})\Rightarrow
u_{i}[F_{d_{i}}(O_{d_{i}})]\neq\Phi$

{RC} = set of roles which are authorized to perform functions
$\\{F_{d_{i}}()\\}$ on object $\\{O_{d_{i}}\\}.$
However,

$(r_{i}\notin\\{RC_{i}\\})\wedge(o_{i}\notin\\{RC_{i}\\})\Rightarrow
u_{i}[F_{d_{i}}(O_{d_{i}})]=\Phi$

{CRC} = set of competing roles classes which are not authorized to perform
functions $\\{F_{d_{i}}()\\}$ on the objects $\\{O_{d_{i}}\\}$ of their
competing class.

###### Definition 5

A Wall is build on top of the statement ”Who can access what?” in a secure
manner and implemented as the object and subject walls. Based on the conflict-
class, we created the binary wall that decidedly separate the conflicting
subjects and objects from each others. The walls within a system are defined
as follows:

{OW} = $\\{\forall{(r_{i},r_{j})}\in R^{2}\mid RC_{i}(r_{i},r_{j})=1\\}$

Set of object walls (OW) is the set of roles $(r_{i},r_{j})\in R^{2}$ such
that $RC_{i}(r_{i},r_{j})=1$ whereas, the $\overline{RC_{i}(r_{i},r_{j})}=0$

{SW} = $\\{\forall{(s_{i}\in r_{i},s_{j}\in r_{j})}\mid
RC_{i}(s_{i},s_{j})=1\\}$

Set of subject walls (SW) is the set of subjects that belong to their relative
roles/ roles classes ${(s_{i}\in r_{i},s_{j}\in r_{j})}$ such that
$RC_{i}(s_{i},s_{j})=1$ whereas, the $\overline{RC_{i}(s_{i},s_{j})}=0$

###  IV-A Object Set Binary Wall (obj)

Object set denotes to the set of all objects, where each object $obj_{i}$
belongs to $RC_{i}$ and associated with two subsets of RC:

$OWRC_{obj_{i}}\subseteq RC_{i}$ $OWCRC_{obj_{i}}\subseteq CRC_{i}$

Simply $OWRC_{i}$ is the set of roles classes, where they have an access right
to the object $obj_{i}$. $OWCRC_{i}$ is the set of conflicting roles classes,
denied to be accessed to the object $obj_{i}$. Therefore, we can represent
each pairwise $\\{OWRC_{obj},OWCRC_{obj}\\}$ by a binary pairwise of arrays
$\\{BinOWRC_{obj},BinOWCRC_{obj}\\}$, named it a binary object’s wall. Each
array has a size of n bits (the name of roles classes). Bits j between 1 and n
are initialized as following:

$BinOWRC_{obj}$ is 1 if $RC_{i}\in OWRC_{obj}$, otherwise 0. $BinOWCRC_{obj}$
is 1 if $RC_{i}\in OWCRC_{obj}$, otherwise 0.

The once operation authorised on bits, is to change value from 0 to 1.

###  IV-B Subject Set Binary Wall (sub)

Subject set denotes to the set of all subjects, where each subject $S_{i}$
associated with two subsets of roles class (RC).

$SWG_{S_{i}}\subseteq RC_{i}$ $SWD_{S_{i}}\subseteq RC_{i}$

Simply $SWG_{i}$ is the access granted set. It is a set of roles classes have
similar objects inside the subject wall of $S_{i}$. $SWD_{i}$ is the denied
set. It is a set of roles classes denied to will be access by the subject
$S_{i}$. For the binary wall creation around the subjects, we can represent
each pairwise $\\{SWG_{s},SWD_{s}\\}$ by a binary pairwise of arrays
$\\{BinSWG_{s},BinSWD_{s}\\}$, named it a binary subject’s wall. Each array
has a size of n bits (the number of roles classes). Bit j between 1 and n are
initialized as following:

$BinSWG_{s}$ is 1 if $RC_{i}\in SWG_{s}$, otherwise 0 $BinSWD_{s}$ is 1 if
$RC_{i}\in SWD_{s}$, otherwise 0

The once operation authorised on bits, is to change value from 0 to 1.

###### Definition 6

Access right is a set of operations (read, write, delete etc.) that a role has
to perform on domain objects is represented in the form of Table I. We
considered the two access rights in our system that are read and write.
However, only the authorized user can perform their assigned operations on
domain objects after their access verification and the walls will be updated
according to the operations explained in section V.

TABLE I: Access Rights | $R_{1}$ | $R_{2}$ |  | $\cdots$ |  | $R_{i}$
---|---|---|---|---|---|---
$Obj_{1}$ |  {r, w} |  {r} |  | $\\{\cdots\\}$ |  | $\\{\cdots\\}$
$Obj_{2}$ |  {w} | $\\{\emptyset\\}$ |  |  |  |
$\vdots$ |  |  |  |  |  |
$Obj_{i}$ |  |  |  |  |  | $\\{\cdots\\}$

##  V Security Check Point

Security check point is based on the data access authorization and subject and
object binary walls updates.

###  V-A Access Authorization

Subject is granted to access the object, if and only if the following access
authorisation condition is verified:

$SWG_{i}\cap OWCRC_{j}=\emptyset$ AND $SWD_{i}\cap OWRC_{j}=\emptyset$

According to the binary walls of subject and object, the access condition is
mapped as following:

$BinSWG_{s}\land BinOWCRC_{o}=\\{00…00\\}$ AND  $BinSWD_{s}\land
BinOWRC_{o}=\\{00…00\\}$

The access is denied, if the objects (data warehouse) of competing roles find
inside the same wall.

###  V-B Wall Updating

There is no need to update the walls if the access is denied. However, if the
subject is granted to access the object the binary walls of subject and object
will be updated immediately according to the access type (reading or writing).

####  V-B1 Read from an object

If the subject is granted to read from an object, the binary subject wall will
be update as following:

$BinSWG_{s}=BinOWRC_{o}\lor BinSWG_{s}$ $BinSWD_{s}=BinOWCRC_{o}\lor
BinSWD_{s}$

####  V-B2 Write to an object

If the subject is granted to write to an object, the binary object wall will
be update as following:

$BinOWRC_{o}=BinOWRC_{o}\lor BinSWG_{s}$ $BinOWCRC_{o}=BinOWCRC_{o}\lor
BinSWD_{s}$

##  VI Data Management Module

In this section, we will present the data storage module by taking into
account the privacy mechanisms as shown in Figure 3.

Figure 3: Data Privacy Management

###  VI-A Sensitivity Level and Protection of Data

There are four common terms that represent the nature of attributes present in
the dataset on the basis of sensitivity level and relation with entity are;
identifiable, Quasi, sensitive and insensitive attributes. Preserving the
privacy of a piece of data that is sensitive, identifiable, or quasi-
identifiable is a critical challenge. Whereas, de-identification and
anonymisation of data maintains a low confidence threshold for linking the
sensitive information to an individual [21]. The most adoptable privacy models
are; k-anonymity, l-diversity, (X,Y)-anonymity, LKC-privacy, t-closeness etc
can be implemented using various anonymisation operations, such as noise
addition, generalisation, shuffling, and perturbation[22] [23] [24].

###  VI-B Confidentiality Measurement and Data Transformation

According to the common laws of confidentiality or data protection (Human
Rights Act 1998 and Data Protection Act 1998), private data cannot be
disclosed without explicit consent, legal requirement and public interest.
However, these laws are based on trust, which can be easily spoiled by
malicious insiders with legitimate access to confidential data. Therefore, we
required data confidentiality or transformation that can prevent an
individual’s personal and sensitive information from someone with legitimate
access to data. Data confidentiality and transformation functions within the
system are defined as follows:

Data Confidentiality = $\alpha\hskip 14.22636pt\therefore 0<=\alpha<=1$

The range of $\alpha$ from 0 to 1 measures the level of data confidentiality.
This clearly indicates that $\alpha$ = 1 is fully confidential data whereas,
$\alpha$ = 0 is less confidential and fully anonymised data.

Data Transformation $(F_{T}())=\\{F_{D_{1}}\cdots F_{D_{n}},F_{A_{1}}\cdots
F_{A_{m}}\\}$ $F_{T}(Obj_{i})_{\alpha_{x}}=(Obj_{j})_{\alpha_{y}}\hskip
14.22636pt\therefore\alpha_{y}<\alpha_{x}$

Data transformation $(F_{T}())$ is the set of de-identification $(F_{D})$ and
anonymisation $(F_{A})$ techniques that decreases the confidentiality level
and transforms the original data into anonymised data.

####  VI-B1 Data De-identification

Firstly, we consider the original data records as an object of the
organization that contains the identifiable and sensitive information of an
individual along with other attributes. This data is to the core confidential
and must be accessible to the minimum number of roles. The Original Data
within the system is defined as follows:

$Obj_{1}=OD_{{\alpha}_{x}}\hskip 14.22636pt\therefore\alpha_{x}=1$ (Fully
Confidential Data)

Secondly, after applying the de-identification function $(F_{D})$, this
original data is converted into a de-identified data object. De-identification
decreases the confidentiality level of the OD.

$F_{D}(OD)_{\alpha_{x}}=(DD)_{\alpha_{y}}\hskip
14.22636pt\therefore\alpha_{y}<\alpha_{x}$ $Obj_{2}=(DD)_{\alpha_{y}}\hskip
14.22636pt$ (Less Confidential Data)

####  VI-B2 Data Anonymisation

Thirdly, we applied anonymisation techniques on the de-identified data so that
the identity of an individual cannot be revealed through linkage attacks. This
data warehouse is fully anonymised and their level of confidentiality is
$\approx 0.$

$F_{A}(DD)_{\alpha_{y}}=(AD)_{\alpha_{z}}\hskip
14.22636pt\therefore\alpha_{z}\approx 0$ $Obj_{3}=(AD)_{\alpha_{z}}\hskip
14.22636pt$ (Fully Anonymised Data)

##  VII Case Study: Model Illustration in Healthcare

Adopting the advanced technology for Electronic Health Records (EHR) not only
revolutionizes the way to treat diseases but also empowers many other sectors
like; insurance companies, law enforcement agencies, pharmaceutical, and other
product-selling companies etc. Based on the sensitivity, usability and
multiple access of EHR, we select the healthcare sector for model
illustration.

###  VII-A Data Objects

The defined data warehouses (ODW, DDW and ADW) are considered as the objects
of healthcare organisation and must be protected from unauthorised access.

###  VII-B Roles and Their Responsibilities

According to the primary functions of the data science life-cycle we define
certain roles and grant them access permissions regarding their
responsibilities. The defined roles and their responsibilities are explained
as follows (Role = R):
R1: Data Collector: Is responsible for the acquisition of patients’ data from
different sources like; IoT devices, sensors, patients’ health documents,
physically attending patients etc. Data collection tasks can be performed by
using mobile apps, web services etc there is no need to access any data
warehouse.
R2: Data Integration Officer: Key concerns refer to the data quality, design
and implementation of data integration applications. This will be responsible
for the integration of data types and formats into a single location known as
the ODW by maintaining the data quality.
R3: Data Privacy Officer: This specific role within the data management system
is required for data protection. Their key responsibility is to apply data de-
identification and anonymisation functions on ODW.
R4: Data Controller: involves managing the data about data (metadata), whereby
this “other data” is generally referred to data models and structures, not the
content. This role is ultimately accountable with regard to the definition,
data quality and value of data in a given subject area.
R5: Data Analyst: By accessing the de-identified data, data analyst is
responsible to structure the raw data, formulate or recognise the various
patterns in the data through the mathematical and computational algorithms,
and by using the standard statistical tools to analyse and interpret the data.
R6: Data Scientist: Apart from the work done by the data analyst, the data
scientist will perform model creation, designing the new algorithms that can
solve the specific research problems of the organisation etc. This role can
collaborate with similar roles of other organisations in order to provide
insights about business performance and support decision-making via sharing
anonymised data.
R7: End Users (Patients and Doctors etc:) End users are not directly involved
in the functions of the data management system but they can access the
individual record through the interfaces defined by the healthcare
organisation such as; mobile or web services, in-person services etc.

$\\{U_{i}\\}\in[R1,R4,R7]\mid U_{i}[F_{d_{i}}(ODW,DDW,ADW)]=\Phi$

###  VII-C Equivalent Roles Classes and Binary Walls Creation

We assigned the roles to their particular class with respect to the access
privileges they have on data objects.Table II clearly represents the model
elements.

####  VII-C1 Data Management Class (DMC)

Data Management Class contains the number of roles who can access the object
(ODW) of this class. $[DMC]=\\{R2,R3\\}$

####  VII-C2 Data Analyst Class (DAC)

Data Analyst Class carry the number of roles who can access the ”De-identified
Data Warehouse (DDW)” to perform their responsibilities. $[DAC]=\\{R5\\}$

####  VII-C3 Data Scientist Class (DSC)

Data Scientist Class holds the number of roles who can fulfill their
responsibilities by accessing the object ADW. $[DSC]=\\{R6\\}$
As their access privileges to the data warehouses, DMC, DAC, and DSC are
conflicting with each others.

TABLE II: Conflict of Interest Relation and Walls Representation

Equivalent Roles Classes |  Conflicting Roles Classes |  Access Objects |  Binary Object Wall {BinOWERC, BinOWCRC} |  Subjects |  Binary Subject Wall {BinSWG, BinSWD}
---|---|---|---|---|---
DMC |  {DAC, DSC} |  ODW |  $\\{100,011\\}$ |  S1 |  $\\{100,011\\}$
DAC |  {DMC, DSC} |  DDW |  $\\{010,101\\}$ |  S2 |  $\\{010,101\\}$
DSC |  {DMC, DAC} |  ADW |  $\\{001,110\\}$ |  S3 |  $\\{001,110\\}$

###  VII-D Illustration Queries

We have following sequence queries to validate our system. All the queries are
performed on binary walls.

Q1: Subject S1 needs to perform writing operation on the object ODW (original
data warehouse).
S1 can access their required object after verifying the access authorisation
condition.

$\\{100\\}\land\\{011\\}=\\{000\\}$ AND $\\{011\\}\land\\{100\\}=\\{000\\}$

After the access granted to S1 and the write operation is performed on ODW,
object walls will be updated as:

$BinOWERC_{ODW}=\\{100\\}\lor\\{100\\}=\\{100\\}$
$BinOWCRC_{ODW}=\\{011\\}\lor\\{011\\}=\\{011\\}$

Q2: Subject S3 wants to perform reading operation on the object DDW. Their
access is denied, because the conflicting subjects cannot access the
conflicting objects. However, there is no need to update the walls because of
access denied.

$\\{001\\}\land\\{101\\}\neq\\{000\\}$ AND
$\\{110\\}\land\\{010\\}\neq\\{000\\}$

##  VIII Conclusion

As the today’s world is knowledge economy, where the more you know is
proportional to more data that you have. In a typical data owning
organizational structure, there are number of roles/actors to whom the
individuals information at each data architectural layer is accessible and the
risks associated with these insiders are far more difficult because of their
legitimate access to the system. Hence, we need to formulate a data privacy
ensuring system to protect the identity and sensitivity of data not only from
external but also from internal malicious intruders. This paper first
regulated the actors/users of the organisation by adopting the strategy of
RBAC. Secondly, the concept of CWSP is implemented as a firewall mechanisms to
restrict the unauthorised access of users. Lastly, we proposed a comprehensive
data management architecture composed of necessary components of data de-
identification, and anonymisation. Therefore, our model is considered as
useful and have significance in the sense that it provides security and
privacy along each functional layer of data architecture. Furthermore, this
system is also not subjected to faulty human intentions because it is an
automatic system that impose limits on the administrators’ roles with respect
to regulating the accessibility of data and exercising their authority in a
malicious manner. Along with this issue, it also encounters with the risk of
mistakes by administrators.

##  IX Acknowledgment

This work is supported by Science Foundation Ireland under grant number
$SFI/12/RC/2289_{P}2-$ Insight Centre for Data Analytics.

## References

  * [1] T. Fisher, _The data asset: How smart companies govern their data for business success_. John Wiley & Sons, 2009\.
  * [2] K. E. Greenaway, Y. E. Chan, and R. E. Crossler, “Company information privacy orientation: a conceptual framework,” _Information Systems Journal_ , vol. 25, no. 6, pp. 579–606, 2015.
  * [3] Y. Chabot, A. Bertaux, C. Nicolle, and T. Kechadi, “An ontology-based approach for the reconstruction and analysis of digital incidents timelines,” _Digital Investigation_ , vol. 15, pp. 83–100, 2015.
  * [4] N. Van der Meulen, “The challenge of countering identity theft: Recent developments in the united states, the united kingdom, and the european union,” _Report Commissioned by the National Infrastructure Cyber Crime program (NICC). Retrieved May_ , vol. 20, p. 2007, 2006.
  * [5] C. L. Huth, D. W. Chadwick, W. R. Claycomb, and I. You, “Guest editorial: A brief overview of data leakage and insider threats,” _Information Systems Frontiers_ , vol. 15, no. 1, pp. 1–4, 2013.
  * [6] L. Zhou, V. Varadharajan, and M. Hitchens, “Achieving secure role-based access control on encrypted data in cloud storage,” _IEEE transactions on information forensics and security_ , vol. 8, no. 12, pp. 1947–1960, 2013.
  * [7] A. Shahid, T.-A. N. Nguyen, and M.-T. Kechadi, “Big data warehouse for healthcare-sensitive data applications,” _Sensors_ , vol. 21, no. 7, p. 2353, 2021.
  * [8] T.-A. Nguyen, N.-A. Le-Khac, M. Kechadi _et al._ , “Privacy-aware data analysis middleware for data-driven ehr systems,” in _International Conference on Future Data and Security Engineering_. Springer, 2017, pp. 335–350.
  * [9] N. H. Tran, T.-A. Nguyen-Ngoc, N.-A. Le-Khac, M. Kechadi _et al._ , “A security-aware access model for data-driven ehr system,” _arXiv preprint arXiv:1908.10229_ , 2019.
  * [10] B. Blobel, “Authorisation and access control for electronic health record systems,” _International journal of medical informatics_ , vol. 73, no. 3, pp. 251–257, 2004.
  * [11] F. H. R. France, “Security of health care records in belgium: Application in a university hospital,” _International Journal of Medical Informatics_ , vol. 73, no. 3, pp. 235–238, 2004.
  * [12] M. Van der Haak, A. C. Wolff, R. Brandner, P. Drings, M. Wannenmacher, and T. Wetter, “Data security and protection in cross-institutional electronic patient records,” _International journal of medical informatics_ , vol. 70, no. 2-3, pp. 117–130, 2003.
  * [13] G. H. Motta and S. S. Furuie, “A contextual role-based access control authorization model for electronic patient record,” _IEEE Transactions on information technology in biomedicine_ , vol. 7, no. 3, pp. 202–207, 2003.
  * [14] X. Xie, I. Ray, R. Adaikkalavan, and R. Gamble, “Information flow control for stream processing in clouds,” in _Proceedings of the 18th acm symposium on access control models and technologies_ , 2013, pp. 89–100.
  * [15] R. Wu, G.-J. Ahn, H. Hu, and M. Singhal, “Information flow control in cloud computing,” in _6th International Conference on Collaborative Computing: Networking, Applications and Worksharing (CollaborateCom 2010)_. IEEE, 2010, pp. 1–7.
  * [16] S. Fehis, O. Nouali, and M.-T. Kechadi, “A new distributed chinese wall security policy model,” _Journal of Digital Forensics, Security and Law_ , vol. 11, no. 4, p. 11, 2016.
  * [17] S. Fehis, O. Nouali, and T. Kechadi, “A new chinese wall security policy model based on the subject’s wall and object’s wall,” in _2015 first international conference on anti-cybercrime (ICACC)_. IEEE, 2015, pp. 1–6.
  * [18] ——, “Secure encryption key management as a secaas based on chinese wall security policy,” _Journal of Information Security and Applications_ , vol. 63, p. 102975, 2021.
  * [19] R. Sandhu, D. Ferraiolo, R. Kuhn _et al._ , “The nist model for role-based access control: towards a unified standard,” in _ACM workshop on Role-based access control_ , vol. 10, no. 344287.344301, 2000.
  * [20] D. F. Brewer and M. J. Nash, “The chinese wall security policy.” in _IEEE symposium on security and privacy_ , vol. 1989. Oakland, 1989, p. 206.
  * [21] B. C. Fung, K. Wang, A. W.-C. Fu, and S. Y. Philip, _Introduction to privacy-preserving data publishing: Concepts and techniques_. Chapman and Hall/CRC, 2010.
  * [22] L. Sweeney, “k-anonymity: A model for protecting privacy,” _International journal of uncertainty, fuzziness and knowledge-based systems_ , vol. 10, no. 05, pp. 557–570, 2002.
  * [23] A. Machanavajjhala, D. Kifer, J. Gehrke, and M. Venkitasubramaniam, “l-diversity: Privacy beyond k-anonymity,” _ACM Transactions on Knowledge Discovery from Data (TKDD)_ , vol. 1, no. 1, pp. 3–es, 2007.
  * [24] A. Faridoon and M. T. Kechadi, “Data privacy in its three forms–a systematic review,” in _International Conference on Future Data and Security Engineering_. Springer, 2020, pp. 425–433.


11institutetext: Kenta Izumi, Hiroki Tanaka, Kazuhiro Shidara and Satoshi
Nakamura 22institutetext: Nara Institute of Science and Technology, Takayama-
cho 8916-5, Ikoma-shi, Nara, Japan.
22email: {izumi.kenta.if6, hiroki-tan, shidara.kazuhiro.sc5, s-nakamura
<EMAIL_ADDRESS>33institutetext: Hiroyoshi Adachi, Daisuke Kanayama, and Takashi
Kudo 44institutetext: Health and Counseling Center, Osaka University,
Yamadaoka 1-1, Suita-shi, Osaka, Japan.

# Response Generation for Cognitive Behavioral Therapy with Large Language
Models: Comparative Study with Socratic Questioning

Kenta Izumi  Hiroki Tanaka  Kazuhiro Shidara  Hiroyoshi Adachi  Daisuke
Kanayama  Takashi Kudo  and Satoshi Nakamura

###### Abstract

Dialogue systems controlled by predefined or rule-based scenarios derived from
counseling techniques, such as cognitive behavioral therapy (CBT), play an
important role in mental health apps. Despite the need for responsible
responses, it is conceivable that using the newly emerging LLMs to generate
contextually relevant utterances will enhance these apps. In this study, we
construct dialogue modules based on a CBT scenario focused on conventional
Socratic questioning using two kinds of LLMs: a Transformer-based dialogue
model further trained with a social media empathetic counseling dataset,
provided by Osaka Prefecture (OsakaED), and GPT-4, a state-of-the art LLM
created by OpenAI. By comparing systems that use LLM-generated responses with
those that do not, we investigate the impact of generated responses on
subjective evaluations such as mood change, cognitive change, and dialogue
quality (e.g., empathy). As a result, no notable improvements are observed
when using the OsakaED model. When using GPT-4, the amount of mood change,
empathy, and other dialogue qualities improve significantly. Results suggest
that GPT-4 possesses a high counseling ability. However, they also indicate
that even when using a dialogue model trained with a human counseling dataset,
it does not necessarily yield better outcomes compared to scenario-based
dialogues. While presenting LLM-generated responses, including GPT-4, and
having them interact directly with users in real-life mental health care
services may raise ethical issues, it is still possible for human
professionals to produce example responses or response templates using LLMs in
advance in systems that use rules, scenarios, or example responses.

##  1 Introduction

The number of people with mental disorders worldwide is estimated to be around
970 million in 2019, having increased by 25% since 2000 with the growth of the
world population GBDmentaldisorder . In particular, anxiety disorders and
depression accounted for approximately 60% of the prevalence of mental
disorders in 2019 GBDprevalence . Under these circumstances, qualified
psychiatrists and other therapists remain in short supply, and thus the
growing number of patients who are unable to receive the care they need has
become a major concern ShortageOfPsychiatry ; BrainDrain . On the other hand,
with the spread of smartphones, various mobile health apps have been
developed, and it’s expected that they will help to provide immediate care to
people who have difficulty accessing needed healthcare mHealthGap ;
mHealthGrowth . Among other solutions, dialogue systems such as Wysa
inkster2018empathy , which encourages users to build cognitive resilience
through empathetic responses based on counseling techniques such as cognitive
behavioral therapy (CBT) beck1979cognitive , are expected to be useful in the
form of mental healthcare apps. Toward this approach, a variety of dialogue
systems have been explored so far vaidyam2019chatbots ; abd2019overview ;
shidara2022automatic ; akiyoshi2021robot .

A review of the literature up to 2019 reported that most of these systems
generate responses based on pre-defined rules, and thus system utterances are
commonly described as inadequate, e.g., “conversations are shallow, confusing
or too short.” Accordingly, the adoption of natural language processing
technology is expected to improve the quality of dialogue abd2021perceptions .
Recently, natural language understanding and generation technology for large
language models (LLMs), as exemplified by the 2022 release of OpenAI’s ChatGPT
chatGPT , has developed rapidly, and CBT dialogue systems that use high-
performance LLMs to generate responses have also been researched and developed
danieli2022assessing ; liu2023chatcounselor . Most previous apps had been
controlled by predefined or rule-based scenarios, mainly due to technological
limitations. Nevertheless, they had the advantage of providing responsible
responses, compared to the LLM-generated responses that the developers cannot
fully control and predict.

To the best of our knowledge, no study has examined how the use of LLM-based
response generation changes the effectiveness of CBT and user experiences
compared to conventionally adopted simple scenario or rule-based systems. We
examined whether the user’s subjective evaluation of the system was improved
by replacing or combining the Socratic questions in a scenario with responses
generated by LLMs.

##  2 Methods

In this study, dialogue scenarios were created with reference to the CBT
dialogue scenarios centered on Socratic questions, considered important in
CBT, as used in Shidara et al. shidara2022automatic . We conducted an
experiment to investigate the effect on the user’s mood and cognitive change
persons1985mechanisms ; schmidt2019putting , which are indices for measuring
the effectiveness of CBT, and other dialogue qualities when the Socratic
questions in a scenario were replaced or combined with responses generated by
an LLM.

###  2.1 Dialogue scenarios

In this study, scenarios consisting of 15 system utterances were created for
the CBT-based dialogue scenario shidara2022automatic  (Table 1). In CBT, when
a certain mood arises, the thoughts that occur automatically, such as
assumptions and interpretations of the situation, are called automatic
thoughts. Patients are encouraged to modify these into more balanced and
adaptive thoughts beck2020cognitive . In the scenario, the system first asks
the user about the situation that is troubling him/her, then asks about the
mood that arises when she/he thinks about it and its intensity, and finally
asks about the thoughts that occur at that time and puts the strongest
thoughts on the agenda as automatic thoughts (utterance numbers 1-7). This
part can be considered questions asking about ABC: A) Activating event, B)
Beliefs, and C) Consequences, which is a basic model in CBT ellis1991revised .
Then, six turns of Socratic questions are asked to encourage the user to
objectively view and reconsider his/her automatic thoughts (utterance numbers
8-13). In addition, the user is again asked about the intensity of her/his
current mood and the scenario ends with a parting expression (utterance
numbers 14 and 15).

Socratic questioning is a questioning technique that allows the therapist to
explore the client’s thoughts and beliefs and to help the client think
critically about them, broaden their perspective, and arrive at a more
realistic and adaptive perspective. This technique has been considered an
important component of CBT for depression vittorio2022using .

Table 1: Dialogue scenario for the dialogue system in SQ condition. Question
items and question order are the same in all experiments. Only questions in
bold differ from the LLM condition; the conditions with LLMs use only
generated sentences or combinations of a Socratic question and a generated
sentence.

| Question item | System’s response
---|---|---
1 | Situation |  Let’s address your concerns together. Are you currently facing any issues or difficulties? If you feel comfortable doing so, please share what’s troubling you.
2 | Mood | How did you feel at that time?
3 |  Initial mood score |  On a scale of 0 to 100, where 0 is no problem at all and 100 is a huge problem, how intense is that feeling?
4 |  Automatic thought | What thoughts came to your mind at that time?
5 |  Automatic thought | I see. Are there any others? Please share anything that comes to mind.
6 |  Automatic thought | Please tell me the strongest thought among those that came to your mind.
7 |  Automatic thought |  The thoughts that come to mind in response to a situation are called automatic thoughts. Today we are going to delve into your thought. Is that okay?
8 |  Socratic question 1 | If the automatic thought is correct, what is the basis for it?
9 |  Socratic question 2 | On the other hand, if your automatic thought is incorrect, what evidence can you think of?
10 |  Socratic question 3 |  Can you think of any other perspectives regarding the situation that’s troubling you, aside from your current automatic thoughts?
11 |  Socratic question 4 | What would be the worst possible outcome in this situation?
12 |  Socratic question 5 | What would be the best possible outcome in this situation?
13 |  Socratic question 6 | What do you think is the most realistic outcome in this situation?
14 |  Mood score at the end |  Now, how intense is the feeling you initially had? Please rate it again on a scale from 0 to 100.
15 |  Closing remark |  If your mood has changed, that’s a sign that you’ve successfully organized your thoughts. That’s it for today. Good job!

In this study, we implemented the following three types of dialogue systems,
where utterances 8-13 of the scenarios were varied according to the system: 1)
asking Socratic questions as in the scenario, 2) uttering the responses
generated by the LLMs, and 3) combining LLM-generated responses with Socratic
questions. We focus on how each dialogue system can help people rethink their
automatic thought, arrive at a more adaptive and balanced thought, and thus
improve their mood after asking ABC questions (in utterance numbers 1-7). In
particular, the dialogue flow of system 2, where the system asks ABC questions
and utters generated responses after that, is roughly similar to TEO (Therapy
Empowerment Opportunity) used in Danieli et al. danieli2022assessing ;
mousavi2021Wouldyoulike . We expected that adding Socratic questions to LLM-
generated responses would help the systems encourage the user to rethink their
automatic thought and not deviate from the context of CBT. In each of the
latter two systems, which generate responses by LLMs, the following two LLMs
were used to implement the dialogue systems: Osaka empathetic counseling
dialogue model (OsakaED) and GPT-4. We chose these two LLMs because they have
different characteristics from each other: OsakaED specializes in counseling
in social media, and GPT-4 is a general-purpose state-of-the-art model. In
this way, five dialogue systems were implemented, each with different
utterances 8-13 of the dialogue scenario. Socratic questions are represented
by SQ and OsakaED, and when Socratic questions are added to the utterances of
each model, they are connected by “+” to form the system name.

Table 2: List of dialogue systems compared in this study.

System name |  Description
---|---
SQ |  Socratic questions based on the scenario are uttered
OsakaED |  Response generated by Osaka social media empathetic dialogue model are uttered
OsakaED+SQ |  Socratic questions based on the scenario are added to the responses generated by Osaka social media empathetic dialogue model and uttered
GPT-4 |  Response generated by GPT-4 are uttered
GPT-4+SQ |  Socratic questions based on the scenario are added to the responses generated by GPT-4 and uttered

#### OsakaED

OsakaED was developed by further fine-tuning the Japanese Transformer Encoder-
Decoder dialogue model with a dataset provided by Osaka Prefecture. This
model, created by Sugiyama et al. sugiyama2023empirical , had already been
fine-tuned with empathic dialogue data and possesses 1.6 billion parameters.
We chose this model as a base model because it’s trained with text chat
conversations via the internet and thought to already have empathetic response
generation ability in a form similar to human speech on social media. Osaka
social media counseling dataset consists of 2077 dialogues and 62,066
utterances of counselors counseling their clients using the messenger
application LINE111https://line.me/. From these, 29,062 pairs of context and
counselor utterances were extracted so as not to include utterances consisting
only of system utterances, which are mainly placed at the beginning of the
dialogues. In training, 90% of the data were training data and 10% were
validation data, with a batch size of 4, cross-entropy as the objective
function, and Adafactor as the optimizer. Here, 11 epochs were trained with a
learning rate of $1.0\times 10^{-5}$, and the validation perplexity used the
model from the 6th epoch with the lowest validation perplexity. Because of the
limitation of the input length, the 1 to 7 user utterances are combined and
are treated as one utterance, which represents an ABC note, and the system
utterance 8 was generated to follow it. System utterances 9 to 13 were
generated by entering the ABC notes and subsequent system and user utterances
as history. The input preprocessing and response generation method was
basically the same as Sugiyama’s implementation sugiyama2021empirical , which
sampled five response candidates by top-p sampling with $p=0.90$ and ranked
them. In the dialogues in the pre-experiment, keywords that were included in
the generated responses, which attempted to end or guide the dialogues
specific to the dataset and were considered inappropriate, were observed and
set as NG (No Good) words. The highest rank response not containing any of NG
words was selected as the response. When NG words are included in all response
candidates, an error message is output, and in this case, the sample is not
treated as valid in the subjective evaluation. In addition, when Socratic
questions were added, the response with the highest rank not containing any of
NG words and not ending with a question was selected to avoid duplicating
questions. If all response candidates were questions, the response with the
highest rank among the candidates that did not contain any of the NG words was
selected as the response.

#### GPT-4

GPT-4 is a large-scale language model developed by OpenAI that shows human-
level performance on tasks across a wide range of domains, including
mathematics, coding, and medicine bubeck2023sparks . We used GPT-4 through
OpenAI’s Chat Completions API. The Chat Completions API pairs a role “role”
with its message content “content” in a dictionary format, and then compiles
these multiple dictionaries into a list in the order of utterances, which is
input as “messages”. If “user” is specified as the “role”, the “content” is
the content of the message from the user (user message), and if “assistant” is
specified, it is the speech generated by GPT-4 (assistant message). when
“system” is specified, it describes how the assistant works in the whole
conversation (system message), such as instructions to the assistant and
his/her personality. Usually, the system message is placed at the beginning of
the history, followed by the user message and the assistant message
alternating openaiapi . In this study, we followed this method to input
messages.

Kumar et al. explored prompt design for mental health chatbots using GPT-3,
focusing on the three dimensions of identity, intent, and behavior, with two
identities and three intents and behaviors, respectively, for a total of 18
different dialogue systems, and thus 18 prompts were created and evaluated.
The prompts with “coach” as the identity and “strong interpersonal skills” as
the behavior received the highest overall evaluation among the prompts with
intent for CBT kumar2022exploring , and it was adopted as the system message.
In this study, the system messages shown in Table 3, described below, were
placed first, and the dialogue history, including subsequent user and system
utterances, was included in the messages as much as possible to generate
responses as input to GPT-4. Temperature was set to 1 the same as the default
setting. When adding a Socratic question to the generated response, as shown
in Table 3, we have “the coach does not ask questions.” This sentence was
inserted after the first sentence to discourage the respondent from
duplicating the question.

Table 3: System prompts for GPT-4 and GPT-4+SQ, based on Kumar et al. kumar2022exploring . The prompts translated into Japanese without indentation were used in the experiment. System | System prompt
---|---
GPT-4 |  | The following is a conversation with a coach.
---
The coach helps the Human understand how their thoughts, feelings,
and behaviors influence each other.
If the Human demonstrates negative thoughts,
the coach helps the Human replace them with more realistic beliefs.
The coach has strong interpersonal skills.
GPT-4+SQ |  | The following is a conversation with a coach.
---
The coach does not ask questions.
The coach helps the Human understand how their thoughts, feelings,
and behaviors influence each other.
If the Human demonstrates negative thoughts,
the coach helps the Human replace them with more realistic beliefs.
The coach has strong interpersonal skills.

Example responses from GPT-4 and OsakaED are shown below. Both examples
consist of the system utterances 8 and 9 and the user utterance 8, in the
context when the user consults about skin trouble and acne.

GPT-4 Thank you. Let’s dig deeper into the thoughts that arise from that “I
want to heal fast” feeling. How do you feel that thought is affecting you? How
does it affect your behavior and daily life? User I check when looking in the
mirror and pay attention to the skin and diet. GPT-4 Your automatic thoughts
strongly express your desire to “heal quickly”. This thought prompts you to
take action and make it easier for you to try to live a healthier life. You
once thought, “Why can’t I get well?” but in fact, that thought may not be in
alignment with reality. I would like to propose a new way of looking at it.
You can help yourself more by looking at it from a realistic perspective, as
in, “I have control over my skin condition,” and “I can improve it over time
if I take proper care of my skin problems.” What do you think about this new
view?

OsakaED It’s hard to have skin problems and acne that won’t go away for a long
time… User Yes, it is. It’s hard. OsakaED Why is it that skin problems and
acne don’t go away all the time…

As shown in these examples, GPT-4 tends to help the user explore their
thoughts in a longer text, while OsakaED show empathy in a shorter text.

###  2.2 Subjective evaluation

A subjective evaluation experiment was conducted using a crowdsourcing
platform, CrowdWorks 222https://crowdworks.jp/, in which 105 crowd
participants (healthy participants) were gathered to evaluate our dialogue
systems. This experiment was conducted under the review and approval of the
research ethics committee of the Nara Institute of Science and Technology
(reference number: 2022-I-52). After reading the explanation of the experiment
and agreeing to participate in the experiment, the participants were asked to
provide their crowdsourcing ID, gender, age, and rating on the Kessler
Psychological Distress Scale (K6) furukawa2003performance , a widely used
measure of psychological distress, to be assigned to each participant at
CrowdWorks. They then read a leaflet about CBT written by a
psychiatrist333https://www.cbtjp.net/downloads/skillup/. The participants
reserved a time to perform a dialogue with the system, started the dialogue at
the reserved time, and ended the dialogue after confirming the last utterance
of the system (utterance number 15). The participants completed the experiment
by answering a questionnaire about cognitive change and dialogue quality, and
free-text descriptions about the strengths, weaknesses, and improvements of
the system. Throughout the experiment, participants were not informed of the
type of dialogue system with which they interacted.

In CBT, the goal is to improve mood by modifying automatic thoughts into
adaptive thoughts. In this experiment, following the work of Persons et al.
persons1985mechanisms , the mood change was calculated by an equation 1 and
then used as an index to evaluate the degree to which mood improved. Here,
$initial\,mood\,score$ and $mood\,score\,at\,end$ represent the intensities of
mood elicited by system utterances 3 and 14 in the scenario, respectively:

$Mood\,change=-\frac{\left(mood\,score\,at\,end\right)-\left(initial\,mood\,score\right)}{\left(initial\,mood\,score\right)}$
(1)

The Cognitive Change immediate scale (CC-immediate) schmidt2019putting  for
measuring immediate cognitive change with single-session CBT was translated
into Japanese and used to assess cognitive change due to interaction with the
system. The CC-immediate consists of six questions, each rated on a 7-point
Likert scale with 0 being “not at all” and 6 being “completely,” and the
responses to all questions were summed to form a 31-point scale from 0 to 30
to evaluate the immediate change in cognition.

Additionally, a questionnaire consisting of 15 items was used to evaluate the
quality of dialogue at the end of the conversation, which has two additional
questions to that created by Sugiyama et al. sugiyama2023empirical , based on
a survey for dialogue between dialogue systems and users conducted by
Fitrianie et al. fitrianie2020 . The Sugiyama et al. study originally included
an item called “empathy,” but because this item asked whether the user could
empathize with the system’s utterances, “I could empathize with the system’s
utterances,” it was changed to an item called user’s empathy, and a new item
called “system’s empathy” was added along with the question “the system showed
empathy” sugiyama2023empirical . This is because assessing the system’s
empathetic ability has been suggested as an effective way of soothing the
user’s mood and, more specifically, improving the outcome de2020effectiveness
; jani012roleofempathy . The item called “initiative” with the question “I was
able to lead the dialogue” was also added. It’s pointed out to be lost in
scenarios and rule-based dialogues in a study by Abd-Arazaq et al.
abd2021perceptions , and we assess this to confirm whether this quality is
improved by using LLMs and its importance for CBT. Continued usability is
measured by the item “re-speak,” which is measured by the question “I would
like to talk to this system again.” Each item was rated on an 11-point Likert
scale, with 0 representing “completely disagree” and 10 representing
“completely agree.” Mann-Whitney’s U-test was used to assess the difference in
scales between the two systems according to the particular interests of
comparison.

##  3 Results

The experiment resulted in a total of 86 valid responses. The number of users
for each system was 24 for SQ, 14 for OsakaED, 13 for OsakaED+SQ, 17 for
GPT-4, and 18 for GPT-4+SQ. The mean and standard deviation of K6 was 7.60 and
5.59, and no significant differences were seen in any two groups’ comparisons
($p>0.27$ in all comparisons). The results of subjective evaluations are
summarized in Table 3.

Table 4: Subjective evaluations of each system (mean and standard deviation).
Bold font represents the highest value in each scale.

|  SQ |  OsakaED |  OsakaED+SQ |  GPT-4 |  GPT-4+SQ
---|---|---|---|---|---
mood change |  0.28 (0.24) |  0.22 (0.16) |  0.19 (0.40) |  0.44 (0.26) |  0.49 (0.14)
CC-immediate |  16.6 (7.07) |  14.4 (5.83) |  16.9 (4.65) |  19.8 (4.93) |  18.8 (5.55)
naturalness |  6.25 (2.97) |  5.93 (2.56) |  6.54 (1.61) |  7.76 (2.19) |  6.56 (2.89)
ease |  7.25 (2.45) |  7.00 (2.94) |  8.31 (1.55) |  8.06 (1.43) |  6.83 (3.13)
enjoyment |  5.13 (2.98) |  5.36 (2.92) |  6.23 (2.24) |  6.53 (2.70) |  6.39 (2.68)
user’s empathy |  5.17 (2.62) |  5.07 (2.62) |  5.85 (1.72) |  7.76 (2.14) |  7.06 (2.10)
attentiveness |  5.79 (2.95) |  6.14 (3.01) |  7.15 (2.15) |  7.59 (2.29) |  7.56 (1.46)
trust |  5.63 (2.65) |  5.43 (2.90) |  5.77 (1.92) |  7.76 (2.19) |  7.44 (1.82)
personality |  3.96 (3.04) |  4.79 (2.91) |  5.00 (2.45) |  6.65 (2.94) |  6.33 (2.33)
agency |  4.29 (3.09) |  4.57 (2.82) |  4.85 (2.64) |  7.18 (2.38) |  6.22 (2.34)
topic |  5.54 (3.20) |  4.21 (2.12) |  5.54 (2.60) |  5.94 (2.84) |  5.72 (2.49)
emotion |  3.42 (2.93) |  4.29 (2.52) |  4.00 (2.38) |  6.41 (2.60) |  5.50 (2.71)
consistency |  8.04 (1.68) |  5.79 (2.83) |  6.77 (1.59) |  8.47 (1.42) |  8.00 (2.00)
engagement |  5.63 (3.21) |  5.43 (2.56) |  5.46 (2.50) |  7.06 (2.44) |  6.39 (2.33)
re-speak |  5.50 (3.12) |  5.93 (2.81) |  6.15 (2.44) |  6.65 (2.74) |  6.50 (2.98)
system’s empathy |  5.42 (2.62) |  6.93 (2.30) |  6.85 (2.70) |  7.94 (2.14) |  7.44 (2.31)
initiative |  4.21 (2.32) |  5.79 (1.85) |  5.46 (2.26) |  5.53 (2.60) |  4.94 (2.96)

An overview of Table 4 shows that GPT-4 received the highest ratings in 14
items except for mood change, ease, and initiative. For mood change, ease, and
initiative, GPT-4+SQ, OsakaED+SQ, and OsakaED received the highest ratings,
respectively. The two systems using GPT-4 received higher ratings than the
other systems for user’s emapthy, trust, personality, agency, emotion,
engagement, and system’s empathy, as well as for mood change, and CC-
immediate. These more emotionally rich and empathetic responses may have led
to improved outcomes such as mood change and CC-immediate. On the other hand,
the two systems using the OsakaED did not score as highly on many items as SQ,
although they did score highly on ease and initiative. The mood change and CC-
immediate did not show improvements compared to SQ, and it is not considered
that the outcome was improved.

In the following, we will analyze the results in detail by focusing on
specific points of view and comparing two groups.

###  3.1 Comparison between SQ and generated responses

To investigate the effect of replacing SQ with generated responses from LLMs,
the differences between SQ and OsakaED and between SQ and GPT-4 were assessed.
In the comparison between SQ and OsakaED, there were significant differences
in consistency and initiative: Consistency decreased ($p<0.05$), while
initiative increased ($p<0.05$). In the comparison between SQ and GPT-4, there
were significant differences in mood change and attentiveness ($p<0.05$), as
well as in user empathy, trust, personality, agency, emotion, and the system’s
empathy ($p<0.01$).

In scenarios where SQ was replaced with generated responses from OsakaED,
there was only one improvement in initiative. Considering that OsakaED
performed better in many of the items, including consistency and user’s
empathy, than with the reported result of the base model (model trained with
EmpatheticDialogue dataset) in the previous work sugiyama2023empirical , it’s
surprising that simply asking Socratic questions in six turns worked
competitively. The increase in initiative suggests that LLM responses may
prompt users to take more control or be more proactive in the interaction.
Conversely, the decrease in consistency indicates that OsakaED-generated
responses may be more variable or less predictable than SQ.

The significant differences observed when SQ was replaced with GPT-4 responses
in terms of mood change and attentiveness, as well as a range of user
experience dimensions, emphasize the sophisticated capabilities of GPT-4 in
affecting user perception, even though the provided prompt is not optimized
for this very well. The positive changes in mood, user and system empathy, and
emotion are particularly noteworthy, since they suggest that interactions with
GPT-4 can lead to an emotionally satisfying user experience. However, in many
of these improvements, re-speak is not significantly improved ($p=0.23$). For
long-time use, this dimension is believed to be important, and thus further
consideration should be given to it. Additionally, despite the prompt being
designed to discourage duplication of questioning for GPT-4+SQ, it was
observed that responses generated by both GPT-4 and GPT-4+SQ frequently
involved Socratic Questions. An annotator assigned to analyze these responses
reported that they consistently involved Socratic Questions in all dialogues.
This indicates that GPT-4 not only understands Socratic Questions but also has
the capability to appropriately ask them in context.

###  3.2 Comparison between SQ and generated responses with SQ

To investigate the effect of adding generated responses from LLMs to SQ, the
differences between SQ alone and SQ combined with OsakaED (OsakaED+SQ), as
well as between SQ alone and SQ combined with GPT-4 (GPT-4+SQ), were assessed.
In the comparison between SQ and OsakaED+SQ, consistency was found to decrease
($p<0.05$). In the comparison between SQ and GPT-4+SQ, there were significant
differences in user empathy, trust, agency, and emotion (p¡0.05), as well as
in mood, personality, and the system’s empathy ($p<0.01$). When SQ was added
to LLM-generated responses, the decrease in consistency remained notable in
the case of OsakaED+SQ, suggesting that the addition of SQ does not
necessarily mitigate the variability introduced by LLM responses. The findings
from GPT-4+SQ comparisons reinforce the idea that GPT-4’s responses
significantly influence user experience, potentially overshadowing the impact
of the added SQ.

###  3.3 Comparison between generated responses and generated responses with
SQ

To investigate the effect of adding SQ to generated responses from LLMs, the
differences between OsakaED and OsakaED+SQ, as well as between GPT-4 and
GPT-4+SQ, were assessed. However, no significant differences were found at any
point. Surprisingly, contrary to our expectations, the addition of SQ to LLM-
generated responses did not yield significant differences in any assessed
dimension. This could indicate that the presence of SQ does not substantially
alter the user’s experience once they are engaged in LLM-generated content.
This may be because the LLM is capable of generating a sufficiently
informative or attractive response that including SQ would be redundant, but
the situation would be different if SQ is added to the OsakaED than if it is
added to GPT-4. In the comparison between OsakaED and OsakaED+SQ, it is
observed that 13 of 17 items were increased by adding SQ, including cognitive
change, naturalness, and ease, although no significant differences were found.
On the other hand, in the comparison between GPT-4 and GPT-4+SQ, 14 items
except for mood change were decreased by adding SQ. This could possibly
indicate that the presence of SQ improves the response of OsakaED but worsens
the response of GPT-4. In particular, it is noteworthy that both GPT-4 and
GPT-4+SQ were observed to generate Socratic Questions. This observation
suggests that the combination with SQ becomes more redundant, as GPT-4 alone
already appears capable of producing such responses.

##  4 Discussion

###  4.1 Principal Findings

In this study, we constructed dialogue systems based on a CBT scenario focused
on Socratic questioning using two kinds of LLMs. By comparing systems that
used LLM-generated responses with those that did not, we investigated the
impact of the generated responses. OsakaED does not necessarily improve
outcomes compared to scenario-based dialogue, even though the dialogue model
used to generate responses was trained with a real counseling dataset. On the
other hand, the results suggest that GPT-4 is highly capable for counseling.
In particular, it’s notable that the outcome and many of the dialogue
qualities were often significantly improved when using GPT-4 generated
responses, even though the user had just a six-turn interaction and the
provided prompt (system message) for this was not so highly optimized. In
addition, It was observed that GPT-4 can ask Socratic Questioning in context.
This may provide many implications for the field in analyzing what features of
the GPT-4 generated responses achieve these improvements in mood and response
qualities.

The fact that OsakaED did not show any particular improvement over SQ, a
simple scenario-based system, indicates that building and using LLMs with a
dataset of human counselors does not necessarily lead to an improvement in the
system. Therefore, when developing a dialogue system that uses LLMs to
generate responses, it would be important to check beforehand whether the
system can be improved compared to older systems such as rules, scenarios, and
example responses.

On the other hand, The number of parameters for OsakaED was 1.6B, which was
relatively small. In recent years, open-source LLMs with a larger number of
parameters, such as LLaMA touvron2023llama  and LLaMA 2 touvron2023llama2  by
Meta, have been released. As there are already studies that have tuned these
in the mental health field liu2023chatcounselor ; yang2023mentalllama , by
producing a model aligned to CBT, it is thought to be possible to develop an
LLM that surpasses the scenario-based dialogue systems in this setting.

###  4.2 Ethical use of LLMs for response generation in mental healthcare
services

While GPT-4 was found to be able to generate high-quality responses, using
GPT-4 in real-world mental health care services and interacting directly with
users will raise ethical issues lee2023benefits ; jeyaraman2023chatgpt ;
amamou2023chatgpt ; gilbert2023large . First of all, even though mental health
care services will be used by people with mental health issues, especially
some of whom may have suicidal ideation, developers cannot control in advance
what responses GPT-4 will generate and lead those users. In other words,
developers cannot fully guarantee the safety of the system’s interaction with
users, and this can be a critical ethical issue. This includes the problem
that LLMs may make up false information due to the “hallucination” phenomenon
they cause. The issue of “hallucination” is a critical shortcoming in this
field because it can fabricate wrong information about medical care
lee2023benefits ; darcy2023generative . Although research on LLM alignment has
been heavily studied, and alignment has been done for GPT-4 ouyang2022training
, it is unlikely that this fundamental concern of not being able to confirm
output in advance will ever be resolved. Actually, Woebot Health has a policy
of outputting responses designed by human experts in advance, without direct
user interaction with the LLMs darcy2023generative ; woebot2023aiatwoebot .

Given the ethical issues, it is reasonable to avoid directly presenting the
response output by the LLMs to the user and to output a pre-designed response,
as in the case of woebot. Nevertheless, one way to take advantage of the LLMs
high response generation capability is for a human expert to work with LLMs in
advance to generate a dictionary of example responses in advance. In this
method, a large number of example responses for various contexts can be
generated in advance using LLMs, and checked and supervised by human experts
to generate some of the responses the systems chose from. If there was no more
appropriate response for the context, a human-in-the-loop strategy in which
the human expert efficiently adds new types of responses using GPT-4 or other
LLMs would allow more appropriate responses to be uttered at each turn that
uses this response strategy. In addition, the context and its response pairs
produced during this cycle could be used to train other open-source LLMs, like
LLaMA, to make better response generation models by instruction tuning, which
will potentially accelerate these cycles peng2023instruction .

Furthermore, in the construction of dialogue systems for mental health care,
understanding user input through LLMs may also be an important application,
which may be used to perform response selection and to estimate and monitor
user states. Recently, there have been studies on the estimation of various
mental states of users focusing on depression, suicidal ideation, cognitive
distortions, etc., using GPT-4 and other LLMs, and there is a possibility that
the mental states of users can be understood more precisely and from various
perspectives xu2023leveraging ; yang2023mentalllama ; levkovich2023suicide ;
chen2023empowering . It is expected that a real-world interactive system for
mental health care will be constructed and operated as a human-in-the-loop
operation, in which human experts provide supervision and intervention to
improve the system while reaping the benefits of these high capabilities of
LLMs.

###  4.3 Limitations

As a limitation, the experiment in this study is a small-scale one with a
limited number of crowdsourced subjects. In particular, CBT is a long-term,
multi-session program, and if its effectiveness as CBT is to be accurately
verified, it is necessary to conduct a long-term randomized controlled trial
with a larger sample size to compare its effectiveness.

Furthermore, in this experiment, the system was set up so that responses with
and without LLMs were not returned at the same speed, but as quickly as
possible from time to time. As a result, responses without LLMs presented the
next response instantaneously, while those with LLMs had a delay of several
seconds and especially longer delays when GPT-4 was used. It is known that the
speed of response changes the user’s evaluation of the system moon1999effects
; holtgraves2007procedure ; holtgraves2007perceiving ; gnewuch2018faster ;
folstad2020users , and it is possible that these responses speeds influenced
this human evaluation. In fact, some answers for free text questions asking
for improvements to the system indicated that the responses were too fast,
which led to comments such as, “When I received a reply the moment I sent my
message, I felt that I was getting a fixed reply without considering what I
had written.” For the GPT-4 and GPT-4+SQ, comments were seen that responses
were too slow, leading to comments such as, “I was worried that the system had
stopped.” In order to more accurately compare only the quality of the
responses, it would be necessary to match the response speeds for all
utterances.

##  5 Conclusion

In this study, we evaluated dialogue systems based on CBT scenarios using two
types of LLMs: OsakaED and GPT-4. We compared systems with LLM-generated
responses to those without, to assess the impact of these responses. The
results suggest that GPT-4 is highly effective in counseling contexts, but it
is also observed that OsakaED does not necessarily yield better outcomes than
scenario-based dialogues, even though LLMs are being trained on real
counseling data. This could suggest that the use of LLMs is not always
effective, compared to the scenario-based systems. In addition, although GPT-4
is considered to be effective for mental healthcare response generation, it
will raise safety problems if a system directly shows responses generated from
LLMs to the users. One possible approach to solving safety issues while taking
advantage of the high response generation capability of LLMs such as GPT-4 is
thought to develop a human-in-the-loop system that uses a dictionary of
example responses or templates created in cooperation between LLMs and human
experts. In the future, we will focus on counseling by GPT-4, investigate
optimally effective prompts for CBT, and analyze the needed counseling skills.
Furthermore, we will verify the achieved counseling skills by the prompts.

###### Acknowledgements.

This research is supported by CREST (grant number: JPMJCR19A5). The OsakaED
dataset was provided by the Public Health and Medical Administration Office,
Department of Public Health and Medical Affairs, Osaka Prefectural Government,
and the Kansai Counseling Center.

## References

  * [1] Alaa A Abd-Alrazaq, Mohannad Alajlani, Ali Abdallah Alalwan, Bridgette M Bewick, Peter Gardner, and Mowafa Househ.  An overview of the features of chatbots in mental health: A scoping review.  International Journal of Medical Informatics, 132:103978, 2019.
  * [2] Alaa A Abd-Alrazaq, Mohannad Alajlani, Nashva Ali, Kerstin Denecke, Bridgette M Bewick, and Mowafa Househ.  Perceptions and opinions of patients about mental health chatbots: scoping review.  Journal of medical Internet research, 23(1):e17828, 2021.
  * [3] Takuto Akiyoshi, Junya Nakanishi, Hiroshi Ishiguro, Hidenobu Sumioka, and Masahiro Shiomi.  A robot that encourages self-disclosure to reduce anger mood.  IEEE Robotics and Automation Letters, 6(4):7925–7932, 2021.
  * [4] Badii Amamou, Amina Aissa, Lina Brahmi, Amjed Ben Haouala, Ismail Dergaa, I Dergaa, F Fekih-Romdhane, S Hallit, AA Loch, JM Glenn, et al.  Chatgpt is not ready yet for use in providing mental health assessment and interventions.  2023\.
  * [5] E Anthes.  Mental health: There’s an app for that.  Nature, 532:20–23, 2016.
  * [6] Aaron T Beck.  Cognitive therapy of depression.  Guilford press, 1979.
  * [7] Judith S Beck.  Cognitive behavior therapy: Basics and beyond.  Guilford Publications, 2020.
  * [8] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al.  Sparks of artificial general intelligence: Early experiments with gpt-4.  arXiv preprint arXiv:2303.12712, 2023.
  * [9] Zhiyu Chen, Yujie Lu, and William Yang Wang.  Empowering psychotherapy with large language models: Cognitive distortion detection through diagnosis of thought prompting.  arXiv preprint arXiv:2310.07146, 2023.
  * [10] Morena Danieli, Tommaso Ciulli, Seyed Mahed Mousavi, Giorgia Silvestri, Simone Barbato, Lorenzo Di Natale, and Giuseppe Riccardi.  Assessing the impact of conversational artificial intelligence in the treatment of stress and anxiety in aging adults: Randomized controlled trial.  JMIR Ment Health, 9(9):e38067, Sep 2022.
  * [11] Alison Darcy.  Why generative ai is not yet ready for mental healthcare, March 2023.
  * [12] Mauro De Gennaro, Eva G Krumhuber, and Gale Lucas.  Effectiveness of an empathic chatbot in combating adverse effects of social exclusion on mood.  Frontiers in psychology, 10:3061, 2020.
  * [13] Albert Ellis.  The revised abc’s of rational-emotive therapy (ret).  Journal of rational-emotive and cognitive-behavior therapy, 9(3):139–172, 1991.
  * [14] Siska Fitrianie, Merijn Bruijnes, Deborah Richards, Andrea Bönsch, and Willem-Paul Brinkman.  The 19 unifying questionnaire constructs of artificial social agents: An iva community analysis.  IVA ’20, New York, NY, USA, 2020. Association for Computing Machinery.
  * [15] Asbjørn Følstad and Petter Bae Brandtzaeg.  Users’ experiences with chatbots: findings from a questionnaire study.  Quality and User Experience, 5(1):3, 2020.
  * [16] Institute for Health Metrics and Evaluation.  Gbd results tool.  http://ghdx.healthdata.org/gbd-results-tool?params=gbd-api-2019-permalink/cb9c37d9454c80df77adaed394d7fc0f, accessed 14 October 2023.
  * [17] Institute for Health Metrics and Evaluation.  Gbd results tool.  http://ghdx.healthdata.org/gbd-results-tool?params=gbd-api-2019-permalink/5066348dc958b095cb6ceb4bfd9c3e07, accessed 14 October 2023.
  * [18] Toshiaki A Furukawa, Ronald C Kessler, Tim Slade, and Gavin Andrews.  The performance of the k6 and k10 screening scales for psychological distress in the australian national survey of mental health and well-being.  Psychological medicine, 33(2):357–362, 2003.
  * [19] Stephen Gilbert, Hugh Harvey, Tom Melvin, Erik Vollebregt, and Paul Wicks.  Large language model ai chatbots require approval as medical devices.  Nature Medicine, 29(10):2396–2398, 2023.
  * [20] Ulrich Gnewuch, Stefan Morana, Marc Adam, and Alexander Maedche.  Faster is not always better: understanding the effect of dynamic response delays in human-chatbot interaction.  2018\.
  * [21] Thomas Holtgraves and Tai-Lin Han.  A procedure for studying online conversational processing using a chat bot.  Behavior research methods, 39(1):156–163, 2007.
  * [22] TM Holtgraves, Stephen J Ross, CR Weywadt, and TL Han.  Perceiving artificial social agents.  Computers in human behavior, 23(5):2163–2174, 2007.
  * [23] Becky Inkster, Shubhankar Sarda, and Vinod Subramanian.  An empathy-driven, conversational artificial intelligence agent (wysa) for digital mental well-being: real-world data evaluation mixed-methods study.  JMIR mHealth and uHealth, 6(11):e12106, 2018.
  * [24] Mercer SW. Jani BD, Blane DN.  The role of empathy in therapy and the physician-patient relationship.  Forsch Komplementmed, 19(5):252–257, 2012.
  * [25] Madhan Jeyaraman, Swaminathan Ramasubramanian, Sangeetha Balaji, Naveen Jeyaraman, Arulkumar Nallakumarasamy, and Shilpa Sharma.  Chatgpt in action: Harnessing artificial intelligence potential and addressing ethical challenges in medicine, education, and scientific research.  World Journal of Methodology, 13(4):170, 2023.
  * [26] Sharanya Srinivasan Joe Gallagher.  Ai at woebot health â our core principles, August 2023.
  * [27] Harsh Kumar, Ilya Musabirov, Jiakai Shi, Adele Lauzon, Kwan Kiu Choy, Ofek Gross, Dana Kulzhabayeva, and Joseph Jay Williams.  Exploring the design of prompts for applying gpt-3 based chatbots: A mental wellbeing case study on mechanical turk.  arXiv preprint arXiv:2209.11344, 2022.
  * [28] Peter Lee, Sebastien Bubeck, and Joseph Petro.  Benefits, limits, and risks of gpt-4 as an ai chatbot for medicine.  New England Journal of Medicine, 388(13):1233–1239, 2023.
  * [29] Inbar Levkovich and Zohar Elyoseph.  Suicide risk assessments through the eyes of chatgpt-3.5 versus chatgpt-4: Vignette study.  JMIR mental health, 10:e51232, 2023.
  * [30] June M Liu, Donghao Li, He Cao, Tianhe Ren, Zeyi Liao, and Jiamin Wu.  Chatcounselor: A large language models for mental health support.  arXiv preprint arXiv:2309.15461, 2023.
  * [31] LÃ³pez-Coronado M MartÃnez-PÃ©rez B, de la Torre-DÃez I.  Mobile health applications for the most prevalent conditions by the world health organization: Review and analysis.  Journal of medical Internet research, 15(6):e120, 2013.
  * [32] Youngme Moon.  The effects of physical distance and response latency on persuasion in computer-mediated communication and human–computer communication.  Journal of Experimental Psychology: Applied, 5(4):379, 1999.
  * [33] Seyed Mahed Mousavi, Alessandra Cervone, Morena Danieli, and Giuseppe Riccardi.  Would you like to tell me more? generating a corpus of psychotherapy dialogues.  In Proceedings of the Second Workshop on Natural Language Processing for Medical Conversations, pages 1–9, Online, June 2021. Association for Computational Linguistics.
  * [34] Christopher JL Murray, Theo Vos, Rafael Lozano, Mohsen Naghavi, Abraham D Flaxman, Catherine Michaud, Majid Ezzati, Kenji Shibuya, Joshua A Salomon, Safa Abdalla, et al.  Disability-adjusted life years (dalys) for 291 diseases and injuries in 21 regions, 1990–2010: a systematic analysis for the global burden of disease study 2010.  The lancet, 380(9859):2197–2223, 2012.
  * [35] Gureje O Oladeji BD.  Brain drain: a challenge to global mental health.  BJPsych International, 13(3):61–63, 2016.
  * [36] OpenAI.  Introducing chatgpt, 2022.  https://openai.com/blog/chatgpt, accessed 15 October 2023.
  * [37] OpenAI.  Chat completions api, 2023.  Accessed: 2024-01-26.
  * [38] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
  * [39] Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao.  Instruction tuning with gpt-4.  arXiv preprint arXiv:2304.03277, 2023.
  * [40] Jacqueline B Persons and David D Burns.  Mechanisms of action of cognitive therapy: The relative contributions of technical and interpersonal interventions.  Cognitive therapy and Research, 9:539–551, 1985.
  * [41] Iony D Schmidt, Benjamin J Pfeifer, and Daniel R Strunk.  Putting the “cognitive” back in cognitive therapy: Sustained cognitive change as a mediator of in-session insights and depressive symptom improvement.  Journal of Consulting and Clinical Psychology, 87(5):446, 2019.
  * [42] Kazuhiro Shidara, Hiroki Tanaka, Hiroyoshi Adachi, Daisuke Kanayama, Yukako Sakagami, Takashi Kudo, and Satoshi Nakamura.  Automatic thoughts and facial expressions in cognitive restructuring with virtual agents.  Frontiers in Computer Science, 4:762424, 2022.
  * [43] Hiroaki Sugiyama, Masahiro Mizukami, Tsunehiro Arimoto, Hiromi Narimatsu, Yuya Chiba, Hideharu Nakajima, and Toyomi Meguro.  Empirical analysis of training strategies of transformer-based japanese chit-chat systems, 2021.
  * [44] Hiroaki Sugiyama, Masahiro Mizukami, Tsunehiro Arimoto, Hiromi Narimatsu, Yuya Chiba, Hideharu Nakajima, and Toyomi Meguro.  Empirical analysis of training strategies of transformer-based japanese chit-chat systems.  In 2022 IEEE Spoken Language Technology Workshop (SLT), pages 685–691. IEEE, 2023.
  * [45] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * [46] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  arXiv preprint arXiv:2307.09288, 2023.
  * [47] Aditya Nrusimha Vaidyam, Hannah Wisniewski, John David Halamka, Matcheri S Kashavan, and John Blake Torous.  Chatbots and conversational agents in mental health: a review of the psychiatric landscape.  The Canadian Journal of Psychiatry, 64(7):456–464, 2019.
  * [48] Lisa N Vittorio, Samuel T Murphy, Justin D Braun, and Daniel R Strunk.  Using socratic questioning to promote cognitive change and achieve depressive symptom reduction: evidence of cognitive change as a mediator.  Behaviour research and therapy, 150:104035, 2022.
  * [49] Xuhai Xu, Bingshen Yao, Yuanzhe Dong, Hong Yu, James Hendler, Anind K Dey, and Dakuo Wang.  Leveraging large language models for mental health prediction via online text data.  arXiv preprint arXiv:2307.14385, 2023.
  * [50] Kailai Yang, Tianlin Zhang, Ziyan Kuang, Qianqian Xie, and Sophia Ananiadou.  Mentalllama: Interpretable mental health analysis on social media with large language models.  arXiv preprint arXiv:2309.13567, 2023.