# Assessing Empathy in Large Language Models with Real-World Physician-Patient
Interactions

Man Luo1,2 Christopher J. Warren1 Lu Cheng3 Haidar M. Abdul-Muhsin1
Imon Banerjee1 1Mayo Clinic  2 Intel Lab  3 University of Illinois Chicago
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

The integration of Large Language Models (LLMs) into the healthcare domain has
the potential to significantly enhance patient care and support through the
development of empathetic, patient-facing chatbots. This study investigates an
intriguing question Can ChatGPT respond with a greater degree of empathy than
those typically offered by physicians? To answer this question, we collect a
de-identified dataset of patient messages and physician responses from Mayo
Clinic and generate alternative replies using ChatGPT. Our analyses
incorporate novel empathy ranking evaluation (EMRank) involving both automated
metrics and human assessments to gauge the empathy level of responses. Our
findings indicate that LLM-powered chatbots have the potential to surpass
human physicians in delivering empathetic communication, suggesting a
promising avenue for enhancing patient care and reducing professional burnout.
The study not only highlights the importance of empathy in patient
interactions but also proposes a set of effective automatic empathy ranking
metrics, paving the way for the broader adoption of LLMs in healthcare.

##  1 Introduction

Figure 1: Given a patient message, we prompt ChatGPT for a response. We
restrict the length of the ChatGPT response to mimic the statistics of the
physician’s response (See Table 2). Both ChatGPT’s and a physician’s responses
are then evaluated using a multi-dimension LLM-EMRank metric for automatic
ranking (LLaMA Empathy Evaluation). In addition. we also conduct a human
empathy evaluation to ensure a thorough and rigorous assessment.

Large Language Models (LLMs) have exhibited exceptional performance in
clinical and biomedical areas Lee et al. (2020); Luo et al. (2022b). They have
been effectively used for extracting information from clinical notes Parmar et
al. (2022); Chiang et al. , providing differential diagnoses based on patient
symptoms Macherla et al. (2023), retrieving biomedical literature information
Luo et al. (2022a), and assisting data labelling He et al. (2024).
underscoring their flexibility and vital role in enhancing healthcare services
and research advancements. A notable achievement, such as with models like
ChatGPT Ray (2023), is the development of patient-facing chatbots built on the
foundation of LLMs Dave et al. (2023). These chatbots aim to deliver instant
answers to patient questions, significantly reducing the waiting period for
doctor’s advice and helping to decrease the burden on healthcare professionals
Wilson and Marasoiu (2022). Beyond meeting immediate patient needs, chatbots
that respond with empathy can greatly boost patient engagement, a key factor
in improving patient quality of life (QoL). An empathetic chatbot is more
effective in maintaining continuous dialogue with patients, creating a
nurturing space even within the digital realm Inkster et al. (2018).

While the creation of empathetic dialogues in general contexts has been
explored Tahir et al. (2023), a comprehensive examination of chatbots’
empathetic abilities when interacting with real patients and comparing their
responses to those of actual physicians remains limited Altamimi et al.
(2023). This study aims to answer an intriguing question, Can ChatGPT respond
with a greater degree of empathy than those typically offered by physicians?
Particularly, we collect a real-world dataset comprising patient messages and
the corresponding responses from physicians at Mayo Clinic. We manually de-
identify patients’ and physicians’ information to protect their privacy.
Subsequently, we utilize the ChatGPT
API111https://platform.openai.com/docs/overview, a leading-edge LLM, to craft
responses to these de-identified patient messages, following detailed
instructions (refer to the green box in Figure 1).

The next step of our research involves assessing the level of empathy in
responses from both physicians and ChatGPT. Previous studies have primarily
focused on detecting empathy through models trained on specific datasets, a
technique that often lacks generalizability to different domains without
domain-specific fine-tuning. Drawing inspiration from the emerging ability of
LLMs Wei et al. (2022), especially the in-context learning, where an LLM can
perform an unseen task with instruction and few examples of the task, we
investigate the effectiveness of in-context learning for empathy ranking in
healthcare domain. We introduce multiple metrics using LLaMA Touvron et al.
(2023), named LLaMA-EMRank. Specifically, we employ LLaMA zero-shot, one-shot,
and few-shot learning capabilities, as well as an ensemble of these methods.
In a zero-shot scenario, we prompt LLaMA to execute the EMRank task through
domain-specific instructions (refer to the blue box in Figure 1). For one-shot
and few-shot scenarios, we collect in-context learning (ICL) examples from
patients (§5.1). Compared with prior metrics Lee et al. (2022a); Kim et al.
(2021); Lee et al. (2022b), LLaMA-EMRank eliminating the need of fine-tuning
an evaluation model and the framework is generalizable to other domains (i.e.
by changing the few-shot ICL examples and domain specific instruction).

We strategically choose a strong model (i.e. ChatGPT) to be the response
generation model and a smaller model (i.e. LLaMA) to be the empathy evaluation
model for two reasons. First, empathetic response generation to a patient
message task is a complex task, necessitating not just an understanding of
empathy but also a comprehensive and accurate grasp of the relevant domain
knowledge, areas where ChatGPT excels as a leading model. Second, given that
ChatGPT is the response generation model, to ensure independence in the
assessment process, minimizing any potential bias that might arise if the same
model (ChatGPT) were used for both generating and evaluating responses, choose
a different LLM that preserves robust linguistic understanding ability. The
concept of weak-to-strong generalization has been explored Burns et al.
(2023). Moreover, our experiments show that the LLaMA-EMRank metrics reach a
desired agreement with humans evaluation, indicating its validity of ranking
the empathy degree of two responses (§6.2).

Overall, our investigation aims to measure the empathetic quality of LLM-
powered chatbot responses in comparison to human physicians and to examine how
well these models’ outputs align with human judgments. Through this work, we
aspire to reveal the capabilities of LLMs in fostering patient care and
support, representing a significant advancement in integrating artificial
intelligence into healthcare. We summarize the three contributions of this
work:

  * •

Our analysis is based on real-patient data, ensuring that the findings and
conclusions are directly relevant to real-world healthcare scenarios.

  * •

We compares the empathetic responses of ChatGPT with those of physicians using
four innovative automatic metrics: LLaMA-EMRank.

  * •

we incorporate patient evaluations to assess the concordance between LLaMA-
EMRank metrics and human perceptions, demonstrating the metrics’ reliability.

  * •

Our findings suggest potential for LLM-powered chatbots to surpass physicians
in delivering empathetic responses, thereby enhancing patient interaction.

##  2 Related Work

Dataset | Domain | Source | Curating Method | Evaluation Metric
---|---|---|---|---
EmpatheticDialogues 2019 | Daily | AMT Conversation | Crowd-sourcing | PPL , BLUE, UE-score, EPITOME,Diff-EPITOME
EMPATHETICPERSONAS 2019 | Mental Health | AMT Conversation | Crowd-sourcing | PPL, Accuracy
IEMPATHIZE 2021 | Oncology | Online Cancer Platform | Graduate Student Annotation | F1, Precision, Recall
EDOS 2021 | Daily | Movie Dialogue | semi-automated | F1, Accuracy
EPITOME 2020 | Mental Health | Online support platforms | Crowd Sourcing | F1, Accuracy
Ours | Oncology | Hospital | Real-patient | PPL, zero-shot, one-shot, few-shots

Table 1: Comparing with Existing Datasets and Metrics for Empathy Evaluation.

#### Empathy Dataset

Several dialogue datasets have been annotated with emotion labels,
demonstrating the diversity in approaches and applications within this
research area Busso et al. (2008); Chatterjee et al. (2019). Notably, Welivita
and Pu (2020) conducted an in-depth analysis of the EmpatheticDialogues
dataset Rashkin et al. (2019), identifying eight empathetic response intents,
such as encouragement and consolation, alongside 32 fine-grained emotions that
range from positive to negative. The process of labeling these datasets
varies, involving either human annotators or semi-automatic methods, which
occasionally results in ambiguous emotion categorizations Chatterjee et al.
(2019). Rashkin et al. (2019) employed Amazon Mechanical Turk (AMT) workers to
gather a dataset enriched with textual expressions spanning four emotional
contexts: sadness, anger, anxiety/fear, and happiness/contentment. This
dataset also features empathetic rewritings of 45 foundational utterances. In
a similar setting, EPITOME Sharma et al. (2020) aggregates conversations from
online mental support platforms, with AMT workers tasked to detect three key
communication mechanisms: emotional reactions, interpretations, and
explorations. Moreover, Hosseini and Caragea (2021) introduced the IEMPATHIZE
dataset, which comprises sentences sourced from an online platform for cancer
patients (specifically those with lung and breast cancer). This dataset was
annotated by two graduate students who classified each sentence into one of
three categories: seeking empathy, providing empathy, or none, highlighting
the nuanced understanding of empathetic communication within specific
contexts. The EDOS dataset is curated through a semi-automatic approach by
Welivita et al. (2021). It begins with a selection of 9,000 movie subtitles
from the OpenSubtitles corpus, which are initially annotated by humans to
match the emotion labels used in EmpatheticDialogues. These annotated
subtitles are then used to train a BERT-based emotion classification model,
which in turn annotates additional movie subtitles, culminating in a dataset
comprising over one million entries.

Our research uniquely contributes by analyzing real conversations between
patients and physicians, with patients comparing chatbot to physician
responses, offering insights closest to real-world healthcare scenarios.
Unlike prior studies focused on emotion detection or dialogue generation, our
work centers on evaluating chatbot versus physician interactions, aligning
with the trend of using LLMs as chatbots without domain-specific tuning. This
direct comparison approach not only differentiates our dataset but also
highlights the practical application of LLMs in healthcare, bridging a
significant gap in existing research.

#### Empathy Evaluation

In the realm of emotion detection tasks, accuracy and F1 scores are commonly
used to assess system performance, as indicated in Table 1. For evaluating
response fluency, BLEU and perplexity (PPL) metrics are employed. These
methods operate independently of fine-tuned models, contrasting with metrics
that necessitate model fine-tuning to gauge empathy levels. Lee et al. (2022a)
introduced the UE-score (utterance entailment), utilizing a model trained on
the SNLI dataset to determine if a hypothesis is entailed by a premise. Kim et
al. (2021) developed an automatic scoring mechanism based on the EPITOME
framework Sharma et al. (2020), employing three RoBERTa models Liu et al.
(2019) to predict emotional reactions, interpretations, and explorations, with
the final empathy score of a sentence being the average of these three scores.
Based on EPITOME, Lee et al. (2022b) crafted the Diff-EPITOME score to compare
the empathy in responses from humans and chatbots, where a lower score
signifies a more human-like response. However, this assumes that humans always
express empathy, which may not hold true under all circumstances. Another
limitation of EPITOME framework’s is that the three categories might not work
in all circumstances. For instance, October et al. (2018) outlined five
empathy evaluation aspects in patient-physician interactions that differ from
the EPITOME framework. Despite the reliance on automatic metrics, human
evaluations play a crucial role in validating whether responses are empathetic
or if automatic metrics align with human judgment. Li et al. (2019) introduced
ACUTE-EVAL for comparing responses in dialogue contexts, using Pearson’s r
score to assess alignment between human and automatic evaluations. Ayers et
al. (2023) compared the empathy degree of a model generated answer and
physician’s answer by human annotation.

Our study introduces unique metrics for ranking empathy levels between
responses without depending on fine-tuning, setting our approach apart from
previous methodologies.

##  3 Data Collection

#### Source of the Data

Our dataset comprises messages collected from a patient portal at Mayo Clinic,
which is used for answering patients’ questions. We collect messages from
individuals diagnosed with prostate cancer. This dataset was compiled by
extracting the conversations of men who underwent radical prostatectomy
between December 2018 and October 2023. We gather messages under the category
of “Patient advice request” and randomly select messages from the pool of
available conversations.

#### De-indefication Process

The de-identification process was meticulously carried out by a team of three
two medical students and one physician who is a post graduate year 3 urology
resident, with the primary goal of ensuring the privacy and confidentiality of
individuals mentioned in the dataset collected from the internal platform.
This process involved a detailed review of each patient message and physician
response to identify and remove or anonymize any personally identifiable
information (PII) including patient names, physician names, dates, phone
numbers, and addresses.

#### Statistic

Table 2 shows the statistics of the dataset. Notably, ChatGPT’s responses tend
to be more lengthy on average than those of human counterparts, yet they align
well with our specified length limit of 100 words. This observation suggests
ChatGPT’s effective management of response length. Examples of patient
messages and physician responses can be found in Figure 2 and Figure 6 in the
Appendix.

Statistic | Value
---|---
Total Number of Questions | 491
Average Length of Question | 94
Average Length of Physician Response | 78
Average Length of ChatGPT Response | 104
Table 2: Empathy Evaluation Dataset Statistics: Analyzing Messages from Real
Prostate Cancer Patients, Responses from Physicians, and ChatGPT’s Replies.

##  4 Chatbot Response

#### ChatGPT

is the state-of-the-art language model developed by OpenAI Achiam et al.
(2023). Its adeptness at mimicking human conversation allows it to cover a
broad spectrum of topics, offering explanations, answers, and creative
outputs. The model has notably captured the interest of the clinical sector,
where its potential to improve patient care, medical education, and
operational efficiency is considerable. Our research focuses on a particularly
intriguing question: Can ChatGPT respond with a greater degree of empathy than
those typically offered by physicians? To explore this, we leverage ChatGPT’s
instructional learning capabilities, employing specific prompts to elicit
responses from the model, as detailed in the subsequent section. Our
experiments are based on the API of ChatGPT, gpt-4-1106-preview version which
is trained on the knowledge up to April 2023. We use the ChatGPT inference
length of 4,096, which is the maximum number of tokens (words, punctuation,
and spaces) that ChatGPT can process or generate in a single inference.

#### Prompt

The specific prompt we used in our study is illustrated in the green box of
Figure 1. We instruct ChatGPT to function as a urology expert, considering our
dataset comprises questions from prostate cancer patients. Additionally, we
guide ChatGPT to respond with empathy and limited the response length to 100
words. Initially, we did not impose a word limit, but we observed that ChatGPT
often produced lengthy responses. To ensure a fair comparison with the
responses from physicians, we decided to implement a word count restriction.
This decision was made after comparing the average length of physician
responses with ChatGPT responses, as presented in Table 2.

##  5 Evaluation

Given that existing evaluation metrics primarily focus on empathy detection
tasks, we explore LLM’s capability for automatic empathy ranking evaluation.
To this end, we propose multiple empathy ranking metric that consists of
several new empathetic factors described by LLMs. For the analysis of
ChatGPT’s responses, we employ a distinct language model for comparison with
those of physicians. We select the LLaMA model Touvron et al. (2023) for this
purpose, as it stands out as one of the best open-source models, having been
extensively researched and applied in various studies. Specifically, we make
use of the LLaMA model’s in-context learning capabilities to assess empathy,
and termed this family metrics as LLaMA-EMRank. Additionally, we complement
this approach with a human study to gauge how well each automated method
aligns with human judgment.

###  5.1 LLaMA-EMRank: Automatic Empathy Ranking Evaluation

#### Zero-shot Evaluation

LLMs have demonstrated a remarkable capacity for zero-shot learning without
the need for task-specific fine-tuning Brown et al. (2020). This capability is
particularly valuable as it eliminates the extensive computational and data
preparation costs associated with fine-tuning models for specific downstream
tasks. We employ a language model to determine the more empathetic response
between two options: one from a physician and the other from the chatbot, as a
reply to a patient’s question. This involves feeding the model both responses
and asking it to identify which one exhibits greater empathy. The blue box in
Figure 1 showcases the inputs given to the model.

#### Few-shot Evaluation

Figure 2: In Context Learning Example: Patient is given two responses and
assesses which response is more empathetic and provides the justification.
Note that patients do not know which response is from ChatGPT or the physician
when evaluating empathy.

LLMs also excel in few-shot learning Luo et al. (2024), where they leverage a
small number of examples to adapt to new tasks. This few-shot capability
significantly enhances the models’ flexibility and efficiency, allowing them
to perform specialized tasks by understanding the context from a few
demonstration examples. In applying this few-shot learning to evaluate empathy
in responses, we start by collecting examples by presenting a patient’s
question alongside two responses, who then determine which response is more
empathetic and explain why. After collecting three examples, we input them
into an LLM. After the three examples, we then ask LLaMA to rate a new message
from the patient and two responses from the physician and ChatGPT which
response is more empathetic. Figure 2 depicts one demonstration example
provided to LLaMA and the other two examples are given in the Appendix.

#### One-shot Evaluation

One-shot learning is a special case of few-shot learning where only one
example is given to the model. Compared to few-shot learning, one-shot
learning is more inference efficient since the input length to LLM is shorter
Luo et al. (2023). Here, rather than randomly choosing one example from the
three examples we collected, we used each one and ran three times inference.
And then we take a majority vote out of three as the final answer.

#### Ensemble Evaluation

As we will analyze in the later section (§6.3) that each of the previous
methods have strengths and weaknesses of each approach, instead of depending
on a singular method, we also introduce an ensemble metric by taking the
majority consensus from zero-shot, one-shot, and few-shot assessments.

#### Language Model Perplexity (PPL)

PPL is a key metric for assessing language model performance, especially in
text generation tasks like dialogue systems Liu et al. (2016), where lower
perplexity signifies more fluent, coherent, and realistic outputs. This metric
also has applications in cognitive science to understand human language
processing Frankenberg et al. (2019); Carrasco-Farré (2022). In our research,
we primarily employ Perplexity (PPL) as a measure of fluency. Additionally,
driven by interest, we investigate the correlation between perplexity and
empathy in language model responses within the experimental section. We adapt
PPL to our study as follows:

$\text{PPL}(A\mid
Q)=\sqrt[N]{\prod_{i=1}^{N}\frac{1}{P(a_{i}|Q,a_{1},a_{2},\ldots,a_{i-1})}},$
(1)

where $Q$ is a patient question, $A$ is an answer from either chatbot or a
real physician, and $a_{i}$ is the $i^{th}$ token in the answer $A$. We
compute the PPL scores for the physician’s answer and the chatbot’s answer and
the one with lower PPL is considered as more empathetic.

###  5.2 Patient Evaluation

We further carried out a human evaluation study that included three male
patients with prostate cancer who had undergone radical prostatectomy within
the same timeframe as our collected patient message dataset. For data
collection, we used Google Forms to gather their
annotations222https://www.google.com/forms/about/. To avoid bias, we
randomized the sequence of responses from both the physician and the chatbot
for every question presented. A snapshot of the questionnaire is available in
Figure 7 in the Appendix. Our study evaluated 70 questions, with each one
being assessed by the three participating patients. To address potential
subjectivity in the evaluations, we established a consensus for each response
based on a majority vote.

##  6 Results and Analysis

###  6.1 Human Evaluation Result

We calculated the inter-annotator agreement rate using Fleiss’ kappa score and
discovered a negative score (i.e. -0.15), highlighting the subjective nature
of feeling empathetic. To overcome this challenge, we select the option with
the majority of votes as the more empathetic choice for each patient message.
In Table 3, the human evaluation revealed that ChatGPT’s responses are more
empathetic than those of humans in 72.85% of cases. This indicates that
ChatGPT is perceived to exhibit more empathy than humans, highlighting its
potential application in the healthcare field by providing patients with more
mental support.

###  6.2 LLaMA-EMRank Evaluation Result

Evaluation | Empathy | Fluency
---|---|---
LLaMA-EMRank | Human Evaluation | PPL
Zero-shot | One-shot | Few-shots | Ensemble
ChatGPT | 92.41% | 79.09% | 91.6% | 91.85% | 72.85% | 92.0%
Physician | 7.59% | 20.91% | 8.40% | 8.15% | 27.15% | 8.00%
Table 3: Comparison of ChatGPT and Physician Responses Using LLaMA-EMRank
Metrics and Human Evaluation.

We report results for automatic evaluation using LLaMA-EMRank in Table 3. We
observe that across all automatic evaluation metrics, ChatGPT demonstrates a
much greater empathy level than the physician. For example, for zero-shot
evaluation in the third column, 92.41% ChatGPT responses are more empathetic
than Physicians. Of particular interest is that one-shot evaluation is the
closest to the ratio of human judgment. However, it’s noteworthy that
ChatGPT’s performance is considered higher on automatic metrics compared to
human evaluations. We further investigate the correlation between human
judgment and automatic metrics through Pearson’s correlation coefficient, with
a 95% confidence interval. Figure 3 shows the results and all LLaMA-EMRank
achieve positive Pearson’r value, suggesting that the introduced EMRank
metrics serve as reliable indicators of the level of empathy. Among all
metrics, ensemble metric aligns most closely with human judgment, followed by
the order of one-shot voting, few-shot voting, and zero-shot. This indicates
the effectiveness of ensemble method.

Besides empathy, we also use PPL to measure the fluency of the response, and
we see that in 92% cases, ChatGPT responses show lower PPL values than the
Physician’s response, indicating that ChatGPT’s responses are fluent. We also
investigate the correlation between PPL scores and empathy. The negative
Pearson’s result in Figure 3 shows that PPL is not a strong indicator of
empathy.

Figure 3: The Pearson values between Automatic Metric and Human Judgement.

###  6.3 A Closer Look at LLaMA-EMRank

Previous findings indicate that each LLaMA-EMRank metric aligns with human
judgment; however, we have encountered certain limitations with each method.
In this section, we will discuss the advantages and disadvantages of each
LLaMA-EMRank metric and outline the strategies we have adopted to enhance
their reliability.

#### Zero shot

In this setting, we observe that for some cases, LLaMA does not generate which
responses are better, rather it says “Please let me know how I can assist
you”. To mitigate this, we have tried different prompts to identify the most
effective prompt. Given the limitless possibilities in prompt design, we
conducted several iterations of prompt exploration before settling on the most
effective one. Still, the model fails to explicitly identify which response
demonstrates greater empathy. We speculate that this ambiguity arises from the
nuanced complexity of the responses, which may be nearly indistinguishable in
terms of empathetic content. Another significant challenge involves the post-
processing of the model’s output, as it has the capability to generate a wide
range of sentences. We attempted to constrain the output’s format through the
prompt, yet the model did not consistently adhere to these guidelines.
Consequently, we manually review the LLaMA output to discover patterns and
employ regular expression tools to systematically extract the evaluations from
the model’s output. The patterns are given in Appendix.

#### Few shots

Evaluation | Order 1 | Order 2 | Order 3 | Majority Vote
---|---|---|---|---
ChatGPT | 89.0% | 94.91% | 90.22% | 91.60%
Physician | 11.0% | 5.09% | 9.78% | 8.40%
Pearson | 0.16 | 0.18 | 0.26 | 0.20

Table 4: Performance of Few-Shot Evaluation Across Three Variations of In-
Context Examples Ordering.

The post-processing of the model’s answer is simplified due to its adherence
to the format provided by in-context examples. Unlike the Zero-shot scenarios,
the explanations are significantly briefer. The performance in Few-shot
settings is influenced by the sequence in which examples are presented to the
model. We shuffle the order of the responses three times, we observed
variability in performance, as shown in Table 4. The concordance between Few-
shot outcomes and human evaluation also varies. Ultimately, we consolidate the
three responses and determine the final assessment based on the majority
opinion.

#### One Shot

Evaluation | Example 1 | Example 2 | Example 3 | Majority Vote
---|---|---|---|---
ChatGPT | 71.89% | 89.61% | 75.76% | 79.09%
Physician | 28.11% | 10.39% | 24.24% | 20.91%
Pearson | 0.13 | 0.20 | 0.20 | 0.23

Table 5: Performance of One-Shot Evaluation Performance with Different One-
shot In-Context Examples.

The post-processing of the one-shot evaluation is also much more
straightforward compared to the zero-shot as the model generates the answer in
the same format as the one-shot in-context example. Nevertheless, the outcomes
of one-shot evaluations fluctuate significantly with changes in the example
used, as evidenced by the large variance detailed in Table 5. The degree of
alignment between various one-shot examples and human judgment also varies.
Notably, employing a majority vote approach enhances the concordance between
human assessments and one-shot evaluations. Therefore, it underscores the
importance of conducting multiple one-shot evaluations and adopting the
majority vote to arrive at the final judgment.

###  6.4 Qualitative Analysis

Figure 4: Given the responses from ChatGPT and Physician, human, and all other
automatic metrics rate ChatGPT’s response as being more empathetic. Note that
patients do not know which response is from ChatGPT or the physician. Figure
5: Given the responses from ChatGPT and Physician, humans rate the physician’s
response as being more empathetic while all other automatic metrics rate
ChatGPT’s response as more empathetic. Note that patients do not know which
response is from ChatGPT or the physician.

In addition to quantitative analyses, we also present qualitative analyses to
support previous findings. In Figure 4, we illustrate an instance where there
is consensus between human and automatic metrics in favor of ChatGPT’s
response. In both the One-shot and Few-shot evaluation scenarios, the response
is simple and closely adheres to the format of in-context learning examples
(Figure 2). Conversely, the Zero-shot evaluation provides a more comprehensive
analysis, initially defining empathy before referencing specific responses to
support the evaluation. On the other hand, we also notice that zero-shot
evaluation is more time-consuming than the one-shot and few-shot evaluation.
Figure 5 presents a contrasting scenario where humans prefer the physician’s
response, yet all automatic metrics favor ChatGPT’s response. Upon examining
responses 1 and 2, it becomes challenging to decisively determine which is
more empathetic. This highlights the significance of employing multi-
dimensional metrics for the assessment of highly subjective empathy.

##  7 Discussion and Conclusion

In this study, we embarked on a novel exploration of the empathetic
capabilities of LLM-powered chatbots, with a particular focus on their
application within the healthcare sector. By meticulously comparing these
chatbots’ responses to those of human physicians, our investigation sheds
light on the potential for LLMs to significantly enhance patient care and
support. Our methodology was grounded in real-world data from physician-
patient interaction, ensuring that our insights and conclusions are deeply
rooted in practical, real-world healthcare scenarios. This approach not only
enhances the validity of our findings but also underscores the relevance of
our research to current healthcare practices.

One of the key contributions of our work lies in the development and
application of innovative automatic metrics specifically designed to assess
empathetic responses. These metrics facilitated a nuanced comparison between
the empathy conveyed by ChatGPT and that of human physicians. Moreover, by
incorporating patient evaluations into our study, we were able to assess the
concordance between these automated metrics and human perceptions. This step
was crucial in validating the reliability and relevance of our metrics,
offering a comprehensive view of both the technological and humanistic aspects
of empathy in patient care. Our findings indicate a promising potential for
LLM-powered chatbots to surpass human physicians in delivering empathetic
responses. This revelation highlights the transformative potential of
integrating such technology into healthcare. Chatbots capable of providing
empathetic interaction could significantly enhance the patient experience,
offering support and care that is both accessible and emotionally resonant.

Our study also underscores the importance of continuous refinement and
validation of the automatic metrics used to assess empathetic responses. The
alignment between these metrics and human judgments is crucial, as it ensures
that the technology truly meets human needs and expectations. Further research
should aim to develop personalized empathy metrics and responsible chatbots
Cheng et al. (2021).

## References

  * Achiam et al. [2023] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.  Gpt-4 technical report.  arXiv preprint arXiv:2303.08774, 2023.
  * Altamimi et al. [2023] Ibraheem Altamimi, Abdullah Altamimi, Abdullah S Alhumimidi, Abdulaziz Altamimi, and Mohamad-Hani Temsah.  Artificial intelligence (ai) chatbots in medicine: A supplement, not a substitute.  Cureus, 15(6), 2023.
  * Ayers et al. [2023] John W. Ayers, Adam Poliak, Mark Dredze, Eric C. Leas, Zechariah Zhu, Jessica B. Kelley, Dennis J. Faix, Aaron M. Goodman, Christopher A. Longhurst, Michael Hogarth, and Davey M. Smith.  Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum.  JAMA Internal Medicine, 183(6):589–596, 06 2023.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * Burns et al. [2023] Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al.  Weak-to-strong generalization: Eliciting strong capabilities with weak supervision.  arXiv preprint arXiv:2312.09390, 2023.
  * Busso et al. [2008] Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan.  Iemocap: Interactive emotional dyadic motion capture database.  Language resources and evaluation, 42:335–359, 2008.
  * Carrasco-Farré [2022] Carlos Carrasco-Farré.  The fingerprints of misinformation: how deceptive content differs from reliable sources in terms of cognitive effort and appeal to emotions.  Humanities and Social Sciences Communications, 9(1):1–18, 2022.
  * Chatterjee et al. [2019] Ankush Chatterjee, Umang Gupta, Manoj Kumar Chinnakotla, Radhakrishnan Srikanth, Michel Galley, and Puneet Agrawal.  Understanding emotions in text using deep learning and big data.  Computers in Human Behavior, 93:309–317, 2019.
  * Cheng et al. [2021] Lu Cheng, Kush R Varshney, and Huan Liu.  Socially responsible ai algorithms: Issues, purposes, and challenges.  Journal of Artificial Intelligence Research, 71:1137–1181, 2021.
  * [10] Chia-Chun Chiang, Man Luo, Gina Dumkrieger, Shubham Trivedi, Yi-Chieh Chen, Chieh-Ju Chao, Todd J Schwedt, Abeed Sarker, and Imon Banerjee.  A large language model-based generative natural language processing framework finetuned on clinical notes accurately extracts headache frequency from electronic health records.  medRxiv.
  * Dave et al. [2023] Tirth Dave, Sai Anirudh Athaluri, and Satyam Singh.  Chatgpt in medicine: an overview of its applications, advantages, limitations, future prospects, and ethical considerations.  Frontiers in Artificial Intelligence, 6:1169595, 2023.
  * Frankenberg et al. [2019] Claudia Frankenberg, Jochen Weiner, Tanja Schultz, Maren Knebel, Christina Degen, Hans-W Wahl, and Johannes Schroeder.  Perplexity–a new predictor of cognitive changes in spoken language?–results of the interdisciplinary longitudinal study on adult development and aging (ilse).  Linguistics Vanguard, 5(s2):20180026, 2019.
  * He et al. [2024] Zeyu He, Chieh-Yang Huang, Chien-Kuang Cornelia Ding, Shaurya Rohatgi, and Ting-Hao Kenneth Huang.  If in a crowdsourced data annotation pipeline, a gpt-4.  In Proceedings of the CHI Conference on Human Factors in Computing Systems, pages 1–25, 2024.
  * Hosseini and Caragea [2021] Mahshid Hosseini and Cornelia Caragea.  It takes two to empathize: One to seek and one to provide.  In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 13018–13026, 2021.
  * Inkster et al. [2018] Becky Inkster, Shubhankar Sarda, Vinod Subramanian, et al.  An empathy-driven, conversational artificial intelligence agent (wysa) for digital mental well-being: real-world data evaluation mixed-methods study.  JMIR mHealth and uHealth, 6(11):e12106, 2018.
  * Kim et al. [2021] Hyunwoo Kim, Byeongchang Kim, and Gunhee Kim.  Perspective-taking and pragmatics for generating empathetic responses focused on emotion causes.  In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2227–2240, 2021.
  * Lee et al. [2020] Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang.  Biobert: a pre-trained biomedical language representation model for biomedical text mining.  Bioinformatics, 36(4):1234–1240, 2020.
  * Lee et al. [2022a] Jing Yang Lee, Kong Aik Lee, and Woon Seng Gan.  Improving contextual coherence in variational personalized and empathetic dialogue agents.  In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7052–7056. IEEE, 2022.
  * Lee et al. [2022b] Young-Jun Lee, Chae-Gyun Lim, and Ho-Jin Choi.  Does gpt-3 generate empathetic dialogues? a novel in-context example selection method and automatic evaluation metric for empathetic dialogue generation.  In Proceedings of the 29th International Conference on Computational Linguistics, pages 669–683, 2022.
  * Li et al. [2019] Margaret Li, Jason Weston, and Stephen Roller.  Acute-eval: Improved dialogue evaluation with optimized questions and multi-turn comparisons.  arXiv preprint arXiv:1909.03087, 2019.
  * Liu et al. [2016] Chia-Wei Liu, Ryan Lowe, Iulian Vlad Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau.  How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation.  In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2122–2132, 2016.
  * Liu et al. [2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  arXiv preprint arXiv:1907.11692, 2019.
  * Luo et al. [2022a] Man Luo, Arindam Mitra, Tejas Gokhale, and Chitta Baral.  Improving biomedical information retrieval with neural retrievers.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11038–11046, 2022.
  * Luo et al. [2022b] Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie-Yan Liu.  Biogpt: generative pre-trained transformer for biomedical text generation and mining.  Briefings in Bioinformatics, 23(6):bbac409, 2022.
  * Luo et al. [2023] Man Luo, Xin Xu, Zhuyun Dai, Panupong Pasupat, Mehran Kazemi, Chitta Baral, Vaiva Imbrasaite, and Vincent Y Zhao.  Dr. icl: Demonstration-retrieved in-context learning.  arXiv preprint arXiv:2305.14128, 2023.
  * Luo et al. [2024] Man Luo, Xin Xu, Yue Liu, Panupong Pasupat, and Mehran Kazemi.  In-context learning with retrieved demonstrations for language models: A survey.  arXiv preprint arXiv:2401.11624, 2024.
  * Macherla et al. [2023] Srija Macherla, Man Luo, Mihir Parmar, and Chitta Baral.  Mddial: A multi-turn differential diagnosis dialogue dataset with reliability evaluation.  arXiv e-prints, pages arXiv–2308, 2023.
  * October et al. [2018] Tessie W October, Zoelle B Dizon, Robert M Arnold, and Abby R Rosenberg.  Characteristics of physician empathetic statements during pediatric intensive care conferences with family members: a qualitative study.  JAMA network open, 1(3):e180351–e180351, 2018.
  * Parmar et al. [2022] Mihir Parmar, Swaroop Mishra, Mirali Purohit, Man Luo, Murad Mohammad, and Chitta Baral.  In-boxbart: Get instructions into biomedical multi-task learning.  In Findings of the Association for Computational Linguistics: NAACL 2022, pages 112–128, 2022.
  * Rashkin et al. [2019] Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau.  Towards empathetic open-domain conversation models: A new benchmark and dataset.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5370–5381, 2019.
  * Ray [2023] Partha Pratim Ray.  Chatgpt: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope.  Internet of Things and Cyber-Physical Systems, 2023.
  * Sharma et al. [2020] Ashish Sharma, Adam Miner, David Atkins, and Tim Althoff.  A computational approach to understanding empathy expressed in text-based mental health support.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5263–5276, 2020.
  * Tahir et al. [2023] Sharjeel Tahir, Syed Afaq Shah, and Jumana Abu-Khalaf.  Artificial empathy classification: A survey of deep learning techniques, datasets, and evaluation scales.  arXiv preprint arXiv:2310.00010, 2023.
  * Touvron et al. [2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  arXiv preprint arXiv:2307.09288, 2023.
  * Wei et al. [2022] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al.  Emergent abilities of large language models.  Transactions on Machine Learning Research, 2022.
  * Welivita and Pu [2020] Anuradha Welivita and Pearl Pu.  A taxonomy of empathetic response intents in human social conversations.  In Proceedings of the 28th International Conference on Computational Linguistics, pages 4886–4899, 2020.
  * Welivita et al. [2021] Anuradha Welivita, Yubo Xie, and Pearl Pu.  A large-scale dataset for empathetic response generation.  In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1251–1264, 2021.
  * Wilson and Marasoiu [2022] Lee Wilson and Mariana Marasoiu.  The development and use of chatbots in public health: scoping review.  JMIR human factors, 9(4):e35882, 2022.

Figure 6: Three in-context learning examples annotated by patients.

##  Appendix A Zero-shot post-processing

As we mentioned, the output of the zero-shot evaluation is free-form and thus
we need to manually check the output and define two regular expressions to
extract the assessment from the model. Table 6 shows the regular expression
patterns.

Pattern | Value
---|---
Pattern 1 | response [12] \w+ (?:slightly )?more empath
Pattern 2 | \w+ that shows more empathy \w+ response [12]

Table 6: Two patterns are used to process the zero-shot evaluation.

##  Appendix B In-context Learning Examples

Figure 6 shows the three in-context learning examples used in few-shot
learning or one-shot learning evaluation.

##  Appendix C Empathy Evaluation Questionnaire

We prepare 7 Google Forms and each one includes 10 questions that ask patients
to rank the empathy degree of two responses. Figure 7 shows a snapshot of the
questionnaire.

Figure 7: Questionnaire to collect the Assessment from Patient.