# Moral Narratives Around the Vaccination Debate on Facebook

Mariano G. Beiró<EMAIL_ADDRESS>0000-0002-5474-0309 Universidad de Buenos
Aires. Facultad de IngenieríaBuenos AiresArgentina CONICET-Universidad de
Buenos Aires. INTECINBuenos AiresArgentina ,  Jacopo D’ Ignazi
<EMAIL_ADDRESS>ISI FoundationTurinItaly ,  María Florencia Prado
Universidad de Buenos Aires. Facultad de IngenieríaBuenos AiresArgentina ,
Victoria Pérez Bustos  Universidad de Buenos Aires. Facultad de
IngenieríaBuenos AiresArgentina and  Kyriaki Kalimeri  0000-0001-8068-5916
<EMAIL_ADDRESS>ISI FoundationTurinItaly

(2023)

###### Abstract.

Vaccine hesitancy is a complex issue with psychological, cultural, and even
societal factors entangled in the decision-making process. The narrative
around this process is captured in our everyday interactions; social media
data offer a direct and spontaneous view of peoples’ argumentation. Here, we
analysed more than 500,000 public posts and comments from Facebook Pages
dedicated to the topic of vaccination to study the role of moral values and,
in particular, the understudied role of the Liberty moral foundation from the
actual user-generated text. We operationalise morality by employing the Moral
Foundations Theory, while our proposed framework is based on recurrent neural
network classifiers with a short memory and entity linking information. Our
findings show that the principal moral narratives around the vaccination
debate focus on the values of Liberty, Care, and Authority. Vaccine advocates
urge compliance with the authorities as prosocial behaviour to protect
society. On the other hand, vaccine sceptics mainly build their narrative
around the value of Liberty, advocating for the right to choose freely whether
to adhere or not to the vaccination. We contribute to the automatic
understanding of vaccine hesitancy drivers emerging from user-generated text,
providing concrete insights into the moral framing around vaccination
decision-making. Especially in emergencies such as the Covid-19 pandemic,
contrary to traditional surveys, these insights can be provided contemporary
to the event, helping policymakers craft communication campaigns that
adequately address the concerns of the hesitant population.

vaccine hesitancy, natural language processing, social media, deep learning,
moral foundations theory

††journalyear: 2023††copyright: acmlicensed††conference: Proceedings of the
ACM Web Conference 2023; April 30-May 4, 2023; Austin, TX, USA††booktitle:
Proceedings of the ACM Web Conference 2023 (WWW ’23), April 30-May 4, 2023,
Austin, TX, USA††doi: 10.1145/3543507.3583865

Vaccines have saved countless lives; among the significant achievements of
vaccines is the eradication of smallpox in the 1960s and 1970s, a disease
estimated to have killed up to 300 million people, including six European
monarchs, in the 20th-century alone (Henderson, 2011). Despite being the most
successful and cost-effective health intervention in human history, vaccine
hesitancy continues to grow in both low- and high-income countries (Larson et
al., 2011; Salmon et al., 2015). In 2019 the World Health Organization (WHO)
declared vaccine hesitancy as one of the top 10 threats to global health
(Organization, 2019).

More recently, a debate mainly concerned parents arguing about children’s
vaccination (Betti et al., 2021) scaled up with the COVID-19 pandemic to a
broader point of argumentation (Mejova and Kalimeri, 2020). Despite the
scientific consensus about the safety of vaccines, objections continue to rise
with familiar narratives that mainly evolve around science denial (Browne et
al., 2015), conspiracy theories (Jolley and Douglas, 2014), and alternative
health practices (Kalimeri et al., 2019; Kata, 2010). The drivers of vaccine
hesitancy are complex, ranging from the individual (i.e. psychological) to
interpersonal (social cycle interactions) and public policy (i.e. national and
local agencies) (Dubé and MacDonald, 2016; MacDonald et al., 2015). Not all
factors received the same attention from researchers and practitioners; still,
there is very little work examining the psychological and, in particular, the
moral values underpinning objection towards vaccination. Even if noted moral
reasoning was related to vaccination opinion formation (Kunda, 1990), it was
the development of the Moral Foundations Theory (MFT) (Haidt and Joseph, 2004)
to provide scientists with a concrete framework to assess moral values
systematically.

MFT focuses on the explanation of the psychological basis of morality, its
origins, development, and cultural variations, and identifying the following
five moral foundations (Haidt and Graham, 2007; Haidt and Joseph, 2004);
Care/Harm: basic concerns for the suffering of others, including virtues of
caring and compassion, Fairness/Cheating: concerns about unfair treatment,
inequality, and more abstract notions of justice, Loyalty/Betrayal: concerns
related to obligations of group membership, such as loyalty, self-sacrifice
and vigilance against betrayal, Authority/Subversion: concerns related to
social order and the obligations of hierarchical relationships such as
obedience, respect, and proper role fulfilment, and Purity/Degradation:
concerns about physical and spiritual contagion, including virtues of
chastity, wholesomeness and control of desires. More recently, the theory was
enhanced with a sixth dimension: Liberty/Oppression (Haidt, 2012). Liberty
expresses the feelings of reactance and resentment towards oppressors.

Importantly, there is a scientific consensus on the moral viewpoints related
to vaccine hesitancy (Amin et al., 2017; Kalimeri et al., 2019; Rossen et al.,
2019). The common denominator of the argumentation narrative against vaccine
uptake includes arguments of “Purity” of the body and soul, advocating for
alternative health practices, notions of “Liberty”, intended as freedom of
choice as an alternative to obligatory vaccination schemes (Betti et al.,
2021; Kalimeri et al., 2019). Till now, most studies have been carried out
employing questionnaires which, even if administered online, still have a
limited reach. Here, we assess the moral narratives of vaccine advocates and
sceptics on social media (SM), particularly on posts and comments on Facebook
Pages related to vaccination. Such an approach can provide additional sources
of information, especially from under-represented countries and populations in
the peer-reviewed literature (Larson et al., 2014).

Here, we assess the moral viewpoints of people analysing user-generated text
on social media. Focusing on the linguistic measurements, we trained a Long
Short-Term Memory (LSTM) deep neural network to infer the expression of moral
values in text (Hochreiter and Schmidhuber, 1997). This study aims to provide
insights into the moral narratives around vaccine hesitancy, analysing a large
volume of user-generated data. Our findings show that people centre their
argumentation around the moral value of liberty, care, and authority. When
focusing on the argumentation of each side, we notice that vaccine advocates
prioritise the need to comply with authorities to protect society as a whole,
while sceptics favour the value of liberty, expressed as freedom of choice on
whether to adhere or not to the vaccination. These insights can be
constructive since public health communication campaigns commonly develop
their narratives around care and the shared responsibility of protecting the
most vulnerable in our community. However, such messaging does not reflect the
concerns of those sceptical about vaccinations who prioritise notions of
freedom of choice and alternative health remedies. We hope these findings will
inform policymakers and practitioners to craft communication campaigns to
bridge the mistrust between health care and the public (Dhaliwal et al., 2020;
Dubé et al., 2015).

##  1\. Related Work

Social science research has shown that vaccination decision-making should be
understood in a broader socio-cultural context (Dubé et al., 2013). Vaccine
hesitancy has a long history, with the role of traditional media and later of
the Internet to offer an opportunity for vocal anti-vaccination activists to
diffuse their message (Zimmerman et al., 2005). SM platforms have further
magnified this phenomenon, allowing individuals to rapidly create and share
content globally without editorial oversight (Puri et al., 2020). SM users are
subject to confirmation bias and selective exposure, interacting only with
like-minded peers in the so-called echo-chamber effect (Cinelli et al., 2021).

Such phenomena can be particularly concerning since often they proliferate
misinformation while have been noticed both on the Twitter platform (Cossard
et al., 2020) and Facebook (Schmidt et al., 2018)111In the light of these
findings, Facebook took several actions to tackle fake news decreasing their
proliferation by adding a quality control
https://about.fb.com/news/2019/03/combatting-vaccine-misinformation/. Even
more concerning findings with broad implications for public health messaging
were brought to light regarding the advertising communication campaigns placed
on the Facebook platform by different stakeholders, including politicians,
entrepreneurs, and public health authorities, in an intense competition for
the audience’s attention (Mejova and Kalimeri, 2020).

Despite this, not all online communication around vaccination is harmful; a
study on a parenting forum showed that users were eager to exchange opinions
and learn from each other (Betti et al., 2021). Applying natural language
processing techniques to a corpus of more than one million comments from
BabyCenter US, a popular parenting forum, the authors showed that hesitant
parents were mainly influenced by personal and peer environments’ adverse
reactions. At the same time, they were more likely to be interested in
alternative medicine and natural lifestyles, suggesting a broader distrust of
healthcare providers and mainstream medicine.

Focusing on the Facebook Platform, Klimiuk et al. (Klimiuk et al., 2021)
manually categorised $\approx 20,000$ comments on vaccination in Poland as per
the main categories of hesitancy. In decreasing order of importance, they
ranked conspiracy theories, falsehoods, concerns regarding the safety of
vaccines, and violations of the human right of liberty, presenting the
compulsory vaccination as an act of “totalitarianism”. A limitation of this
study is that the authors considered only one Facebook Page for three months.
Similar clusters were obtained by Hoffman et al. (Hoffman et al., 2019) from
the manual analysis of Facebook status posts on a sample of $197$ individuals
in the US. Kalimeri et al. (Kalimeri et al., 2019) analysed the opinions of a
cohort of approximately 2,000 participants from Italy who were following
Facebook Pages about vaccination via online administered questionnaires. Their
findings showed that those more sceptical about vaccine uptake showed more
anti-authoritarian attitudes; in particular, they did not trust the presidency
of the country and the current government while also expressing anti-European
opinions. Moreover, they vigorously defend the traditional moral values of
Italian society and, in particular, the religion while explicitly stating that
newer lifestyle leads to the decline of civilisation.

Table 1. Dataset statistics. The last row points out the number of comments effectively used to train and evaluate our models, which were filtered to contain at least five words, excluding mentions. | PV Pages | AV pages
---|---|---
Pages | $101$ | $85$
Original Posts | $52,894$ | $24,615$
Original Comments | $215,341$ | $391,764$
Filtered Comments | $170,954$ | $286,111$

Such findings are emerging from more than online and social media platform
data. Studies employing traditionally administered surveys reach the same
conclusions. In particular, Rossen et al. (Rossen et al., 2019) administered
an online questionnaire to $296$ individuals in Australia. The authors showed
that vaccine-hesitant individuals valued significantly fewer concepts of
authority and more notions of purity, liberty and fairness. Previously, Amin
et al. (Amin et al., 2017) surveyed 1,000 parents in the US, showing that the
higher the vaccine hesitancy, the higher the level of purity and liberty moral
foundations. Interestingly, they also showed that the harm/care foundation,
intensively addressed in traditional pro-vaccination campaigns, did not
significantly impact vaccine hesitancy.

Moral values detection from user-generated text is not a trivial problem since
values are often only implicitly signalled in language. The majority of
predictive models developed to address this task are trained on easily
accessible Twitter data (Araque et al., 2020; Mooijman et al., 2018), while
they are often based on computational linguistics and lexicon-based approaches
(Graham et al., 2009; Araque et al., 2020, 2022). Except for a preliminary
study by Araque et al. (Araque et al., 2021, 2022), none of the well-
established lexicons includes the “Liberty” moral foundation, which plays a
central role in the vaccination debate according to the related literature.
Here, contributing to the current state of the art, we propose an entirely
data-driven approach based on neural networks to infer the moral narratives in
user-generated text.

##  2\. Data Collection

Table 2.  Annotation of Moral values. The content of each comment was annotated with respect to the moral value expressed as Care/Harm, Fairness/Cheating, Loyalty/Betrayal, Authority/Subversion, and Purity/Degradation,Liberty/Oppression, or “Non moral”; Virtue and Vice denote positive and negative polarity in the foundation spectrum. Note that a comment may express more than one moral values.  | Presence | Virtue | Vice
---|---|---|---
Authority | 314 | 110 | 204
Liberty | 205 | 140 | 65
Loyalty | 78 | 40 | 38
Care | 489 | 357 | 132
Fairness | 297 | 174 | 123
Purity | 192 | 80 | 112
Non-moral | 353 | - | -
Non-relevant | 2883 | - | -

Replicating the methodology of (Schmidt et al., 2018), we collected data from
Facebook Pages in the period Jan 2012-Jun 2019, using the Facebook Graph API
(FBa, [n. d.]). We queried the API for public Pages containing the keywords
“vaccine”, “vaccines”, or “vaccination” in their name. Then we excluded from
the analysis Facebook pages that were not in the English language and were not
relevant to the topic of vaccination (e.g., the Facebook page of the music
band “The Vaccines”). The remaining pages were manually categorised according
to their content into supportive (hereafter, PV, for brevity) or contrary
(hereafter, AV) to vaccination. We also collected all public posts and
comments from each Page. Overall, we obtained 607,105 comments in $186$ “Pro”
and “Anti”222Here, we conventionally used the term “Anti” as an abbreviation
to refer to the vaccine-hesitant population. vaccination pages. After removing
comments with less than five words, our final dataset consisted of
approximately 450,000 comments. A summary of the dataset is shown in the upper
part of Table 1.

Since vaccination is a highly controversial topic, and despite the moderation
of Facebook Pages, the published posts may bring up discussions where both
sides are expressed regardless of the primary stance of the Page. Hence, we
annotated 4,498 comments independently of the affiliation of the Page on which
they were written. We employed nine annotators, all familiar with the moral
foundation’s theory. For each comment, we annotated it as supporting (pro-),
opposing (anti-) or not taking any stance (non-relevant) according to its
position towards vaccination. For those comments that did take a stance
towards vaccination, we further annotated the expression of moral values, if
any, indicating the “vice” or the “virtue” of the foundation. Table 2 reports
the distribution of the annotated comments. We assessed the inter-annotator
agreement via Cohen’s kappa coefficient, obtaining .78 (substantial agreement)
for the first part of the annotation concerning the relevance to vaccination
and .32 (fair agreement) for the moral value expressed in the comment. The
latter shows the difficulty of the task also for human annotators.

A limitation of this study is that we need to have information about the
author of the comments; hence, we cannot draw any conclusion about the
adequate size of each community or the users’ engagement in the discussion.

##  3\. Methods

(a) Deep network architecture used for predicting comment types.

(b) Deep network architecture used for predicting moral foundations.

Figure 1.  Overview of the deep network architectures employed for prediction
of relevance to vaccination 1(a) and moral values 1(b). For relevance
prediction 1(a), the embeddings of each comment go through two parallel
pipelines: one processing the text with a bidirectional LSTM, and the other
applying entity recognition through TagMe. For moral values prediction 1(b),
the embeddings of each comment enter a bidirectional LSTM, and the final layer
predicts each moral’s virtue and vice using a softmax activation function.

The complexity of the reasoning and the background knowledge often entailed in
the lengthy comments posted on the Facebook Platform render the above
methodologies inadequate. Here, we propose an approach based on a Recurrent
Neural Network-based classifier with long short-term memory (LSTM)333The
architecture was implemented in Keras (Chollet et al., 2015). inspired by the
work of Lin et al. (Lin et al., 2018).

Initially, we aim to identify the comments that indeed argue about
vaccination. To do so, we build a Relevance Classifier which predicts whether
a specific comment is explicitly favouring (“Pro”), hesitant (“Anti”), or not
expressing a position (Non-relevant, “NR”) towards vaccination. The Relevance
Classifier is composed of three parallel stages (see Fig. 1(a)): an LSTM
neural network, a named entity recognition stage based on TagMe (Ferragina and
Scaiella, 2010) and a page class stage. Then, we further classify the comments
relevant to vaccination concerning the appeal to morality in terms of the
presence of a moral foundation (e.g. “Care-Presence” vs “No Care-Presence”)
and its polarity (e.g. “Care-Virtue”, “Care-Vice”). The structure of the Moral
Classifier consists of a word embedding component and an LSTM neural network
(see Fig. 1(b)). We train the relevance and moral classifiers on the annotated
subset of comments; then, these predictive models are employed to infer the
unlabelled part of the dataset.

In detail, for the Relevance component, we apply basic natural language pre-
processing techniques (i.e. stemming, removing punctuation and stopwords,
tokenisation) and then employ the 100-dimensional pre-trained Glove embeddings
(Pennington et al., 2014) model to obtain the word representation. To
incorporate background knowledge, we apply entity linking to associate
mentions with their referent entities, using TagMe (Ferragina and Scaiella,
2010). TagMe is a powerful tool that identifies on-the-fly meaningful
substrings (called “spots”) in an unstructured text and links each of them to
a relevant Wikipedia page efficiently and effectively. TagMe provides a
confidence score of $\rho$, an estimate of the annotation’s “goodness”
concerning the other entities of the input text. For this study, we threshold
this confidence score to $\rho\geq 0.1$ and utilise them as additional
features to improve the prediction. The bidirectional LSTM takes as input the
sequence of word embeddings $\\{w1,w2,...,wn\\}$ in a text and output hidden
states $\\{h1,h2,...,hn\\}$; the last output $hn$ is then passed to the fully
connected layer. Finally, we concatenate them with the vectors from the
background knowledge and add a final dense layer with a $\tanh$ activation
function. The training dataset was built from a subset sample of the complete
annotations to have a balanced dataset among the Pro, Anti and NR classes; the
training was thus performed on 2100 comments, 700 for each class. To prevent
overfitting, we apply dropout to outputs of the embedding, LSTM, and fully
connected layers, while we also perform 10-fold cross-validation.

For the comments in our dataset classified to express a “Pro” or an “Anti”
vaccination opinion, we trained one LSTM model to predict each moral
foundation’s presence and polarity. The Moral component follows the same pre-
processing and embedding approach (Pennington et al., 2014) model to obtain
the word representation. Similarly, these are fed to the bidirectional LSTM.
In the case of polarity prediction, we perform 5-fold cross-validation due to
the limited amount of labelled data per moral class (see Table 2). We report
the Area Under the Receiver Operating Characteristic curve (AUROC) for all
steps for each classifier. Figure 1(b) depicts the architecture of this
design.

##  4\. Results and Discussion

Figure 2. Monthly percentages of comments classified as (PRO, ANTI, NR)-vax in Pro-vax and Anti-vax pages during time. Table 3.  Cross-validated performance (AUROC) of different models for the prediction of a comment’s position: (PRO, ANTI, NR)-vax. Regression Model, LSTM branch of the model and the full model as described in Section 3. The Regression Model serves as a baseline model. | Regression Model | LSTM branch | LSTM full
---|---|---|---
_PV comments_ | $.51\pm.03$ | $.63\pm.04$ | $.71\pm.06$
_AV comments_ |  $.51\pm.0$4 | $.80\pm.03$ | $.82\pm.02$
_NR comments_ | $.50\pm.04$ | $.71\pm.04$ | $.80\pm.05$
Table 4. Percentages of comments classified as Pro-Vax, Anti-Vax, or Non-Relevant (NR) in regard to vaccination stances. Second and third column express the percentage of comments expressing these stances in Pro-Vax and Anti-Vax Facebook pages in our dataset. | All pages | PV pages | AV pages
---|---|---|---
_PV comments_ | 24.18% | 29.13% | 21.22%
_AV comments_ | 28.94% | 24.91% | 31.35%
_NR comments_ | 46.88% | 45.96% | 47.42%

Initially, from our data collection, we noticed that overall the AV pages are
fewer, with almost half of the posts, which triggered almost double the
comments concerning the PV pages. This observation is confirmed by the
findings reported in the currently related literature regarding the vaccine
sceptics, who appear to be an extremely vocal minority (Cossard et al., 2020).

Stance Classification. Moving to the relevance classification, our model
accurately predicted the stance towards vaccination with an average accuracy
of $.76$ on a 10-fold cross-validation scheme. We conventionally refer to the
AUROC curve (Area Under the Receiver Operating Characteristic) metric as
“accuracy”. We opted for this metric to evaluate our framework due to its
stability in heavily imbalanced classification tasks like ours. This is an
encouraging finding showing that the model can distinguish whether an unseen
comment is supportive (“Pro”) or sceptical (“Anti”) towards vaccination or
whether it does not take any position. When applying this model to predict the
comments’ stance on the entire dataset, we notice that approximately half the
comments do not express a firm position on the debate. The vaccine sceptic
comments are more frequent than the supportive ones (see Table 4), confirming
the characterisation of the vaccine sceptics as a vocal minority. However, we
find that despite the persistent content moderation observed on the Facebook
platform leading to echo-chamber effects (Schmidt et al., 2018), in both PV
and AV pages, there exists an appropriate amount of comments expressing the
opposite stance, which suggests that there is significant interaction and
exposure to different opinions within both communities (Betti et al., 2021).
Figure 2 shows that the evolution in time of the percentages for pro- anti-
and non-relevant comments remain stable.

As a baseline for comparison, we considered two approaches: a benchmark model
implemented as a Logistic Regression classifier trained on a bag-of-words
representation for each comment. Secondly, an ablation study where the LSTM
model is trained only on text features. This latter evaluation step accounts
for the importance of the annotations and page information for the final
prediction task. Table 3 shows that our architecture (third column)
significantly outperforms the Logistic Regression model (first column) as well
as the simple LSTM architecture (second column), highlighting the importance
of the TagMe and pages features for understanding the stance towards
vaccination.

Moral Presence Classification. In the second step of the classification, we
detect the presence of each moral foundation in the vaccine-related comments.
Overall, from the entire dataset, 107,712 comments have been classified as
vaccine supportive, while 128,939 as hesitant; the remaining ones, around
210,000, have instead been classified as non-relevant to the topic of
vaccination. Table 5 depicts the average AUROC for each foundation trained and
tested on the labelled data with a 10-fold cross-validation scheme. We notice
that the loyalty foundation is the hardest to predict; this is a
straightforward limitation of the annotated sample of comments since there the
Loyalty was the rarest foundation to be detected, and hence the training
samples were few. Interestingly, Lin et al. (Lin et al., 2018) also report
LoyaltyLoyalty as the most challenging foundation to predict, despite the
higher availability of labelled training data. The Liberty and care moral
foundations were the ones to be predicted most accurately with .71 and .68,
respectively (see Table 5). Especially for the case of the Liberty foundation,
this finding shows that despite the limited annotated instances for the other
foundations (205 for Liberty versus 489 for Care), the content is quite
consistent, achieving the highest accuracy.

Moral Polarity Classification. Finally, we trained a multi-target classifier
with twelve binary targets, one for each foundation and polarity. Table 5
(Virtue and Vice columns) depict the AUROC scores for the Virtue and the Vice
polarities of each foundation, respectively. Here we find that Authority is
the best-performing one, with an AUROC of $.65$ on either polarity. Overall,
the performance of this model is poorer than that of the model predicting the
presence of each moral foundation, probably due to the lack of training data
(see Table 2) and the fact that the presence prediction model puts together
the annotations regarding Virtue and vice of each foundation, thus gathering
more data.

Table 5. Average and Standard Deviation of the LSTM predictors accuracy for each of the six moral foundations, expressed in terms of the area under the receiver operating characteristic (AUROC). | Presence | Moral Virtue | Moral Vice
---|---|---|---
|  | (Positive Polarity) | (Negative Polarity)
Authority | $.61\pm.07$ | .65 | .65
Liberty | $.71\pm.08$ | .58 | .61
Loyalty | $.63\pm.12$ | .71 | .61
Care | $.68\pm.05$ | .62 | .55
Fairness | $.60\pm.06$ | .56 | .60
Purity | $.61\pm.08$ | .49 | .57
Table 6. Measures of moral content predicted by the moral values classifier. The first three columns represent the percentage of comments that exhibit the presence of a given moral (either positive or negative) in comments of different type: Pro-vax (PV), Anti-vax (AV), and Non-relevant (NR). Ratios in the three remaining columns represent, among those comments, the fraction of positive ones (virtue) over negative ones (vice) in relation to that moral value. | Occurrences | Virtue vs Vice Ratios
---|---|---
Authority | 9.40% | 30.76% | 36.23% | 0.34 | 0.63 | 0.17
Liberty | 8.58% | 12.50% | 16.07% | 0.72 | 1.49 | 2.20
Loyalty | 9.38% | 9.07% | 7.75% | 0.08 | 0.29 | 0.48
Care | 8.98% | 26.44% | 36.38% | 1.14 | 2.82 | 2.02
Fairness | 25.58% | 24.55% | 21.07% | 0.05 | 0.46 | 0.89
Purity | 12.67% | 21.33% | 22.87% | 0.34 | 0.50 | 1.03
| _NR_ | _PV_ | _AV_ | _NR_ | _PV_ | _AV_

To assess the moral narratives in how people reason about vaccination, we
confronted the moral presence of messages that support vaccination as opposed
to the hesitant ones (see Table 6). Note that a comment may express more than
one moral foundation or be non-moral: from the results of the classifier over
all the 12 combinations across moral foundations and polarities, we indeed
find that around $\sim 32\%$ of the comments are assigned more than one moral
value, $\sim 35\%$ just one, and $\sim 33\%$ are assigned no moral foundation
at all.

As a measure of the leaning of each group towards the extremes of the moral
axis, we evaluated the ratio of comments classified as positive and negative
for each foundation. This is:

(1) $VVR(M,S)=\frac{P(Virtue|M,S)}{P(Vice|M,S)}$

where $M$ represents a moral foundation and $S$ is the stance towards
vaccination. A high value of this ratio tells us that group $S$ highly values
foundation $M$ as a virtue. Thus, comparing these ratios between $Pro$ and
$Anti$ comments is a measure of how vaccination support and hesitancy
correlate to moral values.

The analysis of these ratios for the different morals and groups is shown in
Table 6. We observe that comments classified as opposing vaccination express
more intensely the moral value of Liberty (Virtue) ($VVR$ of $2.20$) in
comparison with those supporting vaccination ($VVR$ of $1.49$). Moreover,
vaccine-hesitant individuals are firmly positioned against the authoritarian
actions of those in charge ($VVR$ of $0.17$). Such findings confirm the
current state-of-the-art literature (Amin et al., 2017; Kalimeri et al., 2019;
Betti et al., 2021), which underlines the importance that the vaccine-hesitant
community gives to notions related to freedom of choice. Focusing on the
liberty, care and authority foundations, Figure 3 depicts the Virtue vs Vice
Ratio (VVR) over time. We notice a stable and consistent difference in moral
framing, with the vaccine advocates expressing more concerns about the
protection of others while the hesitant tend to focus more on the freedom of
choice and self-determination argumentation. We also find that comments
supporting vaccination express more familiar notions of the Vice spectrum of
the Purity foundation; such narratives often criticise natural remedies and
holistic alternatives to vaccination in line with the literature (Kata, 2010).

Figure 3. VVR (Virtue vs Vice Ratio) over time for the Care, Liberty and
Authority foundations expressed in predicted Pro-vax and Anti-vax comments. To
denoise we applied a moving average filter per month with a six month window
frame.

The importance of these findings lies in the insights provided into the moral
narratives and argumentation of vaccine advocates and hesitant. In critical
social issues such as vaccine hesitancy, mutual understanding of the other’s
side argumentation is fundamental to crafting communication campaigns
addressing these specific concerns and drivers of hesitancy.

##  5\. Conclusions

The drivers of vaccine hesitancy are complex; scientists and medical doctors
have long been trying to map the causes and dependencies of that factors (Dubé
and MacDonald, 2016; Kata, 2010). Even though various psychological
attributes, such as personality traits, have been related to vaccine
hesitancy, the role of moral values has been largely understudied.

Here, we employ the Moral Foundations Theory to assess the moral narratives in
random, user-generated text from social media. We analysed more than 400,000
comments on Facebook pages regarding vaccination, both from the supporting and
opposing sides. In particular, we systematically assess the role of the
Liberty moral foundation previously omitted. While in the future, we aim to
assess the moral narratives via bidirectional encoder representations from
transformers; here, we build on the work of Lin et al. (Lin et al., 2018). We
trained a series of models based on Recurrent Neural Networks with short-term
memory and entity linking.

Here, we confirmed that the vaccine sceptic community is a vocal minority;
however, despite the platform’s moderation, the users are exposed to the
content of like-minded peers and diverse opinions. The deep architecture
proposed is shown to outperform the baseline models while performing similarly
to existing models in moral values prediction from the user-generated text.
The main contribution of this study is the large-scale, automatic, and in-
depth analysis of the moral framing of vaccine advocates and sceptics,
advancing the understanding of vaccine hesitancy drivers.

Our findings confirm the importance of the liberty foundation to the
discussion around vaccination and feelings of oppression from the authorities.
Such indications are also supported by survey-based social science and public
health studies; however, this is the first time such a narrative has emerged
in a data-driven way from automatically analysing user-generated text.
Understanding the viewpoints of the hesitant community will hopefully lead to
communication campaigns and strategies that avoid eliciting backfiring effects
such as those of an obligatory vaccination act.

Interestingly, in light of the recent COVID-19 pandemic, the vaccine hesitancy
drivers are likely to be rooted in the same moral narratives. Antivaccination
campaigns built around concepts of holistic cures and natural remedies to
substitute vaccines gained publicity, putting at risk the global public health
(Larson, 2018). Such insights are critical during public health emergencies
and can help prevent misinformation attempts that develop their narratives
around people’s concerns (Mejova and Kalimeri, 2020).

###### Acknowledgements.

KK and JI acknowledge financial support from the Lagrange Project of the
Institute for Scientific Interchange Foundation (ISI Foundation) funded by
Fondazione Cassa di Risparmio di Torino (Fondazione CRT).

## References

  * (1)
  * FBa ([n. d.]) [n. d.].  Facebook (August 2013) Using the graph API (Facebook, Menlo Park, CA).  https://developers.facebook.com/docs/graph-api/using-graph-api/.  [Online; accessed 14-June-2019].
  * Amin et al. (2017) Avnika B Amin, Robert A Bednarczyk, Cara E Ray, Kala J Melchiori, Jesse Graham, Jeffrey R Huntsinger, and Saad B Omer. 2017\.  Association of moral values with vaccine hesitancy.  _Nature Human Behaviour_ 1, 12 (2017), 873–880.
  * Araque et al. (2020) Oscar Araque, Lorenzo Gatti, and Kyriaki Kalimeri. 2020\.  MoralStrength: Exploiting a moral lexicon and embedding similarity for moral foundations prediction.  _Knowledge-based systems_ 191 (2020), 105184.
  * Araque et al. (2021) Oscar Araque, Lorenzo Gatti, and Kyriaki Kalimeri. 2021\.  The Language of Liberty: A Preliminary Study. In _Companion Proceedings of the Web Conference 2021_ (Ljubljana, Slovenia) _(WWW ’21)_. Association for Computing Machinery, New York, NY, USA, 623?626.  https://doi.org/10.1145/3442442.3452351
  * Araque et al. (2022) Oscar Araque, Lorenzo Gatti, and Kyriaki Kalimeri. 2022\.  LibertyMFD: A Lexicon to Assess the Moral Foundation of Liberty.. In _Proceedings of the 2022 ACM Conference on Information Technology for Social Good_. 154–160.
  * Betti et al. (2021) Lorenzo Betti, Gianmarco De Francisci Morales, Laetitia Gauvin, Kyriaki Kalimeri, Yelena Mejova, Daniela Paolotti, and Michele Starnini. 2021.  Detecting adherence to the recommended childhood vaccination schedule from user-generated content in a US parenting forum.  _PLoS computational biology_ 17, 4 (2021), e1008919.
  * Browne et al. (2015) Matthew Browne, Patricia Thomson, Matthew Justus Rockloff, and Gordon Pennycook. 2015.  Going against the herd: psychological and cultural factors underlying the ?vaccination confidence gap?.  _PLoS one_ 10, 9 (2015), e0132562.
  * Chollet et al. (2015) François Chollet et al. 2015\.  Keras.  https://github.com/fchollet/keras.
  * Cinelli et al. (2021) Matteo Cinelli, Gianmarco De Francisci Morales, Alessandro Galeazzi, Walter Quattrociocchi, and Michele Starnini. 2021.  The echo chamber effect on social media.  _Proceedings of the National Academy of Sciences_ 118, 9 (2021), e2023301118.
  * Cossard et al. (2020) Alessandro Cossard, Gianmarco De Francisci Morales, Kyriaki Kalimeri, Yelena Mejova, Daniela Paolotti, and Michele Starnini. 2020\.  Falling into the echo chamber: the Italian vaccination debate on Twitter. In _Proceedings of the International AAAI Conference on Web and Social Media_ , Vol. 14. 130–140.
  * Dhaliwal et al. (2020) Dhamanpreet Dhaliwal, Cynthia Mannion, et al. 2020\.  Antivaccine messages on facebook: preliminary audit.  _JMIR public health and surveillance_ 6, 4 (2020), e18878.
  * Dubé et al. (2015) Eve Dubé, Dominique Gagnon, Noni E MacDonald, et al. 2015\.  Strategies intended to address vaccine hesitancy: Review of published reviews.  _Vaccine_ 33, 34 (2015), 4191–4203.
  * Dubé et al. (2013) Eve Dubé, Caroline Laberge, Maryse Guay, Paul Bramadat, Réal Roy, and Julie A Bettinger. 2013\.  Vaccine hesitancy: an overview.  _Human vaccines & immunotherapeutics_ 9, 8 (2013), 1763–1773.
  * Dubé and MacDonald (2016) Eve Dubé and Noni E. MacDonald. 2016.  Chapter 26 - Vaccine Acceptance: Barriers, Perceived Risks, Benefits, and Irrational Beliefs.  In _The Vaccine Book (Second Edition)_ (second edition ed.), Barry R. Bloom and Paul-Henri Lambert (Eds.). Academic Press, 507–528.  https://doi.org/10.1016/B978-0-12-802174-3.00026-6
  * Ferragina and Scaiella (2010) Paolo Ferragina and Ugo Scaiella. 2010.  Tagme: on-the-fly annotation of short text fragments (by wikipedia entities). In _Proceedings of the 19th ACM international conference on Information and knowledge management_. 1625–1628.
  * Graham et al. (2009) Jesse Graham, Jonathan Haidt, and Brian A Nosek. 2009\.  Liberals and conservatives rely on different sets of moral foundations.  _Journal of personality and social psychology_ 96, 5 (2009), 1029.  https://doi.org/10.1037/a0015141
  * Haidt (2012) Jonathan Haidt. 2012\.  _The righteous mind: Why good people are divided by politics and religion_.  Vintage.
  * Haidt and Graham (2007) Jonathan Haidt and Jesse Graham. 2007.  When morality opposes justice: Conservatives have moral intuitions that liberals may not recognize.  _Social Justice Research_ 20, 1 (2007), 98–116.
  * Haidt and Joseph (2004) Jonathan Haidt and Craig Joseph. 2004.  Intuitive ethics: How innately prepared intuitions generate culturally variable virtues.  _Daedalus_ 133, 4 (2004), 55–66.
  * Henderson (2011) Donald A Henderson. 2011\.  The eradication of smallpox–an overview of the past, present, and future.  _Vaccine_ 29 (2011), D7–D9.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ 9, 8 (1997), 1735–1780.
  * Hoffman et al. (2019) Beth L Hoffman, Elizabeth M Felter, Kar-Hai Chu, Ariel Shensa, Chad Hermann, Todd Wolynn, Daria Williams, and Brian A Primack. 2019.  It?s not all about autism: The emerging landscape of anti-vaccination sentiment on Facebook.  _Vaccine_ 37, 16 (2019), 2216–2223.
  * Jolley and Douglas (2014) Daniel Jolley and Karen M Douglas. 2014.  The effects of anti-vaccine conspiracy theories on vaccination intentions.  _PloS one_ 9, 2 (2014), e89177.
  * Kalimeri et al. (2019) Kyriaki Kalimeri, Mariano G. Beiró, Alessandra Urbinati, Andrea Bonanomi, Alessandro Rosina, and Ciro Cattuto. 2019.  Human values and attitudes towards vaccination in social media. In _Companion Proceedings of The 2019 World Wide Web Conference_. 248–254.
  * Kata (2010) Anna Kata. 2010\.  A postmodern Pandora’s box: anti-vaccination misinformation on the Internet.  _Vaccine_ 28, 7 (2010), 1709–1716.
  * Klimiuk et al. (2021) Krzysztof Klimiuk, Agnieszka Czoska, Karolina Biernacka, and Łukasz Balwicki. 2021.  Vaccine misinformation on social media–topic-based content and sentiment analysis of Polish vaccine-deniers? comments on Facebook.  _Human Vaccines & Immunotherapeutics_ (2021), 1–10.
  * Kunda (1990) Ziva Kunda. 1990\.  The case for motivated reasoning.  _Psychological bulletin_ 108, 3 (1990), 480\.
  * Larson (2018) Heidi J Larson. 2018\.  The biggest pandemic risk? Viral misinformation.  _Nature_ 562, 7726 (2018), 309–310.
  * Larson et al. (2011) Heidi J Larson, Louis Z Cooper, Juhani Eskola, Samuel L Katz, and Scott Ratzan. 2011.  Addressing the vaccine confidence gap.  _The Lancet_ 378, 9790 (2011), 526–535.
  * Larson et al. (2014) Heidi J Larson, Caitlin Jarrett, Elisabeth Eckersberger, David MD Smith, and Pauline Paterson. 2014\.  Understanding vaccine hesitancy around vaccines and vaccination from a global perspective: a systematic review of published literature, 2007–2012.  _Vaccine_ 32, 19 (2014), 2150–2159.
  * Lin et al. (2018) Ying Lin, Joe Hoover, Gwenyth Portillo-Wightman, Christina Park, Morteza Dehghani, and Heng Ji. 2018\.  Acquiring background knowledge to improve moral value prediction. In _Proceedings of the 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM)_. Barcelona, Spain, 552–559.
  * MacDonald et al. (2015) Noni E MacDonald et al. 2015\.  Vaccine hesitancy: Definition, scope and determinants.  _Vaccine_ 33, 34 (2015), 4161–4164.
  * Mejova and Kalimeri (2020) Yelena Mejova and Kyriaki Kalimeri. 2020.  COVID-19 on Facebook ads: competing agendas around a public health crisis. In _Proceedings of the 3rd ACM SIGCAS Conference on Computing and Sustainable Societies_. 22–31.
  * Mooijman et al. (2018) Marlon Mooijman, Joe Hoover, Ying Lin, Heng Ji, and Morteza Dehghani. 2018.  Moralization in social networks and the emergence of violence during protests.  _Nature Human Behaviour_ 2, 6 (2018), 389–396.
  * Organization (2019) World Health Organization. 2019\.  Ten threats to global health in 2019.  https://www.who.int/vietnam/news/feature-stories/detail/ten-threats-to-global-health-in-2019/.  [Online; accessed 14-June-2022].
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014.  Glove: Global vectors for word representation. In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_. 1532–1543.
  * Puri et al. (2020) Neha Puri, Eric A Coomes, Hourmazd Haghbayan, and Keith Gunaratne. 2020.  Social media and vaccine hesitancy: new updates for the era of COVID-19 and globalized infectious diseases.  _Human vaccines & immunotherapeutics_ 16, 11 (2020), 2586–2593.
  * Rossen et al. (2019) Isabel Rossen, Mark J Hurlstone, Patrick D Dunlop, and Carmen Lawrence. 2019.  Accepters, fence sitters, or rejecters: Moral profiles of vaccination attitudes.  _Social Science & Medicine_ 224 (2019), 23–27.
  * Salmon et al. (2015) Daniel A Salmon, Matthew Z Dudley, Jason M Glanz, and Saad B Omer. 2015.  Vaccine hesitancy: causes, consequences, and a call to action.  _Vaccine_ 33 (2015), D66–D71.
  * Schmidt et al. (2018) Ana Lucía Schmidt, Fabiana Zollo, Antonio Scala, Cornelia Betsch, and Walter Quattrociocchi. 2018.  Polarization of the vaccination debate on Facebook.  _Vaccine_ 36, 25 (2018), 3606–3612.
  * Zimmerman et al. (2005) Richard K Zimmerman, Robert M Wolfe, Dwight E Fox, Jake R Fox, Mary Patricia Nowalk, Judith A Troy, and Lisa K Sharp. 2005\.  Vaccine criticism on the world wide web.  _Journal of medical internet research_ 7, 2 (2005), e369.


# Digital Diagnostics: The Potential of Large Language Models in Recognizing
Symptoms of Common Illnesses

Gaurav Kumar Gupta1 Aditi Singh2 Sijo Valayakkad Manikandan3 Abul Ehtesham4
1 Youngstown State University  Youngstown  OH
2 Cleveland State University  Cleveland  OH
3 NIKE  Inc  Beaverton  Oregon
4 The Davey Tree Expert Company  Kent  OH
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

The recent swift development of LLMs like GPT-4, Gemini, and GPT-3.5 offers a
transformative opportunity in medicine and healthcare, especially in digital
diagnostics. This study evaluates each models diagnostic abilities by
interpreting symptoms and determining diagnoses that fit well with common
illnesses, and it demonstrates how each of these models could significantly
increase diagnostic accuracy and efficiency. Through a series of diagnostic
prompts based on symptoms from medical databases, GPT-4 demonstrates higher
diagnostic accuracy from its deep and complete history of training on medical
data. Meanwhile, Gemini performs with high precision as a critical tool in
disease triage, demonstrating its potential to be a reliable model when
physicians are trying to make high-risk diagnoses. GPT-3.5, though slightly
less advanced, is a good tool for medical diagnostics. This study highlights
the need to study LLMs for healthcare and clinical practices with more care
and attention, ensuring that any system utilizing LLMs promotes patient
privacy and complies with health information privacy laws such as HIPAA
compliance, as well as the social consequences that affect the varied
individuals in complex healthcare contexts. This study marks the start of a
larger future effort to study the various ways in which assigning ethical
concerns to LLMs task of learning from human biases could unearth new ways to
apply AI in complex medical settings.

_K_ eywords LLMs $\cdot$ Healthcare $\cdot$ AI $\cdot$ Digital Health $\cdot$
Medical Diagnostics $\cdot$ Natural Language Processing (NLP) $\cdot$ Machine
Learning in Medicine $\cdot$ Clinical Decision Support Systems $\cdot$ Health
Informatics $\cdot$ Patient Data Analysis $\cdot$ Artificial Intelligence
Applications in Health $\cdot$ Precision Medicine $\cdot$ Symptom Recognition
AI

##  1 Introduction

The development of Large Language Models (LLMs), such as GPT-4, Gemini, and
GPT-3.5, has revolutionized natural language processing in the recent
past[21][26]. With potential applications across multiple domains, these
models hold particular promise for medicine in healthcare settings. They
possess the ability to parse and produce human-sounding text, which could
serve as diagnostic tools. Considering the high volume of patients seeking
medical help, digital processes that automate and enhance diagnostic output
can boost accessibility and efficiency, ultimately revolutionizing patient
care[7]. However, the adoption of LLMs in diagnosing common, everyday ailments
raises the pertinent question about these models’ diagnostic accuracy[27].
This serves as the premise for this empirical study on the performance of
different LLMs in recognizing symptoms of common health conditions. This
research investigates the extent to which these models can accurately emulate
and diagnose medical illnesses and symptoms, setting the stage for exploring
their potential in healthcare applications. LLMs hold promise for modern
diagnostic facilities, mirroring these questions regarding their diagnostic
accuracy, which stands as a stepping stone to effective healthcare[15]. The
primary purpose of this study is to appraise and evaluate the performance of
the models in diagnosing ailments and to gauge how accurate they are in
detecting symptoms of common health conditions. The ultimate aim is to
establish whether LLMs can be utilized for digital diagnoses and give us
insights into the future of healthcare. The impetus behind this line of
inquiry is to scrutinize what processes defense AI entails in this aspect of
healthcare. Understanding where these models work (and where they don’t) can
essentially guide the refinement of AI technologies into medicine, aiding the
broader endeavor of leveraging other forms of technology for healthcare
purposes. This study will briefly trace the development of LLMs in healthcare,
offer a critical review of relevant literature, and analyze the study’s
methodology, setting the stage for a nuanced exploration of the transformative
desires of LLMs’ involvement in digital diagnostics, healthcare, and the
future of medicine.

##  2 Related Work

The integration of Large Language Models (LLMs) into healthcare marks a
significant transformation in medical diagnostics and patient care [1]. These
models are fundamentally changing how healthcare providers interact with
patients, make decisions, and manage diseases. LLMs enhance the precision and
speed of diagnoses, especially in specialties that rely on detailed data
analysis such as radiology or pathology, which benefits early detection and
treatment planning [20]. These models also refine patient interactions by
offering personalized health consultations and symptom assessments, increasing
both trust and satisfaction. Moreover, LLMs integrate diverse data sources to
provide actionable insights, aiding healthcare professionals in making
informed decisions [17]. Their application spans several domains: they support
mental health professionals by analyzing speech patterns and behavior, aid
chronic disease management through data from wearable technology, and assist
in vision care by evaluating imaging for early signs of conditions like
diabetic retinopathy [18]. However, deploying LLMs involves addressing
significant challenges such as ensuring data privacy, maintaining model
transparency, and managing the ethical implications of automated decision-
making in healthcare [12]. Despite these hurdles, the potential of LLMs to
improve healthcare outcomes is immense, provided there is careful
consideration of the technical, ethical, and regulatory issues involved [30]
[23].

###  2.1 Advancements in LLMs Capabilities

The incorporation of Large Language Models (LLMs) into healthcare is ushering
in a new era of medical diagnostics and patient care, characterized by
significantly enhanced decision-making capabilities and patient interaction.
Researchers like Meskó et al.[19] have highlighted the profound impact of
LLMs, such as GPT-4, which are adept at processing extensive datasets and
delivering responses that are remarkably human-like. These models facilitate a
multidimensional approach to healthcare by seamlessly integrating diverse data
streams—textual, visual, and sensor-based. This integration enhances
diagnostic accuracy and improves patient outcomes by enabling more precise and
comprehensive health assessments.

Expanding on the practical applications of these capabilities, Yuan et al.[13]
have demonstrated how LLMs can optimize the efficiency of clinical trials
through enhanced patient-trial matching. Their method utilizes the natural
language processing strengths of LLMs to improve compatibility between
Electronic Health Records (EHRs) and clinical trial requirements, tackling
prevalent issues such as data standardization and patient data privacy. The
implementation of a privacy-aware data augmentation strategy has been
particularly effective, resulting in a 7.32% improvement in the matching
accuracy of patients to clinical trials. This not only speeds up the process
but also ensures greater precision in selecting suitable candidates for
trials, thereby accelerating medical research and potential therapeutic
discoveries.

In the realm of continuous health monitoring and predictive healthcare, the
work of Jin et al.[14] and Kim et al.[16] with the Health-LLM system
exemplifies the advanced capabilities of LLMs. This system utilizes complex
algorithms to analyze data from wearable sensors in real-time, offering
personalized health predictions that adapt to the changing parameters of a
patient’s condition. By integrating and interpreting multifaceted data sets,
the Health-LLM system provides targeted advice for chronic disease management,
enhancing preventive care and patient adherence to prescribed health regimens.

Moreover, the expansion of LLM applications into mental health care by Xu et
al.[29] represents a significant breakthrough. Their research on Mental-LLM
explores the utility of LLMs in analyzing behavioral patterns and textual data
to predict mental health outcomes. By employing advanced fine-tuning
techniques, these models have shown remarkable proficiency in understanding
the nuances of language related to mental health symptoms. This allows for
more accurate assessments and personalized treatment plans, addressing the
complex needs of mental health patients with unprecedented precision[18][24].

These detailed explorations and applications of LLMs across various healthcare
domains underscore the immense potential of this technology to revolutionize
both the methodology and efficacy of medical care. However, the deployment of
such sophisticated AI tools must be managed with a keen awareness of the
ethical implications, including patient privacy, data security, and the
potential biases inherent in AI models[12]. As LLMs continue to evolve and
their integration into clinical practice deepens, it is crucial to maintain
rigorous standards for data handling and model transparency to fully realize
their benefits in a responsible and ethical manner[15].

###  2.2 Challenges in Healthcare Integration

The integration of generative AI and Large Language Models (LLMs) into
healthcare systems unveils a spectrum of challenges that are as diverse as
they are complex. Yu et al. [30] meticulously detail these challenges, placing
a strong emphasis on the critical need for robust data privacy measures,
precise model fine-tuning, and thorough system implementation strategies to
ensure that AI deployment achieves its intended outcomes without compromising
security or efficiency. Their research highlights the indispensable role of
collaborative co-design, which involves both clinicians and patients in the
development process. This collaborative approach not only ensures that AI
tools are finely attuned to specific medical needs but also mitigates risks
related to data security and breaches of patient privacy, which are paramount
concerns in digital healthcare[12].

Expanding on these foundational issues, Singh et al. [25] address the
psychological implications of AI integration within clinical settings,
particularly focusing on the cognitive biases that LLMs might exhibit, such as
overconfidence and underestimation. These biases have significant
ramifications for clinical decision-making, potentially leading to diagnostic
errors or misguided patient care strategies. The authors advocate for the
establishment of advanced mechanisms that can assess and adjust the confidence
levels of AI systems. Ensuring that LLM outputs are rigorously scrutinized and
verified by medical professionals is crucial to maintaining the integrity and
trustworthiness of AI-assisted diagnostics.

In a similar vein, Ullah et al. [27] delve deeper into the technical and
interpretative challenges that arise when LLMs are applied to diagnostic
medicine, especially in fields requiring nuanced analysis like digital
pathology. They identify key obstacles such as the absence of deep contextual
understanding and the interpretability issues of complex medical data
processed by LLMs. Moreover, biases embedded in the training datasets can lead
to skewed AI diagnostics, reflecting the "black-box" nature of these systems.
This opacity in AI decision-making processes poses significant hurdles to
clinical acceptance and reliability[15].

Furthermore, the Health-LLM system, discussed by Kim et al. [16], serves as a
case study for the practical challenges associated with deploying LLMs in a
healthcare setting focused on leveraging wearable sensor data for health
predictions. While the system demonstrates considerable potential for
enhancing personalized healthcare through sophisticated data analysis, it also
highlights significant technical challenges in accurately processing and
interpreting multimodal, domain-specific data. These challenges necessitate
continuous advancements in AI technologies to improve their accuracy,
reliability, and applicability in diverse clinical environments[11].

Lastly, Montagna et al. [20] highlight the logistical and ethical challenges
in implementing LLM-based chatbot systems for managing chronic diseases. Their
focus is on the intricacies of maintaining data privacy within decentralized
architectures, crucial for the secure handling of sensitive patient data. They
propose an architectural framework that not only supports chronic disease
management but is also adaptable to various medical conditions and compliant
with strict privacy regulations. However, the practical deployment of such
systems faces significant challenges, including the reliability of AI
applications in medical settings, the imperative for extensive clinical
trials, and ongoing concerns about the confidentiality and security of patient
data[3].

Addressing these multidimensional challenges is essential for the successful
integration of LLMs into healthcare. It requires a concerted effort to enhance
technical capabilities, develop ethical AI practices, and enforce strict
regulatory standards[31]. By tackling these issues head-on, the medical
community can better harness the transformative potential of AI, ultimately
leading to improved patient outcomes and more efficient healthcare delivery
systems.

###  2.3 LLMs Integration in Healthcare Diagnostics

The research highlighted in this review underscores the dual nature of
integrating Large Language Models (LLMs) into healthcare diagnostics. Meskó et
al. [19] and Kusa et al. [17] provide foundational perspectives on both the
potential and pitfalls of employing LLMs in medical contexts. Meskó et al.
illuminate the extensive capabilities of LLMs, particularly their ability to
manage and synthesize vast amounts of multimodal data—combining text, images,
and possibly real-time data—to enhance diagnostic accuracy and patient
outcomes. Their discussion extends beyond traditional data handling and
explores how these models can adapt to the nuanced demands of healthcare by
offering personalized diagnostic insights and treatment options.

On the other hand, Kusa et al. delve into the challenges that arise due to the
sensitivity of LLMs to user input variations. Their detailed analysis reveals
how differing symptom descriptions and entrenched patient beliefs can skew the
outcomes of health diagnostics. The study highlights the critical need for
systems that can discern and adjust to such variations, thereby avoiding
potential misdiagnoses or errors that could arise from automated processes.
This sensitivity to input underscores a significant challenge: the LLMs’
reliance on the quality and nature of the data fed into them, which can
significantly impact their effectiveness and reliability.

The insights from these studies are instrumental for ongoing research aimed at
evaluating the diagnostic accuracy of LLMs in healthcare. They not only
underscore the immense potential of these technologies to transform medical
diagnostics through enhanced data processing capabilities but also highlight
the complexities involved in their implementation. The integration of LLMs
into clinical settings demands the development of robust methodologies that
can effectively handle the variability in user inputs and the inherent
sensitivities of the models[30].

The effective deployment of LLMs in healthcare diagnostics depends crucially
on their ability to provide accurate, reliable medical advice that aligns with
professional standards. This necessitates a balanced approach to technology
integration, where the benefits of LLM capabilities in processing and
analyzing complex datasets are harmonized with the need for precision and
reliability in medical diagnostics[25].

While the promise of LLMs in revolutionizing healthcare diagnostics is
undeniable, the path to their full integration is layered with technical and
ethical challenges. It requires a strategic and meticulous approach to ensure
that the deployment of these technologies enhances, rather than complicates,
the diagnostic process. The insights provided by Meskó et al. [19] and Kusa et
al. [17] are crucial in guiding this integration, advocating for a careful and
informed application of LLMs in healthcare to truly harness their potential in
improving patient care and treatment outcomes. This calls for continued
research, development, and refinement of LLM technologies to meet the
stringent demands of healthcare applications, ensuring that the future of
medical diagnostics is both innovative and ethically sound.

##  3 Methodology

###  3.1 Description of the LLMs Evaluated

This study assesses the diagnostic precision of three user-friendly Large
Language Models (LLMs) likely to be most useful in clinical settings – GPT-4,
Gemini, and GPT-3.5. Each of these models has been shown to be adept at
generating novel and clinically useful information, and able to perform not
only clinical prediction and diagnosis, but also aiding in the maintenance and
recovery of health through data-driven insights. These models are easily
accessible, widely used by the general public, and demand rapid evaluation for
their suitability and usefulness in clinical tasks and healthcare research.
Each LLMs model has a unique computational strength in working with clinical
datasets, and all three stand to improve diagnostic reliability in healthcare
settings[17].

GPT-4: Developed by OpenAI, GPT-4 stands as a leading entity in the realm of
language understanding and generation. This model is distinguished by its
advanced capability to interpret complex inquiries. Notably, its architectural
design is optimally aligned for the assessment of diagnostic accuracy based on
descriptions of medical symptoms. Demonstrating profound efficacy within the
medical sphere, GPT-4 has attained a remarkable accuracy rate of 75 percent on
the Medical Knowledge Self-Assessment Program. This achievement not only
highlights GPT-4’s intricate understanding of complex medical queries but also
accentuates its critical role in enhancing diagnostic precision from symptom
narratives [21].

Gemini: Gemini represents a significant advancement in the landscape of Large
Language Models (LLMs), particularly with its tailored design for domain-
specific applications, including healthcare. The architecture of Gemini is
meticulously crafted to enhance the understanding and generation of nuanced
responses within these specialized domains. This makes Gemini an invaluable
tool for a wide range of tasks, notably in medical diagnostics and healthcare
inquiries, where precision and accuracy are paramount. Its ability to
seamlessly integrate and reason across multimodal inputs further underscores
its potential to revolutionize how medical information is processed and
interpreted, setting a new benchmark for AI in healthcare[26].

GPT-3.5: Serving as a foundational benchmark, GPT-3.5, the precursor to GPT-4,
possesses substantial capabilities in language understanding and generation,
albeit with slightly lesser proficiency compared to its successor. Its
utilization furnishes a comparative framework to evaluate progress in Large
Language Models (LLMs) and their applicability in medical diagnostics. Despite
its predecessor status, GPT-3.5 has demonstrated commendable performance in
the medical domain, notably achieving a 53 percent accuracy rate on the
Medical Knowledge Self-Assessment Program. This metric underscores its
capability to process and understand clinical and healthcare-related
inquiries, thus marking a significant step towards leveraging AI in the
enhancement of diagnostic accuracy [21].

Figure 1: Data Collection Process

###  3.2 Data collection methods:

The foundational dataset for this study was constructed from data extracted
from reputable medical sources, including the Centers for Disease Control and
Prevention (CDC), World Health Organization (WHO), Mayo Clinic, Cleveland
Clinic, and Johns Hopkins Hospital. The selection criterion for diseases
focused on conditions frequently encountered in daily life to evaluate the
potential utility of Large Language Models (LLMs) in providing diagnostic
insights for common ailments. Seasonal allergies, the common cold, and food-
related issues such as allergies or diarrhea were included due to their
prevalence among the general population, while chronic and complex diseases
like cancer were deliberately excluded due to ethical considerations and the
current limitations of LLMs in handling such diagnoses accurately. For each
selected disease, a detailed list of symptoms was compiled, forming a
comprehensive dataset that pairs these symptoms with their corresponding
disease names. This data was organized to support the creation of diagnostic
prompts, designed to query disease predictions based on symptom descriptions.
These diagnostic prompts were meticulously crafted to present the symptoms of
each disease, asking for disease predictions along with a confidence score for
each diagnosis. The prompts were uniformly applied to maintain consistency in
the evaluation process. Subsequently, the responses were manually verified to
assess the accuracy and reliability of disease prediction, grounding the
study’s findings in a robust methodological framework. This methodology
underscores the study’s aim to investigate the applicability of LLMs as a
supportive tool for individuals in recognizing common health conditions. By
focusing on diseases frequently encountered, this research contributes
valuable insights into the capabilities and limitations of AI technologies in
everyday health applications.

Prompt for models:
The following dialogue presents a prompt used to test the diagnostic
capabilities of various language models.

Based on these symptoms: [Symptoms], identify the only one disease based on
the symptoms that match closely. Provide me confidence level—Low, Medium,
High—to each, based on how closely the symptoms align with the diseases you
have predicted based on the Symptoms. No explanation needed, provide me exact
one-word disease name.

###  3.3 Evaluation Metrics for Diagnosing Diseases through LLMs

The efficacy of Language Learning Models (LLMs) in diagnosing diseases from
descriptions of medical symptoms was evaluated using a detailed, multi-step
manual process. This approach incorporated the use of precision, recall, and
the F1 score, metrics renowned for their ability to offer a rounded
perspective on the accuracy of predictive models in both identifying correct
diagnoses and highlighting the omission of relevant diagnoses.

In our study, we scrutinized the LLMs’ outputs for each dataset entry,
systematically classifying each response based on its diagnostic accuracy. The
classifications were as follows:

  * •

True Positive (TP): Instances where the LLM correctly identified the disease,
showcasing the model’s capability to accurately match symptom descriptions
with the correct disease diagnosis.

  * •

False Positive (FP): Instances where the LLM incorrectly identified a disease,
attributing a condition to the symptom descriptions that did not align with
the actual disease present, thereby overestimating the model’s diagnostic
accuracy.

  * •

False Negative (FN): Instances where the LLM either attributed a different
disease than the one actually present based on the symptom descriptions or
failed to recognize the presence of a disease altogether, thus underestimating
the model’s diagnostic sensitivity.

We then proceeded to compute the following metrics:

  * •

Precision: This metric evaluates the exactness of the model’s positive
predictions (i.e., the proportion of TP observations among all positive
diagnoses made by the model), offering insight into the accuracy of the
model’s disease identification.

$\text{Precision}=\frac{TP}{TP+FP}$ (1)

  * •

Recall: This metric assesses the model’s ability to identify all pertinent
instances (i.e., the ratio of TP observations to all actual positives within
the dataset), providing a measure of the model’s comprehensiveness in disease
detection.

$\text{Recall}=\frac{TP}{TP+FN}\ $ (2)

  * •

F1 Score: This metric serves as a balanced measure of both precision and
recall, particularly valuable when the contributions of both metrics are of
equal importance. It is calculated as the harmonic mean of precision and
recall, furnishing a singular measure of the model’s overall diagnostic
performance.

$\text{F1
Score}=2\cdot\frac{\text{Precision}\cdot\text{Recall}}{\text{Precision}+\text{Recall}}$
(3)

Employing these metrics enables a comprehensive evaluation of the LLMs’
diagnostic capabilities, providing nuanced insights into the precision of
correct diagnoses and the models’ overall efficacy in disease identification.

###  3.4 Experimental Setup

The experimental setup involved an intricate process where symptom-based
prompts, meticulously crafted from our curated dataset, were presented to the
LLMs under study. Each model was subjected to an identical sequence of
prompts, ensuring that the variability in responses could solely be attributed
to the models’ interpretative capabilities. This methodical approach
facilitated a standardized comparison of query presentation and the verbatim
recording of responses. The recorded predictions were then methodically
compared against the actual diseases linked with the given symptom
descriptions. This comparison allowed for an evaluative analysis of each
model’s diagnostic accuracy, providing a clear framework for assessing
performance within a controlled and consistent environment. This level of
detail in the experimental setup was crucial for ensuring the reliability and
validity of our findings, underscoring the study’s contribution to
understanding the potential of LLMs in medical diagnostics.

Table 1: Model Performance and Characteristics

Model |  Number of Parameters |  Training Data Size |  Primary Application Areas |  Unique Features |  Precision |  Recall |  F1 Score
---|---|---|---|---|---|---|---
GPT-4 |  175 billion |  >800 GB |  General-purpose language tasks, content creation, question-answering |  Improved context understanding, Multitasking capabilities |  0.95 |  0.91 |  0.92
GPT-3.5 |  13 billion |  570 GB |  Text completion, language translation, coding assistance |  Fine-tuning capabilities for specific tasks |  0.90 |  0.84 |  0.86
Gemini |  50 million |  100 GB |  Niche applications (e.g., medical, legal), Low-resource languages |  Optimized for efficiency and specific domain knowledge |  0.97 |  0.69 |  0.80

##  4 Results

###  4.1 Overview of Findings

Our comprehensive evaluation delved into the diagnostic abilities of three
state-of-the-art Language Learning Models (LLMs) — GPT-4, GPT-3.5, and Gemini.
The goal was to assess how effectively these models can analyze and diagnose
medical conditions based on detailed descriptions of symptoms. The findings,
illustrated in Figure 1, reveal significant differences in the diagnostic
accuracy and capabilities of each model, shedding light on their potential
utility in clinical settings. GPT-4 emerged as the standout performer in our
study, demonstrating exceptional diagnostic accuracy. This model’s success is
attributable to its extensive training on a vast array of medical literature
and patient data, which has equipped it with a profound understanding of
medical symptomatology. GPT-4’s ability to consistently and accurately
identify diseases from symptom descriptions showcases its advanced algorithmic
structure and sophisticated data processing capabilities. It sets a benchmark
in the realm of AI-driven medical diagnostics, proving to be a robust tool
that could revolutionize how healthcare providers approach diagnosis and
treatment planning. Close behind in terms of performance, GPT-3.5 displayed
robust diagnostic skills as well. Although it did not surpass GPT-4, its
effectiveness in converting complex symptom data into accurate health
assessments makes it a valuable asset in the medical field. GPT-3.5 supports
clinical decision-making by providing reliable interpretations of medical
contexts, which can greatly aid physicians in diagnosing and understanding
patient conditions more effectively. Its solid performance underlines the
reliability of well-trained LLMs in handling medical diagnostic tasks,
highlighting the potential for AI to assist significantly in everyday
healthcare operations. Gemini, although it did not achieve as many correct
diagnoses as its counterparts, was noted for its extraordinary diagnostic
precision. This model adopts a conservative approach, prioritizing accuracy
over quantity in its outputs. Such high-confidence predictions make Gemini
especially suitable for use in clinical scenarios where accuracy is critical,
such as in the diagnosis of complex or rare conditions where the cost of a
misdiagnosis can be particularly high. Gemini’s approach to minimizing false
positives is vital in clinical practices where precision is paramount, and the
margin for error is minimal. The collective performance of these LLMs paints
an optimistic picture of the role of AI in enhancing medical diagnostics. The
integration of these advanced models into healthcare could lead to faster,
more accurate, and highly reliable diagnostic processes. GPT-4’s broad
diagnostic capabilities, GPT-3.5’s dependable performance, and Gemini’s
meticulous precision collectively embody the advancement of artificial
intelligence within the healthcare sector. These findings from our study not
only underscore the substantial progress that AI technology has made in
unraveling and understanding the complexities of human health but also pave
the way for their future applications in medical practice. By enhancing the
efficiency and accuracy of diagnostics, these models could serve as invaluable
tools for medical practitioners, enabling better patient outcomes and
transforming the landscape of healthcare delivery. As we continue to explore
and refine these technologies, their integration into clinical workflows holds
the promise of making healthcare more effective, personalized, and accessible
to all.

Figure 2: Model Prediction Correctness

###  4.2 Comparative Analysis

The comparative evaluation of GPT-4, GPT-3.5, and Gemini through our study
provides an illuminating overview of their diagnostic abilities, each
underscored by unique strengths as revealed by the performance metrics
summarized in Table 2 and visually elucidated in Figure 2. GPT-4, with its
outstanding number of correct answers, stands as a testament to its
comprehensive training regimen, encompassing a wide array of medical data.
This extensive preparation is reflected in its superior F1 score, indicative
of the model’s proficiency in deciphering complex medical language and
accurately mapping symptoms to diagnoses.

Figure 2 graphically displays the comparative accuracy of the models,
highlighting GPT-4’s dominance in correctly answering questions, thereby
showcasing its exceptional capability to navigate the complexities inherent in
the symptom-diagnosis correlation. This visual representation provides a clear
perspective on each model’s performance, with GPT-4 leading in precision and
understanding.

Further enhancing our analytical perspective, Figure 3 delves into the
confidence levels associated with each model’s predictions. Here, the
confidence distributions of GPT-4 and GPT-3.5 are primarily classified under
the ’High’ category, underscoring their robust assertiveness and reliability
in diagnostic conclusions. In stark contrast, Gemini’s inclination towards
high-confidence responses, despite a lower overall number of predictions,
spotlights its unparalleled precision. This trait is particularly crucial in
healthcare contexts where the stakes of misdiagnosis are high, emphasizing the
need for accuracy and high confidence in diagnostics.

However, Gemini’s impressive precision comes at the cost of recall, as
evidenced by its performance metrics. This suggests a cautious approach to
diagnosis, where the model opts for certainty over breadth, potentially
overlooking certain conditions in the process. Meanwhile, GPT-3.5, although
not as advanced as GPT-4, demonstrates significant diagnostic utility,
balancing precision and recall effectively. Its solid performance affirms its
value in scenarios where cutting-edge models like GPT-4 might not be available
or necessary.

Integrating these insights, the study underscores the multifaceted diagnostic
capabilities inherent in these LLMs. GPT-4 emerges as a versatile asset across
various medical domains, whereas Gemini’s precision earmarks it as an
invaluable resource for confirming diagnoses with high confidence. On the
other hand, GPT-3.5’s reliable performance ensures its continued relevance in
the evolving landscape of AI in healthcare.

The nuanced findings from our study advocate for a strategic incorporation of
LLMs within healthcare settings, leveraging each model’s distinct strengths.
Such an approach not only augments the diagnostic process but also enhances
the overall quality of care, paving the way for an AI-integrated healthcare
ecosystem that prioritizes accuracy, efficiency, and patient safety. This
comprehensive analysis and the subsequent recommendations aim to foster a
deeper understanding of how LLMs can be optimally utilized in healthcare,
marking a significant step towards the realization of AI’s potential in
improving diagnostic outcomes and patient care.

Table 2: Comparative Performance of LLMs in Digital Diagnostics Model | Precision | Recall | F1 Score
---|---|---|---
GPT-4 | 0.95 | 0.91 | 0.92
Gemini | 0.97 | 0.69 | 0.80
GPT-3.5 | 0.90 | 0.84 | 0.86
Figure 3: Model Confidence Category

###  4.3 Concluding Insights

This study offers insights into the diagnostic capabilities of GPT-4, Gemini,
and GPT-3.5. GPT-4 is noted for its high accuracy in diagnosing common
illnesses, making it the top performer among the models tested. Gemini, with
its high precision, shows great promise as a supplemental tool for digital
diagnostics, especially in tasks that require pinpoint accuracy. GPT-3.5,
ranking second in disease prediction accuracy, remains a reliable option
despite being slightly less advanced. It showcases the rapid advances in LLM
technology and its tangible benefits for healthcare.

These findings highlight the substantial potential of integrating LLMs into
digital healthcare solutions. They also emphasize the need for ongoing
development to improve the accuracy and reliability of these models, ensuring
they can effectively meet clinical needs in digital healthcare environments.

##  5 Discussion

###  5.1 Interpretation of Results

This study carefully examined the capabilities of three advanced Language
Learning Models (LLMs)—GPT-4, GPT-3.5, and Gemini—in diagnosing everyday
illnesses based on symptom descriptions. The standout performer, GPT-4,
showcases the potential of AI in medical diagnostics through its ability to
understand and process complex medical data. This model’s effectiveness
highlights its extensive training across diverse medical scenarios, making it
exceptionally good at matching symptoms with the correct medical conditions.
Although not as advanced as GPT-4, GPT-3.5 still demonstrated significant
capability in making accurate medical diagnoses. Its ability to deliver
reliable assessments makes it a useful tool in healthcare, particularly where
newer technologies might not yet be available. Gemini, while it ranked third
in the number of correct diagnoses, excelled in precision. This high level of
accuracy when making a diagnosis shows that Gemini is very careful about
ensuring its predictions are correct, which is crucial in medical settings
where errors can have serious consequences

###  5.2 Enhancing Diagnostic Processes with Large Language Models

The introduction of Large Language Models (LLMs) into the medical context is
likely to change the way healthcare is provided at the very first stages of
contact between a patient and a healthcare professional. In the future, LLMs
will likely improve the speed and quality of first medical consultations,
allowing more rapid assessments of patient information, thereby taking much
pressure off medical experts or allowing other professionals to fill the role.
In settings where resources are scarce, especially when access to first human
consultations is unfeasible, LLMs could quickly analyze the data provided by
symptoms and tell the patient what they might be suffering from[23][15].

In triaging these cases, LLMs could help to prioritize patients based on
urgency and route them to the appropriate level of care. All of this could
lead to improved patient outcomes, such as faster interventions, fewer missed
appointments, and reduced wait times before treatment. Patient education is
also one of the benefits of using these models. LLMs can also be used to
provide patients with additional information about their symptoms and
potential diagnoses—additional knowledge that empowers people to better
understand their health[17].

Yet, such incorporation of AI-enabled tools in healthcare systems is bound to
be fraught, and we shall have to navigate several hefty issues. For example,
adherence to high healthcare regulations, such as the Health Insurance
Portability and Accountability Act (HIPAA) in the United States, this is
comprehensive federal legislation that sets metalegal standards and outlines
the protection and secure transfer of patients’ data in healthcare
systems[12]. All the information gathered by LLMs during healthcare
interactions must be strictly HIPAA compliant; they should master the vital
data and conversation with full security and confidentiality.

In addition, LLMs might prove valuable, but if integrated, their use must
complement, rather than substitute, the human side of delivering care,
augmenting physician sensibilities of the human system over replacing the
physician. Error rates of LLMs for complex diagnoses need to be constantly
validated against medical benchmarks[25]. Deviating from the norm could result
in a misdiagnosis. Moreover, the models also need to be kept up-to-date
through regular checking and flagging.

That commitment to multidisciplinary work among technologists, clinicians, and
regulators is also crucial to using LLMs in healthcare effectively. From
conceiving AI tools that are both architecture- and design-wise solid and
clinically useful to providing right-size validation and ensuring ethical use,
stringent workflows across disciplines are necessary. Testing each model goes
beyond mere optimization of technical performance to confirm that the tool is
capable of effectively helping clinicians in actual practice[10].

In conclusion, incorporating LLMs into healthcare is a promising avenue for
improving diagnosis and patient care. However, to address these challenges, we
must implement careful planning and robust processes. Doing so will help to
make the most of AI in healthcare and improve outcomes for patients and the
medical system.

###  5.3 Limitations of the Study

The LLMs in this study were primarily evaluated on common, less complex
illnesses, which do not fully represent the broader, more challenging aspects
of diagnosing chronic or severe conditions. Chronic diseases often involve
complex symptoms that are difficult to interpret without a comprehensive
understanding of an individual’s medical history and additional diagnostic
tests. Moreover, the reliance on text analysis in our study ignores the
multimodal nature of traditional medical diagnostics, which often include
visual elements like scans, detailed patient histories, and physical
examinations. Future improvements in AI models for healthcare should aim to
incorporate these various data types to provide more accurate and holistic
diagnoses.

###  5.4 Future Research Directions

Future research should focus on creating LLMs that can analyze various types
of medical information beyond just text. This includes integrating visual data
from scans and other medical images to create more comprehensive diagnostic
tools. Also, developing AI systems that can understand and process information
across different languages and cultural contexts will be crucial for their
global applicability. It’s also important to ensure these systems are
developed with strict adherence to ethical standards, particularly regarding
patient privacy. Exploring ways to securely integrate AI into healthcare while
respecting patient confidentiality will be essential. Additionally, putting
these AI models into real-world clinical settings to see how they perform and
what impact they have on healthcare efficiency and patient outcomes will
provide critical insights into their practical value and limitations.

This research demonstrates the significant potential and challenges of using
advanced AI models in healthcare diagnostics. With careful development and
ethical considerations, these tools could greatly enhance the ability to
diagnose and treat patients more efficiently and accurately. Continued
exploration and improvement of these technologies could lead to their
successful integration into everyday medical practice, benefiting both
healthcare providers and patients.

##  6 Conclusion

The exploration into the capabilities of Large Language Models (LLMs) like
GPT-4, Gemini, and GPT-3.5 has culminated in a comprehensive understanding of
their potential to enhance digital diagnostics in healthcare. This study has
rigorously evaluated the performance of these models, shedding light on their
strengths in identifying symptoms of common illnesses and their limitations.
The core findings demonstrate that LLMs like GPT-4 offer considerable promise
in processing medical language with high accuracy. They herald a significant
step forward in providing immediate, accessible healthcare guidance. The
specialized Gemini model’s remarkable precision points towards the feasibility
of creating niche, domain-focused LLMs that can provide precise diagnostic
support. GPT-3.5, while slightly overshadowed by its successors, still
displays commendable capabilities, indicative of the rapid advancements within
the field of AI in healthcare.These results reinforce the transformative
potential of LLMs in digital diagnostics, suggesting that they can complement
conventional diagnostic methods, enhancing the quality and accessibility of
patient care. However, the journey from potential to actualized utility in a
clinical setting will require overcoming substantial hurdles, including the
integration of multimodal data, ethical considerations, and ensuring adherence
to stringent healthcare standards.The limitations of this study, predominantly
its scope restricted to textual analysis and the manual evaluation process,
set a clear directive for future research. The next phase should aim to expand
the capabilities of LLMs beyond text, incorporating visual and empirical data
to align closely with comprehensive clinical diagnostics. Moreover, the study
emphasizes the necessity to build ethically aligned, culturally sensitive, and
linguistically diverse LLMs to serve global healthcare needs effectively. In
the quest to harness AI for healthcare, LLMs emerge not as standalone
solutions but as part of a collaborative toolset augmenting the expertise of
medical professionals. As research progresses, it will be paramount to embed
these models within real-world clinical workflows to fully assess their
practicality and reliability. With continued development and responsible
implementation, LLMs are poised to play a pivotal role in shaping the future
of healthcare, making diagnostics more accessible, accurate, and patient-
centric.

## Acknowledgments

Thank you Dr. Aditi Singh as mentor for helping me in the process of
experiment, writing the paper.

## Data availability

The dataset associated in this manuscript, sourced from reputable medical
institutions including the CDC, WHO, Mayo Clinic, Cleveland Clinic, and Johns
Hopkins Hospital, are publicly available, focusing on common conditions to
evaluate the diagnostic capabilities of Large Language Models (LLMs).

## Declaration conflict of interest

The author declares no conflict of interest.

## References

  * [1] Mahyar Abbasian, Iman Azimi, Amir M. Rahmani and Ramesh Jain  “Conversational Health Agents: A Personalized LLM-Powered Agent Framework”, 2024  arXiv:2310.02374 [cs.CL]
  * [2] N. Baharudin, M.. Mohamed-Yassin and A.. Daher  “Prevalence and factors associated with lipid-lowering medications use for primary and secondary prevention of cardiovascular diseases among Malaysians: the REDISCOVER study”  In _BMC Public Health_ 22, 2022, pp. 228  DOI: 10.1186/s12889-022-12595-1
  * [3] J Baric-Parker and E.E. Anderson  “Patient Data-Sharing for AI: Ethical Challenges, Catholic Solutions”  In _The Linacre Quarterly_ 87.4, 2020, pp. 471–481  DOI: 10.1177/0024363920922690
  * [4] John A. Batsis, Todd A. Mackenzie, Rebecca T. Emeny, Francisco Lopez-Jimenez and Stephen J. Bartels  “Low Lean Mass With and Without Obesity, and Mortality: Results From the 1999–2004 National Health and Nutrition Examination Survey”  In _The Journals of Gerontology: Series A_ 72, 2017, pp. 1445–1451  DOI: 10.1093/gerona/glx002
  * [5] Yu Ying Chiu, Ashish Sharma, Inna Wanyin Lin and Tim Althoff  “A Computational Framework for Behavioral Assessment of LLM Therapists”, 2024  arXiv:2401.00820 [cs.CL]
  * [6] Avishek Choudhury and Zaira Chaudhry  “Large Language Models and User Trust: Focus on Healthcare (Preprint)”  In _Journal of Medical Internet Research_ JMIR Publications Inc., 2024  DOI: 10.2196/56764
  * [7] Hejie Cui, Xinyu Fang, Ran Xu, Xuan Kan, Joyce C. Ho and Carl Yang  “Multimodal Fusion of EHR in Structures and Semantics: Integrating Clinical Records and Notes with Hypergraph and LLM”, 2024  arXiv:2403.08818 [cs.LG]
  * [8] J. Curtò, I. Zarzà, Gemma Roig, Juan Carlos Cano, Pietro Manzoni and Carlos T. Calafate  “LLM-Informed Multi-Armed Bandit Strategies for Non-Stationary Environments”  In _Electronics_ 12.13, 2023  DOI: 10.3390/electronics12132814
  * [9] Uttam Dhakal et al.  “GPT-4’s assessment of its performance in a USMLE-based case study”, 2024  arXiv:2402.09654 [cs.AI]
  * [10] Christos A. Frantzidis and Panagiotis D. Bamidis  “Description and future trends of ICT solutions offered towards independent living: the case of LLM project”  In _Proceedings of the 2nd International Conference on PErvasive Technologies Related to Assistive Environments_ , PETRA ’09  Corfu, Greece: Association for Computing Machinery, 2009  DOI: 10.1145/1579114.1579173
  * [11] Akash Ghosh, Arkadeep Acharya, Raghav Jain, Sriparna Saha, Aman Chadha and Setu Sinha  “CLIPSyntel: CLIP and LLM Synergy for Multimodal Question Summarization in Healthcare”, 2023  arXiv:2312.11541 [cs.AI]
  * [12] Barry A. Humphrey  “Data Privacy vs. Innovation: A Quantitative Analysis of Artificial Intelligence in Healthcare and Its Impact on HIPAA regarding the Privacy and Security of Protected Health Information”  ProQuest Dissertations Publishing, 2021
  * [13] Yuan J, Tang R, Jiang X and Hu X  “LLM for Patient-Trial Matching: Privacy-Aware Data Augmentation Towards Better Performance and Generalizability”  In _American Medical Informatics Association (AMIA) Annual Symposium_ URL: https://par.nsf.gov/biblio/10448809
  * [14] Mingyu Jin et al.  “Health-LLM: Personalized Retrieval-Augmented Disease Prediction System”, 2024  arXiv:2402.00746 [cs.CL]
  * [15] Eunkyung Jo, Daniel A. Epstein, Hyunhoon Jung and Young-Ho Kim  “Understanding the Benefits and Challenges of Deploying Conversational AI Leveraging Large Language Models for Public Health Intervention”  In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ , CHI ’23  <conf-loc>, <city>Hamburg</city>, <country>Germany</country>, </conf-loc>: Association for Computing Machinery, 2023  DOI: 10.1145/3544548.3581503
  * [16] Yubin Kim, Xuhai Xu, Daniel McDuff, Cynthia Breazeal and Hae Won Park  “Health-LLM: Large Language Models for Health Prediction via Wearable Sensor Data”, 2024  arXiv:2401.06866 [cs.CL]
  * [17] Wojciech Kusa, Edoardo Mosca and Aldo Lipani  ““Dr LLM, what do I have?”: The impact of user beliefs and prompt formulation on health diagnoses”  In _Journal of Medical Internet Research_ JMIR Publications Inc., Toronto, Canada, 2021
  * [18] Tin Lai et al.  “Psy-LLM: Scaling up Global Mental Health Psychological Services with AI-based Large Language Models”, 2023  arXiv:2307.11991 [cs.CL]
  * [19] Bertalan Meskó, Georgina Hetényi and Zsuzsanna Győrffy  “The role of artificial intelligence in precision medicine”  In _Expert review of precision medicine and drug development_ 2.5 Taylor & Francis, 2017, pp. 239–241
  * [20] Sara Montagna, Stefano Ferretti, Lorenz Cuno Klopfenstein, Antonio Florio and Martino Francesco Pengo  “Data Decentralisation of LLM-Based Chatbot Systems in Chronic Disease Self-Management”  In _Proceedings of the 2023 ACM Conference on Information Technology for Social Good_ , GoodIT ’23  Lisbon, Portugal: Association for Computing Machinery, 2023, pp. 205–212  DOI: 10.1145/3582515.3609536
  * [21] OpenAI  “GPT-4 Technical Report”, 2023  arXiv:2303.08774 [cs.CL]
  * [22] W. Peng, Y. Feng and C. Yao  “Evaluating AI in Medicine: A Comparative Analysis of Expert and ChatGPT Responses to Colorectal Cancer Questions”  In _Sci Rep_ 14, 2024, pp. 2840  DOI: 10.1038/s41598-024-52853-3
  * [23] Malik Sallam  “ChatGPT Utility in Healthcare Education, Research, and Practice: Systematic Review on the Promising Perspectives and Valid Concerns”  In _Healthcare_ 11.6, 2023  DOI: 10.3390/healthcare11060887
  * [24] Aditi Singh, Abul Ehtesham, Saifuddin Mahmud and Jong-Hoon Kim  “Revolutionizing Mental Health Care through LangChain: A Journey with a Large Language Model”  In _2024 IEEE 14th Annual Computing and Communication Workshop and Conference (CCWC)_ , 2024, pp. 0073–0078  DOI: 10.1109/CCWC60891.2024.10427865
  * [25] Aniket Kumar Singh, Bishal Lamichhane, Suman Devkota, Uttam Dhakal and Chandra Dhakal  “Do Large Language Models Show Human-like Biases? Exploring Confidence—Competence Gap in AI”  In _Information_ 15.2, 2024, pp. 92  DOI: 10.3390/info15020092
  * [26] Gemini Team  “Gemini: A Family of Highly Capable Multimodal Models”, 2024  arXiv:2312.11805 [cs.CL]
  * [27] E. Ullah, A. Parwani and M.. Baig  “Challenges and Barriers of Using Large Language Models (LLM) Such as ChatGPT for Diagnostic Medicine with a Focus on Digital Pathology – A Recent Scoping Review”  In _Diagn Pathol_ 19, 2024, pp. 43  DOI: 10.1186/s13000-024-01464-7
  * [28] P. Webster  “Six Ways Large Language Models are Changing Healthcare”  In _Nat Med_ 29, 2023, pp. 2969–2971  DOI: 10.1038/s41591-023-02700-1
  * [29] Xuhai Xu et al.  “Mental-LLM: Leveraging Large Language Models for Mental Health Prediction via Online Text Data”  In _Proc. ACM Interact. Mob. Wearable Ubiquitous Technol._ 8.1 New York, NY, USA: Association for Computing Machinery, 2024  DOI: 10.1145/3643540
  * [30] Katherine Yu, Andrew L Beam and Isaac S Kohane  “Artificial intelligence in healthcare: A critical analysis of the legal and ethical implications”  In _International journal of medical informatics_ 141 Elsevier, 2020, pp. 104431
  * [31] Jiayi Yuan  “Improving patient-trial matching with a data-driven approach”  In _International Conference on Medical Informatics_ , 2021

Figure 4: Performance Metrics