# The Ethics of ChatGPT in Medicine and Healthcare: A Systematic Review on
Large Language Models (LLMs)

Joschka Haltaufderheide & Robert Ranisch Joschka Haltaufderheide is with the
Juniorprofessorship for Medical Ethics with a Focus on Digitization, Faculty
of Health Sciences Brandenburg, University of Potsdam, Germany, e–mail:
<EMAIL_ADDRESS>ORCID: 0000–0002–5014–4593.Robert
Ranisch is Juniorprofessor for Medical Ethics with a Focus on Digitization,
Faculty of Health Sciences Brandenburg, University of Potsdam, Germany,
e–mail<EMAIL_ADDRESS>ORCID: 0000–0002–1676–1694

###### Abstract

Background: With the introduction of ChatGPT, Large Language Models (LLMs)
have received enormous attention in healthcare. Despite their potential
benefits, researchers have underscored various ethical implications. While
individual instances have drawn much attention, the debate lacks a systematic
and comprehensive overview of practical applications currently researched and
ethical issues connected to them. Against this background, this work aims to
map the ethical landscape surrounding the current stage of deployment of LLMs
in medicine and healthcare.

Methods: Electronic databases and commonly used preprint servers were queried
using a comprehensive search strategy which generated 796 records. Studies
were screened and extracted following a modified rapid review approach.
Methodological quality was assessed using a hybrid approach. For 53 records, a
meta-aggregative synthesis was performed.

Results: Four general fields of applications emerged and testify to a vivid
phase of exploration. Advantages of using LLMs are attributed to their
capacity in data analysis, personalized information provisioning, and support
in decision-making or mitigating information loss and enhancing medical
information accessibility. However, our study also identifies recurrent
ethical concerns connected to fairness, bias, non-maleficence, transparency,
and privacy. A distinctive concern is the tendency to produce harmful
misinformation or convincingly but inaccurate content. A recurrent plea for
ethical guidance and human oversight is evident.

Discussion: Given the variety of use cases, it is suggested that the ethical
guidance debate be reframed to focus on defining what constitutes acceptable
human oversight across the spectrum of applications. This involves considering
the diversity of setting, varying potentials for harm, and different
acceptable thresholds for performance and certainty in diverse healthcare
settings. In addition, a critical inquiry is necessary to determine the extent
to which the current experimental use of LLMs is both necessary and justified.

###### Index Terms:

Large Language Model, LLM, ChatGPT, Healthcare, Medicine, Ethics,

††publicationid: pubid:

##  I Introduction

Large language models (LLMs) have emerged as a transformative force in
artificial intelligence (AI), generating significant interest across various
sectors. The 2022 launch of OpenAI’s ChatGPT demonstrated their groundbreaking
capabilities, revealing the current state of development to a wide audience.
Public availability and scientific interest, since then, have resulted in a
flood of scientific papers considering possible areas of application[1] as
well as their ethical and social implications from a practical perspective.[2]
A burgeoning field for LLMs lies within medicine and healthcare,[3]
encompassing clinical, educational and research applications.[3, 4, 5, 6, 7,
8, 9] In a brief span, a significant number of publications has investigated
the potential uses of LLMs in these domains,[10] indicating a positive
trajectory for the integration of medical AI. Present-day LLMs, such as
ChatGPT, are considered to have a promising accuracy in clinical decision-
making,[11, 12] diagnosis,[13] symptom-assessment, and triage-advise.[14] In
patient-communication, it has been posited that LLMs can also generate
empathetic responses.[15] LLMs specifically trained on biomedical corpora
forebode even further capacities for clinical application and patient care[16]
in the foreseeable future.

Conversely, the adoption of LLMs is entwined with ethical and social
concerns.[17] In their seminal work, Bender et al. anticipated real-world
harms that could arise from the deployment of LLMs.[18] Scholars have
delineated potential risks across various application domains.[19, 20] The
healthcare and medical field, being particularly sensitive and heavily
regulated, is notably susceptible to ethical dilemmas. This sector is also
underpinned by stringent ethical norms, professional commitments, and societal
role recognition. Despite the potential benefits of employing advanced AI
technology, researchers have underscored various ethical implications
associated with using LLMs in healthcare and health-related research.[4, 6,
21, 22, 7, 23, 24] Paramount concerns include the propensity of LLMs to
disseminate inadequate information, input of sensitive health information or
patient data, raising concerns regarding privacy,[22] and perpetuating harmful
gender, cultural or racial biases,[25, 26, 27, 28] well known from machine
learning algorithms[29] especially in healthcare.[30] Case reports have
documented that ChatGPT has already led to actual real damages, potentially
life-threatening for patients.[31]

While individual instances have drawn attention to ethical concerns
surrounding the use of LLMs in healthcare, there appears to be a deficit in
comprehensive, systematic overviews addressing these ethical considerations.
This gap is significant, given the ambitions to rapidly integrate LLMs and
foundational models into healthcare systems.[32] Our intention is to bridge
this lacuna by mapping out the ethical landscape that surrounds the deployment
of LLMs in this field. To achieve this, we will provide an encompassing
overview of applications and ethical considerations pertinent to the
utilization of LLMs in medicine and healthcare. We will, first, provide an
encompassing overview of applications and ethical considerations pertinent to
the utilization of LLMs in medicine and healthcare by delineating the
ethically relevant applications, interventions, and contexts where LLMs have
been tested or proposed within the field. Secondly, we aim to identify the
principal outcomes as well as the opportunities, risks, benefits, and
potential harms associated with the use of LLMs, as deemed significant from an
ethical standpoint. With this, we aspire to not only to outline the current
ethical discourse but also to inform future dialogues and policy-making in the
intersection of LLMs and healthcare ethics.

##  II Methods

A review protocol focusing on practical applications and ethical
considerations grounded in experience was designed by the authors and
registered in the international prospective register of systematic
reviews.[33] Relevant publication databases and preprint servers were queried.
Inclusions were screened and extracted in a two-staged process following a
modified rapid review approach.[34] Inclusion and exclusion criteria were
based on the three key concepts of intervention, application setting, and
outcomes (see Supplement 1). No additional inclusion or exclusion criteria
(e.g. publication type) were applied. However, we excluded work that was
solely concerned with (ethical) questions of medical education, academic
writing, authorship and plagiarism. While we recognize that these issues are
affected by the use of LLMs in significant ways[6, 35, 36] these, challenges
are not specific to health-related applications. Database searches were
conducted in July 2023 (see Table I for sources). Subsequently, the authors
independently screened titles and abstracts of 10% of all database hits (73
records) to test and refine inclusion and exclusion criteria. After a joint
discussion of the results, the remaining 90% were screened by the first
author.

TABLE I: Overview on Sources and Searchstring Sources
---
Databases |  MEDLINE via PubMed
|  CINAHL
|  Embase
|  Philosophers’ Index
|  PsychInfo
|  IEEEX Xplore
Preprint Servers |  arXiv
|  MedRxiv
|  BioRxiv
|  TechRxiv
|  OSF Preprints
Search
Searchstring |  1\. ChatGPT [Text Word]
|  2\. LLM [Text Word]
|  3\. Large Language Model [Text Word]
|  4\. 1 OR 2 OR 3
|  4\. Ethics [Text Word]
|  5\. Moral [Text Word]
|  6\. 4 OR 5
|  7\. 3 AND 6
Wildcards and database-specific truncations (e.g. ethic*, moral*) where used
where appropriate and applicable.

Data was extracted using a self-designed extraction form (see Supplement 2).
The extraction categories were transformed into a coding tree using MaxQDA.
Both authors independently coded 10% of the material to develop and refine the
coding scheme in more detail. The remaining material was extracted by J.H.
Results were iteratively discussed in three joint coding sessions.

A final synthesis was conducted following a meta-aggregative approach. Based
on our extraction fields, we, first, developed preliminary categories
encompassing actors, values, device properties, arguments, recommendations and
conclusions. These categories were, then, iteratively refined and aggregated
through additional coding until saturation was reached.

Given the constraints of normative quality appraisal[37] and in line with our
research goal to portrait the landscape of ethical discussions, we decided to
take a hybrid approach to the quality question. We descriptively report on
procedural quality criteria (see Table II) to distinguish material that
underwent processual quality control (such as peer review) from other
material. In addition, we critically engage with the findings during reporting
to appraise comprehensiveness and validity of the extracted information
pieces.

##  III Results

Our search yielded a total of 796 database hits. After removal of duplicates,
738 records went through title/abstract screening. 158 full-texts were
assessed. 53 records were included in the dataset, encompassing 23 original
articles, [23, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53,
54, 55, 56, 57, 58, 59] including theoretical or empirical work, 11
letters,[60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70] six editorials,[71, 72,
73, 74, 75, 76] four reviews,[77, 8, 78, 79] three comments,[22, 80, 81] one
report[82] and five unspecified articles.[83, 84, 85, 86, 87] Most works focus
on applications utilizing ChatGPT across various healthcare fields, as
indicated in Table II.

TABLE II: Overview on Included Records Publication |  Procedural Quality Control |  Setting
---|---|---
Title |  Type of Work |  Peer Reviewed |  COI |  Device |  Field of Application
[80][80] |  Commentary |  Unclear |  Unclear |  ChatGPT; ChatGPT 4 |  Nursing education, research and practice
[38][38] |  Empirical Article |  Yes |  None disclosed |  GPT 3 |  Neurology
[60][60] |  Letter |  No |  None disclosed |  ChatGPT |  Emergency Medicine
[39][39] |  Theoretical Article |  Preprint |  Unclear |  ChatGPT, Google Bard, Meta LLaMA |  Healthcare
[40][40] |  Empirical Article |  Yes |  Unclear |  ChatGPT 4 |  Pediatric Palliative Care
[41][41] |  Empirical Article |  Preprint |  Unclear |  ChatGPT; GPT 3.5 |  Ophtalmology
[61][61] |  Letter |  No |  None disclosed |  ChatGPT |  Obesity Treatment
[62][62] |  Letter |  No |  Conflict disclosed |  ChatGPT |  Dermatology
[63][63] |  Letter |  No |  Conflict disclosed |  ChatGPT |  Aesthetic Medicine
[43][43] |  Empirical Article |  Yes |  None disclosed |  ChatGPT |  Epidemiological Research
[64][64] |  Letter |  No |  None disclosed |  ChatGPT; GPT 3 |  Infectiology
[42][42] |  Theoretical Article |  Preprint |  Unclear |  ChatGPT; ChatDoctor; Google BARD |  Sport Science and Medicine
[83][83] |  Unspecified |  Yes |  None disclosed |  ChatGPT; GPT 3.5 |  Nuclear Medicine and Radiology
[8][8] |  Review |  Yes |  None disclosed |  ChatGPT |  Medicine
[49][49] |  Theoretical Article |  Yes |  Conflict disclosed |  GPT; BERT; GPT 2; GPT 3; GPT 4; Instruct GPT; BioBERT; BioGPT; PubMedGPT; Med-PaLm; CORD-19 |  Public Health
[84][84] |  Unspecified |  Unclear |  None disclosed |  ChatGPT |  Dentistry
[44][44] |  Theoretical Article |  Preprint |  Unclear |  ChatGPT |  Healthcare
[81][81] |  Commentary |  Unclear |  None disclosed |  ChatGPT |  Dermatology
continued on the next page
TABLE II: Overview on Included Records (Continued) Publication |  Procedural Quality Control |  Setting
---|---|---
Title |  Type of Work |  Peer Reviewed |  COI |  Device |  Field of Application
[85][85] |  Unspecified |  Yes |  None disclosed |  ChatGPT |  Emergency Medicine
[45][45] |  Empirical Article |  Preprint |  Conflict disclosed |  ChatGPT; GPT 3; NeuroGPT-X |  Neurosurgery
[82][82] |  Report |  Preprint |  Unclear |  ProteinChat |  Protein Research
[65][65] |  Letter |  No |  None disclosed |  ChatGPT |  Aesthetic Surgery
[86][86] |  Unspecified |  Yes |  Conflict disclosed |  ChatGPT; LaMDA; BARD; Med-Palm |  Healthcare
[46][46] |  Empirical Article |  Preprint |  None disclosed |  ChatGPT; InstructGPT |  Cardiopulmonary Medicine
[47][47] |  Empirical Article |  Preprint |  Unclear |  ChatGPT; GPT 4; Elicit; Med-PaLM |  Education, Research and Healthcare
[66][66] |  Letter |  No |  Conflict disclosed |  ChatGPT |  Infection Medicine
[71][71] |  Editorial |  No |  Unclear |  ChatGPT |  Pharmacy
[72][72] |  Editorial |  No |  None disclosed |  ChatGPT |  Surgery
[48][48] |  Empirical Article |  Preprint |  None disclosed |  ChatGPT; GPT 3 |  Ophtalmology
[67][67] |  Letter |  No |  No |  ChatGPT |  Surgery
[22][22] |  Commentary |  Unclear |  No |  ChatGPT; BioGPT; LaMDA; Sparrow; Pangu Alpha; OPT-IML; Megataron Turing MLG |  Medicine and Medical Research
[50][50] |  Empirical Article |  Preprint |  None disclosed |  ChatGPT |  Occupational Medicine
[73][73] |  Editorial |  No |  Conflict disclosed |  ChatGPT 4 |  Microbial genomics research
[51][51] |  Empirical Article |  Preprint |  Unclear |  BERT; BioBERT; BioClinicalBERT; SciBERT; UMLS-BERT |  Medicine
[68][68] |  Letter |  Preprint |  Conflict disclosed |  ChatGPT 4 |  Psychopharmacology
[52][52] |  Empirical Article |  Preprint |  Unclear |  ChatGPT; GPT 3.5 Turbo; accGPT |  Radiology
continued on the next page
TABLE II: Overview on Included Records (Continued) Publication |  Procedural Quality Control |  Setting
---|---|---
Title |  Type of Work |  Peer Reviewed |  COI |  Device |  Field of Application
[77][77] |  Review |  Preprint |  None disclosed |  ChatGPT |  Healthcare
[53][53] |  Theoretical Article |  Yes |  None disclosed |  GPT 2 |  Public Health
[54][54] |  Theoretical Article |  Preprint |  None disclosed |  ChatGPT; BERT |  Healthcare
[74][74] |  Editorial |  No |  Unclear |  ChatGPT |  Mental Health
[87][87] |  Unspecified |  No |  Unclear |  ChatGPT |  Pharmacy
[55][55] |  Theoretical Article |  Preprint |  None disclosed |  BERT; various Natural language processing models |  Healthcare
[56][56] |  Empirical Article |  Preprint |  None disclosed |  ChatGPT; GPT 4 |  Otolaryngology
[57][57] |  Empirical Article |  Preprint |  Unclear |  ChatGPT; GPT 3.5 |  Medicine
[78][78] |  Review |  Yes |  None disclosed |  ChatGPT |  Healthcare and Health Research
[75][75] |  Editorial |  No |  Unclear |  ChatGPT |  Mental Health Nursing
[69][69] |  Letter |  No |  None disclosed |  ChatGPT; GPT 4 |  Opthalmology
[79][79] |  Review |  Preprint |  None disclosed |  BERT; BioBERT; BlueBERT; PubMedBERT; ChatGPT; GPT 4; BioGPT; Med-PaLM |  Healthcare and Medicine
[59][59] |  Empirical Article |  Preprint |  None disclosed |  ChatGPT; GPT 3.5 |  Hepatology
[58][58] |  Empirical Article |  Preprint |  None disclosed |  ChatGPT; GPT 4 |  Hepatology
[26][26] |  Empirical Article |  Preprint |  Unclear |  ChatGPT; Foresight; PaLM, Gopher; Chinchilla |  Medicine
[76][76] |  Editorial |  No |  None disclosed |  ChatGPT |  Nursing
[70][70] |  Letter |  No |  None disclosed |  ChatGPT |  Neuropsychiatric practice and research

During analysis, four general themes emerged in our dataset, which we use to
structure reporting. These themes include clinical applications, patient
support applications, support of health professionals, and public health
perspectives. Table III provides exemplary scenarios for each theme derived
from the dataset.

TABLE III: Exemplary applications of LLMs Predictive Analysis and Risk
Assessment
---
[42][42] |  Supporting initial diagnose and triaging of patients by fine-tuning LLMs on a specialised dataset of electronic medical records, clinical notes, Sports science and medicine literature.
[55][55] |  Using traditional and modern natural language processing to triage patients on arrival based on structured data and unstructered free-text history of presenting complaint to predict risk stratification. This includes predictions on the likelihood of admission to hospital, prediction of critical illness, prediction of triage score, prediction of provider-assigned chief complaint, prediction of investigation, and prediction of infection.
Patient Consultation and Communication
[63][63] |  Enhancing patient consultations by providing accurate and reliable information on aesthetic procedures, their risks, benefits and potential outcomes, enabling well-informed decisions and improved treatment outcomes.
[83][83] |  Providing language translation and helping health professionals to communicate with patients speaking foreign languages; helping health professionals to educate their patients and empower patients to take an active role.
TABLE III: Exemplary applications of LLMs (Continued) Diagnosis
---
[38][38] |  Using GPT 3 to distinguish individuals with Alzheimers Disease from healthy controls and to infer cognitive testing scores based on linguistic features. It is shown that the approach outperforms conventional approaches and performs comparable to specifically fine-tuned models. Usable as a web app in the doctors office.
[52][52] |  Supporting radiologists diagnostic performance by providing imaging recommendations in accordance with recent guidelines.
Treatment Planning
[61][61] |  Using ChatGPT to provide personalized recommendations on topics such as nutrition, exercise and psychological support in obesity treatment.
[64][64] |  Using ChatGPT to provide treatment recommendations based on patients clinical presentation, disease severity, and comorbidities.
Patient Support
[58][58] |  Using ChatGPT as an informational plattform to comprehend and to respond to cirrhosis related questions in different languages, adressing barriers that may impact patient care.
[48][48] |  Using ChatGPT for the assessment of acute ophtalmological conditions with regard to triage accurracy and recommendations of preclinical measures.
Professional Support and Reserach
[47][47] |  Using LLMs to increase efficiency in note-taking through prepopulation of forms, voice recording and morphing into clinical notes or synthesizing existing patient notes to save clinicians time.
[85][85] |  Using Conversational AI to create study documents by translating complex concepts into simpler ones or designing informed consent documents for patients.
[82][82] |  Using a ChatGPT-like (ProteinGPT) systems to accelerate protein research. The model is aimed at learning and understanding protein 3D structures. ProteinGPT enables users to upload proteins, ask questions, and engage in interactive conversations to gain insights.
Public Health
[53][53] |  Using LLMs to create an AI-guided message creation system to disseminate health related information via social media.
[64][64] |  Using ChatGPT to monitor news and social media platforms for signs of outbreaks of disease clusters and to alert health professionals to potential threats.

###  III-A Clinical applications

####  III-A1 Predictive analysis and risk assessment

To support initial diagnose and triaging of patients,[42, 55] several authors
discuss the use of LLMs in the context of predictive patient analysis and risk
assessment in or prior to clinical situations as a potentially transformative
application.[77, 83] The role of LLMs in this scenario is described as that of
a “co-pilot” using available patient information to flag areas of concern or
to predict diseases and risk factors.[47]

[83], in line with most authors, notes that predicting health outcomes and
relevant patterns is very likely to improve patient outcomes and contributes
to patient benefit.[83] For example, overcrowded emergency departments present
a serious issue worldwide and have a significant impact on patient outcomes.
From a perspective of avoidance of harm, using LLMs with triage notes could
lead to reduced length of stay and a more efficient utilization of time in the
waiting room.[55]

All authors note, however, that such applications might also be problematic
and require close human oversight.[42, 47, 54, 83] Although LLMs might be able
to reveal connections between disparate knowledge,[43] generating inaccurate
information would have severe negative consequences.[47, 77] It might lead to
direct harm to patients or provide clinicians with false and dangerous
justifications and rationales for their decisions.[77] These problems are
tightly connected to inherent biases in LLMs, their tendency to “hallucinate”
and their intransparency.[55] In addition, uncertainties are increased by use
of unstructured data. Medical notes often differ from the data pretrained
models utilise. This makes it difficult to predict accuracy of output when
inputting such data or using it to fine-tune LLMs.[55] Interpretability of
results and recommendations introduce additional complexity and sources of
potential harm.[55] [83] notes that despite such difficulties, the use of LLMs
proceeds largely in absence of guidelines, recommendations and control. The
outcome, hence, ultimately depends on clinicians’ ability to interpret
findings and identify inaccurate information.[83]

####  III-A2 Patient consultation and communication

LLMs can offer a novel approach in patient-provider interaction, where they
can facilitate informational exchange and bridge gaps between clinical and
preclinical settings such as self-management measures or community aids.[8]
This includes easing the transition between settings by removing barriers to
communication[47, 63, 83, 86] or removing barriers in the clinical workflow to
facilitate timely and efficient support. As is suggested, LLMs can collect
information from patients or provide additional information, enabling well-
informed decisions and increasing satisfaction in patients.[59, 63, 83]
Provision of language translation and simplification of medical jargon may
allow patients to become more engaged in the process and enhance patient-
provider communication.[83, 86] However, it remains unclear in our dataset how
such applications would look like in practice — specifically where, when and
how LLMs actually could be integrated.

These suggestions require to consider ethically relevant boundaries regarding
the protection of patient data, and safety,[39, 63, 80, 86] potentially unjust
disparities,[39, 63, 86] and the wider dimensions of care such as the
therapeutic relationship.[39, 62, 64, 67, 80] Robust measures to avoid
incorrect information in technological mediation of communication and the need
to strike a balance with “the human touch” of care[63] are stressed. With
regard to the former, [63] argue for robust expert oversight. Regarding the
latter, [67] note a potential shift in power dynamics between patients and
providers in which providers might lose their authoritative position and might
be seen as less knowledgeable.[67] Others fear a loss of personal care that
should be avoided[39, 64, 80] and the lack of contextual content of individual
health challenges.[80, 45] Open communication and consent to technical
mediation of patient-provider communication is required to promote trust but
might be difficult to achieve.[72, 81]

####  III-A3 Diagnosis

Many studies in our dataset discusses the possible use of LLMs for
diagnosis.[8, 39, 42, 47, 62, 64, 69, 70, 77, 78, 81, 83] It is suggested that
the LLMs’ ability to analyze large amounts of unstructured data provides
pathways to timely, efficient and more accurate diagnosis to the benefit of
patients.[38, 39, 70, 78, 81] It might also enable the discovery of hidden
patterns[42] and reduce healthcare costs.[39, 52]

An ethical problem emerges with potentially negative effects on patient
outcomes due to biases in the training data,[39, 42, 44, 77, 78, 81]
especially with the lack of diverse datasets risking underrepresentation of
marginalized or vulnerable groups. Biased models may result in unfair
treatment of disadvantaged groups, leading to disparities in access,
exacerbating existing inequalities, or harming persons through selective
accuracy.[44] Based on an experimental study setup, [26] deliver an insightful
example showing that ChatGPT and Foresight NLP exhibit racial bias towards
black patients.[26] Problems of interpretability, hallucinations, and
falsehood mimicry pile onto this issue and increase these risks.[38, 39, 47,
77] With regard to transparency, two sources suggest that LLM-supported
diagnosis hamper the process of providing adequate justification due to their
opacity.[39, 77] This is understood to threaten the authoritative position of
professionals, leaving them at risk of not being able to provide a rationale
for a diagnosis[38] and might lead to an erosion of trust between both
parties. This is in line with others noting that LLMs are not able to
replicate a process of clinical reasoning in general and, hence, fail to
comprehend the complexity of the process.[47, 62, 78] Based on the principle
of avoidance of harm, it is an important requirement to subject each generated
datum to clinical validation as well as to develop “ethical and legal systems”
to mitigate these problems.[39, 42, 62]

It needs to be noted, however, that the technically unaided process of
diagnoses is also known to be subjective and prone to error.[70] This implies
that an ethical evaluation should be carried out in terms of relative
reliability and effectiveness compared to existing alternatives. Whether and
under what circumstances this might be the case is a question that is not
addressed.

####  III-A4 Treatment planning

Six studies in our dataset highlight the use of LLMs in providing personalized
recommendations for treatment regimens or to support clinicians in treatment
decisions based on electronic patient information or history,[61, 63, 64, 69,
70, 83] providing a quick and reliable course of action to clinicians and
patients. However, as with diagnostic applications, biases and perpetuating
existing stereotypes and disparities is a constantly discussed theme.[63, 64,
70] [44] also cautions that LLMs will likely prioritize certain types of
treatments or interventions over others, disproportionately benefiting certain
groups and disadvantaging others.[44]

Additionally, it is highlighted that inputting patient data raises ethical
questions regarding confidentiality, privacy, and data security.[61, 63, 64,
69, 70] This especially applies to commercial and publicly available models
such as ChatGPT. Inaccuracies in potential treatment recommendations are also
noted as a concerning source for harm.[61, 63, 64, 69, 70] In a broader
context, several authors suggest that for some LLMs the absence of internet
access, insufficient domain-specific data, limited access to treatment
guidelines, knowledge on local or regional characteristics of the healthcare
system, and outdated research significantly heighten the risk of inaccurate
recommendations.[22, 40, 41, 43, 50, 58]

###  III-B Patient support applications

####  III-B1 Patient information and education

Almost all authors concerned with patient-facing applications highlight the
benefits of rapid and timely information access that users experience with
state-of-the-art LLMs. [72] compare patients’ use of chatbots with shifts that
have accompanied the development of the internet as patient information
source.[72] Such access can improve laypersons’ health literacy by providing a
needs-oriented access to comprehensible medical information,[71] which is
regarded as an important precondition of autonomy to allow more independent,
health-related decisions.[8, 77] In their work on the use of ChatGPT 4 in
overcoming language barriers, [59] highlight an additional benefit, as LLMs
could provide cross-lingual translation and thus contribute to equalizing
healthcare and racial disparities.[59]

Regarding ethical concerns and risks, biases are seen as an significant source
of harm.[8, 42, 77, 78] The literature also highlights a crucial difference in
the ethical acceptability of using patient support applications, leading to
more critical stance when LLMs are used by laypersons compared to health
professionals.[26, 56] However, ethical acceptability varies across fields;
for instance, otolaryngology and infectious disease studies find ChatGPT’s
responses to patients lack detail but aren’t harmful,[56] whereas pharmacology
and mental health indicate greater potential risks.[70, 71]

####  III-B2 Symptom assessment and health management

LLMs can offer laypersons personalized guidance, such as lifestyle adjustments
during illness,[83] self-assessment of symptoms,[64, 66] self-triaging, and
emergency management steps.[8, 60] Although current arrangements seem to
perform well and generate compelling responses,[8, 50, 66] a general lack of
situational awareness is noted as a common problem that might lead to severe
harm.[8, 64, 66] Situational awareness means the ability to generate responses
based on contextual criteria such as the personal situation, medical history
or social situation. The inability of most current LLMs to seek clarifications
by asking questions and their lack of sensitivity to query variations can lead
to imprecise answers.[48, 66] For instance, research by [48] on self-triaging
in ophthalmologic emergencies indicates that ChatGPT’s responses can’t
reliably prioritize urgency, reducing their usefulness.[48]

###  III-C Support of health professionals and researchers

####  III-C1 Documentation and administrative tasks

LLMs could automate tasks like medical reporting,[83] or summarizing patient
interactions[8] including automatic population of forms or discharge
summaries. The consensus is that LLMs could streamline clinical workflows,[8,
39, 46, 54, 55, 63, 71, 77, 83, 84, 86] offering time savings for health
professionals currently burdened with extensive administrative duties.[71, 86]
By automating these repetitive tasks, professionals could dedicate more time
to high-quality medical tasks.[86] Crucially, such applications would require
the large-scale integration of LLMs into existing clinical data systems.[52]

####  III-C2 Research

In health research, LLMs are suggested to support text, evidence or data
summarization,[57, 67, 85] identify research targets,[8, 64, 75, 86] designing
experiments or studies,[75, 86] or facilitate knowledge sharing between
collaborators,[40, 73, 83] and to communicate results.[77] This highlights the
potentials for accelerating research[49, 82] and relieving researchers of
workload,[8, 43, 67, 77, 78, 86] leading to more efficient research workflows
allowing researchers to spend less time on burdensome routine work.[8, 83] To
certain authors, this could involve condensing crucial aspects of their work,
like crafting digestible research documents for ethics reviews or consent
forms.[85] However, LLMs capacities are also critically examined, with [57]
Tang et al. emphasizing ChatGPT’s tendency to produce attribution and
misinterpretation errors, potentially distorting original source information,
echoing concerns over interpretability, reproducibility, uncertainty handling,
and transparency.[57, 77]

Some authors fear that using LLMs could compromise research integrity by
disrupting traditional trust factors like source traceability, factual
consistency, and process transparency.[22] Additionally, concerns about
overreliance and deskilling are raised, as LLMs might diminish researchers’
skills and overly shape research outcomes.[49] Given that using such
technologies inevitably introduces biases and distortions to the research
flow, [73] suggest researchers must maintain vigilance to prevent undue
influence from biases introduced by these technologies, advocating for strict
human oversight and revalidation of outputs.[73]

Figure 1: Fields of Application of LLMs in Medicine

###  III-D Public Health perspectives

The dataset encompasses studies that explore the systemic implications of
LLMs, especially from a public health perspective.[53, 64, 78] This includes
using LLMs in public health campaigns, for monitoring news and social media
for signs of disease outbreaks[64] and targeted communication strategies.[53]
Additionally, research examines the potential for improving health literacy or
access to health information, especially in low-resource settings. Access to
health information through LLMs can be maintained free of charge or at very
low costs for laypersons.[58] Considering the case of mental health,
especially low and middle income countries might benefit.[74] These countries
often have a huge treatment gap driven by a deficit in professionals or
inequitable resource distribution. Using LLMs could mitigate accessibility and
affordability issues, potentially offering a more favorable alternative to the
current lack of access.[74]

However, a number of authors raise doubts about overly positive expectations.
[53] highlight the risks of a dual use of LLMs.[53] While they might further
equal access to information, malicious actors can and seem to be using LLMs to
spread fake information and devise health messages at an unprecedented scale
that is harmful to societies.[53, 54, 78] [49] take this concern one step
further, presenting the concept of an AI-driven infodemic[49] in which the
overwhelming spread of imprecise, unclear, or false information leads to
disorientation and potentially harmful behavior among recipients. Health
authorities have often seen AI technologies as solutions to information
overload. However, the authors caution that an AI-driven infodemics could
exacerbate future health threats. While infodemic issues in social media and
grey literature are noted, AI-driven infodemics could also inundate scientific
journals with low-quality, excessively produced content.[49]

The commercial nature of most current LLMs systems present another critical
consideration. The profit-driven nature of the field can lead to
concentrations of power among a limited number of companies and a lack of
transparency. This economic model, as highlighted by several studies, can have
negative downstream effects on accessibility and affordability.[22, 39, 46]
Developing, using, or refining models can be expensive, limiting accessibility
and customization for marginalized communities. Power concentration also means
pricing control lies with LLM companies, with revenues predominantly directed
towards them.[47] These questions are also mirrored in the selection of
training data and knowledge bases[22] which typically encompass knowledge from
well-funded, English speaking countries and, thus, significantly
underrepresents knowledge from other regions. This could exacerbate health
disparities by reinforcing biases rather than alleviating them.

##  IV Discussion

Our analysis has unveiled an extensive range of LLM applications currently
under investigation in medicine and healthcare (see Figure 1). This surge in
LLMs was largely caused by the advent and ease of use of ChatGPT, a platform
not originally tailored for professional healthcare settings, yet widely
adopted within it.[10, 86] This presents a rather unique instance where a
general-purpose technology has rapidly permeated the sector of healthcare and
research to an unprecedented extent.

Figure 2: Discussed Dimensions of Impact of LLMs

Our review highlights a vivid testing phase of LLMs across various healthcare
domains.[10] Despite the lack of real-world applications, especially in the
clinic, there is an overarching sentiment of the promise LLMs hold. It is
posited that these tools could increase the efficiency of healthcare delivery
and research, with the potential to benefit patient outcomes while alleviating
burdensome workload of healthcare professionals. These advantages of LLMs are
largely attributed to their capabilities in data analysis, personalized
information provisioning, and support in decision-making, particularly where
quick analysis of voluminous unstructured data is paramount. Moreover, by
mitigating information loss and enhancing medical information accessibility,
LLMs stand to significantly bolster healthcare quality.

However, our study has also surfaced recurrent ethical concerns associated
with LLMs. These concerns echo the wider discourse on AI ethics,[88, 89, 90]
particularly in healthcare,[91] and touch on issues of fairness, bias, non-
maleficence, transparency, and privacy. Yet, LLMs introduce a distinctive
concern linked to a dimension of epistemic values, that is, their tendency to
produce harmful misinformation or convincingly but inaccurate content through
hallucinations as illustrated in Figure 2.[92] The effects of such
misinformation are particularly severe in healthcare, where the outcome could
be dire. The inherent statistical and predictive architecture combined with
the intransparency of LLMs presents significant hurdles in validating the
clinical accuracy and reliability of their outputs.[93, 94, 95]

The inclination of LLMs to output erroneous information underscores the need
for human oversight and continual validation of machine-generated output, as
our dataset demonstrates. This need is accentuated by the lack of professional
guidelines or regulatory oversight within this field.[21] Consequently, there
is a noticeable demand for ethical guidelines, evidenced within the literature
surrounding healthcare applications of LLMs.[49, 63, 67, 73, 74, 77, 78, 81]

While we concur with the need for such guidance, our analysis suggests that
the real challenge lies not in the articulation of such a need but in
comprehending the scope of what this entails. There are inherent and
contextual limitations and benefits associated with LLMs that warrant
consideration. Inherently, state-of-the-art LLMs carry the risks of biases,
hallucinations, and challenges in validity assessment, reliability testing,
and reproducibility. Contextually, the effectiveness of LLM usage hinges on
various situational factors, including the user utilizing LLMs, their level of
expertise as well as their epistemic position (e.g. expert versus layperson),
the specific domain of application, the risk profile of the application, and
potential alternatives that the LLM is compared against.

A nuanced ethical discourse must recognize the multilayered nature of
LLM–usage, from the epistemic stance of the user to the potential for harm,
the varying degrees of potential harm due to misinformation or bias, and the
diverse normative benchmarks for performance and acceptable levels of
uncertainty. Our recommendation is to reframe the ethical guidance debate to
focus on defining what constitutes acceptable human oversight and validation
across the spectrum of applications and users. This involves considering the
diversity of epistemic positions of users, the varying potentials for harm,
and the different acceptable thresholds for performance and certainty in
diverse healthcare settings.

Given these questions, a critical inquiry is necessary to the extent to which
the current experimental use of LLMs is both necessary and justified. Our
dataset exemplifies a diversity of perspectives, methodologies, and
applications of LLMs, revealing a significant degree of ambiguity and
uncertainty about the appropriate engagement with this technology. Notably, a
portion of current research seems propelled more by a sense of experimental
curiosity than by well-defined methodological rigor, at times pushing the
boundaries of ethical acceptability, particularly when sensitive real patient
data are utilized to explore capabilities of systems like ChatGPT.

To frame these developments, it is instructive to adopt the perspective of
viewing the implementation of LLMs as akin to a “social experiment”.[96, 97]
We employ this concept in a descriptive sense to denote a situation in which –
according to van der Poel – the benefits, risks and ethical issues of a
technology can only fully manifest subsequent to its widespread
introduction.[97] Such a stance recognizes that the novelty of LLMs, combined
with its inherent complexity and opacity, necessitates an iterative process of
diminishing uncertainties and learning through which consequences only
gradually emerge. By the same time, framing the current developments as social
experiment also reinforces the need to establish and respect ethical limits –
especially within the healthcare domain, where professional duties and
responsibilities towards patients are foundational.

With this in mind we suggest that understanding how we acquaint ourselves with
disruptive technologies must be central to any future ethical discourse. There
is a compelling need for additional research to ascertain the conditions under
which LLMs can be appropriately utilized in healthcare, but also to establish
conditions of gradual experimentation and learning that align with principles
of health ethics.

##  V Conclusion

This review addresses ethical considerations of using LLMs in healthcare at
the current developmental stage. However, serval limitations are import to
acknowledge. Ethical examination of LLMs in healthcare is still nascent and
struggles to keep pace with rapid technical advancements. Thus, the review
offers a starting point for further discussions. A significant portion of the
source material originated from preprint servers and did not undergo rigorous
peer review, potentially introducing limitations in their quality and
generalizability. Additionally, the findings’ generalizability may be limited
due to variations in researched settings, applications, and interpretations of
LLMs.

## Declarations

### Author Contributions

JH and RR designed the protocol. JH conducted the search. Screening data
extraction and analysis was jointly conducted by JH and RR as described in the
methods section. JH wrote the first draft of the manuscript which was, then,
revised by RR. RR is the PI of the project from which this article derives.
All authors read and approved the final version.

### Declaration of Funding

This study was funded by the VolkswagenStiftung as part of the Digital Medical
Ethics Network (grant number 9B233). The funder played no role in study
design, data collection, analysis and interpretation of data, or the writing
of this manuscript.

### Conflict of Interest

All authors declare no financial or non-financial competing interests.

### Data Availability Statement

The datasets used and/or analysed during the current study available from the
corresponding author on reasonable request.

### Acknowledgement

Fig.1 and Fig. 2 created with BioRender.com

## References

  * [1] Jean Kaddour et al.  “Challenges and Applications of Large Language Models”  arXiv, 2023  DOI: 10.48550/ARXIV.2307.10169
  * [2] Rishi Bommasani et al.  “On the Opportunities and Risks of Foundation Models”  arXiv, 2022  DOI: 10.48550/arXiv.2108.07258
  * [3] Peter Lee, Carey Goldberg and Isaac Kohane  “The AI Revolution in Medicine: GPT-4 and Beyond”  Hoboken: Pearson, 2023
  * [4] Peter Lee, Sebastien Bubeck and Joseph Petro  “Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine”  In _New England Journal of Medicine_ 388.13 Massachusetts Medical Society, 2023, pp. 1233–1239  DOI: 10.1056/NEJMsr2214184
  * [5] Arun James Thirunavukarasu et al.  “Large Language Models in Medicine”  In _Nature Medicine_ 29.8, 2023, pp. 1930–1940  DOI: 10.1038/s41591-023-02448-8
  * [6] Jan Clusmann et al.  “The Future Landscape of Large Language Models in Medicine”  In _Communications Medicine_ 3.1, 2023, pp. 141  DOI: 10.1038/s43856-023-00370-1
  * [7] Malik Sallam  “ChatGPT Utility in Healthcare Education, Research, and Practice: Systematic Review on the Promising Perspectives and Valid Concerns”  In _Healthcare_ 11.6, 2023, pp. 887  DOI: 10.3390/healthcare11060887
  * [8] Tirth Dave, Sai Anirudh Athaluri and Satyam Singh  “ChatGPT in Medicine: An Overview of Its Applications, Advantages, Limitations, Future Prospects, and Ethical Considerations”  In _Frontiers in artificial intelligence_ 6, 2023, pp. 1169595–1169595  DOI: 10.3389/frai.2023.1169595
  * [9] Diane M. Korngiebel and Sean D. Mooney  “Considering the Possibilities and Pitfalls of Generative Pre-trained Transformer 3 (GPT-3) in Healthcare Delivery”  In _npj Digital Medicine_ 4.1, 2021, pp. 93  DOI: 10.1038/s41746-021-00464-x
  * [10] Jianning Li et al.  “ChatGPT in Healthcare: A Taxonomy and Systematic Review”  In _Computer Methods and Programs in Biomedicine_ 245, 2024, pp. 108013  DOI: 10.1016/j.cmpb.2024.108013
  * [11] Arya Rao et al.  “Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow”  medRxiv, 2023  DOI: 10.1101/2023.02.21.23285886
  * [12] Hao Liu, Yifan Peng and Chunhua Weng  “How Good Is ChatGPT for Medication Evidence Synthesis?”  In _Studies in Health Technology & Informatics_ 302, 2023, pp. 1062–1066  DOI: 10.3233/SHTI230347
  * [13] Hirotaka Takita et al.  “Diagnostic Performance Comparison between Generative AI and Physicians: A Systematic Review and Meta-Analysis”  medRxiv, 2024, pp. 2024.01.20.24301563  DOI: 10.1101/2024.01.20.24301563
  * [14] Jae Hyuk Kim, Sun Kyung Kim, Jongmyung Choi and Youngho Lee  “Reliability of ChatGPT for Performing Triage Task in the Emergency Department Using the Korean Triage and Acuity Scale”  In _DIGITAL HEALTH_ 10 SAGE Publications Ltd, 2024  DOI: 10.1177/20552076241227132
  * [15] John W. Ayers et al.  “Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum”  In _JAMA Internal Medicine_ 183.6, 2023, pp. 589–596  DOI: 10.1001/jamainternmed.2023.1838
  * [16] Karan Singhal et al.  “Large Language Models Encode Clinical Knowledge”  In _Nature_ 620.7972, 2023, pp. 172–180  DOI: 10.1038/s41586-023-06291-2
  * [17] Thilo Hagendorff  “Mapping the Ethics of Generative AI: A Comprehensive Scoping Review”  arXiv, 2024  DOI: 10.48550/ARXIV.2402.08323
  * [18] Emily M. Bender, Timnit Gebru, Angelina McMillan-Major and Shmargaret Shmitchell  “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?”  In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ , 2021, pp. 610–623  DOI: 10.1145/3442188.3445922
  * [19] Laura Weidinger et al.  “Taxonomy of Risks Posed by Language Models”  In _2022 ACM Conference on Fairness, Accountability, and Transparency_ , 2022, pp. 214–229  DOI: 10.1145/3531146.3533088
  * [20] Partha Pratim Ray  “ChatGPT: A Comprehensive Review on Background, Applications, Key Challenges, Bias, Ethics, Limitations and Future Scope”  In _Internet of Things and Cyber-Physical Systems_ 3, 2023, pp. 121–154  DOI: 10.1016/j.iotcps.2023.04.003
  * [21] Stephen Gilbert et al.  “Large Language Model AI Chatbots Require Approval as Medical Devices”  In _Nature Medicine_ 29.10, 2023, pp. 2396–2398  DOI: 10.1038/s41591-023-02412-6
  * [22] Hanzhou Li et al.  “Ethics of Large Language Models in Medicine and Medical Research”  In _The Lancet Digital Health_ 5.6, 2023, pp. e333–e335  DOI: 10.1016/S2589-7500(23)00083-3
  * [23] Changyu Wang et al.  “Ethical Considerations of Using ChatGPT in Health Care”  In _Journal of Medical Internet Research_ 25, 2023, pp. e48009  DOI: 10.2196/48009
  * [24] Thomas Grote and Philipp Berens  “A Paradigm Shift?: On the Ethics of Medical Large Language Models.”  In _Bioethics_ , forthcoming
  * [25] Abubakar Abid, Maheen Farooqi and James Zou  “Large Language Models Associate Muslims with Violence”  In _Nature Machine Intelligence_ 3.6, 2021, pp. 461–463  DOI: 10.1038/s42256-021-00359-2
  * [26] Joshua Au Yeung et al.  “AI Chatbots Not yet Ready for Clinical Use”  medRxiv, 2023  DOI: 10.1101/2023.03.02.23286705
  * [27] Jesutofunmi A. Omiye et al.  “Large Language Models Propagate Race-Based Medicine”  In _npj Digital Medicine_ 6.1, 2023, pp. 195  DOI: 10.1038/s41746-023-00939-z
  * [28] Travis Zack et al.  “Assessing the Potential of GPT-4 to Perpetuate Racial and Gender Biases in Health Care: A Model Evaluation Study”  In _The Lancet Digital Health_ 6.1, 2024, pp. e12–e22  DOI: 10.1016/S2589-7500(23)00225-X
  * [29] Harini Suresh and John V. Guttag  “A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle”  In _EAAMO ’21: Equity and Access in Algorithms, Mechanisms, and Optimization_ , 2021, pp. 1–9  DOI: 10.1145/3465416.3483305
  * [30] Ziad Obermeyer, Brian Powers, Christine Vogeli and Sendhil Mullainathan  “Dissecting Racial Bias in an Algorithm Used to Manage the Health of Populations”  In _Science_ 366.6464, 2019, pp. 447–453  DOI: 10.1126/science.aax2342
  * [31] Jonathan A. Saenger, Jonathan Hunger, Andreas Boss and Johannes Richter  “Delayed Diagnosis of a Transient Ischemic Attack Caused by ChatGPT”  In _Wiener klinische Wochenschrift_ , 2024  DOI: 10.1007/s00508-024-02329-1
  * [32] Michael Moor et al.  “Foundation Models for Generalist Medical Artificial Intelligence”  In _Nature_ 616.7956 Nature Publishing Group, 2023, pp. 259–265  DOI: 10.1038/s41586-023-05881-4
  * [33] Robert Ranisch and Joschka Haltaufderheide  “Ethics of ChatGPT: A Systemic Review of Large Language Models in Healthcare and Medicine”  In _PROSPERO 2023. CRD42023431326_
  * [34] Chantelle Garritty et al.  “Updated Recommendations for the Cochrane Rapid Review Methods Guidance for Rapid Reviews of Effectiveness”  In _BMJ_ 384 British Medical Journal Publishing Group, 2024, pp. e076335  DOI: 10.1136/bmj-2023-076335
  * [35] Alaa Abd-alrazaq et al.  “Large Language Models in Medical Education: Opportunities, Challenges, and Future Directions”  In _JMIR Medical Education_ 9, 2023, pp. e48291  DOI: 10.2196/48291
  * [36] Michael Liebrenz et al.  “Generating Scholarly Content with ChatGPT: Ethical Challenges for Medical Publishing”  In _The Lancet Digital Health_ 5.3 Elsevier, 2023, pp. e105–e106  DOI: 10.1016/S2589-7500(23)00019-5
  * [37] Marcel Mertz  “How to Tackle the Conundrum of Quality Appraisal in Systematic Reviews of Normative Literature/Information? Analysing the Problems of Three Possible Strategies (Translation of a German Paper)”  In _BMC Medical Ethics_ 20.1, 2019, pp. 81  DOI: 10.1186/s12910-019-0423-5
  * [38] Felix Agbavor and Hualou Liang  “Predicting Dementia from Spontaneous Speech Using Large Language Models.”  In _PLoS Digital Health_ 1.12, 2022, pp. e0000168  DOI: 10.1371/journal.pdig.0000168
  * [39] Hazrat Ali, Junaid Qadir and Zubair Shah  “ChatGPT and Large Language Models (LLMs) in Healthcare: Opportunities and Risks”  Techrxiv, 2023  DOI: 10.36227/techrxiv.22579852.v2
  * [40] Mohammed Almazyad et al.  “Enhancing Expert Panel Discussions in Pediatric Palliative Care: Innovative Scenario Development and Summarization with ChatGPT-4”  In _Curēus_ 15.4, 2023, pp. e38249  DOI: 10.7759/cureus.38249
  * [41] Fares Antaki et al.  “Evaluating the Performance of ChatGPT in Ophthalmology: An Analysis of Its Successes and Shortcomings”  medRxiv, 2023  DOI: 10.1101/2023.01.22.23284882
  * [42] Mark Connor and Michael O’Neill  “Large Language Models in Sport Science & Medicine: Opportunities, Risks and Considerations”  arXiv, 2023  DOI: 10.48550/arXiv.2305.03851
  * [43] G. Carullo, F. Toscano, N. Nante and D. Golinelli  “A Step-by-Step Researcher’s Guide to the Use of an AI-based Transformer in Epidemiology: An Exploratory Analysis of ChatGPT Using the STROBE Checklist for Observational Studies”  In _Journal of Public Health_ , 2023  DOI: 10.1007/s10389-023-01936-y
  * [44] Emilio Ferrara  “Should ChatGPT Be Biased? Challenges and Risks of Bias in Large Language Models”  arXiv, 2023  DOI: 10.48550/arXiv.2304.03738
  * [45] Edward Guo et al.  “neuroGPT-X: Towards an Accountable Expert Opinion Tool for Vestibular Schwannoma”  medRxiv, 2023  DOI: 10.1101/2023.02.25.23286117
  * [46] Ralf E Harskamp and Lukas De Clercq  “Performance of ChatGPT as an AI-assisted Decision Support Tool in Medicine: A Proof-of-Concept Study for Interpreting Symptoms and Management of Common Cardiac Conditions (AMSTELHEART-2)”  medRxiv, 2023  DOI: 10.1101/2023.03.25.23285475
  * [47] Mohammad Hosseini et al.  “An Exploratory Survey about Using ChatGPT in Education, Healthcare, and Research”  medRxiv, 2023  DOI: 10.1101/2023.03.31.23287979
  * [48] Dominik Knebel et al.  “Assessment of ChatGPT in the Preclinical Management of Ophthalmological Emergencies – an Analysis of Ten Fictional Case Vignettes”  medRxiv, 2023  DOI: 10.1101/2023.04.16.23288645
  * [49] L. De Angelis et al.  “ChatGPT and the Rise of Large Language Models: The New AI-driven Infodemic Threat in Public Health”  In _Frontiers in public health_ 11.(De Angelis, Baglivo, Arzilli, Privitera, Rizzo) Department of Translational Research and New Technologies in Medicine and Surgery, University of Pisa, Pisa, Italy, 2023, pp. 1166120–1166120  DOI: 10.3389/fpubh.2023.1166120
  * [50] Martina Padovan et al.  “ChatGPT in Occupational Medicine: A Comparative Study with Human Experts”  medRxiv, 2023  DOI: 10.1101/2023.05.17.23290055
  * [51] Ridam Pal, Hardik Garg, Shashwat Patel and Tavpritesh Sethi  “Bias Amplification in Intersectional Subpopulations for Clinical Phenotyping by Large Language Models”  medRxiv, 2023  DOI: 10.1101/2023.03.22.23287585
  * [52] A Rau et al.  “A Context-Based Chatbot Surpasses Trained Radiologists and Generic ChatGPT in Following the ACR Appropriateness Guidelines”  medRxiv, 2023  DOI: 10.1101/2023.04.10.23288354
  * [53] Ralf Schmälzle and Shelby Wilcox  “Harnessing Artificial Intelligence for Health Message Generation: The Folic Acid Message Engine.”  In _Journal of Medical Internet Research_ 24.1, 2022, pp. e28858  DOI: 10.2196/28858
  * [54] Sakib Shahriar and Kadhim Hayawi  “Let’s Have a Chat! A Conversation with ChatGPT: Technology, Applications, and Limitations”  arXiv, 2023  DOI: 10.48550/arXiv.2302.13817
  * [55] Jonathon Stewart et al.  “Applications of Natural Language Processing at Emergency Department Triage: A Systematic Review”  medRxiv, 2022  DOI: 10.1101/2022.12.20.22283735
  * [56] Krish Suresh et al.  “Utility of GPT-4 as an Informational Patient Resource in Otolaryngology”  medRxiv, 2023  DOI: 10.1101/2023.05.14.23289944
  * [57] Liyan Tang et al.  “Evaluating Large Language Models on Medical Evidence Summarization”  medRxiv, 2023  DOI: 10.1101/2023.04.22.23288967
  * [58] Yee Hui Yeo et al.  “Assessing the Performance of ChatGPT in Answering Questions Regarding Cirrhosis and Hepatocellular Carcinoma”  medRxiv, 2023  DOI: 10.1101/2023.02.06.23285449
  * [59] Yee Hui Yeo et al.  “GPT-4 Outperforms ChatGPT in Answering Non-English Questions Related to Cirrhosis”  medRxiv, 2023  DOI: 10.1101/2023.05.04.23289482
  * [60] Chiwon Ahn  “Exploring ChatGPT for Information of Cardiopulmonary Resuscitation.”  In _Resuscitation_ 185, 2023, pp. 109729  DOI: 10.1016/j.resuscitation.2023.109729
  * [61] S. Arslan  “Exploring the Potential of Chat GPT in Personalized Obesity Treatment”  In _Annals of Biomedical Engineering_ 51.9 United States: Springer, 2023, pp. 1887–1888  DOI: 10.1007/s10439-023-03227-9
  * [62] E.J. Beltrami and J.M. Grant-Kels  “Consulting ChatGPT: Ethical Dilemmas in Language Model Artificial Intelligence”  In _Journal of the American Academy of Dermatology_ , 2023  DOI: 10.1016/j.jaad.2023.02.052
  * [63] Giovanni Buzzaccarini, Rebecca Susanna Degliuomini and Marco Borin  “The Artificial Intelligence Application in Aesthetic Medicine: How ChatGPT Can Revolutionize the Aesthetic World”  In _Aesthetic plastic surgery_ 47.5, 2023, pp. 2211–2212  DOI: 10.1007/s00266-023-03416-w
  * [64] Kunming Cheng et al.  “Potential Use of Artificial Intelligence in Infectious Disease: Take ChatGPT as an Example”  In _Annals of biomedical engineering_ 51.6, 2023, pp. 1130–1135  DOI: 10.1007/s10439-023-03203-3
  * [65] Rohun Gupta, Kazimir Bagdady and Brian A. Mailey  “Ethical Concerns amidst Employment of ChatGPT in Plastic Surgery”  In _Aesthetic surgery journal_ 43.8, 2023, pp. NP656–NP657  DOI: 10.1093/asj/sjad108
  * [66] Alex Howard, William Hope and Alessandro Gerada  “ChatGPT and Antimicrobial Advice: The End of the Consulting Infection Doctor?”  In _Lancet Infectious Diseases_ 23.4, 2023, pp. 405–406  DOI: 10.1016/S1473-3099(23)00113-5
  * [67] Wenbo Li, Yinxu Zhang and Fengmin Chen  “ChatGPT in Colorectal Surgery: A Promising Tool or a Passing Fad?”  In _Annals of biomedical engineering_ 51.9, 2023, pp. 1892–1897  DOI: 10.1007/s10439-023-03232-y
  * [68] Roy H Perlis  “Research Letter: Application of GPT-4 to Select next-Step Antidepressant Treatment in Major Depression”  medRxiv, 2023  DOI: 10.1101/2023.04.14.23288595
  * [69] Ethan Waisberg et al.  “GPT-4: A New Era of Artificial Intelligence in Medicine”  In _Irish journal of medical science_ 192.6, 2023  DOI: 10.1007/s11845-023-03377-8
  * [70] Yi Zhong et al.  “The Artificial Intelligence Large Language Models and Neuropsychiatry Practice and Research Ethic”  In _Asian journal of psychiatry_ 84, 2023, pp. 103577  DOI: 10.1016/j.ajp.2023.103577
  * [71] Ammar Abdulrahman Jairoun et al.  “ChatGPT: Threat or Boon to the Future of Pharmacy Practice?”  In _Research in Social & Administrative Pharmacy_ 19.7, 2023, pp. 975–976  DOI: 10.1016/j.sapharm.2023.03.012
  * [72] Joseph A. Kavian, Hannah L. Wilkey, Parth A. Patel and Carter J. Boyd  “Harvesting the Power of Artificial Intelligence for Surgery: Uses, Implications, and Ethical Considerations”  In _The American surgeon_ 89.12, 2023, pp. 5102–5104  DOI: 10.1177/00031348231175454
  * [73] A.J. Page, N.M. Tumelty and S.K. Sheppard  “Navigating the AI Frontier: Ethical Considerations and Best Practices in Microbial Genomics Research”  In _Microbial genomics_ 9.6 United Kingdom: NLM (Medline), 2023  DOI: 10.1099/mgen.0.001049
  * [74] Om P. Singh  “Artificial Intelligence in the Era of ChatGPT - Opportunities and Challenges in Mental Health Care.”  In _Indian Journal of Psychiatry_ 65.3, 2023, pp. 297–298  DOI: 10.4103/indianjpsychiatry.indianjpsychiatry_112_23
  * [75] Sandra P. Thomas  “Grappling with the Implications of ChatGPT for Researchers, Clinicians, and Educators.”  In _Issues in Mental Health Nursing_ 44.3, 2023, pp. 141–142  DOI: 10.1080/01612840.2023.2180982
  * [76] Patricia S. Yoder-Wise  “This Is a Real Editorial or Is It?”  In _Journal of Continuing Education in Nursing_ 54.3, 2023, pp. 99–100  DOI: 10.3928/00220124-20230214-01
  * [77] M. Sallam  “The Utility of ChatGPT as an Example of Large Language Models in Healthcare Education, Research and Practice: Systematic Review on the Future Perspectives and Potential Limitations”  United States: medRxiv, 2023  DOI: 10.1101/2023.02.19.23286155
  * [78] Omar Temsah et al.  “Overview of Early ChatGPT’s Presence in Medical Literature: Insights from a Hybrid Literature Review by ChatGPT and Human Experts”  In _Curēus_ 15.4, 2023, pp. e37281  DOI: 10.7759/cureus.37281
  * [79] Qianqian Xie and Fei Wang  “Faithful AI in Healthcare and Medicine”  medRxiv, 2023  DOI: 10.1101/2023.04.18.23288752
  * [80] A.-F. Abdulai and L. Hung  “Will ChatGPT Undermine Ethical Values in Nursing Education, Research, and Practice?”  In _Nursing inquiry_ 30.3 Australia: NLM (Medline), 2023, pp. e12556  DOI: 10.1111/nin.12556
  * [81] A.L. Ferreira and J.B. Lipoff  “The Complex Ethics of Applying ChatGPT and Language Model Artificial Intelligence in Dermatology”  In _Journal of the American Academy of Dermatology_ 89.4 United States: NLM (Medline), 2023, pp. e157–e158  DOI: 10.1016/j.jaad.2023.05.054
  * [82] Han Guo, Mingjia Huo, Ruiyi Zhang and Pengtao Xie  “ProteinChat: Towards Achieving ChatGPT-Like Functionalities on Protein 3D Structures”  Techrxiv, 2023  DOI: 10.36227/techrxiv.23120606.v1
  * [83] Geoffrey M. Currie  “Academic Integrity and Artificial Intelligence: Is ChatGPT Hype, Hero or Heresy?”  In _Seminars in nuclear medicine_ 53.5, 2023, pp. 719–730  DOI: 10.1053/j.semnuclmed.2023.04.008
  * [84] Florin Eggmann and Markus B Blatz  “ChatGPT: Chances and Challenges for Dentistry.”  In _Compendium of Continuing Education in Dentistry_ 44.4, 2023, pp. 220–224
  * [85] Michael Gottlieb, Jeffrey A. Kline, Alexander J. Schneider and Wendy C. Coates  “ChatGPT and Conversational Artificial Intelligence: Friend, Foe, or Future of Research?”  In _The American journal of emergency medicine_ 70, 2023, pp. 81–83  DOI: 10.1016/j.ajem.2023.05.018
  * [86] S. Harrer  “Attention Is Not All You Need: The Complicated Case of Ethically Using Large Language Models in Healthcare and Medicine”  In _eBioMedicine_ 90.(Harrer) Digital Health Cooperative Research Centre, Melbourne, Australia, 2023, pp. 104512–104512  DOI: 10.1016/j.ebiom.2023.104512
  * [87] Centaine L. Snoswell, Nazanin Falconer and Aaron J Snoswell  “Pharmacist vs Machine: Pharmacy Services in the Age of Large Language Models.”  In _Research in Social & Administrative Pharmacy_ 19.6, 2023, pp. 843–844  DOI: 10.1016/j.sapharm.2023.03.006
  * [88] Emre Kazim and Adriano Soares Koshiyama  “A High-Level Overview of AI Ethics”  In _Patterns_ 2.9, 2021, pp. 100314  DOI: 10.1016/j.patter.2021.100314
  * [89] Thilo Hagendorff  “The Ethics of AI Ethics: An Evaluation of Guidelines”  In _Minds and Machines_ 30.1, 2020, pp. 99–120  DOI: 10.1007/s11023-020-09517-8
  * [90] Anna Jobin, Marcello Ienca and Effy Vayena  “The Global Landscape of AI Ethics Guidelines”  In _Nature Machine Intelligence_ 1.9 Nature Publishing Group, 2019, pp. 389–399  DOI: 10.1038/s42256-019-0088-2
  * [91] Jessica Morley et al.  “The Ethics of AI in Health Care: A Mapping Review”  In _Social Science & Medicine_ 260, 2020, pp. 113172  DOI: 10.1016/j.socscimed.2020.113172
  * [92] Ziwei Xu, Sanjay Jain and Mohan Kankanhalli  “Hallucination Is Inevitable: An Innate Limitation of Large Language Models”  arXiv, 2024  DOI: 10.48550/arXiv.2401.11817
  * [93] Thomas Grote and Philipp Berens  “On the Ethics of Algorithmic Decision-Making in Healthcare”  In _Journal of medical ethics_ 46.3, 2020, pp. 205–211  DOI: 10.1136/medethics-2019-105586
  * [94] Thomas Grote  “Trustworthy Medical AI Systems Need to Know When They Don’t Know”  In _Journal of Medical Ethics_ 47.5 Institute of Medical Ethics, 2021, pp. 337–338  DOI: 10.1136/medethics-2021-107463
  * [95] Dina Babushkina and Athanasios Votsis  “Epistemo-Ethical Constraints on AI-human Decision Making for Diagnostic Purposes”  In _Ethics and Information Technology_ 24.2, 2022, pp. 22  DOI: 10.1007/s10676-022-09629-y
  * [96] Ibo van de Poel  “Why New Technologies Should Be Conceived as Social Experiments”  In _Ethics, Policy & Environment_ 16.3 Routledge, 2013, pp. 352–355  DOI: 10.1080/21550085.2013.844575
  * [97] Ibo van de Poel  “An Ethical Framework for Evaluating Experimental Technology”  In _Science and Engineering Ethics_ 22.3, 2016, pp. 667–686  DOI: 10.1007/s11948-015-9724-3

##  Supplement A Key concepts for the search

  1. 1.

LLM are machines which use computational methods, partially under human
supervision, to extract statistical relationships from large corpora of data
such as text documents, databases and websites. This allows them to predict a
number of tokens or words given a certain input and thereby to generate text
that follows the learned statistical conventions to an extent that most humans
find to mimick human language use.

  2. 2.

Healthcare is the practice of restoring, maintaining or improving a physical
or mental well-being, including prevention, diagnosis, treatment or cure of
disease or injury. Healthcare is often performed in structured settings
involving the help of trained health professionals such as medical doctors,
nurses etc.

  3. 3.

Ethical issues can be defined as a state of affairs in which moral
implications of a given situation cannot be determined without much
reservation, disagreement with regard to the right course of action exists, or
conflicting moral obligations presenting themselves. This may include a
variety of different situations, such as occurrence of unclear or undetermined
benefits, chances, risks or harms, unclear, undetermined or conflicting views
about addressees of moral complaints as well as situations in which applicable
moral principles obviously should have been considered but weren’t. We
determined that that an ethical issue exists whenever it is understood as
such.

##  Supplement B Overview on data extraction fields

  1. 1.

Bibliographic data [author; title; journal/server; funding; coi]

  2. 2.

Article type [orientation;type]

  3. 3.

Aims of publication

  4. 4.

Field of application [name of device; device description; anticipated
development; technical intentionality/capabilities; technical limitations;
medical domain; applications and outcomes; benchmarks for comparison;

  5. 5.

Value dimension [value; normative background; ethical methods]

  6. 6.

Findings [user-interactions; concerns; perception of hype/fears; systemic
perspectives]

  7. 7.

Recommendations

  8. 8.

Implications for practice

  9. 9.

Future research directions