# Artificial General Intelligence
for Medical Imaging Analysis

Xiang Li, Lu Zhang, Zihao Wu, Zhengliang Liu, Lin Zhao, Yixuan Yuan, Jun Liu,
Gang Li, Dajiang Zhu, Pingkun Yan, Quanzheng Li, Wei Liu, Tianming Liu , and
Dinggang Shen (Corresponding authors: Xiang Li, Tianming Liu, Dinggang
Shen)Xiang Li and Quanzheng Li are with the Department of Radiology,
Massachusetts General Hospital and Harvard Medical School, Boston 02115, USA.
(e-mail: {xli60,li.quanzheng}@mgh.harvard.edu).Lu Zhang and Dajiang Zhu are
with the Department of Computer Science and Engineering, The University of
Texas at Arlington, Arlington 76019, USA. (e-mail<EMAIL_ADDRESS>and
dajiang.zhu@uta.edu).Zihao Wu, Zhengliang Liu, Lin Zhao and Tianming Liu are
with the School of Computing, The University of Georgia, Athens 30602, USA.
(e-mail: {zihao.wu1,zl18864,lin.zhao,tliu}@uga.edu).Yixuan Yuan is with the
Department of Electronic Engineering, Chinese University of Hong Kong, Hong
Kong. (e-mail: yxyuan@ee.cuhk.edu.hk).Jun Liu is with the Department of
Radiology, Second Xiangya Hospital, Changsha 410011, China. (e-mail:
junliu123@csu.edu.cn).Gang Li is with the Department of Radiology at the
University of North Carolina at Chapel Hill, Chapel Hill 27599, USA. (e-mail:
gang_li@med.unc.edu).Pingkun Yan is with the Department of Biomedical
Engineering at Rensselaer Polytechnic Institute, Troy, New York 12180, USA.
(e-mail: yanp2@rpi.edu).Wei Liu is with the Department of Radiation Oncology,
Mayo Clinic, Scottsdale 85259, USA. (e-mail: liu.wei@mayo.edu).Dinggang Shen
is with the School of Biomedical Engineering, ShanghaiTech University,
Shanghai 201210, China; Shanghai United Imaging Intelligence Co., Ltd.,
Shanghai 200230, China; Shanghai Clinical Research and Trial Center, Shanghai,
201210, China. (e-mail: Dinggang.Shen@gmail.com).

###### Abstract

In this review, we explore the potential applications of Artificial General
Intelligence (AGI) models in healthcare, focusing on foundational Large
Language Models (LLMs), Large Vision Models, and Large Multimodal Models. We
emphasize the importance of integrating clinical expertise, domain knowledge,
and multimodal capabilities into AGI models. In addition, we lay out key
roadmaps that guide the development and deployment of healthcare AGI models.
Throughout the review, we provide critical perspectives on the potential
challenges and pitfalls associated with deploying large-scale AGI models in
the medical field. This comprehensive review aims to offer insights into the
future implications of AGI in medical imaging, healthcare and beyond.

###### Index Terms:

ChatGPT, GPT-4, LLM, AGI, Medical Imaging

##  I Introduction

AGI models such as LLMs have demonstrated success [1] in general domains.
However, applying them directly to healthcare can pose significant challenges,
potentially leading to decreased performance or even making it impossible [2].
These challenges primarily stem from the medical domain’s unique
characteristics, such as the specialized nature of clinical text and medical
imaging, and the expertise needed for accurate interpretation. For example,
medical imaging data, which encompasses images obtained from various
modalities [3, 4] such as X-rays, computed tomography (CT), magnetic resonance
imaging (MRI), ultrasound, microscope, and more, is primarily acquired for
diagnostic or therapeutic purposes, aiming to provide vital insights into a
patient’s health condition [5, 6]. Interpreting these images demands
specialized knowledge and expertise in anatomy, pathology, and radiology. In
contrast, natural images typically consist of everyday objects that can be
accurately recognized by most individuals based on common sense. This
distinction presents a notable challenge when it comes to annotating a
substantial volume of high-quality medical imaging data, as it necessitates
the involvement of numerous experts, which is often impractical or unfeasible.
Consequently, the medical domain faces a scarcity of extensive training data
required for directly training large models from scratch, in contrast to the
abundant availability of such data in the general domain. Moreover, due to the
heterogeneity between medical data and general data, it is not optimal to
directly apply LLMs (e.g., GPT-4 [7]) and large multimodal models (e.g.,
Kosmos-1 [8]) trained on general data to the medical domain. In addition,
clinical data usually contains sensitive patient information. Strict
regulations, such as HIPAA (Health Insurance Portability and Accountability
Act) [9, 10] in the United States, govern the storage, transmission, and
handling of patient data. As a result, applying publicly available
foundational models, such as GPT-4, to medical data becomes exceedingly
difficult. In general, adapting AGI models to the medical domain requires
careful consideration of the unique challenges posed by medical data.

To unlock the potential benefits of AGI in improving healthcare outcomes,
researchers in the medical field are dedicated to developing effective
approaches to tailor these models to the specific requirements of the medical
domain. These efforts primarily concentrate on three key areas: data,
knowledge, and models. In the realm of data, the primary aim is to address two
key challenges: data scarcity and the presence of sensitive information. For
example, one publication proposed the ChatAug method [11], which is built on
ChatGPT and can rephrase each sentence in the training samples into multiple
conceptually similar but semantically different samples. This augmentation
technique is particularly valuable in clinical report studies, where different
clinicians may provide personalized reports for the same clinical symptom. By
training LLMs with the augmented data samples generated by ChatAug, the models
can generate diagnostic reports in diverse styles and accurately capture the
typical symptoms of diseases without being influenced by specific expression
styles. In the medical imaging domain, a recent study utilized diffusion
models [12] to produce high quality synthetic data for skin disease
classification that incurs no classification degradation on test data. In
another interesting work [13], a novel framework was proposed to leverage
GPT-4 for the de-identification of clinical notes. This framework can
automatically identify and remove Protected Health Information (PHI) from
unstructured medical texts while preserving the original structure and meaning
of the texts. Compared to the traditional de-identification methods, this
approach demonstrates higher accuracy and remarkable reliability in masking
private information, showcasing the immense potential of LLMs in handling
medical text data.

To better guide AGI models, researchers try to develop effective strategies to
incorporate expert knowledge and domain knowledge in the learning process.
Several notable studies in this topic deserve attention [14, 15, 16]. In
medical imaging, a recent study demonstrated that simple MLP-based SAM-
adapters could be used to inject task and domain-specific knowledge into the
Segment Anything Model (SAM) [16]. Another work in NLP [14] introduces a
dynamic prompt generation approach that iteratively improves the quality of
prompts during the training process by incorporating domain knowledge and
expert guidance. This iterative approach allows the prompts to evolve and
adapt, resulting in improved performance of LLMs in medical tasks. By
leveraging domain-specific knowledge and expert input, the generated prompts
can effectively guide the models towards more accurate and contextually
relevant outputs. Another noteworthy work [15] proposes the use of multimodal
prompts, which effectively integrate multi-modal information [17] for
computer-aided diagnosis (CAD). As we have previously discussed, the medical
domain is characterized by its multimodality [3, 4], where the diagnosis of
diseases often requires the integration of information from various sources,
including different medical imaging modalities, clinical context, patient
history, and other relevant information. By incorporating multimodal prompts,
models can better leverage the diverse and complementary information
available, leading to more comprehensive and accurate analyses and
predictions.

Alongside the research on data and knowledge, there have been efforts to
explore and examine the design principles of models in both highly specialized
domain (e.g., radiology and radiation oncology) and the general domain. One
recent study [18] discussed the question of whether model design should be
generic or domain-specific. This work specifically focused on radiology
language inference, a highly specialized task, and provided a detailed
comparison between generic models and local domain-specific models in terms of
performance and cost (as large models can be expensive). The study yielded
insightful results that shed light on the advantages and disadvantages of each
approach. Another review paper [19] took a more fundamental discussion and
explored the relationship between AGI and the human intelligence. The authors
discussed the essential characteristics that an AGI system should possess
based on the fundamental features of human intelligence.

In addition to the technical discussions, another important area of research
focuses on the practical applications of AGI models [20, 21]. Maximizing the
potential of these models in various fields has become a hot topic of
investigation. In the medical domain, aside from conventional tasks such as
medical image segmentation, disease diagnosis, and medical report generation,
researchers are exploring the possibility of utilizing these large models for
more meaningful applications. For example, LLMs can be used to help educate
the next generation of medical professionals in radiation oncology [22]. The
goal is to leverage these models to alleviate the scarcity of medical
resources and provide more personalized and human-centric healthcare. By
harnessing the capabilities of large language models, there is great potential
to revolutionize the healthcare field.

##  II LLMs and AGI: characteristics and technological foundations

To help readers understand the technical contents of this paper, we first
provide a summary of the enabling technologies of LLMs. In a previous study
[21], a comprehensive summary was provided on the development and
categorization of language models using an evolutionary tree. In this review
paper, we will focus on analyzing the key characteristics and essential
technological foundations of LLMs and AGI system compared to traditional
language models.

###  II-A Key characteristics of LLMs/AGI

Current LLMs and AGI possess several key characteristics that distinguish them
from traditional deep models. These characteristics contribute to their
capabilities and potential impact.

  * •

Emergent Ability: Emergence, as described by Nobel Physics prize laureate, Dr.
Philip Anderson [23], occurs when quantitative changes in a system lead to
qualitative changes in its behavior. Recently, it has been observed that LLMs
exhibit a multitude of “emergent” abilities that larger models can accomplish
but smaller models cannot [24, 25]. Interestingly, many of these abilities
extend beyond text analysis, encompassing tasks such as performing
mathematical operations, generating executable code, and even decoding movies
using emojis. Analyses indicate that certain tasks and models go beyond a
complexity threshold where the model’s functionality starts to increase
exponentially. Presently, researchers are engaged in a race to not only
identify additional emergent abilities but also to understand the underlying
reasons and mechanisms behind their occurrence. Their aim is to predict and
comprehend this unpredictability. Gaining a comprehensive understanding of
emergence holds the potential to address fundamental questions in AGI
development and evolution. It can shed light on whether complex models are
genuinely engaging in novel processes or simply excelling in statistical
analyses. Furthermore, this understanding can assist researchers in harnessing
the potential benefits of emergent abilities while mitigating associated
risks.

Figure 1: Illustration of a) in-context learning and b) prompt engineering.

  * •

Multimodal Learning Ability: The multimodal learning ability of AGI models
refers to their ability to comprehend and generate text in conjunction with
other modalities, such as images, audio, or video. This attribute is crucial
towards developing advanced AGI systems that surpass human intelligence, as it
enables a more holistic and comprehensive understanding of information across
different modalities. Notably, several models have made significant
contributions to multimodal learning. CLIP [26], DALL-E [27], GLIDE [28],
VisualGPT [29] are among the prominent models recognized for their
contributions to image-to-text and text-to-image generation tasks. These
models have demonstrated remarkable capabilities in bridging the gap between
images and textual descriptions. Additionally, METER [30] stands out as a
representative model for Visual Question Answering (VQA), a task involving
answering text-based questions based on visual content. More recently, models
such as PaLM-E [31] and Kosmos-1 [8] have pushed multimodal capabilities to
new highs.For instance, PaLM-E is capable of guiding robots and demonstrating
generalist abilities, and BiomedGPT [32] has demonstrated good
generalizability and robustness for a series of down-stream tasks as a pre-
trained model for multi-modal medical data. These models represent significant
advancements in multimodal learning, showcasing the potential of foundational
models to comprehend and generate text in conjunction with other modalities.
The integration of multimodal capabilities contributes to the development of
more comprehensive and versatile AI systems that can process and understand
information across different modalities, bringing us closer to the realization
of advanced AGI.

###  II-B Technological foundations of LLMs/AGI

Current LLMs and AGI rely on several technological foundations that provide
the necessary tools and techniques to enhance their capabilities and drive
their advancements. We will introduce these key technological foundations in
this section.

  * •

Transformer Architecture: The Transformer model, introduced in 2017 by Vaswani
et al. [33], has revolutionized the field of Natural Language Processing (NLP)
and has become the standard neural network architecture of LLMs [34, 35, 36,
37, 38, 39]. The significance of the Transformer architecture lies in its
ability to address the limitations of prior models, such as Recurrent Neural
Networks (RNNs) [40, 41, 42, 6, 3], in handling variable-length sequences and
context awareness. Unlike RNNs, the Transformer model does not rely on
sequential processing, making it more efficient and suitable for parallel
computation. This parallelism enables the Transformer to process and
understand large amounts of text simultaneously, significantly improving its
training and inference speed. At the core of the Transformer is the multi-head
attention mechanism, which allows the model to assign different weights to
tokens based on their relevance. This attention mechanism enables the
Transformer to capture long-term dependencies in a more effective manner,
enhancing its performance across a wide range of NLP tasks. By attending to
different parts of the input sequence, the Transformer can effectively
understand the relationships between tokens and capture contextual
information.

Additionally, the parallelizable nature of the Transformer architecture makes
it highly adaptable and scalable. This flexibility enables large-scale pre-
training, where the model is trained on vast amounts of data to learn general
language patterns and representations. These pre-trained models can then be
fine-tuned for specific downstream tasks, such as text generation or language
translation. This approach allows the Transformer to leverage its learned
knowledge and generalize well to various domains and tasks.

Although previous models can already be very deep, the Transformer’s design
allows models to take precedence over inductive biases by fully benefiting
from the large size of the network to scale up the training. In addition,
unlike some traditional models that heavily rely on hand-crafted rules or
prior linguistic knowledge, the Transformer learns directly from the data,
making it more flexible and capable of capturing complex patterns and nuances
[43].

  * •

In-context Learning: In-context learning [44] is a powerful technique employed
by language models to learn and make predictions based on a limited context
provided during inference or interaction (see Figure 1 part (a) for an
example). Unlike traditional approaches that heavily rely on extensive pre-
training or fine-tuning using large labeled datasets, in-context learning
allows models to leverage specific examples or instructions within the input
prompt to guide their behavior and generate contextually relevant outputs. The
key idea of in-context learning is to learn from analogy. To facilitate this
process, in-context learning starts by constructing a demonstration context
using a few examples. These examples showcase the desired input-output
behavior that the model should learn. Subsequently, the demonstration context
is combined with a query question, resulting in a prompt that contains both
the query and the demonstration context. This prompt is then fed into the
language model for prediction. In contrast with supervised learning, where
model parameters are updated through backpropagation during a training stage,
in-context learning does not involve explicit parameter updates. Instead, it
relies on pretrained language models that directly perform predictions based
on the provided examples and prompts. The model is expected to discern the
underlying patterns within the demonstration context and generate accurate
predictions accordingly. With the remarkable scaling of model size and the
availability of large-scale corpora, LLMs have demonstrated their proficiency
in in-context learning. These models possess the capability to extract
meaningful information from a few examples within the given context. By
leveraging the extensive knowledge encoded in their pretrained weights, LLMs
excel at learning from limited context and making informed predictions.

  * •

Prompt Engineering: Traditionally, the process of collecting and labeling
responses for training or fine-tuning NLP models has been both time-consuming
and costly. However, prompt engineering [45, 46, 47, 48, 49] offers a more
efficient alternative. A prompt, in this context, refers to a set of
instructions that customize the behavior and guide the subsequent interactions
and outputs of an LLM (see Figure 1 (b) for a taxonomy of prompt design).
Recent studies have demonstrated the effectiveness of prompt engineering in
adapting large-scale pre-trained language models to specific downstream tasks,
eliminating the need for extensive fine-tuning. Prompt engineering allows for
customization and tailoring of LLM capabilities. By designing appropriate
prompts, developers can shape the LLM’s behavior to align with specific
objectives, tasks, or user requirements. This level of customization enhances
the practical usability and applicability of LLMs across various domains and
use cases. In general, prompt engineering plays a vital role in current LLMs
and provides a promising way for developing advanced AGI system. By designing
appropriate prompts, developers can leverage the power of pre-trained models
and optimize their performance, making LLMs and AGI systems more effective,
efficient, and adaptable in the specific domain.

  * •

RLHF: The alignment of an AGI system is a critical process that ensures its
behavior is in line with desired principles, values, and objectives. This
alignment is essential to ensure that AGI systems operate ethically,
responsibly, and in accordance with human intentions and societal norms. In
current LLMs, a novel strategy known as Reinforcement Learning from Human
Feedback (RLHF) has been adopted to achieve alignment [50]. RLHF involves
training LLMs by incorporating human feedback to guide their learning process.
RLHF has been employed in LLMs such as InstructGPT [50] and ChatGPT, to train
these models by incorporating feedback from human supervisors. By leveraging
human knowledge and preferences, RLHF ensures that the learned behaviors of
LLMs align with human values and requirements. In traditional reinforcement
learning, an agent learns through trial and error, receiving rewards or
penalties based on its actions [51, 52]. RLHF takes this process further by
actively involving humans in the training loop. Human supervisors play a
pivotal role by providing feedback to the learning agent, shaping its
behavior, and guiding it towards desired outcomes. This approach is
particularly valuable in domains where human expertise and intuition are
essential, such as complex decision-making tasks or domains with limited or
imperfect simulations. RLHF is not only pertinent to LLMs but also an
important strategy in the development of AGI. It contributes to achieving
alignment between AGI systems and human values. By incorporating human
feedback, RLHF helps ensure that AGI systems learn and operate in a manner
that aligns with human intentions and desires.

##  III Large Language Models for Medical Imaging

Large language models are at the forefront of AGI development. The advancement
of LLMs is bringing a revolution to medical imaging. In this section, we
highlight key points of consideration that facilitate the integration and
adaptation of LLMs in healthcare. Medical practitioners and healthcare
organizations are now envisioning a future where AI-powered systems assist in
disease diagnosis, patient outcome prediction, medical education, and
streamlining administrative work. Simultaneously, these models could provide a
unified knowledge space integrating various data modalities, promoting a
comprehensive approach to patient care. While these possibilities paint an
exciting picture for the future of healthcare, it’s also imperative to address
the associated challenges, such as data privacy, regulations, and imbalances,
along with the technical obstacles in prompt crafting and healthcare data
preparation. This section will delve into the detailed potential applications
and the confronting challenges of LLMs in healthcare.

###  III-A Roadmaps

  * •

Expert-in-the-loop: The incorporation of expert knowledge is vital towards
harnessing the full potential of artificial intelligence (AI) within
specialized medical fields. While LLMs like ChatGPT and its successors have
demonstrated proficiency in general medical knowledge [53, 1], they might
encounter challenges when performing domain-specific tasks due to a deficiency
in specialized knowledge [22, 20] and generate erroneous content.

A promising approach to overcome these obstacles is the adoption of
Reinforcement Learning with Expert Feedback (RLEF). LLMs like ChatGPT are
trained with RLHF [50], where generated responses during training are ranked
to enhance and refine the model’s outputs. Yet, the application of this
approach to specific medical specialties presents unique complexities due to
the nuanced nature of these disciplines.

For example, in cardiology, an AI model might be tasked with interpreting
electrocardiogram (ECG) readings [54]. While the model could identify basic
patterns, accurately diagnosing complex cardiac conditions like myocardial
infarctions or arrhythmias necessitates the expertise of a cardiologist. In
psychiatry, a model might struggle to differentiate between overlapping mental
health conditions based on patient-reported symptoms, demanding the nuanced
understanding of a psychiatrist [55]. In radiology, distinguishing between a
benign lesion and a malignant tumor on a CT scan requires the specialized
knowledge that a radiologist provides [56]. Likewise, in radiation oncology,
determining the precise dosage and target area for radiation therapy calls for
insights from radiation oncologists [57, 58].

By integrating RLEF within the reinforcement learning loop, AGI models can
mirror the nuances of diverse medical specialties more accurately, leading to
clinically relevant and precise outputs. With sufficient expert feedback,
ChatGPT could potentially interpret ECG readings, help in distinguishing
between complex psychiatric conditions, accurately identify malignancies in
radiological scans, and suggest more nuanced radiation therapy plans.

Further refinement of these LLMs requires a strategy that incorporates
iterative, learning-based approaches deeply rooted in various medical
specialties. This strategy entails incorporating more ”human-in-the-loop” [59]
interventions during training, acknowledging their significant contribution in
enhancing the model’s precision in clinical scenarios.

Indeed, employing semi-supervised [60] and active learning methodologies [61]
within a “human in the loop learning” paradigm [59] holds promise for
enhancing the model’s proficiency. Despite data scarcity in specialized
fields, strategically training LLMs within these domains is crucial for
progress. This approach not only allows the models to benefit from a diverse
set of clinical scenarios but also helps them generalize this learning across
various medical specialties.

Recognizing the potential technical limitations among medical practitioners in
effectively utilizing ChatGPT, it becomes evident that interdisciplinary
collaboration is of utmost importance. To unlock the full potential of AI in
specialized medical fields, a concerted, multidisciplinary effort is needed.
This collaborative approach not only fosters innovation but also ensures the
development of AI tools that are reliable, accurate, and beneficial for real-
world clinical applications across a broad range of medical specialties.

  * •

Tailoring to the Medical Domain: It is important to tailor language models to
specific domains [62, 63, 64]. There’s a strong need to consider domain-
specific data [65, 66] and adopt more refined ways of prompting, including
learning how to ask questions more effectively in a clinical context. In
addition, proper domain adaptation involves learning from multiple data
modalities [67], a crucial aspect of applying LLMs to medical imaging or
medicine in general.

The current reward modeling and simple ranking system used by ChatGPT is
considered somewhat rudimentary, especially if the model is to be applied
across multiple modalities or specialized fields. Hence, future work should
progress towards a more complex and multi-dimensional ranking approach,
enabling a more nuanced understanding and interpretation of various medical
scenarios.

A major challenge in this domain is the difficulty in generating a large
medical data database. Hospitals are often reluctant to share their data due
to privacy and legal concerns. Therefore, finding a legal and ethical way to
access and utilize such data is of paramount importance for training LLMs like
ChatGPT in the future. This would allow these models to better understand and
adapt to the intricacies of specific medical fields, thereby improving their
performance and applicability in these areas.

  * •

Prompt tuning: Prompt tuning is an effective and low-cost approach to adapt
Large Language Models (LLMs) to specific domains [68], data, and tasks [45]
without updating the massive model parameters of large foundational LLMs.
Prompts with task instructions and demonstrations were first utilized in GPT-3
[36] for adapting a pretrained model to various downstream tasks without
modifying any pretrained parameters. Prompt tuning further refines the
pretrained model by adding prompt tokens and solely training the prompt
embedding, without updating the original parameters [69]. In language models,
prompt tuning exhibits superior performance, even surpassing full fine-tuning
[70, 68]. Recently, multimodal prompts have been also introduced into visual
models to further enhance their performance on downstream tasks [15].

  * •

Multi-modal Modeling: The importance of multi-modal learning in the field of
artificial intelligence, especially in the context of LLMs like ChatGPT, has
been highlighted by the emergence of models such as Flamingo [71] and PaLM-E
[31]. These innovative models leverage visual transformers (ViT) [72] and its
variations to encode image information, seamlessly integrating it with
linguistic data within the LLM. This multi-modal approach allows for a
simultaneous interpretation of textual and visual information, thereby
enhancing the model’s comprehension and response accuracy. As the evolution of
LLMs continues, it is imperative to focus on strategies that can improve the
integration of diverse data types. Specifically, refining methods for
transmitting image information to these models could potentially augment their
capacity to handle tasks requiring a simultaneous understanding of visual and
textual inputs, further extending the boundaries of their applicability in
complex, multi-faceted domains such as medicine and healthcare.

  * •

Integrating Medical Informatics: The successful integration of LLMs, such as
ChatGPT, into clinical workflows is fundamentally dependent on the robustness
of medical informatics. The swift, efficient, and precise collection and
integration of data is integral for effectively deploying or updating these
models in healthcare environments. As a result, there is a burgeoning need to
investigate approaches for sourcing textual or other pertinent data from
complex [73] hospital information systems. This access to data is imperative
for incorporating LLMs or multi-modal models into areas such as clinical
diagnostics, treatment, radiation oncology, and radiology.

In this context, informatics plays a vital role, serving as a crucial bridge
connecting vast, intricate data sets with AGI applications in healthcare. The
discipline’s function in enabling data accessibility, deciphering it, and
applying it subsequently is invaluable in the forward march of AI in clinical
environments. This emphasizes the call for scientific exploration to pivot
towards the further refinement and enhancement of medical informatics, thereby
nurturing a more potent relationship between this field and AGI. This
harmonious integration promises not only to boost the operational efficiency
of healthcare systems but also to augment patient outcomes via more precise
and prompt diagnoses and treatments.

  * •

Knowledge Graph: Knowledge graphs can bolster the utility of LLMs like ChatGPT
in the healthcare sector. A medical knowledge graph, with its expansive reach
beyond simple language generation, can effectively enhance the input and
prompts to LLMs [74] and validate the outputs [75] of LLMs. The deployment of
such a tool could bring significant advancements to the field by offering a
structured and contextually meaningful representation of data.

Building a comprehensive knowledge graph is a complex task [76] due to the
sophistication required in identifying entities and extracting relationships,
which are integral components of such a graph. The current research frontier
in this context is focused on devising strategies that facilitate automatic
extraction of knowledge from large data corpora. The objective is to create a
foundational knowledge base that can be readily deployed in clinical practice.

By automating the process of knowledge extraction, the speed of development
and implementation of knowledge graphs in healthcare can be significantly
accelerated. This would consequently enhance the performance, precision, and
efficiency of AI-driven tools in healthcare, thereby promoting improved
patient outcomes and operational excellence in clinical settings.

  * •

Collaboration: Collaborative endeavors are pivotal in the deployment and
optimization of LLMs in healthcare, particularly in addressing complex issues
such as data diversity and data privacy [20, 13]. Effective modeling of
medical data requires careful consideration of patient demographics, socio-
economic diversity, and cultural factors [77, 78]. It is necessary to develop
models that are trained and adapted to data representative of diverse
hospitals and institutions across a nation or region. This cannot be fulfilled
without multidisciplinary collaboration and multi-institutional cooperation.

Patient privacy is also important during collaboration. One potential approach
to uphold patient confidentiality is the deployment of localized LLMs within
individual hospitals, thus confining sensitive patient information within the
premises. An alternative method to balance the requirements of data sharing
and privacy protection involves sharing encoded data rather than the original
data. Given the prolific use and development of encoding embeddings, such a
strategy could be feasible. For instance, techniques like privacy-aware word
embeddings [79] can transform raw data into a less sensitive format. This
transformed data, devoid of personally identifiable information, can then be
shared more freely for model training purposes. The potential also exists for
shared encoders that can process both text and image data, ensuring consistent
encoding across diverse data types. The exploration of such strategies
underlines the ongoing efforts to make effective use of data for model
improvement while upholding the vital principle of patient privacy. Further
examination could illuminate whether sharing encoded data could be more
effective and secure than sharing the original, potentially sensitive data.
Some other methodologies such as federated learning [80] and multi-institution
blockchains [81] are also potential solutions.

###  III-B Current and potential applications

  * •

Disease Diagnosis and Patient Outcome Prediction: LLMs such as ChatGPT offer
promising capabilities in determining the cause of diseases and predicting
patient outcomes. By harnessing the power of LLMs, we can transform text data
into a suitable format for predicting outcomes and diagnosing patients
exhibiting similar symptoms. This concept can be exemplified by feeding case
studies of patients with identical symptoms to ChatGPT to predict potential
disease outcomes based on patterns from the provided data.

  * •

Application in Medical Education and Patient Consultation: LLMs could be an
instrumental tool in educating the next generation of healthcare practitioners
and even in patient consultations [82]. For instance, medical students could
interact with ChatGPT for scenario-based learning, and patients may use it as
an accessible source of information to better understand their medical
conditions.

  * •

Streamlining Clinical and Administrative Work: LLMs such as ChatGPT and GPT-4
could help physicians with writing clinical notes [1, 13] or producing
radiology reports [14]. In addition, LLMs can significantly reduce the burden
of clinical administrative tasks such as drafting administrative documents.
freeing up physicians’ time for more important stuffs in patient care.

  * •

Unified Knowledge Space: Expressing information in a unified space that
encompasses text, images, and genetic data can revolutionize fields such as
bioinformatics and genomics. By creating a unified knowledge graph, relevant
information from diverse sources can be mapped into a graphical space, which
could potentially change the field of bioinformatics, clinical informatics,
medical imaging, and genomics.

A consolidated sphere of knowledge can serve as a springboard for
breakthroughs. For example, future models can leverage this vast data
repository. They could comprehensively analyze a patient’s textual clinical
history, diagnostic imaging findings, laboratory test outcomes, and individual
genetic information. In doing so, they would be equipped to render more
insightful, well-rounded decisions, thereby improving patient outcomes.

  * •

Integrating text data and other modalities: The significance of integrating
different types of data into a unified space for efficient decoding and
analysis is pivotal across various medical domains. In cardiology, for
example, text-based patient reports, imaging data from ultrasound or MRIs,
audio data like heart sounds, and echocardiograms (ECG) signals could be
organized as multimodal inputs to LLMs. An advanced language model could then
analyze these multimodal inputs to provide more comprehensive and accurate
diagnosis of heart conditions, thus supporting informed clinical decision-
making. Similarly, in telemedicine and mental health, the amalgamation of
patient-reported textual data, audio-visual inputs, wearable device metrics,
biometric data, and even environmental sensor data is beneficial. LLMs such as
Kosmos-1 [8], with their capacity to process and make sense of such multimodal
inputs, can potentially revolutionize these fields by providing more accurate,
personalized, and nuanced health assessments and treatment suggestions.

###  III-C Challenges and pitfalls

  * •

Prompting Challenges: The construction of standardized and appropriate prompt
words stands as a significant hurdle for the optimal performance of LLMs like
ChatGPT [1, 83]. The effectiveness of these prompts directly influences the
quality of responses, thus crafting the right prompt is as pivotal as refining
the model itself.

  * •

Data Privacy and Regulations: Medical data utilization in AGI models brings
about unique challenges in terms of data privacy and compliance with
regulatory bodies [13]. Protecting personal health information and meeting the
stringent requirements of institution review boards (IRBs) and HIPAA
guidelines [84] is paramount in any healthcare-related AI deployment. A
potential solution could be the application of local LLMs within hospitals,
which may mitigate privacy concerns while allowing models to learn from a vast
array of clinical data.

  * •

Data Accessibility: Accessing medical data for AGI models raises not only
technical but also legal and ethical complexities [85]. Balancing the
necessity of access for model training and operation against the privacy
rights of patients and healthcare providers is a complex undertaking.

  * •

Preparation and Curation of Healthcare Data: The need for image and
corresponding text data in healthcare poses a significant challenge. The
availability of comprehensive datasets akin to the MIMIC series [86, 87] is
scarce but critical for the effective training and utilization of LLMs in
healthcare applications.

  * •

Deployment in Local Hospitals: The execution of LLMs locally in hospitals
necessitates substantial computational resources. This could prove to be a
limiting factor, particularly for smaller hospitals with limited access to
high-performance computational infrastructure.

  * •

Data Imbalance: Real-world clinical data often exhibits imbalance, with
disproportionate representation of certain conditions, demographics, or other
variables [66, 88]. This poses a challenge for certain models that rely on
balanced, representative data for robust and unbiased outputs.

##  IV Large Vision Models for Medical Imaging

In recent years, deep learning has made significant strides in the field of
medical imaging, revolutionizing the way medical images are analyzed and
interpreted. Various deep learning models, such as Convolutional neural
networks (CNNs) [89, 90] and Vision Transformer (ViT) [72], have shown
remarkable success in a wide range of tasks like medical image reconstruction,
segmentation, and classification. Many of these models have been deployed to
assist radiologists and clinicians in tasks such as identifying abnormalities,
localizing tumors, and quantifying disease progression [90]. Recently, the
emergence of large vision models, such as the notable Segment Anything (SAM)
model [91], may further propel the advancement in medical imaging field,
ultimately leading to improved patient outcomes and more efficient healthcare
practices. In this section, we will delve into the roadmaps for deploying and
adapting large vision models in the medical domain, explore the current and
potential applications, and discuss the challenges and pitfalls associated
with their integration and utilization of large vision models.

###  IV-A Roadmaps

  * •

Large-Scale Datasets: Large-scale, high-quality and diverse medical imaging
datasets play a crucial role in deploying and adapting large vision models in
the medical domain. Gathering and curating such datasets require
collaborations between researchers, medical professionals, and institutions.
These datasets need to cover a wide range of pathologies, modalities, and
patient populations to capture the full spectrum of medical conditions and
ensure the generalizability of the models. Additionally, data privacy and
security measures must be in place to protect patient information and comply
with ethical standards. Federated learning [92, 93] is a promising approach
for addressing privacy and data security concerns in the deployment of large
vision models in the medical imaging field. By leveraging federated learning,
large vision models can be trained using a vast amount of distributed medical
imaging data while preserving data privacy. This approach is particularly
beneficial in scenarios where data cannot be easily shared or centralized due
to legal, ethical, or logistical constraints. Federated learning also provides
an avenue for harmonizing diverse datasets from different institutions,
capturing the variability and heterogeneity in medical imaging. This can
improve the generalization capability of large vision models and their
performance across multiple healthcare settings, enhancing the model’s
robustness and adaptability. Approaches for ensuring fair and secure use of
data would also be vital for a sustainable and reliable ecosystem in medical
image data usage [94, 95].

  * •

Model Adaption: The adaptation of large vision models to the medical domain
involves fine-tuning or transfer learning techniques [96]. Pretrained models
on large-scale general image datasets, such as ImageNet [97], may serve as a
valuable starting point. However, to leverage their learned representations
effectively, these models need to be further trained on medical imaging data.
Fine-tuning requires careful optimization and regularization strategies to
adapt the models to specific medical imaging tasks while preventing
overfitting and preserving their generalization capabilities. Recently, adding
adapters [16] to large vision models has indeed gained popularity as a
flexible and efficient approach to model customization and transfer learning.
Adapters allow for the integration of task-specific information without
modifying the entire model architecture, enabling faster and more cost-
effective model development.

  * •

Multimodal imaging: Medical imaging often involves multiple modalities, such
as ultrasound, MRI, CT, and PET, etc., each providing unique information. The
adaption of large vision models should consider the combination and fusion of
information from these multiple modalities to extract complementary features
and enhance diagnostic accuracy. Exploring fusion techniques, such as late
fusion [98, 99], early fusion [98, 99], or cross-modal attention mechanisms
[66, 100], may be crucial to leverage multi-modality data effectively and
improve the performance of the adapted models.

  * •

Interpretability: Large models often operate as complex black boxes, making it
challenging to understand the reasoning behind their predictions [19, 101,
102]. To establish trust and facilitate clinical decision-making, efforts
should be underway to develop techniques for explaining the decision-making
process of these models. Interpretability methods such as attention maps
[103], saliency maps [104], and Grad-CAM [105] have been explored to provide
insights into which regions of an image contribute most to the model’s
decision, aiding in understanding and verifying the model’s outputs. However,
the interpretability of large vision models has not been fully explored, which
requires further efforts of the community.

  * •

Few-shot/Zero-shot Learning: Few-shot and zero-shot learning hold significant
potential in medical imaging field, where obtaining extensive annotated
datasets can be time-consuming, expensive and even infeasible [106, 66, 65].
LLMs, such as GPT-3 [36] and its successors [1], have demonstrated impressive
zero-shot learning capabilities, where they can generate meaningful responses
or perform tasks without explicit training on specific examples. Translating
this concept to the medical imaging domain, large vision models can
potentially [91] exhibit zero-shot learning abilities, allowing them to
recognize and analyze new medical conditions or imaging modalities for which
no labeled training data is available. This capability could enable them to
adapt to previously unseen diseases, imaging techniques, or even cross-modal
tasks. For example, a large vision model trained on a diverse range of medical
imaging data could potentially infer and interpret new types of images or
identify rare conditions with limited labeled data, based on the analogies and
knowledge learned from similar cases.

  * •

Scalability: Training and deploying large vision models demand significant
computational resources, including high-performance computing infrastructure
and efficient parallel processing capabilities. However, real-time
applications may require optimizations to meet the time constraints of
clinical settings. For example, real-time image analysis is required for rapid
decision-making in emergency situations or during surgical procedures. The
scalability of large vision models is crucial to ensure their practical
applicability and efficiency in these time-sensitive scenarios. To address the
scalability challenge, advancements in hardware acceleration, such as
specialized graphical processing units (GPUs) [107] or tensor processing units
(TPUs) [108], can significantly boost the computational efficiency of large
vision models. Parallel processing techniques [109], distributed computing
[110], model compression [111], and model distillation methods [112] can also
enhance scalability by optimizing memory utilization and reducing
computational overhead.

###  IV-B Current and Potential Applications

  * •

Segment Anything: Large vision models have demonstrated significant potential
in various medical imaging applications, enabling improved diagnostic
accuracy, treatment planning, and disease monitoring. One notable large vision
model is the Segment Anything (SAM) model [91], which was trained on SA-1B
dataset with over 1 billion masks on 11 millions images. SAM supports
promptable segmentation over various segmentation tasks and demonstrates
impressive and powerful zero-shot generalization ability. Recently, SAM has
attracted a lot of attention in the medical imaging field. A group of studies
explored SAM’s ability in medical image segmentation tasks [113, 114, 115,
116, 117, 118]. For example, He et at. evaluated the SAM’s accuracy on 12
medical image segmentation datasets, showing that SAM underperforms the state-
of-the-art methods in these datasets and suggesting the necessity to adapt SAM
for medical imaging application [113]. To improve the performance of SAM on
medical image segmentation, another groups of studies explored adapting SAM
for specific tasks [119, 16, 120, 121, 122]. Ma and Wang et al. utilized a
straightforward way by fine-tuning SAM to general medical image
segmentation[119]. And the results indicate a great improvement compared with
default SAM model. Wu et al. proposed Med SAM Adapter by adding adapter to
both image encoder and mask decoder with comparable and even superior
performances over state-of-the-art methods [120]. Zhang and Liu et al.
customized SAM for medical image segmentation by applying the low-rank-based
(LoRA) finetuning strategy to SAM [122]. SAM has also been employed to medical
image annotation. Liu et al. integrated SAM with 3D Slicer, an open-source
software for visualization, segmentation, and analysis of medical images, to
support interactive annotation [123]. Despite the aforementioned progresses,
SAM has great potential in enabling precise quantitative measurements,
volumetric analyses, and 3D reconstructions and supporting clinicians in
making informed decisions and providing personalized patient care.

  * •

AI-Generated Content (AIGC): Stable Diffusion (SD) and Generative Adversarial
Networks (GAN) AIGC aims to generate digital content based on human input
(e.g., instructions and exemplars) [124]. [125] has gained recognition for its
high-performance capabilities in generating detailed images by leveraging a
latent diffusion model conditioned on text. This model has exhibited
remarkable efficacy in various tasks, including image inpainting and super-
resolution. Additionally, the DALL·E 2 model introduces a groundbreaking
ability to generate synthetic images from textual descriptions [126]. In the
context of the medical domain, the SD and DALL·E 2 models hold immense
potential for addressing prevalent challenges. For example, medical images
often suffer from issues such as noise, artifacts, and low contrast, which can
significantly impact diagnostic accuracy. The SD model can be harnessed to
mitigate these problems by effectively removing noise and artifacts while
enhancing the overall image quality in medical scans. This enhancement
facilitates clearer visualization of anatomical structures and pathologies,
aiding healthcare professionals in accurate diagnosis and treatment planning.
DALL·E 2 can be utilized to generate synthetic medical images for training
purposes, allowing medical professionals to simulate rare or challenging
clinical scenarios and improve their diagnostic skills. Moreover, these
synthetic images can be employed to augment limited datasets, providing a
valuable resource for training large vision models in medical imaging.

  * •

LLM-enhanced Imaging: The GPT-4 model also holds great promise for medical
imaging applications. GPT-4 builds upon its predecessors’ language
understanding capabilities and can be adapted to interpret clinical notes,
radiology reports, and other textual data associated with medical images. This
integration of text understanding and image analysis enables context-aware
diagnosis, personalized treatment recommendations, and efficient retrieval of
relevant medical literature. By leveraging the vast amount of textual
information in healthcare, GPT-4 has the potential to enhance clinical
decision-making and improve patient outcomes.

###  IV-C Challenges and Pitfalls

While the adaptation and utilization of large vision models in the medical
domain demonstrate great promise, there are several challenges and pitfalls
that may be encountered and need to be effectively addressed.

  * •

The Availability of High-quality Annotations: Training large vision models
requires large-scale, accurately annotated datasets. However, medical imaging
datasets are often limited in size and quality due to factors like data
scarcity, privacy concerns, and variations in imaging modalities. The scarcity
of labeled data can hinder the performance and generalizability of large
vision models. To overcome this challenge, approaches such as active learning
[127], data augmentation, and transfer learning from related tasks can be
employed. Collaboration among researchers, healthcare institutions, and
regulatory bodies is also essential to foster the sharing and creation of
standardized, annotated datasets that cover a wide range of medical conditions
and patient demographics.

  * •

High Accuracy Standards: One of the critical challenges in deploying large
vision models in the medical imaging field is the requirement for higher
accuracy standards compared to general image applications. While a certain
level of accuracy may be acceptable in general computer vision tasks, there is
a higher demand for accuracy, reliability, and precision in medical imaging
applications where decisions regarding diagnosis and treatment are made based
on the results provided by these models. A lower classification accuracy or an
inaccurate segmentation in medical images could lead to misdiagnosis,
incorrect treatment plans, delayed interventions, or missed critical findings,
potentially compromising patient safety and well-being.

  * •

Long-tail Problem: Another significant challenges that large vision models
face is the long-tail distribution of diseases and conditions. The long-tail
phenomenon refers to the uneven distribution of labeled data, where a few
common diseases or conditions have abundant training examples, while a vast
majority of rare or less prevalent conditions have limited labeled data
available. This long-tail challenge poses difficulties for large vision models
as they may struggle to accurately classify and diagnose rare or uncommon
medical conditions due to limited exposure during training. The lack of
sufficient labeled examples for these conditions can result in a performance
bias towards the more prevalent diseases, leading to suboptimal accuracy and
sensitivity in real-world clinical settings.

  * •

Ethical Issues: The ethical implications of using large vision models in
healthcare must be carefully considered. Issues such as data privacy, patient
consent, and potential biases in the models require attention. Large vision
models heavily rely on extensive datasets for training, which raises concerns
about the privacy and security of patient information, as well as
vulnerabilities of the healthcare systems to cybersecurity hazards such as
backdoor attacks [128, 129]. Strict data governance policies, anonymization
techniques [13], and adherence to regulatory frameworks such as HIPAA are
crucial to protect patient privacy. Furthermore, biases present in the
training data [130, 1] can inadvertently propagate into the predictions of
large vision models, leading to disparities in healthcare outcomes. Addressing
and mitigating these biases is vital to ensure fair and equitable deployment
of these models across diverse patient populations.

In addition to the challenges mentioned above, there are also other notable
challenges in deploying large vision models in the medical imaging field, such
as the interpretability of the large vision models, the deployment in clinical
scenario, etc. In conclusion, while leveraging large vision models for medical
imaging holds tremendous potential, addressing the aforementioned challenges
is essential for realizing the full benefits of large vision models and
ensuring their safe, effective, and ethical integration into clinical
practice, and leading to improved healthcare outcomes and enhanced patient
care.

##  V Large Multimodal Models for Medical Imaging

Data in the real world is captured across various modalities such as text,
images, videos, and audio, mirroring the perceptual capabilities of the human
brain. In the medical field, data encompassing medical reports, clinical
notes, radiology images, physician dictations, audio recordings of physician-
patient dialogues, and surgical videos [86, 87, 131, 132] paint a
comprehensive picture of the hospital’s operational processes. Typically,
these data can be effortlessly consolidated at the patient level as they all
pertain to specific patients and/or diseases, paving the way for multimodal
models to leverage these rich data for a broad range of applications in the
medical domain. Recently, rapid development has been observed in large
multimodal models, fueled by the emergence of LLMs and expansive vision
models. Newly released models, such as Meta’s Segment Anything model [91],
OpenAI’s DALL·E 2 [126], and Stability AI’s Stable Diffusion [125] model, all
exhibit text-to-image capabilities to prompt segmentation or generation tasks.
As a result, we will mainly focus on Vision Language models in the medical
domain in this section. Specifically, we will describe the roadmap for
deploying and adapting large multimodal models in the medical domain,
introduce current and potential applications, and discuss challenges that
could further catalyze the use of large multimodal models in the medical
domain.

###  V-A Roadmaps

  * •

Scaling and Quality of Paired Data: In the medical field, paired multimodal
data are essential for adapting large multimodal models; however, they are
never sufficient for data-hungry large-scale models, regardless of whether
they are in the general or medical domain. In the general domain, it is
relatively easy to collect text-image pairs from web images that are
accompanied by contextual content or human-annotated captions, which don’t
necessitate specialist knowledge for captioning [26]. New released multimodal
dataset in general domain such as LAION-5B [133] contains billions of text-
image pairs. Conversely, in the medical domain, multimodal datasets are
significantly smaller compared to the general domain, reducing from billions
to thousands, exemplified by MIMIC-CXR [87] and Open-I [134], with 227,835 and
3,955 radiology image-report pairs respectively.

Despite their considerably smaller scale in comparison to the general domain,
the quality of multimodal data in the medical domain is inherently superior.
For instance, in MIMIC-CXR, radiology reports are accurately and concisely
written by radiologists to describe the findings and impression from image. As
a tradeoff, the amount of multimodal data in the medical domain is very
limited, as it requires experts with medical knowledge to annotate and must
adhere to certain privacy rules before being released for public use.

  * •

Pretraining Multimodal Foundational Models: Given the fact that multimodal
data in the general domain consists of a relatively large amount of data but
with sparse information and varying quality, while data in the medical domain
is limited but of high quality, an efficient and effective approach to
deploying and adapting a large multimodal model in the medical domain is to
first pretrain a foundation model on large-scale general domain dataset and
then fine-tune the pretrained foundation model on high-quality medical
dataset. Leveraging the zero-shot in-context learning abilities of pretrained
foundational models, such as OpenClip [135] for image and text retrieval and
Segment Anything Model for image segmentation, we can feasibly adapt these
models to the medical field. They can be readily used for tasks such as
medical report generation, radiology image comprehension, and medical image
segmentation, even when only limited fine-grained medical multimodal data is
available.

  * •

Parameter-Efficient Fine-Tuning: Recently released foundation models possess
two primary characteristics: 1) a large-scale of parameters and 2) large
pretraining datasets. For instance, GPT-3 [36] has 175 billion parameters
trained on 45 TB of data. LLaMA [136] models range from a scale of 7 B to 65 B
trained on 1.4 trillion tokens. This expansive scale leads to high
computational costs for full fine-tuning. Hence, Parameter-Efficient Fine
Tuning (PEFT) strategies [137, 138] can serve as a potent method to rapidly
adapt these pretrained foundation models to the medical domain. Below are some
prevalent PEFT strategies for tuning large pretrained models:

Adapter tuning [139] introduces lightweight modules, referred to as
”adapters”, within the layers of the pretrained large models. When these
models are fine-tuned on downstream tasks, only the parameters of the adapter
are updated, while the pretrained parameters remain fixed. This method allows
the fine-tuned model to retain knowledge from the pretraining phase while
efficiently adapting to the specific downstream task.

LoRA [140] introduces a low-rank bottleneck by decomposing the weight matrix
of the pretrained model into low-rank matrices. During fine-tuning, only these
low-rank matrices are updated, significantly reducing computational costs.
QLoRA [141] further enhances the LoRA approach to ensure that fine-tuning a
4-bit quantized pretrained LLM with LoRA does not compromise performance
compared to 16-bit full fine-tuning.

Utilizing these parameter-efficient fine-tuning strategies allows pretrained
foundation models to be quickly adapted to the medical domain. They deliver
expert-level performance and impressive generalizability across multimodal
medical imaging, including CT scans, MRIs, and X-rays, among others.

###  V-B Current and potential applications

  * •

Image-to-Text Models: At present, the principal deployment of multimodal
models in the medical domain is primarily within Visual Question Answering
(VQA) models [142, 143, 144, 145]. These models enhance diagnostic efficiency
by facilitating the effective analysis of medical images such as X-rays, CT
scans, and MRI scans. By posing queries to the VQA model regarding features,
abnormalities, or regions of interest within an image, medical practitioners
can receive pertinent and detailed answers, aiding in the interpretation of
complex medical images. This technology has the potential to reduce diagnostic
errors, enhance accuracy, and speed up the diagnostic process, ultimately
leading to improved patient outcomes.

CLIP (Contrastive Language-Image Pretraining) [26] is a vision-language model
in the general domain developed by OpenAI. It is trained by learning to
associate images and their corresponding textual descriptions using
contrastive learning. The pretrained image encoder can be utilized to obtain
image embeddings that facilitate tasks such as image classification, and
image-text retrieval. Due to its versatility and impressive cross-modal
understanding capabilities, CLIP is a robust tool for combining visual and
textual information. MedCLIP [146] enhances and extends the general domain
CLIP model to the medical domain by decoupling images and texts for multimodal
contrastive learning to employ unpaired medical image and text.

In conjunction with the newly released open-source Language Learning Models
(LLMs) such as LLaMA [136], Alpaca [147], and Vicuna [148] which serve as text
decoders, MedCLIP [146] can function as an image encoder in VQA models.
XrayGPT [149], a medical domain VQA model similar to miniGPT-4 [142] in the
general domain, automates the analysis of chest radiographs based on provided
X-ray images. Specifically, the pretrained LLM Vicuna [148] is first fine-
tuned using medical conversation data comprising approximately 100k real
dialogues between patients and doctors and 30k radiology conversations to
acquire domain-specific knowledge in the medical domain. Subsequently, a high-
quality dataset of 217k chest X-ray findings report from two datasets (MIMIC-
CXR and OpenI) are used for fine-tuning the XrayGPT [149], which aligns the
frozen MedCLIP as medical image encoder and Vicuna as text decoder to generate
chest radiographs summarization from given X-ray images.

  * •

Text-to-Image Models: While significant progress has been made in the field of
text-prompt image analysis in general domains with models such as Stable
Diffusion and DALL·E 2, it remains a formidable challenge in the medical
domain [150]. Medical images, characterized by complex features, varying
intensities, and region-specific content, demand a heightened level of
precision and strict adherence to specific rules and principles in comparison
to general images. The advent of multimodal models has opened prospective
applications focusing on text-prompt image analysis. Below are three potential
applications of Text-to-Image models.

  * •

Computer-Aided Diagnosis: Text-prompt image analysis could enhance diagnostic
capabilities. By prompting text targeting specific medical image analysis
tasks in clinical scenarios, such as emergency rooms, it can bolster the
efficiency of diagnosis by doctors and radiologists [17]. For instance, the
’Segment Anything’ model could be deployed in the medical domain by utilizing
text prompts that describe the targeted abnormality feature to segment the
aberrant lesion. With SAM’s zero-shot capability, it is highly adaptable for
application in medical imaging across different modalities without the need
for specific fine-tuning for each. Furthermore, computer-aided image analysis
could be employed for second opinions, comparisons, and decision support, thus
leading to more accurate and confident diagnoses.

  * •

Medical Education and Training: Text-prompt image generation could be employed
to generate realistic training scenarios for medical students and healthcare
professionals. By simulating a diverse array of cases, rare conditions, and
complex anatomical variations, synthetic medical image generation could enrich
educational programs and offer invaluable hands-on experience in a controlled
environment.

  * •

Surgical Planning: Precise and detailed preoperative planning is critical for
complex surgical procedures. Text-prompt medical image generation can
facilitate the creation of realistic 3D anatomical models that focus on
abnormalities. This aids surgeons in understanding complex structures,
optimizing surgical approaches, and simulating the expected outcomes of
surgical interventions. Such a system could potentially reduce risks, improve
surgical precision, and enhance patient outcomes.

  * •

Use of Eye-Tracking Technology: The use of eye-tracking technology can provide
valuable data for training LLMs [17]. By recording where a physician’s
attention is focused during case study reviews, models like ChatGPT could
learn to emulate this prioritization of information, potentially improving the
model’s diagnostic reasoning capabilities.

###  V-C Challenges and pitfalls

  * •

Lack of Paired Data: One of the most significant challenges in this context is
the scarcity of large-scale, paired medical image-text data. This is
predominantly due to the heterogeneous and complex nature of medical data. To
advance the development and refinement of multimodal models in the medical
domain, a large-scale, paired dataset combining visual and textual medical
information is essential. A potential solution could be the application of
pseudo-labels, such as generated radiology image reports or synthesized
medical images. These can serve as synthetic datasets or automatic annotations
that may enhance model training and performance. Nevertheless, this approach
should be applied cautiously as the quality of synthetic data might fluctuate,
and any inaccuracies or inconsistencies could potentially affect the efficacy
and robustness of the resulting model.

  * •

Backdoor Attacks: The issue of cybersecurity represents another critical
concern when deploying large foundational models in healthcare. These models
may be vulnerable to backdoor attacks, where malicious actors manipulate the
model during training or inference, posing a substantial security threat to
downstream users [151]. This flaw could result in a compromised model within a
hospital setting, thereby exposing the institution to data leakage and
potential security violations. Hence, cybersecurity must be accorded top
priority during the deployment and adaptation of multimodal models in the
medical domain.

##  VI Future Directions

Although we have discussed some of the future directions under each section,
it is worth providing summarized discussions of the relevant points to have a
holistic view of the potential future works.

###  VI-A Federated Learning in the Era of LLMs

Federated learning [152] is an innovative machine learning approach that
addresses privacy, data ownership, and scalability challenges by enabling the
training of models across decentralized devices or data sources. In the
medical domain, federated learning holds great promise as it allows the
development of powerful and accurate machine learning models while maintaining
the privacy and security of sensitive healthcare data [92]. Recently, the
advancement of transformer-based LLMs has opened up new possibilities for
federated learning in the medical domain. LLMs can provide a common foundation
for different healthcare centers participating in federated learning, enabling
them to have a standardized architecture. This standardization makes it easier
to share and collaborate on model development. Furthermore, the tokenization
process used in LLMs contributes to the effectiveness of federated learning.
By adopting a standardized token space, models from different centers can
maintain a shared understanding of the input data. This shared understanding
allows for effective data sharing and collaboration, as information exchange
can be conducted at the token level. It enables the models to exchange
relevant information while effectively shielding the original and private
data, ensuring data privacy is preserved. Moreover, when a large multimodal
model is used, the tokenization process also facilitates the fusion of
multimodal data at the token level. Different modalities of data can be mapped
to a unified token space, enabling the integration of diverse data types and
enhancing the completeness and richness of information within the federated
learning framework. In general, effective combination of LLMs and federated
learning has significant potential in developing more powerful AGI systems.

###  VI-B Adaption of Large Models

The adaptation of large models to new tasks and domains may involve several
key techniques: prompt fine-tuning, reinforcement learning, adapter modules
(e.g., LoRA), and model distillation. These techniques can be applied
interchangeably across language, vision, and multimodal tasks, allowing models
to adapt effectively to diverse domains.

Prompt fine-tuning is a crucial step in model adaptation. By gradually
adjusting the model’s parameters using specific examples and expected outputs,
the model becomes better aligned with the target task. This iterative process
enables the model to capture task-specific nuances and improve its performance
by fine-tuning its responses and predictions. Reinforcement learning
techniques play a significant role in optimizing the adapted model’s
performance. By incorporating feedback signals, such as task-specific rewards,
the model can learn from trial and error, adjusting its behavior to maximize
the desired outcome. This reinforcement-based optimization enhances the
model’s decision-making capabilities and overall performance. Adapter modules,
like LoRA, offer a flexible and efficient approach to adapt large models.
These modules allow task-specific adjustments to be introduced without
extensively modifying the entire model architecture. By selectively modifying
certain parts of the model, the adapters enable the model to specialize for
specific tasks while minimizing any potential negative impact on the overall
model’s performance and capabilities. Model distillation is an effective
technique for transferring knowledge from larger models to smaller, more
resource-efficient ones. By distilling the knowledge from a larger model into
a compact representation, the adapted model can retain the essential insights
and generalization capabilities of the original model while being more
suitable for deployment in resource-constrained environments. This knowledge
transfer enhances the performance and efficiency of the adapted model.

Despite the effectiveness of the aforementioned methods in adapting large
models to new tasks and domains, it is important to acknowledge that there may
be other approaches and techniques. The field of model adaptation is
constantly evolving, and researchers continue to explore novel methods to
improve the adaptability of large models. These alternative methods may
include techniques such as meta-learning, transfer learning, domain
adaptation, or even domain-specific architecture design. Each of these methods
may have its own advantages and applicability depending on the specific task
and domain at hand. Therefore, while prompt fine-tuning, reinforcement
learning, adapter modules, and model distillation serve as valuable tools for
adaptation, researchers and practitioners should remain open to exploring and
experimenting with alternative methods to further advance the field of large
model adaptation.

###  VI-C Learning from experts

  * •

Direct Expert Involvement in Reinforcement Learning with Expert Feedback
(RLEF): This strategy involves inviting clinicians to participate in the
process of rating the outputs of the AGI models during their training phase.
Clinicians can rank the generated responses based on clinical validity and
usefulness, which can enhance the proficiency of the models in delivering
clinically accurate information. For instance, in the training phase of an AGI
model specialized in cardiology, a cardiologist could provide feedback and
rank the model’s interpretation of an electrocardiogram. This feedback guides
the model in refining its capability to analyze electrocardiograms accurately
and mimic the nuanced decision-making process of a real cardiologist.

  * •

Multidimensional Ranking Approach: In contrast to a simple ranking system, a
more complex and multidimensional approach could better capture the nuances of
medical scenarios. For example, consider an AGI model trained to suggest
differential diagnoses based on a collection of patient symptoms. A panel of
physicians could then evaluate the model’s output based on several factors -
accuracy of diagnosis, appropriateness to the presented symptoms,
consideration of patient medical history, and the potential severity or
urgency of each suggested condition. This multi-faceted feedback could provide
a more comprehensive training signal for the AGI, thereby enhancing its
ability to provide valuable input in complex, real-world clinical situations.

  * •

Development of Interdisciplinary Teams: Assemble teams comprising AI
specialists, healthcare professionals from diverse specialties, ethicists, and
legal experts. These teams can provide well-rounded feedback during the
development and validation phases of the models, thus ensuring the AGI tools
are accurate, ethical, and comply with regulations [153, 154].

  * •

Active Learning Sessions: Clinicians can engage in active learning [127]
sessions with the AGI models. This strategy allows clinicians to guide the AGI
models through a series of (real or simulated) patient scenarios, ensuring
that they learn and adapt based on the expert’s interpretation and response to
the same scenarios. For instance, a psychiatrist could engage with a Large
Multimodal Model (LMM) trained on patient responses and gestures to assist in
mental health consultations. Through these sessions, the psychiatrist can
provide expert responses to simulated patient scenarios, allowing the model to
learn and improve its understanding and handling of complex psychiatric cases

  * •

Refinement of Model Prompting and Tailoring to the Medical Domain: AGI models
can benefit from the expertise of clinicians in refining the prompting
mechanism for specific medical contexts. For example, an LLM could be trained
to generate more clinically relevant prompts for a neurological exam under the
guidance of neurologists. Prompts can be tailored to elicit detailed symptom
descriptions, like refining the question ”Can you describe your headache?” to
”Can you describe the location, duration, intensity, and any associated
symptoms of your headache?”.

###  VI-D Leveraging Multimodal Approaches

Multimodal models entail the synergy of multiple data modalities, including
but not limited to text, images, and audio, to enhance the understanding and
reasoning capabilities of medical models.

For instance, the language reasoning ability of multimodal models can be
employed to facilitate image understanding by extracting and aligning
meaningful information from both medical images and reports to improve
diagnostic accuracy. Specifically, language reasoning can provide rich,
context-specific annotations that aid in illustrating the unique features in
medical images, in order to analyze radiological images more effectively,
interpret clinical notes, and provide more comprehensive assessments.

Additionally, the use of Artificial Intelligence-Generated Content (AIGC) for
generating synthetic data with ground truth in text and image formats presents
an innovative pathway towards improving model accuracy [11]. This direction
involves the utilization of multimodal models to autonomously generate diverse
and validated content, be it in the form of text-based reports or annotated
images, that serves as training data in the medical domain. This could
significantly improve accuracy and also generalizability as the diverse
generated data in different modalities ensures the system’s robustness and
efficiency to adapt to diverse tasks and use-cases such as disease diagnosis,
medical imaging analysis, and treatment recommendations. This capability is
particularly valuable in scenarios where annotated medical data is scarce or
inaccessible.

Moreover, the multimodal model could close the loop from image generation to
the generation of comprehensive analytical reports. It provides a framework
for developing an auto reconstruction as feedback loop that not only produces
fine-grained image generation with text-guided prompts but also facilitates
the generation of precise, comprehensive medical analysis reports from these
images, to better learn the image and text embedding in the medical domain.
This bi-directional process is essential for effectively translating the
insights gained from the texts and images into actionable clinical
recommendations.

###  VI-E Safeguarding healthcare: Mitigation strategies for AGI in clinical
practice

Given the array of potential risks associated with AGI models, there is a need
for comprehensive mitigation strategies to ensure their safe and effective use
in the medical domain. This section outlines some proactive measures to
counteract the risks for LLMs, large vision models (LVM), and large multimodal
models (LMM).

  * •

Adopting Robust Supervision and Validation Frameworks: Regularly validating
the performance of AGI models in a controlled environment can help identify
areas of weakness or misinformation. This process should involve
interdisciplinary teams of AI specialists, medical professionals, ethicists,
and legal experts to assess the models from different perspectives.

  * •

Training on Recent and Representative Data: Continuously updating the training
data to include recent advancements in medical knowledge and guidelines can
help keep the models’ advice aligned with current best practices. The training
data should also represent diverse demographics to minimize biases in the
generated advice [130].

  * •

Establishing a Two-way Communication: Allowing the model to ask clarifying
questions when necessary can help generate more individualized advice. This
can partially address the issue of AGI models not having direct access to a
patient’s medical records and minimize potential harm.

  * •

Implementing Safety Guardrails: Employing a system that alerts users when they
are soliciting or being given medical advice is essential. A disclaimer
stating that AGI models cannot replace professional medical advice would
promote safer use.

  * •

Strengthening Privacy Measures: Using robust privacy-preserving techniques
such as differential privacy [155], federated learning [92], and de-
identification [13] in the training process can help safeguard sensitive data.

  * •

Bias Detection and Mitigation: Regular audits should be conducted to detect
and correct biases [154] in AGI models. Machine learning techniques that can
detect and mitigate bias in AI models can be incorporated into their
development and validation processes [154].

  * •

Educating Users: Increasing public awareness about the limitations of AGI
models is crucial. Users should be informed about the potential risks
associated with relying on AI for medical advice.

  * •

Regulatory Oversight: Regulatory bodies need to provide guidelines and
policies on the use of AGI models in the healthcare domain. Compliance with
regulations such as GDPR and HIPAA should be ensured.

  * •

Collaborative Models: Designing AI models that can work in collaboration with
human doctors, rather than replacing them, can help maximize the benefits of
AI while minimizing its risks. This also incentivizes wide acceptance of AGI
in the health sector.

These counteractive measures are integral to promoting the responsible use of
AGI in the medical field, and to maintain the integrity of medical practice
and the privacy of patients. It is essential that we continue to investigate
these models, raising concerns and finding solutions to ensure that their
potential is harnessed in a safe, ethical, and beneficial manner.

###  VI-F Combining Large-Scale and Local Small-Scale Models

The combination of large-scale models and local small-scale models could
significantly enhance the scope of computer-aided diagnosis and analysis,
enabling its application across diverse scenarios and locations. Large-scale
models, such as ChatGPT, GPT-4 [156], and Med-PaLM [157] which pretrained on
vast amounts of data, possess extensive knowledge and robust representations.
These large foundation models are typically housed in central institutions or
hospitals that possess large in-house training data, high computational
resources, and substantial funding to drive cutting-edge medical image
analysis and NLP research and applications. However, deploying these models
directly in resource-constrained hospital settings or scenarios requiring
real-time inference may not be practical due to computational limitations.

Knowledge distillation [158, 159] could be a potential solution to this
challenge. Large-scale foundation models can act as teachers to train local
small-scale models that are more suitable for deployment in constrained
settings. These small-scale models can leverage the knowledge distilled from
their larger teachers while being optimized for real-time inference and
limited computational resources. This approach strikes a balance between model
capacity and efficiency, ensuring that medical practitioners can benefit from
the power of large models while catering to the practical constraints of
healthcare environments.

By leveraging these local small-scale models, real-time medical artificial
intelligence systems designed for scenarios with limited computational
resources or urgent need for rapid or even instantaneous inference will
transition from theoretical construction to practical application. For
instance, these models can be deployed in emergency rooms, where quick
decision-making is critical, or they can be integrated into surgical tools
such as endoscopes for real-time inference during surgery. Furthermore, small-
scale models could play a crucial role in facilitating real-time dialogues and
guidance with home-hospital AI assistants, thereby extending the reach of
high-quality medical care.

##  VII Conclusion

The potential of Artificial General Intelligence (AGI) in healthcare,
particularly through Large Language Models, Large Vision Models, and Large
Multimodal Models, is promising. AGI models have the capability to completely
revolutionize healthcare, especially when they include experts in the loop,
incorporate medical domain knowledge, and cater to the multimodal nature of
real clinical data. Strategies such as Reinforcement Learning with Expert
Feedback (RLEF), federated learning and the use of semi-supervised and active
learning methodologies offer valuable methods to enhance AGI models for
medical applications.

However, deploying AGI in healthcare is not without challenges. Ethical
considerations, data accessibility challenges, the significant demand for
large-scale datasets, risks of erroneous outputs, data privacy and security
concerns, and regulatory hurdles underscore the need for careful handling of
these transformative technologies. Interdisciplinary collaboration and close
cooperation between scientists and clinicians will be key to unlocking the
full potential of AGI in healthcare.

As we venture deeper into the AGI era, the future of healthcare appears to be
closely tied with the development of AGI technologies. Despite the challenges,
the opportunities offered by AGI in improving patient care and outcomes are
vast. This review hopes to serve as a reference point and catalyst for further
exploration into this dynamic and rapidly evolving field.

## References

  * [1] Y. Liu, T. Han, S. Ma, J. Zhang, Y. Yang, J. Tian, H. He, A. Li, M. He, Z. Liu _et al._ , “Summary of chatgpt/gpt-4 research and perspective towards the future of large language models,” _arXiv preprint arXiv:2304.01852_ , 2023.
  * [2] M. Cascella, J. Montomoli, V. Bellini, and E. Bignami, “Evaluating the feasibility of chatgpt in healthcare: an analysis of multiple clinical and research scenarios,” _Journal of Medical Systems_ , vol. 47, no. 1, pp. 1–5, 2023.
  * [3] L. Zhang, A. Zaman, L. Wang, J. Yan, and D. Zhu, “A cascaded multi-modality analysis in mild cognitive impairment,” in _Machine Learning in Medical Imaging: 10th International Workshop, MLMI 2019, Held in Conjunction with MICCAI 2019, Shenzhen, China, October 13, 2019, Proceedings 10_. Springer, 2019, pp. 557–565.
  * [4] A. Zaman, L. Zhang, J. Yan, and D. Zhu, “Multi-modal image prediction via spatial hybrid u-net,” in _Multiscale Multimodal Medical Imaging: First International Workshop, MMMI 2019, Held in Conjunction with MICCAI 2019, Shenzhen, China, October 13, 2019, Proceedings 1_. Springer, 2020, pp. 1–9.
  * [5] L. Zhang, L. Wang, J. Gao, S. L. Risacher, J. Yan, G. Li, T. Liu, D. Zhu, A. D. N. Initiative _et al._ , “Deep fusion of brain structure-function in mild cognitive impairment,” _Medical image analysis_ , vol. 72, p. 102082, 2021.
  * [6] L. Zhang, L. Wang, and D. Zhu, “Jointly analyzing alzheimer’s disease related structure-function using deep cross-model attention network,” in _2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI)_. IEEE, 2020, pp. 563–567.
  * [7] R. OpenAI, “Gpt-4 technical report,” _arXiv_ , 2023.
  * [8] S. Huang, L. Dong, W. Wang, Y. Hao, S. Singhal, S. Ma, T. Lv, L. Cui, O. K. Mohammed, Q. Liu _et al._ , “Language is not all you need: Aligning perception with language models,” _arXiv preprint arXiv:2302.14045_ , 2023\.
  * [9] Fares Antaki, Samir Touma, Daniel Milad, Jonathan El-Khoury, Renaud Duval, “Evaluating the performance of chatgpt in ophthalmology: An analysis of its successes and shortcomings,” _medRxiv (2023): 2023-01_ , 2023.
  * [10] Maad Mijwil, Mohammad Aljanabi, Ahmed Hussein Ali, “ChatGPT: Exploring the Role of Cybersecurity in the Protection of Medical Information,” _Mesopotamian journal of cybersecurity_ , pp. 18–21, 2023.
  * [11] H. Dai, Z. Liu, W. Liao, X. Huang, Z. Wu, L. Zhao, W. Liu, N. Liu, S. Li, D. Zhu _et al._ , “Chataug: Leveraging chatgpt for text data augmentation,” _arXiv preprint arXiv:2302.13007_ , 2023.
  * [12] M. Akrout, B. Gyepesi, P. Holló, A. Poór, B. Kincső, S. Solis, K. Cirone, J. Kawahara, D. Slade, L. Abid _et al._ , “Diffusion-based data augmentation for skin disease classification: Impact across original medical datasets to fully synthetic images,” _arXiv preprint arXiv:2301.04802_ , 2023.
  * [13] Z. Liu, X. Yu, L. Zhang, Z. Wu, C. Cao, H. Dai, L. Zhao, W. Liu, D. Shen, Q. Li _et al._ , “Deid-gpt: Zero-shot medical text de-identification by gpt-4,” _arXiv preprint arXiv:2303.11032_ , 2023.
  * [14] C. Ma, Z. Wu, J. Wang, S. Xu, Y. Wei, Z. Liu, L. Guo, X. Cai, S. Zhang, T. Zhang _et al._ , “Impressiongpt: an iterative optimizing framework for radiology report summarization with chatgpt,” _arXiv preprint arXiv:2304.08448_ , 2023.
  * [15] Z. Xiao, Y. Chen, L. Zhang, J. Yao, Z. Wu, X. Yu, Y. Pan, L. Zhao, C. Ma, X. Liu _et al._ , “Instruction-vit: Multi-modal prompts for instruction learning in vit,” _arXiv preprint arXiv:2305.00201_ , 2023.
  * [16] T. Chen, L. Zhu, C. Ding, R. Cao, S. Zhang, Y. Wang, Z. Li, L. Sun, P. Mao, and Y. Zang, “Sam fails to segment anything?–sam-adapter: Adapting sam in underperformed scenes: Camouflage, shadow, and more,” _arXiv preprint arXiv:2304.09148_ , 2023.
  * [17] S. Wang, Z. Zhao, X. Ouyang, Q. Wang, and D. Shen, “Chatcad: Interactive computer-aided diagnosis on medical image using large language models,” _arXiv preprint arXiv:2302.07257_ , 2023.
  * [18] Z. Wu, L. Zhang, C. Cao, X. Yu, H. Dai, C. Ma, Z. Liu, L. Zhao, G. Li, W. Liu _et al._ , “Exploring the trade-offs: Unified large language models vs local fine-tuned models for highly-specific radiology nli task,” _arXiv preprint arXiv:2304.09138_ , 2023.
  * [19] L. Zhao, L. Zhang, Z. Wu, Y. Chen, H. Dai, X. Yu, Z. Liu, T. Zhang, X. Hu, X. Jiang _et al._ , “When brain-inspired ai meets agi,” _arXiv preprint arXiv:2303.15935_ , 2023.
  * [20] M. Sallam, “Chatgpt utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns,” in _Healthcare_ , vol. 11, no. 6. MDPI, 2023, p. 887.
  * [21] J. Yang, H. Jin, R. Tang, X. Han, Q. Feng, H. Jiang, B. Yin, and X. Hu, “Harnessing the power of llms in practice: A survey on chatgpt and beyond,” _arXiv preprint arXiv:2304.13712_ , 2023.
  * [22] J. Holmes, Z. Liu, L. Zhang, Y. Ding, T. T. Sio, L. A. McGee, J. B. Ashman, X. Li, T. Liu, J. Shen _et al._ , “Evaluating large language models on a highly-specialized topic, radiation oncology physics,” _arXiv preprint arXiv:2304.01938_ , 2023.
  * [23] P. W. Anderson, “More is different: broken symmetry and the nature of the hierarchical structure of science.” _Science_ , vol. 177, no. 4047, pp. 393–396, 1972.
  * [24] J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler _et al._ , “Emergent abilities of large language models,” _arXiv preprint arXiv:2206.07682_ , 2022.
  * [25] A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonso _et al._ , “Beyond the imitation game: Quantifying and extrapolating the capabilities of language models,” _arXiv preprint arXiv:2206.04615_ , 2022.
  * [26] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark _et al._ , “Learning transferable visual models from natural language supervision,” in _International conference on machine learning_. PMLR, 2021, pp. 8748–8763.
  * [27] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever, “Zero-shot text-to-image generation,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 8821–8831.
  * [28] A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen, “Glide: Towards photorealistic image generation and editing with text-guided diffusion models,” _arXiv preprint arXiv:2112.10741_ , 2021.
  * [29] J. Chen, H. Guo, K. Yi, B. Li, and M. Elhoseiny, “Visualgpt: Data-efficient adaptation of pretrained language models for image captioning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2022, pp. 18 030–18 040.
  * [30] Z.-Y. Dou, Y. Xu, Z. Gan, J. Wang, S. Wang, L. Wang, C. Zhu, P. Zhang, L. Yuan, N. Peng _et al._ , “An empirical study of training end-to-end vision-and-language transformers,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 18 166–18 176.
  * [31] D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu _et al._ , “Palm-e: An embodied multimodal language model,” _arXiv preprint arXiv:2303.03378_ , 2023.
  * [32] K. Zhang, J. Yu, Z. Yan, Y. Liu, E. Adhikarla, S. Fu, X. Chen, C. Chen, Y. Zhou, X. Li _et al._ , “Biomedgpt: A unified and generalist biomedical generative pre-trained transformer for vision, language, and multimodal tasks,” _arXiv preprint arXiv:2305.17100_ , 2023.
  * [33] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [34] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [35] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” _arXiv preprint arXiv:1907.11692_ , 2019.
  * [36] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell _et al._ , “Language models are few-shot learners,” _Advances in neural information processing systems_ , vol. 33, pp. 1877–1901, 2020.
  * [37] A. Radford, K. Narasimhan, T. Salimans, I. Sutskever _et al._ , “Improving language understanding by generative pre-training,” _OpenAI_ , 2018.
  * [38] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever _et al._ , “Language models are unsupervised multitask learners,” _OpenAI blog_ , vol. 1, no. 8, p. 9, 2019.
  * [39] S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin _et al._ , “Opt: Open pre-trained transformer language models,” _arXiv preprint arXiv:2205.01068_ , 2022.
  * [40] T. Mikolov, M. Karafiát, L. Burget, J. Cernockỳ, and S. Khudanpur, “Recurrent neural network based language model.” in _Interspeech_ , vol. 2, no. 3. Makuhari, 2010, pp. 1045–1048.
  * [41] A. Graves and A. Graves, “Long short-term memory,” _Supervised sequence labelling with recurrent neural networks_ , pp. 37–45, 2012.
  * [42] R. Dey and F. M. Salem, “Gate-variants of gated recurrent unit (gru) neural networks,” in _2017 IEEE 60th international midwest symposium on circuits and systems (MWSCAS)_. IEEE, 2017, pp. 1597–1600.
  * [43] C. Zhou, Q. Li, C. Li, J. Yu, Y. Liu, G. Wang, K. Zhang, C. Ji, Q. Yan, L. He _et al._ , “A comprehensive survey on pretrained foundation models: A history from bert to chatgpt,” _arXiv preprint arXiv:2302.09419_ , 2023.
  * [44] Q. Dong, L. Li, D. Dai, C. Zheng, Z. Wu, B. Chang, X. Sun, J. Xu, and Z. Sui, “A survey for in-context learning,” _arXiv preprint arXiv:2301.00234_ , 2022\.
  * [45] P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig, “Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing,” _ACM Computing Surveys_ , vol. 55, no. 9, pp. 1–35, 2023.
  * [46] J. White, Q. Fu, S. Hays, M. Sandborn, C. Olea, H. Gilbert, A. Elnashar, J. Spencer-Smith, and D. C. Schmidt, “A prompt pattern catalog to enhance prompt engineering with chatgpt,” _arXiv preprint arXiv:2302.11382_ , 2023\.
  * [47] X. Liu, Y. Zheng, Z. Du, M. Ding, Y. Qian, Z. Yang, and J. Tang, “Gpt understands, too,” _arXiv preprint arXiv:2103.10385_ , 2021.
  * [48] T. Schick and H. Schütze, “It’s not just size that matters: Small language models are also few-shot learners,” _arXiv preprint arXiv:2009.07118_ , 2020\.
  * [49] T. Shin, Y. Razeghi, R. L. Logan IV, E. Wallace, and S. Singh, “Autoprompt: Eliciting knowledge from language models with automatically generated prompts,” _arXiv preprint arXiv:2010.15980_ , 2020.
  * [50] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray _et al._ , “Training language models to follow instructions with human feedback,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 27 730–27 744, 2022.
  * [51] X. Li, Y. Yang, Y. Sun, and L. Zhang, “A developmental actor-critic reinforcement learning approach for task-nonspecific robot,” in _2016 IEEE Chinese Guidance, Navigation and Control Conference (CGNCC)_. IEEE, 2016, pp. 2231–2237.
  * [52] Y. Yang, X. Li, and L. Zhang, “Task-specific pre-learning to improve the convergence of reinforcement learning based on a deep neural network,” in _2016 12th World Congress on Intelligent Control and Automation (WCICA)_. IEEE, 2016, pp. 2209–2214.
  * [53] T. H. Kung, M. Cheatham, A. Medenilla, C. Sillos, L. De Leon, C. Elepaño, M. Madriaga, R. Aggabao, G. Diaz-Candido, J. Maningo _et al._ , “Performance of chatgpt on usmle: Potential for ai-assisted medical education using large language models,” _PLoS digital health_ , vol. 2, no. 2, p. e0000198, 2023.
  * [54] K. C. Siontis, P. A. Noseworthy, Z. I. Attia, and P. A. Friedman, “Artificial intelligence-enhanced electrocardiography in cardiovascular disease management,” _Nature Reviews Cardiology_ , vol. 18, no. 7, pp. 465–478, 2021\.
  * [55] D. Bzdok and A. Meyer-Lindenberg, “Machine learning for precision psychiatry: opportunities and challenges,” _Biological Psychiatry: Cognitive Neuroscience and Neuroimaging_ , vol. 3, no. 3, pp. 223–230, 2018.
  * [56] B. Van Ginneken, C. M. Schaefer-Prokop, and M. Prokop, “Computer-aided diagnosis: how to move from the laboratory to the clinic,” _Radiology_ , vol. 261, no. 3, pp. 719–732, 2011.
  * [57] J. Unkelbach, M. Alber, M. Bangert, R. Bokrantz, T. C. Chan, J. O. Deasy, A. Fredriksson, B. L. Gorissen, M. Van Herk, W. Liu _et al._ , “Robust radiotherapy planning,” _Physics in Medicine & Biology_, vol. 63, no. 22, p. 22TR02, 2018.
  * [58] J. Y. Chang, X. Zhang, A. Knopf, H. Li, S. Mori, L. Dong, H.-M. Lu, W. Liu, S. N. Badiyan, S. Both _et al._ , “Consensus guidelines for implementing pencil-beam scanning proton therapy for thoracic malignancies on behalf of the ptcog thoracic and lymphoma subcommittee,” _International Journal of Radiation Oncology* Biology* Physics_ , vol. 99, no. 1, pp. 41–50, 2017.
  * [59] H. Wu, M. Wang, J. Wu, F. Francis, Y.-H. Chang, A. Shavick, H. Dong, M. T. Poon, N. Fitzpatrick, A. P. Levine _et al._ , “A survey on clinical natural language processing in the united kingdom from 2007 to 2022,” _NPJ digital medicine_ , vol. 5, no. 1, p. 186, 2022.
  * [60] A. Chebli, A. Djebbar, and H. F. Marouani, “Semi-supervised learning for medical application: A survey,” in _2018 International Conference on Applied Smart Systems (ICASS)_. IEEE, 2018, pp. 1–9.
  * [61] S. Budd, E. C. Robinson, and B. Kainz, “A survey on active learning and human-in-the-loop deep learning for medical image analysis,” _Medical Image Analysis_ , vol. 71, p. 102062, 2021.
  * [62] Y. Gu, R. Tinn, H. Cheng, M. Lucas, N. Usuyama, X. Liu, T. Naumann, J. Gao, and H. Poon, “Domain-specific language model pretraining for biomedical natural language processing,” _ACM Transactions on Computing for Healthcare (HEALTH)_ , vol. 3, no. 1, pp. 1–23, 2021.
  * [63] S. Rezayi, Z. Liu, Z. Wu, C. Dhakal, B. Ge, C. Zhen, T. Liu, and S. Li, “Agribert: knowledge-infused agricultural language models for matching food and nutrition.” IJCAI, 2022.
  * [64] Z. Liu, X. He, L. Liu, T. Liu, and X. Zhai, “Context matters: A strategy to pre-train language model for science education,” _arXiv preprint arXiv:2301.12031_ , 2023.
  * [65] S. Rezayi, H. Dai, Z. Liu, Z. Wu, A. Hebbar, A. H. Burns, L. Zhao, D. Zhu, Q. Li, W. Liu _et al._ , “Clinicalradiobert: Knowledge-infused few shot learning for clinical notes named entity recognition,” in _Machine Learning in Medical Imaging: 13th International Workshop, MLMI 2022, Held in Conjunction with MICCAI 2022, Singapore, September 18, 2022, Proceedings_. Springer, 2022, pp. 269–278.
  * [66] Z. Liu, M. He, Z. Jiang, Z. Wu, H. Dai, L. Zhang, S. Luo, T. Han, X. Li, X. Jiang _et al._ , “Survey on natural language processing in medical image analysis.” _Zhong nan da xue xue bao. Yi xue ban= Journal of Central South University. Medical Sciences_ , vol. 47, no. 8, pp. 981–993, 2022\.
  * [67] M. Moor, O. Banerjee, Z. S. H. Abad, H. M. Krumholz, J. Leskovec, E. J. Topol, and P. Rajpurkar, “Foundation models for generalist medical artificial intelligence,” _Nature_ , vol. 616, no. 7956, pp. 259–265, 2023.
  * [68] X. Chen, L. Li, Q. Fei, N. Zhang, C. Tan, Y. Jiang, F. Huang, and H. Chen, “One model for all domains: Collaborative domain-prefix tuning for cross-domain ner,” _arXiv preprint arXiv:2301.10410_ , 2023.
  * [69] B. Lester, R. Al-Rfou, and N. Constant, “The power of scale for parameter-efficient prompt tuning,” _arXiv preprint arXiv:2104.08691_ , 2021\.
  * [70] X. Liu, K. Ji, Y. Fu, W. Tam, Z. Du, Z. Yang, and J. Tang, “P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks,” in _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , 2022, pp. 61–68.
  * [71] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds _et al._ , “Flamingo: a visual language model for few-shot learning,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 23 716–23 736, 2022.
  * [72] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” _arXiv preprint arXiv:2010.11929_ , 2020.
  * [73] V. P. Aggelidis and P. D. Chatzoglou, “Methods for evaluating hospital information systems: a literature review,” _EuroMed Journal of Business_ , 2008.
  * [74] R. Brate, M.-H. Dang, F. Hoppe, Y. He, A. Meroño-Peñuela, and V. Sadashivaiah, “Improving language model predictions via prompts enriched with knowledge graphs,” in _Workshop on Deep Learning for Knowledge Graphs (DL4KG@ ISWC2022)_ , 2022.
  * [75] B. Peng, M. Galley, P. He, H. Cheng, Y. Xie, Y. Hu, Q. Huang, L. Liden, Z. Yu, W. Chen _et al._ , “Check your facts and try again: Improving large language models with external knowledge and automated feedback,” _arXiv preprint arXiv:2302.12813_ , 2023.
  * [76] H. Cai, W. Liao, Z. Liu, X. Huang, Y. Zhang, S. Ding, S. Li, Q. Li, T. Liu, and X. Li, “Coarse-to-fine knowledge graph domain adaptation based on distantly-supervised iterative training,” _arXiv preprint arXiv:2211.02849_ , 2022.
  * [77] A. Grünebaum, J. Chervenak, S. L. Pollet, A. Katz, and F. A. Chervenak, “The exciting potential for chatgpt in obstetrics and gynecology,” _American Journal of Obstetrics and Gynecology_ , 2023.
  * [78] C. M. P. Jacoba, L. A. Celi, A. C. Lorch, W. Fickweiler, L. Sobrin, J. W. Gichoya, L. P. Aiello, and P. S. Silva, “Bias and non-diversity of big data in artificial intelligence: Focus on retinal diseases: “massachusetts eye and ear special issue”,” in _Seminars in Ophthalmology_. Taylor & Francis, 2023, pp. 1–9.
  * [79] M. Abdalla, M. Abdalla, F. Rudzicz, and G. Hirst, “Using word embeddings to improve the privacy of clinical notes,” _Journal of the American Medical Informatics Association_ , vol. 27, no. 6, pp. 901–907, 2020.
  * [80] M. J. Sheller, B. Edwards, G. A. Reina, J. Martin, S. Pati, A. Kotrotsou, M. Milchenko, W. Xu, D. Marcus, R. R. Colen _et al._ , “Federated learning in medicine: facilitating multi-institutional collaborations without sharing patient data,” _Scientific reports_ , vol. 10, no. 1, pp. 1–12, 2020\.
  * [81] P. D. Thai, M. Doan, W. Liu, T. Liu, S. Li, H.-s. Zhou, and T. N. Dinh, “Blockchain peer-to-peer network: Performance and security,” in _Handbook on Blockchain_. Springer, 2022, pp. 55–83.
  * [82] H. Lee, “The rise of chatgpt: Exploring its potential in medical education,” _Anatomical Sciences Education_ , 2023.
  * [83] J. Wang, E. Shi, S. Yu, Z. Wu, C. Ma, H. Dai, Q. Yang, Y. Kang, J. Wu, H. Hu _et al._ , “Prompt engineering for healthcare: Methodologies and applications,” _arXiv preprint arXiv:2304.14670_ , 2023.
  * [84] T. L. Murray, M. Calhoun, and N. C. Philipsen, “Privacy, confidentiality, hipaa, and hitech: implications for the health care practitioner,” _The Journal for Nurse Practitioners_ , vol. 7, no. 9, pp. 747–752, 2011.
  * [85] D. S. Char, M. D. Abràmoff, and C. Feudtner, “Identifying ethical considerations for machine learning healthcare applications,” _The American Journal of Bioethics_ , vol. 20, no. 11, pp. 7–17, 2020.
  * [86] A. E. Johnson, T. J. Pollard, L. Shen, L.-w. H. Lehman, M. Feng, M. Ghassemi, B. Moody, P. Szolovits, L. Anthony Celi, and R. G. Mark, “Mimic-iii, a freely accessible critical care database,” _Scientific data_ , vol. 3, no. 1, pp. 1–9, 2016.
  * [87] A. E. Johnson, T. J. Pollard, S. J. Berkowitz, N. R. Greenbaum, M. P. Lungren, C.-y. Deng, R. G. Mark, and S. Horng, “Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports,” _Scientific data_ , vol. 6, no. 1, p. 317, 2019.
  * [88] S. Ghosh, E. S. Baranowski, M. Biehl, W. Arlt, P. Tino, and K. Bunte, “Interpretable models capable of handling systematic missingness in imbalanced classes and heterogeneous datasets,” _arXiv preprint arXiv:2206.02056_ , 2022.
  * [89] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_. Springer, 2015, pp. 234–241.
  * [90] S. M. Anwar, M. Majid, A. Qayyum, M. Awais, M. Alnowami, and M. K. Khan, “Medical image analysis using convolutional neural networks: a review,” _Journal of medical systems_ , vol. 42, pp. 1–13, 2018.
  * [91] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo _et al._ , “Segment anything,” _arXiv preprint arXiv:2304.02643_ , 2023.
  * [92] N. Rieke, J. Hancox, W. Li, F. Milletari, H. R. Roth, S. Albarqouni, S. Bakas, M. N. Galtier, B. A. Landman, K. Maier-Hein _et al._ , “The future of digital health with federated learning,” _NPJ digital medicine_ , vol. 3, no. 1, p. 119, 2020.
  * [93] P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings _et al._ , “Advances and open problems in federated learning,” _Foundations and Trends® in Machine Learning_ , vol. 14, no. 1–2, pp. 1–210, 2021\.
  * [94] Y. Cui, J. Ren, H. Xu, P. He, H. Liu, L. Sun, and J. Tang, “Diffusionshield: A watermark for copyright protection against generative diffusion models,” _arXiv preprint arXiv:2306.04642_ , 2023.
  * [95] Y. Liu, H. Hu, X. Zhang, and L. Sun, “Watermarking text data on large language models for dataset copyright protection,” _arXiv preprint arXiv:2305.13257_ , 2023.
  * [96] A. Esteva, A. Robicquet, B. Ramsundar, V. Kuleshov, M. DePristo, K. Chou, C. Cui, G. Corrado, S. Thrun, and J. Dean, “A guide to deep learning in healthcare,” _Nature medicine_ , vol. 25, no. 1, pp. 24–29, 2019.
  * [97] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _2009 IEEE conference on computer vision and pattern recognition_. Ieee, 2009, pp. 248–255.
  * [98] C. G. Snoek, M. Worring, and A. W. Smeulders, “Early versus late fusion in semantic video analysis,” in _Proceedings of the 13th annual ACM international conference on Multimedia_ , 2005, pp. 399–402.
  * [99] K. Gadzicki, R. Khamsehashari, and C. Zetzsche, “Early vs late fusion in multimodal convolutional neural networks,” in _2020 IEEE 23rd international conference on information fusion (FUSION)_. IEEE, 2020, pp. 1–6.
  * [100] Y. Wang, Y. Xie, J. Zeng, H. Wang, L. Fan, and Y. Song, “Cross-modal fusion for multi-label image classification with attention mechanism,” _Computers and Electrical Engineering_ , vol. 101, p. 108002, 2022.
  * [101] M. Böhle, M. Fritz, and B. Schiele, “Holistically explainable vision transformers,” _arXiv preprint arXiv:2301.08669_ , 2023.
  * [102] S. Stevens and Y. Su, “An investigation of language model interpretability via sentence editing,” _arXiv preprint arXiv:2011.14039_ , 2020.
  * [103] M. Hassanin, S. Anwar, I. Radwan, F. S. Khan, and A. Mian, “Visual attention methods in deep learning: An in-depth survey,” _arXiv preprint arXiv:2204.07756_ , 2022.
  * [104] K. Simonyan, A. Vedaldi, and A. Zisserman, “Deep inside convolutional networks: Visualising image classification models and saliency maps,” _arXiv preprint arXiv:1312.6034_ , 2013.
  * [105] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-cam: Visual explanations from deep networks via gradient-based localization,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 618–626.
  * [106] N. Tajbakhsh, H. Roth, D. Terzopoulos, and J. Liang, “Guest editorial annotation-efficient deep learning: the holy grail of medical imaging,” _IEEE transactions on medical imaging_ , vol. 40, no. 10, pp. 2526–2533, 2021\.
  * [107] J. Hagerty, R. J. Stanley, and W. V. Stoecker, “Medical image processing in the age of deep learning,” in _Proceedings of the 12th international joint conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP)_ , 2017, pp. 306–311.
  * [108] N. Jouppi, C. Young, N. Patil, and D. Patterson, “Motivation for and evaluation of the first tensor processing unit,” _ieee Micro_ , vol. 38, no. 3, pp. 10–19, 2018.
  * [109] M. Garland, S. Le Grand, J. Nickolls, J. Anderson, J. Hardwick, S. Morton, E. Phillips, Y. Zhang, and V. Volkov, “Parallel computing experiences with cuda,” _IEEE micro_ , vol. 28, no. 4, pp. 13–27, 2008.
  * [110] G. A. Kaissis, M. R. Makowski, D. Rückert, and R. F. Braren, “Secure, privacy-preserving and federated machine learning in medical imaging,” _Nature Machine Intelligence_ , vol. 2, no. 6, pp. 305–311, 2020.
  * [111] T. Choudhary, V. Mishra, A. Goswami, and J. Sarangapani, “A comprehensive survey on model compression and acceleration,” _Artificial Intelligence Review_ , vol. 53, pp. 5113–5155, 2020.
  * [112] A. Polino, R. Pascanu, and D. Alistarh, “Model compression via distillation and quantization,” _arXiv preprint arXiv:1802.05668_ , 2018.
  * [113] S. He, R. Bao, J. Li, P. E. Grant, and Y. Ou, “Accuracy of segment-anything model (sam) in medical image segmentation tasks,” _arXiv preprint arXiv:2304.09324_ , 2023.
  * [114] P. Shi, J. Qiu, S. M. D. Abaxi, H. Wei, F. P.-W. Lo, and W. Yuan, “Generalist vision foundation models for medical imaging: A case study of segment anything model on zero-shot medical segmentation,” _arXiv preprint arXiv:2304.12637_ , 2023.
  * [115] D. Cheng, Z. Qin, Z. Jiang, S. Zhang, Q. Lao, and K. Li, “Sam on medical images: A comprehensive study on three prompt modes,” _arXiv preprint arXiv:2305.00035_ , 2023.
  * [116] M. A. Mazurowski, H. Dong, H. Gu, J. Yang, N. Konz, and Y. Zhang, “Segment anything model for medical image analysis: an experimental study,” _arXiv preprint arXiv:2304.10517_ , 2023.
  * [117] Y. Huang, X. Yang, L. Liu, H. Zhou, A. Chang, X. Zhou, R. Chen, J. Yu, J. Chen, C. Chen _et al._ , “Segment anything model for medical images?” _arXiv preprint arXiv:2304.14660_ , 2023.
  * [118] C. Mattjie, L. V. de Moura, R. C. Ravazio, L. S. Kupssinskü, O. Parraga, M. M. Delucis, and R. C. Barros, “Exploring the zero-shot capabilities of the segment anything model (sam) in 2d medical imaging: A comprehensive evaluation and practical guideline,” _arXiv preprint arXiv:2305.00109_ , 2023\.
  * [119] J. Ma and B. Wang, “Segment anything in medical images,” _arXiv preprint arXiv:2304.12306_ , 2023.
  * [120] J. Wu, R. Fu, H. Fang, Y. Liu, Z. Wang, Y. Xu, Y. Jin, and T. Arbel, “Medical sam adapter: Adapting segment anything model for medical image segmentation,” _arXiv preprint arXiv:2304.12620_ , 2023.
  * [121] Z. Qiu, Y. Hu, H. Li, and J. Liu, “Learnable ophthalmology sam,” _arXiv preprint arXiv:2304.13425_ , 2023.
  * [122] K. Zhang and D. Liu, “Customized segment anything model for medical image segmentation,” _arXiv preprint arXiv:2304.13785_ , 2023.
  * [123] Y. Liu, J. Zhang, Z. She, A. Kheradmand, and M. Armand, “Samm (segment any medical model): A 3d slicer integration to sam,” _arXiv preprint arXiv:2304.05622_ , 2023.
  * [124] Y. Cao, S. Li, Y. Liu, Z. Yan, Y. Dai, P. S. Yu, and L. Sun, “A comprehensive survey of ai-generated content (aigc): A history of generative ai from gan to chatgpt,” _arXiv preprint arXiv:2303.04226_ , 2023.
  * [125] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 10 684–10 695.
  * [126] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, “Hierarchical text-conditional image generation with clip latents,” _arXiv preprint arXiv:2204.06125_ , 2022.
  * [127] R. M. Monarch, _Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI_. Simon and Schuster, 2021.
  * [128] J. Shi, Y. Liu, P. Zhou, and L. Sun, “Badgpt: Exploring security vulnerabilities of chatgpt via backdoor attacks to instructgpt,” _arXiv preprint arXiv:2304.12298_ , 2023.
  * [129] Z. Yuan, Y. Liu, K. Zhang, P. Zhou, and L. Sun, “Backdoor attacks to pre-trained unified foundation models,” _arXiv preprint arXiv:2302.09360_ , 2023.
  * [130] D. Cirillo, S. Catuara-Solarz, C. Morey, E. Guney, L. Subirats, S. Mellino, A. Gigante, A. Valencia, M. J. Rementeria, A. S. Chadha _et al._ , “Sex and gender differences and biases in artificial intelligence for biomedicine and healthcare,” _NPJ digital medicine_ , vol. 3, no. 1, p. 81, 2020.
  * [131] G. Zeng, W. Yang, Z. Ju, Y. Yang, S. Wang, R. Zhang, M. Zhou, J. Zeng, X. Dong, R. Zhang, H. Fang, P. Zhu, S. Chen, and P. Xie, “MedDialog: Large-scale medical dialogue datasets,” in _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Online: Association for Computational Linguistics, Nov. 2020, pp. 9241–9250. [Online]. Available: https://aclanthology.org/2020.emnlp-main.743
  * [132] A. Ben Abacha and D. Demner-Fushman, “A question-entailment approach to question answering,” _BMC bioinformatics_ , vol. 20, no. 1, pp. 1–23, 2019\.
  * [133] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman _et al._ , “Laion-5b: An open large-scale dataset for training next generation image-text models,” _arXiv preprint arXiv:2210.08402_ , 2022.
  * [134] D. Demner-Fushman, M. D. Kohli, M. B. Rosenman, S. E. Shooshan, L. Rodriguez, S. Antani, G. R. Thoma, and C. J. McDonald, “Preparing a collection of radiology examinations for distribution and retrieval,” _Journal of the American Medical Informatics Association_ , vol. 23, no. 2, pp. 304–310, 2016\.
  * [135] G. Ilharco, M. Wortsman, N. Carlini, R. Taori, A. Dave, V. Shankar, H. Namkoong, J. Miller, H. Hajishirzi, A. Farhadi _et al._ , “Openclip,” _Zenodo_ , vol. 4, p. 5, 2021.
  * [136] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar _et al._ , “Llama: Open and efficient foundation language models,” _arXiv preprint arXiv:2302.13971_ , 2023.
  * [137] R. Dutt, L. Ericsson, P. Sanchez, S. A. Tsaftaris, and T. Hospedales, “Parameter-efficient fine-tuning for medical image analysis: The missed opportunity,” _arXiv preprint arXiv:2305.08252_ , 2023.
  * [138] J. Chen, A. Zhang, X. Shi, M. Li, A. Smola, and D. Yang, “Parameter-efficient fine-tuning design spaces,” _arXiv preprint arXiv:2301.01821_ , 2023.
  * [139] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly, “Parameter-efficient transfer learning for nlp,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 2790–2799.
  * [140] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” _arXiv preprint arXiv:2106.09685_ , 2021.
  * [141] T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “Qlora: Efficient finetuning of quantized llms,” _arXiv preprint arXiv:2305.14314_ , 2023.
  * [142] D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, “Minigpt-4: Enhancing vision-language understanding with advanced large language models,” _arXiv preprint arXiv:2304.10592_ , 2023.
  * [143] H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” _arXiv preprint arXiv:2304.08485_ , 2023.
  * [144] X. Zhang, C. Wu, Z. Zhao, W. Lin, Y. Zhang, Y. Wang, and W. Xie, “Pmc-vqa: Visual instruction tuning for medical visual question answering,” _arXiv preprint arXiv:2305.10415_ , 2023.
  * [145] J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” _arXiv preprint arXiv:2301.12597_ , 2023.
  * [146] Z. Wang, Z. Wu, D. Agarwal, and J. Sun, “Medclip: Contrastive learning from unpaired medical images and text,” _arXiv preprint arXiv:2210.10163_ , 2022\.
  * [147] R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto, “Stanford alpaca: An instruction-following llama model,” https://github.com/tatsu-lab/stanford_alpaca, 2023.
  * [148] W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez _et al._ , “Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality,” _See https://vicuna.lmsys.org_ , 2023.
  * [149] O. Thawakar, A. Shaker, S. S. Mullappilly, H. Cholakkal, R. M. Anwer, S. Khan, J. Laaksonen, and F. S. Khan, “Xraygpt: Chest radiographs summarization using medical vision-language models,” https://github.com/mbzuai-oryx/XrayGPT, 2023.
  * [150] A. Kazerouni, E. K. Aghdam, M. Heidari, R. Azad, M. Fayyaz, I. Hacihaliloglu, and D. Merhof, “Diffusion models for medical image analysis: A comprehensive survey,” _arXiv preprint arXiv:2211.07804_ , 2022.
  * [151] Z. Guan, M. Hu, Z. Zhou, J. Zhang, S. Li, and N. Liu, “Badsam: Exploring security vulnerabilities of sam via backdoor attacks,” _arXiv preprint arXiv:2305.03289_ , 2023.
  * [152] K. Bonawitz, V. Ivanov, B. Kreuter, A. Marcedone, H. B. McMahan, S. Patel, D. Ramage, A. Segal, and K. Seth, “Practical secure aggregation for privacy-preserving machine learning,” in _proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security_ , 2017, pp. 1175–1191.
  * [153] W. Liao, Z. Liu, H. Dai, S. Xu, Z. Wu, Y. Zhang, X. Huang, D. Zhu, H. Cai, T. Liu _et al._ , “Differentiate chatgpt-generated and human-written medical texts,” _arXiv preprint arXiv:2304.11567_ , 2023.
  * [154] I. Y. Chen, E. Pierson, S. Rose, S. Joshi, K. Ferryman, and M. Ghassemi, “Ethical machine learning in healthcare,” _Annual review of biomedical data science_ , vol. 4, pp. 123–144, 2021.
  * [155] F. K. Dankar and K. El Emam, “Practicing differential privacy in health care: A review.” _Trans. Data Priv._ , vol. 6, no. 1, pp. 35–67, 2013.
  * [156] OpenAI, “Gpt-4 technical report,” 2023.
  * [157] K. Singhal, S. Azizi, T. Tu, S. S. Mahdavi, J. Wei, H. W. Chung, N. Scales, A. Tanwani, H. Cole-Lewis, S. Pfohl _et al._ , “Large language models encode clinical knowledge,” _arXiv preprint arXiv:2212.13138_ , 2022.
  * [158] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” _arXiv preprint arXiv:1503.02531_ , 2015.
  * [159] V. Sanh, L. Debut, J. Chaumond, and T. Wolf, “Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter,” _arXiv preprint arXiv:1910.01108_ , 2019.