by horizontally partitioning the model. Each partition is then executed on a
separate device, and micro-batching is employed to conceal any pipeline bubble
[278]. The major drawback of MP and PP is the significant amount of
communication time required between different devices, which can be regarded
as memory-efficient but compute-inefficient methods.

ZeRO. Based on the parallelism scenarios mentioned above, a series of ZeRO-
related studies are introduced [247, 279, 280], presents a set of memory
optimization techniques. This series includes ZeRO, ZeRO-Offload, and ZeRO-
Infinity, which aim to eliminate redundant parameters, utilize CPU and Random
Access Memory (RAM), and introduce NVMe for improved performance.

The ZeRO [247] comprised three stages: ZeRO-1, ZeRO-2, and ZeRO-3, each
corresponding to the partitioning of different components of the model.
Specifically, ZeRO-1 partitions the optimizer states, ZeRO-2 partitions both
the gradients and optimizer states, and ZeRO-3 partitions all model states.

However, the ZeRO [247] is essentially a parallelism solution. Different from
this, the core idea of ZeRO-Offload [279] is using RAM as the supplement of
VRAM. Originally, with a single V100, a 1.4B model can be trained with PyTorch
and the throughput is 30 TFLOPS, and with ZeRO-Offload augmentation, a 10B
model can be trained and the throughput is 40 TFLOPS.

ZeRO-Offload is more focused on single card scenarios, while ZeRO-Infinity
[280] is typical of the industrial field and goes for very large-scale
training. Both designed for Offload, ZeRO-Infinity pays more attention to
communications between multiple GPUs.

All the above functions are integrated into the library DeepSpeed in
Huggingface101010https://huggingface.co/docs/transformers/main/main_classes/deepspeed.
More details and related training tools can be seen in Section IV-E.

Quantization. The definition of quantization is approximating the weights or
activation values represented by the high bit widths (Float32) with the low
bit widths (INT16, INT8, INT4), in which the form of the external display is
the discretization of continuous values. It should be noted that quantization
acceleration needs compatible hardware support.

Quantization can be employed to further downsize LLMs (If an un-quantified LLM
is quantified to int4, its overall size was reduced to a quarter) and
accelerate the computational efficiency. The inclusion of this feature holds
significance not only in the deployment of Healthcare LLMs but also in
providing substantial support to diverse Healthcare mobile devices equipped
with AI cores. For example, the study [217] utilized 8-bit matrix
multiplication combined with an 8-bit optimizer for the feed-forward and
attention projection layers. This approach enabled the utilization of
OPT-175B/BLOOM on a single server equipped with consumer GPUs.

TABLE VII: Healthcare data can be used to train LLMs.

Data | Type | size | Link
---|---|---|---
MIMIC-III | EHR | 58,976 hospital admissions for 38,597 patients | Homepage
MIMIC-IV | EHR | covering a decade of admissions between 2008 and 2019 | Homepage
CPRD [281] | EHR | over 2,000 primary care practices and include 60 million patients | Homepage
PubMed | Scientific Literature | 35M citations and abstracts of biomedical literature | Data Link
PMC | Scientific Literature | 8 million full-text article records | Data Link
RCT [282] | Scientific Literature | 4,528 abstract | Data Link
MS$\hat{~{}}$2 [283] | Scientific Literature | 470,402 abstract | Data Link
CDSR [284] | Scientific Literature | 7,805 abstract | Data Link
SumPubMed [285] | Scientific Literature | 33,772 abstract | Data Link
The Pile | Scientific Literature | 825 GB English text | Data Link
S2ORC [286] | Scientific Literature | 63,709 abstract | Data Link
CORD-19 [287] | Scientific Literature | 1M papers | Data Link
MeQSum [288] | Medical Question Summarization | 1000 instances | Data Link
CHQ-Sum [289] | Medical Question Summarization | 1507 instances | Data Link
UMLS | Knowledge Base | 2M entities for 900K concepts | Homepage
COMETA [290] | Web Data (social media) | 800K Reddit posts | Homepage
MedDialog [291] | Dialogue | 3.66 million conversations | Homepage
CovidDialog [292] | Dialogue | 603 consultations | Homepage
Medical Flashcards [186] | Dialogue | 33955 instances | Data Link
Wikidoc [186] | Dialogue | 67704 instances | Data Link
Wikidoc Patient Information [186] | Dialogue | 5942 instances | Data Link
MEDIQA [293] | Dialogue | 2208 instances | Data Link
CORD-19 [287] | Dialogue | 1056660 instances | Data Link
MMMLU [287] | Dialogue | 3787 instances | Data Link
Pubmed Causal [294] | Dialogue | 2446 instances | Data Link
ChatDoctor [295] | Dialogue | 215000 instances | Data Link
Alpaca-EN-AN [214] | English Instructions | 52K instructions | Data Link
Alpaca-CH-AN [214] | Chinese Instructions | 52K instructions | Data Link
ShareGPT | Conversations | 61653 long conversations | Data Link
WebText | Web Data | 40 GB of text | Data Link
OpenWebText | Web Data | 38 GB of text | Data Link
Colossal Clean Crawled Corpus | Web Data | 806 GB of text | Data Link
OpenI | EHR, Multimodel | 3.7 million images from about 1.2 million papers | Homepage
U-Xray [296] | Multimodel | 3,955 reports and 7,470 images | Homepage
ROCO [297] | Multimodel | 81,000 radiology images and corresponding captions | Homepage
MedICaT [298] | Multimodel | 17,000 images includes captions | Homepage
PMC-OA [244] | Multimodel | 1.6M image-caption pairs | Homepage
CheXpert [299] | Multimodel | 224,316 chest radiographs with associated reports | Homepage
PadChest [300] | Multimodel | 160,000 images with related text | Homepage
MIMIC-CXR | Multimodel | 227,835 imaging studies for 64,588 patients | Homepage
PMC-15M [239] | Multimodel | 15 million Figure-caption pairs | Homepage
OpenPath [301] | Multimodel | 208,414 pathology images related descriptions | Homepage

  * •

✰ Although there are datasets available for Instruction Fine-Tuning, such as
MultiMedQA and the USMLE test, we have opted not to include them in this list.
These datasets are typically employed for evaluation purposes rather than
serving as primary resources for Instruction Fine-Tuning.

###  IV-E Useful Resources

####  IV-E1 OpenBMB

OpenBMB (Open Lab for Big Model Base)111111https://github.com/OpenBMB founded
by TsinghuaNLP & ModelBest Inc, aiming to build foundation models and systems
towards AGI. OpenBMB has published four main projects, namely CPM-Live, BMInf,
BMTrain, and BMCook. CPM-Live121212https://openi.pcl.ac.cn/OpenBMB/CPM-Live
focuses on live training LLMs and includes three LLM training plans as
milestones: CPM-Ant, CPM-Ant+, and CPM-Bee. The project provides real-time
recording of training dynamics on the official website, which includes
information such as loss function, learning rate, learned data, throughput,
gradient size, cost curve, and mean and standard deviation of internal model
parameters. This enables users to diagnose any issues during the training
process more efficiently.

BMCook and BMInf toolkits allow users to utilize LLMs (specifically CPM-Ant)
with limited computational resources. With BMInf, users can perform large
model inference on a single GPU, including consumer graphics cards like the
GTX 1060, replacing the need for a compute cluster. The compressed models (7B,
3B, 1B, 300M) provided by these toolkits can be adapted to various low-
resource scenarios.

The BMTrain toolkit facilitates the efficient training of large models by
leveraging distributed computing resources. The training of CPM-Ant took 68
days and cost 430,000 RMB, which is only 1/20th of Google’s cost for training
the T5-11B model, estimated to be around $130 million. Besides, OpenBMB’s
solution contributes to approximately 1/10th of the carbon dioxide emissions
compared to training the T5-11B model.

####  IV-E2 DeepSpeed Chat

DeepSpeed Chat [302] is a fast, affordable, and scalable open-source framework
that enables end-to-end RLHF training to generate various chatGPT-like models.
There are four core features of DeepSpeed Chat, including easy-breezy
training, a high-performance system, accessible LLM support, and universal
acceleration backed for RLHF. DeepSpeed Chat implements the pattern of
InstructGPT [73], which includes SFT, training the reward model, and the final
reinforcement learning-based tuning. By using DeepSpeed Chat, the above
process can be easily achieved.

Furthermore, DeepSpeed Chat incorporates a DeepSpeed Hybrid Engine,
facilitating a smooth transition of RLHF between inference and training
phases. This functionality optimally harnesses a spectrum of optimizations
tailored for either training or inference processes.

####  IV-E3 Training Data

as mentioned earlier, the transition from PLMs to LLMs brings a significant
shift from a model-centered approach to a data-centered approach. Increasing
the volume of pre-training data has become a key factor in enhancing the
general capabilities of LLMs. In line with this, we have gathered and
organized various datasets for training Healthcare LLMs, as presented in Table
VII. Besides the medical training data, we also list three Github projects
which integrate many general instruction and RLHF training data, including
Awesome Instruction Datasets131313https://github.com/jianzhnie/awesome-
instruction-datasets, Awesome-text/visual-instruction-tuning-
dataset141414https://github.com/yaodongC/awesome-instruction-dataset, and
Awesome-instruction-tuning151515https://github.com/zhilizju/Awesome-
instruction-tuning. Our aim is to assist those interested in training or fine-
tuning Healthcare LLMs in easily identifying the appropriate datasets.

In general, the most common sources of data for Healthcare LLMs include EHR,
scientific literature, web data, and public knowledge bases. When considering
the data structure, QA and dialogue data are the most frequently encountered.
Additionally, apart from the conventional text data used in LLMs, it is
crucial to acknowledge the significance of multimodal data. Given that the
Healthcare domain inherently involves text, images, and time series data,
multimodal LLMs offer a promising direction for further research. We
anticipate that multimodal LLMs will receive expedited attention in future
studies. Following, we briefly introduce some representative data set to
provide a general view.

EHR. The Medical Information Mart for Intensive Care III dataset (MIMIC III)
is widely recognized as one of the most widely used EHR datasets. It
encompasses a comprehensive collection of data from 58,976 unique hospital
admissions involving 38,597 patients who were treated in the intensive care
unit at the Beth Israel Deaconess Medical Center between 2001 and 2012.
Furthermore, the dataset includes 2,083,180 de-identified notes that are
associated with these admissions. MIMIC III provides valuable and extensive
information for research and analysis in the field of Healthcare, which
facilitates many PLMs and LLMs developments, such as MIMIC-BERT [131],
GatorTron [181], and MedAGI [192].

Scientific Literature. PubMed is a freely accessible search engine that
provides access to the MEDLINE database, which contains references and
abstracts related to life sciences and biomedical topics. It serves as a
comprehensive resource with over 32 million citations for biomedical
literature, including content from MEDLINE, life science journals, and online
books. These citations may also include links to full-text content available
on PubMed Central and publisher websites. The PubMed abstracts alone contain
approximately 4.5 billion words, while the full-text articles available on
PubMed Central (PMC) contribute around 13.5 billion words. These datasets
consist of high-quality academic and professional text, making them
particularly suitable for training Healthcare LLMs. Various PLM and LLM
models, such as BioBERT [91], BioELECTRA [303], GatorTron [181], and MedAlpaca
[186], have been trained using PubMed data. PubMed’s vast collection of
biomedical literature serves as a valuable foundation for advancing research
and development in the Healthcare domain.

Web Data. Web data includes any text we can obtain from the Internet. Social
media is one of the most commonly used data types. Reddit is a popular online
platform that combines social news aggregation, content rating, and discussion
features. Users can contribute various types of content, including links, text
posts, images, and videos. The platform is organized into user-created boards
called “communities” or “sub-reddits”, covering a broad range of topics.
Popular posts with more up-votes rise to the top of their respective sub-
reddits and can even make it to the site’s front page. Overall, Reddit offers
a diverse and dynamic space for users to engage in discussions, share content,
and explore a wide range of interests. The study [304] crawled health-themed
forums on Reddit to form COMETA corpus as LLMs training data. Tweets are also
usually employed to collect data, and COVID-twitter-BERT [140], Twitter BERT
[305], and TwHIN-BERT [306] are trained with these data.

Public Knowledge Bases. There exist many Healthcare-related knowledge bases,
such as UMLS [307], CMeKG [308], BioModels [309], and DrugBank [310]. Among
them, UMLS is one of the most popular, which is a repository of biomedical
vocabularies developed by the US National Library of Medicine. The UMLS has
over 2 million names for 900,000 concepts from more than 60 families of
biomedical vocabularies, as well as 12 million relations among these concepts.
Based on this structured data, the US Medical Licensing Examination (USMLE) is
organized and usually employed to test Healthcare LLMs.

CMeKG [308] is a Chinese medical knowledge graph that has been constructed by
referring to authoritative international medical standards and a wide range of
sources, including clinical guidelines, industry standards, medical textbooks,
and other medical texts. This knowledge graph serves as a comprehensive
resource for medical information. Building upon the CMeKG, HuaTuo [187]
utilizes diverse instructional data for its instruction tuning process. HuaTuo
employs Reinforcement Learning from AI Feedback (RLAIF) to refine its
instructions and enhance its performance. The combination of CMeKG and HuaTuo
demonstrates the application of public knowledge bases for developing
Healthcare LLMs. More details can be seen in Section IV-B3.

Figure 11: An example of instruction instance. The instance comes from the
study [311]. TABLE VIII: The statistics of computation cost for existing
Healthcare LLM.

Model Name | Total data size | epoch | Batch size | GPU type | GPU number | GPU time
---|---|---|---|---|---|---
Visual Med-Alpaca | 54k data points | 3 | 128 | A100-80G | 4 | 2.51 hours
GatorTron | >90 billion words | 10 | - | A100 | 992 | 6 days
Galactica | - | - | - | A100-80G | 128 | -
ChatDoctor | 100k conversations | 3 | 192 | A100 | 6 | 3 hours
DoctorGLM | 3.5G | 1 | 4 | A100-80G | 1 | 8 hours
PMC-LLaMA | 75B tokens | 5 | 128 | A100 | 8 | 7 days
Visual Med-Alpaca | 44.8MB* (without images) | - | 128 | A100-80G | 4 | 2.51 hours
BianQue 1.0 | 9 million samples | 1 | - | RTX 4090 | 8 | 16 days
GatorTronGPT | 277B tokens |  | 1,120/560 | A100-80G | 560 | 26 days
HuatuoGPT | 226,042 instances | 3 | 128 | A100 | 8 | -
LLaVA-Med | 15 million figure-caption pairs | - | - | A100 | 8 | 15 hours
Med-Flamingo | 1.3M image-caption pairs | - | 400 | A100-80G | 8 | 6.75 days

TABLE IX:  Estimated FLOPs and Training Tokens for Different Model Sizes.

Parameters | FLOPs | FLOPs (in Gopher unit) | Tokens
---|---|---|---
400 Million | 1.92e+19 | 1/29, 968 | 8.0 Billion
1 Billion | 1.21e+20 | 1/4, 761 | 20.2 Billion
10 Billion | 1.23e+22 | 1/46 | 205.1 Billion
67 Billion | 5.76e+23 | 1 | 1.5 Trillion
175 Billion | 3.85e+24 | 6.7 | 3.7 Trillion
280 Billion | 9.90e+24 | 17.2 | 5.9 Trillion
520 Billion | 3.43e+25 | 59.5 | 11.0 Trillion
1 Trillion | 1.27e+26 | 221.3 | 21.2 Trillion
10 Trillion | 1.30e+28 | 22515.9 | 216.2 Trillion

  * •

$\bigstar$This estimation comes from the study [37]$\bigstar$. Gopher is
another LLM study [312] used to compare.

Data for Instruction Fine-Tuning. The aforementioned data typically consists
of general text that is commonly used for pretraining PLMs or LLMs. However,
when transitioning from PLMs to LLMs, instruction data becomes crucial in
order to equip LLMs with the capability of following instructions effectively.
Unlike PLMs, which primarily focus on next-word prediction, LLMs place greater
emphasis on responding to specific instructions.

To illustrate, an instruction instance is presented in Figure 11. In this
example, the LLM is tasked with identifying chemical-disease relations and
understanding that its response should align with the given instruction,
rather than predicting the next word. By leveraging a sufficient amount of
instruction data for fine-tuning, an LLM can appropriately generate the
desired output, as demonstrated in Figure 11. This emphasizes the importance
of instruction-based training for LLMs to achieve accurate and contextually
relevant responses.

####  IV-E4 Summary

in Section IV-D and Section IV-E, we present a comprehensive overview of two
fundamental resources crucial for LLMs – the training framework and data.
Specifically, Section IV-D2 highlights compute-efficient and memory-efficient
methods, such as Parallelism, ZeRO, and Quantization, that have been proven to
substantially reduce the overall cost associated with LLM training or fine-
tuning. These cutting-edge technologies hold significant value as they
effectively lower the entry barrier for researchers and practitioners
interested in exploring the realm of LLMs. Building upon these advancements,
numerous training frameworks have emerged, offering integrated solutions that
encompass a wide range of acceleration techniques and enhanced support,
thereby facilitating more convenient and streamlined LLM training processes.

The choice of a suitable training framework holds great significance in
accelerating the development of LLMs. Among the available options, DeepSpeed-
chat is a training framework propsoed by Microsoft, which has gained
recognition. Additionally, PyTorch’s official distributed training tool,
Accelerate, offers stability and ease of use for small to medium-sized
training tasks. Another noteworthy open-source LLM training framework is
veGiantModel, developed by ByteDance, which provides valuable support in LLM
development. You can find more information about veGiantModel at their GitHub
repository161616https://github.com/volcengine/veGiantModel.

When it comes to the data used for training LLMs, the volume often surpasses
the capacity of human teams to manually perform quality checks. Consequently,
data collection processes heavily rely on heuristic rules for selecting data
sources and applying filters. In the context of LLM training, there are
various data challenges to address, including the high cost of Healthcare
data, near-duplicates, contamination in benchmark data, personally
identifiable information, and the mixture of domains during pre-training and
fine-tuning tasks.

Based on the above information, one of the primary concerns in developing an
LLM – the computational cost, is involved. By considering the training
framework, data requirements, and the size of the LLM itself, an estimation of
the overall computational cost can be obtained. We have summarized the
relevant computation costs from existing studies in Table VIII. Table IX comes
from the study [37], which estimates the relation among the model size, the
dataset size, and the training FLOPs when we need to train an LLM from
scratch. These data can serve as a helpful reference for those seeking to
estimate the expenses associated with LLM development.

TABLE X: The general evaluation of LLMs.

Categories | Studies |  Evaluation Tasks and Conclusions
---|---|---
Content generation | [313] |  Question answering. ChatGPT is a knowledgeable but inexperienced solver, and GPTs can achieve good accuracy in commonsense question answering.
[314] |  Affective computing. ChatGPT is a good generalist model without any specialized training.
[315] |  Text summarization. ChatGPT occasionally produces longer summaries than the input, and the fine-tuned BART performs better than the zero-shot ChatGPT.
[316] |  Dialogue. Claude and ChatGPT are optimized for chat applications.
[317] |  Translation. ChatGPT and GPT-4 achieve superior performance compared to commercial translation systems.
[318] |  Code synthesis. ChatGPT and GPT-4 achieve superior performance.
Logical reasoning | [319] |  Mathematical reasoning. ChatGPT performed much worse than high-ability students.
[320] |  Deductive, inductive and abductive reasoning. LLMs perform worse than the fine-tuned state-of-the-art model.

  * •

✰ The general evaluation mainly contains two categories: content generation
and logical reasoning.

TABLE XI: The Healthcare evaluation of LLMs.

Categories | Studies |  Models | Scenarios | #Num |  Conclusions
---|---|---|---|---|---
Medical Ex. | [321] |  ChatGPT | Primary Care | 674 |  Average performance of ChatGPT is below the mean passing mark in the last 2 years.
[322] |  ChatGPT | Medical licensure | 220 |  ChatGPT performs at the level of a third-year medical student.
[323] |  ChatGPT | Medical licensure | 376 |  ChatGPT performs at or near the passing threshold.
Medical Q&A. | [324] |  ChatGPT | Physician queries | 284 |  ChatGPT generates largely accurate information to diverse medical queries.
[325] |  ChatGPT, GPT-4, Bard, BLOOMZ | Radiation oncology | 100 |  Each LLM generally outperforms the non-expert humans, while only GPT-4 outperforms the medical physicists.
[103] |  ChatGPT, Claude | Patient-specific EHR | – |  Both models are able to provide accurate, relevant, and comprehensive answers.
[326] |  ChatGPT | Bariatric surgery | 151 |  ChatGPT usually provides accurate and reproducible responses to common questions related to bariatric surgery.
[327] |  ChatGPT | Genetics questions | 85 |  ChatGPT does not perform significantly differently than human respondents.
[328] |  ChatGPT | Fertility counseling | 17 |  ChatGPT could produce relevant, meaningful responses to fertility-related clinical queries.
[329] |  GPT-3.5, GPT-4 | General surgery | 280 |  GPT-3.5 and, in particular, GPT-4 exhibit a remarkable ability to understand complex surgical clinical information.
[330] |  GPT-3.5, GPT-4 | Dementia diagnosis | 981 |  GPT-3.5 and GPT-4 cannot outperform traditional AI tools in dementia diagnosis and prediction tasks.
Medical Gen. | [331] |  ChatGPT | Gastroenterology | 20 |  ChatGPT would generate relevant and clear research questions, but not original.
[332] |  ChatGPT, GPT-4 | Radiology report | 138 |  ChatGPT performs well and GPT-4 can significantly improve the quality.
Medical Ce. | [333] |  ChatGPT | Benchmark tasks | 34.4K |  Zero-shot ChatGPT outperforms the state-of-the-art fine-tuned models in datasets that have smaller training sets.
[334] |  ChatGPT | Clinical and research | – |  ChatGPT could potentially exhibit biases or be susceptible to misuse.

  * •

✰ The Healthcare evaluation of LLMs includes Medical examination (Ex.),
medical question answering (Q&A), medical generation (Gen.), and medical
comprehensive evaluation (Ce.).

##  V Evaluation Method

Presently, there is a wide range of LLMs available for general NLP tasks and
Healthcare applications. Selecting the appropriate model as a benchmark for
intelligent applications is of utmost importance. Consequently, evaluating the
performance of LLMs holds significant value for both the NLP and Healthcare
communities. According to the survey from the study [15], most LLMs were
evaluated based on downstream tasks, where the tasks can be categorized as
testing language and reasoning ability, and scientific knowledge. They found
that LLMs present notable proficiency in comprehending and generating human
language, facilitating interactive exchanges with users through dialogues.
This enables them to effectively address a wide array of Natural Language
Processing (NLP) tasks and furnish elucidative responses. Nevertheless, it is
important to note that their present capabilities do not categorize them as
comprehensive AI systems. They still face performance limitations,
particularly when compared to expert models, across multiple domains that
necessitate domain-specific knowledge. On the other hand, the state-of-the-art
LLMs demonstrate commendable performance in grasping general scientific
knowledge and are capable of generating open-ended responses to science-
related inquiries. Nevertheless, they are susceptible to errors, particularly
when tackling questions that necessitate intricate multi-step reasoning. The
exceptional proficiency in language presents a hurdle for users to accurately
evaluate the factual correctness of information, thereby giving rise to a
spectrum of ethical considerations.

In this section, we will begin by introducing studies on the evaluation of
general NLP tasks. Subsequently, we will review studies focusing on Healthcare
evaluation, discussing aspects such as robustness, bias, and ethics. Finally,
we will conclude by highlighting future directions for health evaluation and
providing a summary.

###  V-A General NLP tasks Evaluation

To provide a comprehensive exposition of LLM evaluation studies in NLP tasks,
we propose two evaluation categories in response to the need for enhanced
intelligence, namely content generation and logical reasoning. The typical
evaluation studies and their main conclusions are summarized in Table X. The
evaluation study for content generation is to generate answers for general NLP
tasks, such as question answering, affective computing, text summarization,
dialogue, translation, and code synthesis. The generation form typically
manifests as either natural language or code. Overall, the results demonstrate
that LLMs have made significant advancements in such general NLP tasks.

However, the question remains whether LLMs can still achieve superior
performance, considering that complex logical reasoning tasks demand planning,
abstraction, and inference abilities. Under this circumstance, several studies
for evaluating logical reasoning abilities are proposed. From the reasoning
form, they can be divided into deductive, inductive, abductive, and
mathematical views. The relevant results all show that LLMs do not perform
well in logical reasoning. For example, ChatGPT incorrectly answers almost all
questions about probability and statistics, permutation and combination, and
geometry [319]. The overall performance is much worse than high-ability
students for mathematical reasoning. Meanwhile, the study [320] provides a
comprehensive evaluation of LLMs for deductive, inductive and abductive
reasoning, and demonstrates LLMs perform worse than the fine-tuned state-of-
the-art model. Meanwhile, LLMs often exhibit noticeable logical flaws and
hallucinations, which pose significant challenges for the practical
application of LLMs in logical reasoning scenarios.

###  V-B Healthcare Evaluation

Different from general NLP tasks, the field of Healthcare is characterized by
its high level of specialization. Evaluating LLMs in this domain necessitates
assessing their capacity to comprehend and utilize medical knowledge and
terminology. The evaluation process may involve designing test cases tailored
to specific tasks and challenges within the medical field. According to the
different forms of evaluation, we categorize the current relevant work into
four folds: medical examination, medical question answering, medical
generation, and medical comprehensive evaluation, which are summarized in
Table XI. The medical examination form involves verifying model performance
through standard medical tests or examinations. Differently, medical question
answering involves utilizing questions posed or collected by human experts to
make assessments. Medical generation focuses on generating new medical
descriptions or knowledge based on a given input. The studies on medical
comprehensive evaluation aim to provide assessments across various application
scenarios rather than focusing on a single aspect.

In the form of medical examination, the study [321] evaluated the strengths
and weaknesses of ChatGPT in primary care using the Membership of the Royal
College of General Practitioners Applied Knowledge Test (AKT). It is observed
that ChatGPT’s average performance (60.17%) is below the mean passing mark in
the last 2 years (70.42%), demonstrating further development is required to
match the performance of qualified primary care physicians. The study [322]
evaluated ChatGPT’s performance on the medical licensing exams utilizing
AMBOSS171717https://www.amboss.com/ and the National Board of Medical
Examiners (NBME), which shows that ChatGPT performs at the level of a third-
year medical student on the question sets examined and its responses to
questions provide interpretable context to justify models written response in
most cases. Similarly, the study [323] tested the performance characteristics
of ChatGPT on USMLE. They certified that ChatGPT is able to perform several
intricate tasks relevant to handling complex medical and clinical information,
as ChatGPT performed at or near the passing threshold of 60% accuracy.

The ability of QA plays a crucial role in the application of models, and as a
result, there have been numerous studies focusing on the evaluation of medical
question answering. To explore the accuracy and completeness of ChatGPT for
medical queries, the study [324] collected 284 medical questions from 33
physicians across 17 specialties. After that, these physicians graded ChatGPT-
generated answers to these questions for accuracy, showing that ChatGPT
achieved relatively high accuracy and completeness scores. Utilizing 100
multiple-choice questions on radiation oncology physics created by an
experienced medical physicist, the study [325] investigated LLMs’ capacity in
answering radiation oncology physics questions. Four LLMs (ChatGPT, GPT-4,
Bard181818https://bard.google.com/, and
BLOOMZ191919https://github.com/bigscience-workshop/xmtf) are utilized to
compare with medical physicists and non-experts. The results demonstrate that
all these LLMs generally outperform the non-expert humans and only GPT-4
outperforms the medical physicists. However, it is not allowed for GPT-4 to
improve performance when scoring based on a majority vote across trials, while
a team of medical physicists is able to greatly outperform GPT-4 using a
majority vote. The study [325] investigated the use of LLMs (ChatGPT and
Claude) for patient-specific question answering from EHRs. On both experiment
settings of one question per session and one topic per session, ChatGPT and
Claude are able to provide accurate, relevant, and comprehensive answers to
general questions, specific questions, and nonanswerable questions. To examine
the accuracy and reproducibility of LLMs in answering patient questions
regarding bariatric surgery, the study [326] gathered 151 questions from
nationally regarded professional societies and health institutions as well as
Facebook support groups. Using ChatGPT, accurate and reproducible responses to
common questions could be provided. The study [327] assessed ChatGPT in the
field of genetics involving 13,636 responses to 85 questions. Although ChatGPT
is significantly better on memorization-type questions versus critical-
thinking questions, it does perform significantly differently than human
respondents. And it would generate plausible explanations for both correct and
incorrect answers. The study [328] explored ChatGPT’s ability for fertility
counseling and only 6.12% ChatGPT factual statements were categorized as
incorrect. The study showcases the capacity of LLMs to generate pertinent and
meaningful responses to clinical queries related to fertility. Nevertheless,
there are certain limitations to consider, including the challenges in
providing reliable source citations and the unpredictable potential for
generating fabricated information. The study [329] aimed to assess the
performance of GPT-3.5 and GPT-4 in understanding complex surgical clinical
information and its potential implications for surgical education and
training. GPT-3.5 achieved an overall accuracy of 46.8%, while GPT-4
demonstrated a significant improvement with an overall accuracy of 76.4%.
GPT-3.5 and GPT-4 specifically have a remarkable ability to understand complex
surgical clinical information. The study [330] explored the potential of
GPT-3.5 and GPT-4 to surpass traditional AI tools in dementia diagnosis. The
experimental results, obtained from two real clinical datasets, indicate that
while LLMs show promise for future advancements in dementia diagnosis. They
currently do not outperform traditional AI tools in terms of performance.

The evaluation of medical generation can provide further insights into the
level of control that LLMs have over medical knowledge. It is significant to
pinpoint the most pressing and important research questions. To this end, the
study [331] evaluated the potential of chatGPT for identifying research
priorities in gastroenterology from four key topics. Several experienced
experts reviewed and rated the generated research questions. It seems ChatGPT
would generate relevant and clear research questions. However, the generated
questions were not considered original. The study [332] investigated the
feasibility of using ChatGPT and GPT-4 to translate radiology reports into
plain language. According to the evaluation by radiologists, ChatGPT performs
well and can successfully translate radiology reports into plain language with
an average score of 4.27 in the five-point system. Further, GPT-4 can
significantly improve the quality of translated reports.

Several studies evaluate the comprehensive capability of LLMs. For example,
the study [333] provides a comprehensive evaluation of ChatGPT’s zero-shot
performance on various benchmark biomedical tasks, i.e., relation extraction,
document classification, question answering, and summarization. Zero-shot
ChatGPT achieves comparable performance to fine-tuned generative transformers
such as BioGPT and BioBART. Additionally, when evaluated on datasets with
limited training data, zero-shot ChatGPT outperforms these fine-tuned models.
These results indicate that ChatGPT exhibits a high degree of specialization
even within the biomedical domain. The study [334] conducted a concise
investigation to assess the potential applications of ChatGPT in four clinical
and research scenarios: support of clinical practice, scientific production,
misuse in medicine and research, and reasoning about public health topics. The
study draws the following conclusions: ChatGPT demonstrates the capability to
offer valuable suggestions; it accurately identifies the context and
summarizes findings; potential misuse is identified; and ChatGPT exhibits
significant potential for expediting scientific progress.

###  V-C Evaluation of Robustness, Bias, and Ethics

To assess how well a model performs when faced with uncertainties,
perturbations, or unexpected inputs, researchers have been studying robustness
evaluation techniques. For instance, in the field of general NLP tasks,
studies have explored the robustness of LLMs in areas such as semantic parsing
[335] and vision-language tasks [336]. In the Healthcare domain, the
evaluation of LLMs’ robustness is relatively limited. One notable example is
the evaluation of ChatGPT’s robustness in translating radiology reports [332].
In this work, the original radiology reports were divided into 25 key
information points, and the correctness and completeness of each point were
evaluated in a point-by-point manner in the translated reports. The overall
translation quality was found to be satisfactory for only 55.2% of the
translated points, indicating ample room for improvement in the robustness of
LLMs in Healthcare settings.

LLMs are generated through training on extensive text datasets, which can
inherently contain various biases and imbalances. When the model is
consistently exposed to specific biases or particular points of view during
training, it tends to learn and reflect those biases, leading to biased
outputs during text generation. In the manual evaluation process, the presence
of biases can also arise due to the diverse academic backgrounds and
perspectives of the experts involved. Each expert may have their own
subjective interpretation or evaluation criteria, which can introduce
deviations in the evaluation results [323].

Furthermore, during the evaluation process, LLMs may require the uploading of
personal privacy data, such as patient-specific EHR [325]. This introduces a
significant privacy risk that demands careful attention. Consequently, ethic
issues related to data privacy and protection [324, 333] must be thoroughly
considered in the evaluation

###  V-D Future Directions for Health Evaluation

The study [15] found that present evaluation methodologies heavily rely on
prompt engineering and established benchmark datasets. Different prompt
formulations can lead to contrasting evaluation outcomes. Furthermore, the
assessment of expert systems frequently hinges on utilizing (in-domain)
datasets that were originally employed for training those systems. An
ambiguity persists regarding potential inadvertent exposure of the scrutinized
data, such as publicly available datasets and established scientific
knowledge, during the training of Large Language Models (LLMs). These aspects
could introduce bias into the comparison between LLMs and their corresponding
baselines, impeding a fair assessment.

According to the current studies of Healthcare evaluation, we conclude the
following four future directions.

Increase the evaluation of faithfulness. Healthcare professionals and patients
place significant trust in the accuracy and reliability of information
provided by LLMs. However, due to the unique nature of the medical domain,
there is a risk that LLMs may generate false knowledge or hallucinations,
which could potentially lead to serious accidents or harm. Therefore,
evaluating the faithfulness of LLMs becomes crucial in order to identify
instances where these models may generate hallucinations and mitigate their
impact.

Towards comprehensive and multitask evaluation. The current evaluation
practices predominantly concentrate on assessing the performance of LLMs on
one specific medical task, which might not provide a comprehensive
understanding of their capabilities across the entire medical applications.
Consequently, there is a clear need for a multitask evaluation system that can
comprehensively evaluate the performance of LLMs across various medical tasks.

Towards multi-dimensional evaluation. While current evaluation efforts have
primarily centered around accuracy, there is a growing recognition of the need
for a multidimensional evaluation framework. It should consider various
aspects beyond accuracy, such as the correctness of interpretation,
robustness, hallucination ratio, content redundancy, biased description, and
ICL capability.

Increase privacy protection in the evaluation process. Medical applications
inherently involve sensitive data privacy concerns that surpass those of other
NLP tasks. Consequently, safeguarding privacy during the evaluation process
becomes of utmost importance. One potential solution to address this challenge
is the adoption of federated learning approaches [337], which enable the
implementation of large-scale evaluation systems while preserving privacy.

###  V-E Summary

In conclusion, while LLMs demonstrate strong performance in general NLP tasks,
they often fall short when it comes to tackling complex logical reasoning
problems. For Healthcare evaluation, LLMs tend to perform below or just meet
the threshold in medical examination scenarios. For the medical question
answering part that has the most studies, LLMs exhibit underperformance in
genetics questions and dementia diagnosis. For medical generation and
comprehensive evaluation, LLMs usually perform well and have a positive impact
despite the existence of non-original generations, biases, or instances of
misuse. From these evaluation studies, we have discovered that LLMs hold
significant potential for various applications in the health field. However,
there are several pressing issues that need to be addressed to enhance their
utilization in this domain.

##  VI Improving Fairness, Accountability, Transparency, and Ethics

Fairness, accountability, transparency, and ethics are four important concerns
in the AI domain. According to the study [338], fairness holds paramount
significance in guaranteeing that AI does not perpetuate or exacerbate
established societal disparities; Accountability plays an important role in
ensuring that individuals responsible for the conception and execution of AI
can be held answerable for their decisions; Transparency assumes a critical
role in ensuring that AI remains open to scrutiny and amenable to audits for
possible biases or inaccuracies; Ethics, similarly, assumes a pivotal role in
guaranteeing that AI is constructed and utilized in manners that align with
prevailing social values and norms.

In the Healthcare domain, we believe that these four aspects are even more
critical because the primary focus is on patient well-being and safety. In
this context, the utmost importance lies in ensuring patients receive optimal
Healthcare marked by equitable access to medical services. Additionally, the
transparent and trustworthy nature of Healthcare decisions, the accountability
in delivering accurate medical diagnoses and treatments, the safeguarding of
patient confidentiality, and the adherence to elevated ethical standards
emerge as distinct and noteworthy considerations, setting Healthcare apart
from AI applications in other domains and more.

In the following subsections, we will survey the common fairness,
accountability, transparency, and ethics issues related to using AI for
Healthcare. Then, we will propose possible mitigation for these issues.

###  VI-A Fairness

Fairness within the context of LLMs and NLP refers to the principle of
equitably treating all users and preventing any form of unjust discrimination.
This essential concept revolves around the mitigation of biases, aiming to
guarantee that the outcomes produced by an AI system do not provide undue
advantages or disadvantages to specific individuals or groups. These
determinations should not be influenced by factors such as race, gender,
socioeconomic status [17], or any other related attributes, e.g., different
input languages [339] and processing tasks [340], striving for an impartial
and balanced treatment of all users. This fundamental tenet aligns with the
broader objective of promoting equality and inclusivity within the
applications of LLMs and NLP.

In an empirical study, the study [340] found that PLMs may generate biased
outcomes given different tasks, prompts, and label word selection methods.
They evaluated both small and large versions of four PLMs, showing that PLMs
can yield huge accuracy gaps in sentiment analysis and emotion detection
tasks, even though the prompts, label word selection, and input text have been
well controlled. This finding goes against human intuition because
cognitively, sentiment and emotions are divisions of subjective expressions
into different granularities. Polarized sentiment can be thought of as a
2-dimensional summary of positive and negative emotions. The accuracy gap
between sentiment and emotion classification tasks indicates that the
performance of PLMs can be significantly impacted by how the label space is
divided in a specific task. For sentiment analysis whose label space is evenly
divided into two classes, e.g., positive and negative classes, PLMs tend to
achieve better performance than the tasks whose label space is unevenly
divided, e.g., angry, fear, sad, and joyful emotions.

The study [39] noticed that when comparing sentiment scores, the fine-tuned
LLaMA 2-Chat exhibits a more positive sentiment compared to pretrained
versions, whereas ChatGPT tends to generate responses with a more neutral
sentiment. In terms of gender, LLMs tend to express a more positive sentiment
towards American female actresses than male actors. Regarding race, Asian
Americans and Hispanic/Latino Americans tend to have relatively higher
sentiment scores compared to other racial subgroups. In the religious ideology
domain, Islam and Sikhism groups display the most significant increase in
sentiment scores after fine-tuning. In the political ideology domain, both
Liberalism and Conservatism groups tend to have the most positive sentiment
scores, while Fascism group scores are predominantly negative. Lastly, in the
profession domain, there is notably positive sentiment towards occupational
categories like “Corporate titles” and “Computer”, while sentiment is most
neutral towards “Professional driver types”.

The biases from LLMs can be attributed to the uneven distribution of
demographic attributes in pre-training corpora [39]. Such an argument also
holds for the Healthcare sector [341]. As an example, CNNs trained on publicly
accessible chest X-ray datasets tend to exhibit underdiagnosis tendencies in
marginalized communities, including female patients, Black patients, Hispanic
patients, and those covered by Medicaid insurance [342]. These specific
patient groups often experience systemic underrepresentation within the
datasets, resulting in biased algorithms that may be susceptible to shifts in
population demographics and disease prevalence. Furthermore, several global
disease classification systems display limited intra-observer consensus,
implying that an algorithm trained and assessed in one country may undergo
evaluation under a dissimilar labeling framework in another country [343,
344].

Current common practices to improve AI fairness in the Healthcare domain focus
on pre-processing, in-processing, and post-processing [341]. Importance
weighting is a pre-processing technique, which involves adjusting the
significance of less frequent samples from protected subgroups. Similarly,
resampling endeavors to rectify sample-selection bias by acquiring more
equitable subsets of the initial training dataset and can be naturally
employed to address the under-representation of specific subgroups. In the
case of tabular-structured data, methods like blinding, data transformation,
and others can be utilized to directly remove proxy variables that encode
protected attributes.

To alleviate the impact of confounding variables, an anti-discrimination
component (an in-processing technique) can be integrated into the model to
discourage the learning of discriminatory attributes related to a protected
attribute. For example, in the case of a logistic regression model,
modifications can be made to include anti-discrimination elements by
evaluating the covariance between the protected attribute and the signed
distance from the sample’s feature vectors to the decision boundary. Another
approach involves adjusting the parameters of the decision boundary to enhance
fairness (by minimizing disparate impact or mistreatment), while still
adhering to accuracy constraints [345]. Deep learning models, e.g., CNNs with
adversarial-loss terms, render the internal feature representations invariant
to variations in protected subgroups [346]. Tuning loss weights for different
classes is also common for unbalanced label learning [347]. Furthermore,
adjustments to the stochastic gradient descent technique can be implemented to
incorporate fairness constraints within online learning frameworks [348].

Post-processing involves methodologies that alter the output of a trained
model, such as probability scores or decision thresholds, to adhere to group
fairness criteria. In aiming for equalized odds, one approach is to establish
appropriate thresholds for each group, ensuring that the model attains a
consistent operating point across all groups. However, in scenarios where the
receiver operating curves do not intersect or where the desired operating
point does not align with an intersection point, implementing this strategy
necessitates deliberately degrading performance for specific subgroups using a
randomized decision rule. Essentially, this signifies that the model’s
performance for certain groups may have to be intentionally diminished to
fulfill the criteria of equalized odds.

For LLMs, bias mitigation methods are frequently studied in the context of
instruction fine-tuning and prompt engineering [349]. The representative
technique for instruction fine-tuning is RLHF. In the case of InstructGPT,
GPT-3 is refined through a process involving RLHF, specifically aimed at
adhering to human instructions. The procedure involves three sequential steps:
firstly, gathering human-authored demonstration data to guide GPT-3’s
learning; secondly, assembling comparative data consisting of model-generated
outputs assessed by annotators to construct a reward model that predicts
outputs preferred by humans; and lastly, fine-tuning policies based on this
reward model using the Proximal Policy Optimization algorithm [350].

###  VI-B Accountability

LLMs possess the propensity to magnify the inherent social biases embedded
within their training data because they can generate hallucinatory or
counterfactual information and present a deficiency in robustness. These
limits imply that LLMs are susceptible to perturbations and deviations from
their intended performance, particularly when exposed to diverse inputs or
scenarios. Thus, ensuring accountability emerges as a pivotal concern when
integrating LLMs within the Healthcare domain.

The study [351] identify two primary factors that significantly contribute to
the performance and instances of hallucination in generative Large Language
Models (LLMs). Firstly, a major influence is attributed to the model’s
memorization of the training data. Additionally, the authors demonstrate that
named entity IDs serve as “indices” for accessing the memorized data.
Secondly, the authors illustrate that LLMs employ an additional heuristic
based on corpus-derived patterns involving word frequencies. They provide
evidence that NLI test samples deviating from these patterns result in
significantly lower scores compared to those adhering to them. Hallucinations
are not a unique flaw of LLMs They are also common in large vision language
models (LVLMs). The study [352] further confirmed that objects that have a
high frequency of occurrence in the visual instructions or co-occur with the
objects present in the image are evidently more susceptible to hallucination
by LVLMs.

Generated counterfactual speech presents an additional obstacle to accountable
AI. In the evaluation conducted by the study [353], ChatGPT was evaluated
using fact-based question-answering datasets, revealing that its performance
did not exhibit enhancements in comparison to earlier versions. Consequently,
the reliability of ChatGPT in tasks necessitating faithfulness is called into
question. For instance, its potential fabrication of references in the context
of scientific article composition [354] and the invention of fictitious legal
cases within the legal domain [355] accentuate the potential risks associated
with its use in critical domains.

The research [356] uncovered a diverse range of viewpoints among Healthcare
professionals regarding the impact of AI on their workload and decision-making
processes. Certain practitioners believed that AI had the potential to
alleviate their workload, enhance clinical decision-making, and ultimately
enhance patient safety by assisting in diagnostics. Conversely, other
practitioners voiced apprehensions about a heightened workload, encompassing
the effort required to learn and manage the technology alongside patient care,
as well as potential risks to patients if the AI system provided unsuitable
recommendations. The study also underscored the participants’ anxieties about
the diminishing interpersonal connection following the integration of AI.

Numerous studies have noted that the apparent scientific style of language
used by ChatGPT can mislead human observers regarding the reliability of its
outputs [357, 358]. The study [15] contended that enabling users to access
human-generated source references is crucial for enhancing the reliability of
the model’s responses. The study [359] advocated for the involvement of both
AI developers and system safety engineers in evaluating the moral
accountability concerning patient harm. Additionally, they recommend a
transition from a static assurance model to a dynamic one, recognizing that
ensuring safety is an ongoing process and cannot be entirely resolved during
the initial design phase of the AI system prior to its deployment.

The study [356] proposed a solution to tackle the issue of accountability,
advocating for the education and training of prospective AI users to discern
the appropriateness of relying on AI recommendations. However, imparting this
knowledge to practitioners demands a considerable investment of effort.
Healthcare professionals frequently grapple with overwhelming workloads and
burnout, making comprehensive training on AI a significant challenge.
Moreover, not all Healthcare practitioners possess adequate statistical
training to comprehend the underlying mechanics of AI algorithms. In addition
to education, the study [356] recommended the establishment of policies and
mechanisms to ensure the protection of both clinicians and AI within the
Healthcare domain.

###  VI-C Transparency

The limited transparency of neural networks has been widely criticized,
presenting significant obstacles to their application in the Healthcare
domain. LLMs and PLMs are complex neural network models, which further
exacerbate the challenges associated with interpretability. In recent years,
there have been efforts to understand the inner workings of PLMs in Healthcare
contexts. Probing PLMs have been extensively employed to uncover the
underlying factors contributing to their performance [360]. For example, [361]
examined PLMs’ disease knowledge, while [362] conducted in-depth analyses of
attention in protein Transformer models, yielding valuable insights into their
mechanisms.

In the general meaning learning domain, a transparent model is typically
characterized by decision-making processes akin to those of white-box models,
e.g., decision tree-based models or linear regression models. It often
encompasses post hoc explanations [363], model-specific explanations [364] or
model-agnostic explanations [365]. Sometimes, the explanation insights are
derived from feature maps [366], generated natural language [367], factual and
counterfactual examples [368], or decision-making evidence [369].

LLMs normally rely on Transformer structures. However, the conventional
Transformer is not intrinsically explainable, as it is a stack of multiple
layers of multi-head attention, skip connections, and non-linear
transformation. The study [370] introduced an innovative approach for
assessing relevancy in Transformer networks in the computer vision domain.
Their method involves assigning local relevance using the Deep Taylor
Decomposition principle and subsequently propagating these relevance scores
through the network’s layers. This propagation incorporates attention layers
and skip connections, introducing a novel challenge compared to existing
methods. The authors’ solution is grounded in a unique formulation that has
demonstrated the ability to preserve the overall relevancy across different
layers of the network.

The study [363] introduced an innovative method accompanied by quantitative
metrics aimed at mitigating the limitations observed in existing post hoc
explanation approaches, as outlined in the literature. These drawbacks include
reliance on human judgment, the necessity for retraining, and issues related
to data distribution shifts during the occlusion of samples. The method
proposed in this study allows for a quantitative assessment of
interpretability methods without the need for retraining and effectively
addresses distribution shifts between training and evaluation sets.
Furthermore, the authors have developed novel metrics and indices to
quantitatively evaluate time-series interpretability methods, offering a
comprehensive evaluation of how closely an interpretability method aligns with
the learned representation of the model in focus.

In the era of LLMs, CoT prompting [33] has emerged as a potential method for
providing a certain level of interpretability by generating reasoning steps.
The technique empowers LLMs to break down complex, multi-step problems into
more manageable intermediate steps. This enables the allocation of additional
computational resources to problems demanding deeper reasoning steps.
Moreover, it offers a transparent view of the LLM’s behavior, shedding light
on its potential process of arriving at a specific answer and offering
insights for identifying and rectifying errors in the reasoning path.
Essentially, a chain of thought can be perceived as a systematic, step-by-step
thought process leading to the derivation of an answer. However, this approach
faces two primary challenges: the high cost of annotations required for CoT
and the evaluation of interpretability. Acquiring demonstrations with
annotated reasoning steps is an expensive task, particularly in professional
fields such as Healthcare. Additionally, evaluating the generated reasoning
results as explainable justifications and ensuring their usability pose
significant challenges.

###  VI-D Ethics

The ethical concerns about using LLMs in the Healthcare domain have been
widely discussed. The study [29] argued that a primary concern relates to the
potential perpetuation of misinformation and biases. Furthermore, the
inaccurate results from LLMs inhibit their autonomous deployment, although
their utilization in an assisting capacity could significantly enhance
efficiency. Domain-specific fine-tuning may enhance their performance, as
evidenced by variants like PubMedBERT and BioBERT derived from the BERT model.
Addressing accountability issues involves ensuring that clinicians and
researchers utilizing these tools take responsibility for the generated
output. Lastly, evaluating clinical interventions utilizing LLMs should
ideally involve randomized controlled trials to assess their impact on
mortality and morbidity. However, determining the appropriate benchmark for
such costly and risky trials remains an open question.

Healthcare LLMs typically possess a wide range of patient characteristics,
including clinical measurements, molecular signatures, demographic
information, and even behavioral and sensory tracking data. It is crucial to
acknowledge that these models are susceptible to the problem of memorizing
training data and simply reproducing it for users. As mentioned in Section
IV-E3, EHRs serve as important training data for Healthcare LLMs, alongside
public scientific literature and web data. However, it is worth noting that
some EHRs remain private due to organizations’ concerns about data exposure.
For instance, clinical records may contain sensitive information such as
patient visits and medical history, and exposing such data could lead to
physical and mental harm to patients. It is important to recognize that de-
identification techniques employed in EHR records (e.g., MIMIC III) may not
always guarantee complete safety. Recent studies have shown that there can be
instances of data leakage from PLMs in the general domain, allowing for the
recovery of personal health information from models trained on such data
sources [371, 372]. Additionally, approaches such as KART [373] have been
proposed to assess the vulnerability of sensitive information in pre-trained
biomedical language models using various attack strategies. The Federated
Learning [374] is a promising technology to alleviate such a problem.

The study [15] brought attention to a pressing issue regarding the potential
misuse of AI-generated content for training subsequent models. They emphasized
that due to the probable presence of biases in content generated by LLMs, any
sequential models trained using this content could inadvertently inherit and
perpetuate those biases. This highlights a significant ethical concern, as the
biases within AI-generated content could be inadvertently propagated,
emphasizing the critical importance of addressing bias mitigation strategies
during training processes involving LLM-generated data.

Complex software designed to aid in the diagnosis, prevention, monitoring,
prediction, prognosis, treatment, or alleviation of diseases falls under the
classification of a medical device, necessitating adherence to regulatory
controls. These controls encompass the development of these tools within a
quality management system to ensure their reliability and safety. Within the
medical domain, incorporating LLMs raises notable ethical considerations due
to their intricate nature. The study [375] highlighted that in the European
Union, stringent post-market surveillance and clinical follow-up are mandated,
presenting specific challenges when applied to LLMs. Notably, LLMs lack
inherent quality assurance directly from their developers, making their
integration as external “plug-in” components of medical devices, for instance,
through an API, impractical. This raises ethical concerns, as the use of LLMs
in critical medical contexts necessitates stringent quality validation and
assurance to ensure patient safety and uphold ethical standards within the
Healthcare domain. The study [376] believed that LLMs are frequently trained
using extensive data sources, which, given their opaque nature, raises
apprehensions regarding potential violations of existing intellectual property
rights. This concern, coupled with the limited transparency of current LLMs,
has recently influenced amendments to the proposed Artificial Intelligence Act
within the European Union. This amendment mandates that companies utilizing
generative AI tools must provide disclosures concerning any copyrighted
material employed in the development of their systems.

##  VII Future work and Conclusion

We have discussed fairness, accountability, transparency, and ethics for
Healthcare in Section VI. Further from technology aspects, we summarize four
points that are the most significant for Healthcare LLMs, including medical
knowledge enhancement, integration with the Healthcare process, effective
interaction with patients and doctors, hallucinations, misunderstandings, and
prompt brittleness.

###  VII-A Future Work

####  VII-A1 Medical knowledge enhancement

the integration of medical knowledge into PLMs has been a prominent topic of
research for several years [160, 377]. In the era of PLMs, there have been
attempts to consider PLMs as a form of soft knowledge base capable of
capturing knowledge. Studies [378, 379] have explored explicit methods to
inject knowledge into PLMs. In the knowledge-intensive Healthcare domain,
models infused with medical knowledge hold tremendous potential for future
applications. An example study [380] illustrates the integration of domain-
specific knowledge (UMLS) into the pre-training process of PLMs. This
integration is accomplished through a knowledge augmentation strategy that
connects words sharing the same underlying “concept” in UMLS. Furthermore, the
study leverages the semantic type knowledge available in UMLS to generate
input embeddings with clinical significance. Another study [381] employs
mention-neighbour hybrid attention to learn heterogeneous entity information.
It infuses the semantic representations of entity types as external medical
knowledge, enhancing the model’s capabilities. While general PLMs like GPT-4
exhibit significant competence in answering medical questions, explicit
injection of medical knowledge remains a challenge, especially when dealing
with smaller-sized PLMs [234, 187]. For both LLMs and PLMs, the most common
approach to injecting medical knowledge is fine-tuning the model parameters
using medical data. However, a major drawback of this method is that the
knowledge remains fixed once training is complete, making it difficult to
incorporate specific knowledge or update the overall knowledge without
retraining. Retrieval-based LLMs [382] may offer a solution to these
challenges, allowing for more flexible and updatable knowledge integration.

####  VII-A2 Integration with Healthcare process

is the application of artificial intelligence in the medical field just an
“old myth”, or can it really change the status quo? Clearly, although current
AI solutions are fragmented and mostly experimental without widespread
adoption, there exist such problems because we believe they are mainly caused
by the following three reasons based on the existing study [383]. First, it is
difficult to integrate with existing hospital information technology (IT)
systems. AI solutions require large amounts of data for training, and most of
this data is currently stored in hospitals’ own information systems.
Retrieving and integrating this data requires upgrades and modifications to
existing systems, which will have an impact on hospitals’ daily operations. In
addition, different hospitals use different data formats and standards, lack
standardized interfaces, and have relatively complex workflows in the
Healthcare domain. AI systems find it difficult to adapt to different
interfaces, which also increases the difficulty of integration. Second,
fragmentation of IT systems due to hospital consolidations. With the increase
in hospital mergers and acquisitions, the original hospitals may use
completely different IT systems. After consolidation, it is necessary to unify
their respective clinical and management systems, which requires huge
investment and a long transition period. Introducing new AI systems during
this process will face great technical challenges. Third, regulations are
unclear and challenging. Currently, laws and regulations for AI medical
applications are incomplete. Key issues such as information security, privacy
protection, and liability attribution lack clear provisions. In addition,
regulations differ across countries and regions. These will bring
uncertainties to the development and application of AI systems. At the same
time, the application of AI in the medical industry involves complex ethical
issues that are also difficult to resolve.

####  VII-A3 Effective Interaction with Patients and Doctors

despite the existing fluency of LLMs in human communication, the unique nature
of the medical domain necessitates specific requirements for the interaction
between LLMs and their users, namely doctors and patients. These requirements
include the ability of LLMs to proactively inquire about symptoms, pose
targeted questions, and effectively manage the pace and flow of conversations.
Additionally, it is desirable for LLMs to perceive and appropriately address
patient emotions such as anxiety and fear, thereby providing suitable
emotional support. Moreover, an augmentation to the dialogue system could
involve incorporating a virtual human design. This design would enable the
model to portray a doctor’s image, encompassing elements such as tone, speech
speed, and facial expressions, with the intention of enhancing rapport in
communication. Simultaneously, we aim to establish a continuous learning
mechanism that enables the model to learn from doctor-patient dialogues and
continually enhance its communication capabilities. This entails automatic
pragmatic learning, ensuring the model optimizes its ability to effectively
communicate with doctors and patients, ultimately leading to more successful
interactions.

####  VII-A4 Hallucinations, Misunderstandings and Prompt Brittleness

hallucinations, misunderstandings, and prompt brittleness are three
fundamental challenges encountered by both general LLMs and Healthcare LLMs.
Hallucinations refer to instances where LLMs generate responses that lack
coherence or relevance to the given input. These “hallucinations” can pose
significant issues, particularly when users are unfamiliar with the discussed
concepts, as they may struggle to identify the inaccuracies in the model’s
output. Misunderstandings represent a misalignment problem where the behavior
of LLMs fails to align with human values, objectives, and expectations. In
other words, LLMs may provide incorrect actions or responses despite receiving
proper instructions. Prompt brittleness signifies that even minor
modifications to the input prompt can yield dramatically different outputs, as
first observed in the study by [384]. In the Healthcare context, these issues
could lead to unacceptable consequences. While additional instructions or
reinforcement learning from human feedback can partially mitigate these
challenges, they do not fully satisfy the stability requirements within the
Healthcare domain. Regarding prompt brittleness, the current state of prompt
engineering heavily relies on extensive experimentation, with a limited
theoretical understanding of why a specific phrasing or formulation of a task
is more sensible beyond achieving improved empirical results. Consequently,
the development of LLMs that exhibit robustness to different prompt styles and
formats remains an unsolved problem.

###  VII-B Conclusion

Recently, there has been a growing interest in LLMs and their potential
applications across various fields. In this study, we aim to provide a
comprehensive survey specifically focusing on Healthcare LLMs. Our survey
encompasses an extensive examination of data, technologies, applications,
fairness, accountability, transparency, ethics, and limitations associated
with Healthcare LLMs. A noteworthy transformation has been observed from
Discriminative AI to Generative AI, as well as from model-centered to data-
centered approaches, marking a significant shift from PLMs to LLMs. This
transition has enabled Healthcare LLMs to support more advanced applications
beyond conventional NLP-based fundamental tasks. Consequently, the emergence
of these advanced applications has inspired numerous related studies.

To facilitate the development of Healthcare LLMs, various instruction datasets
and training and inference technologies have been proposed. These resources
have played a crucial role in accelerating the progress of LLMs, particularly
within the Healthcare domain. Our objective is to summarize these existing
resources, providing valuable support to researchers intending to embark on
the development of their own Healthcare LLMs.

However, despite the opportunities presented by Healthcare LLMs, several
significant challenges persist, impeding their implementation in Healthcare
settings. Issues pertaining to interpretability, privacy protection, medical
knowledge enhancement, integration with Healthcare processes, and effective
interaction with patients and doctors pose substantial obstacles. These
challenges hinder the translation of innovative LLMs into practical adoption
within the Healthcare field. Consequently, physicians and other Healthcare
professionals must carefully consider the potential benefits and limitations
associated with LLMs as they navigate the selection and integration of these
models into their medical practice.

## Acknowledgments

This work has been supported by the National Research Foundation Singapore
under AI Singapore Programme (Award Number: AISG-GC-2019-001-2A and
AISG2-TC-2022-004); The RIE2025 Industry Alignment Fund (I2101E0002 – Cisco-
NUS Accelerated Digital Economy Corporate Laboratory).

## References

  * [1] M. Ge, R. Mao, and E. Cambria, “A survey on computational metaphor processing techniques: From identification, interpretation, generation to application,” _Artificial Intelligence Review_ , 2023.
  * [2] R. Liu, R. Mao, A. T. Luu, and E. Cambria, “A brief survey on advances in coreference resolution,” _Artificial Intelligence Review_ , 2023.
  * [3] R. Mao, K. He, X. Zhang, G. Chen, J. Ni, Z. Yang, and E. Cambria, “A survey on semantic processing techniques,” _Information Fusion_ , p. 101988, 2023.
  * [4] X. Zhang, R. Mao, and E. Cambria, “A survey on syntactic processing techniques,” _Artificial Intelligence Review_ , vol. 56, p. 5645–5728, 2023. [Online]. Available: https://link.springer.com/article/10.1007/s10462-022-10300-7
  * [5] J. Li _et al._ , “Recent advances in end-to-end automatic speech recognition,” _APSIPA Transactions on Signal and Information Processing_ , vol. 11, no. 1, 2022.
  * [6] Y. Peng, S. Dalmia, I. Lane, and S. Watanabe, “Branchformer: Parallel mlp-attention architectures to capture local and global context for speech recognition and understanding,” in _International Conference on Machine Learning_. PMLR, 2022, pp. 17 627–17 643.
  * [7] R. Mao, X. Li, K. He, M. Ge, and E. Cambria, “Metapro online: A computational metaphor processing online system,” in _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)_ , 2023, pp. 127–135.
  * [8] H. Bao, K. He, X. Yin, X. Li, X. Bao, H. Zhang, J. Wu, and Z. Gao, “Bert-based meta-learning approach with looking back for sentiment analysis of literary book reviews,” in _Natural Language Processing and Chinese Computing: 10th CCF International Conference, NLPCC 2021, Qingdao, China, October 13–17, 2021, Proceedings, Part II 10_. Springer, 2021, pp. 235–247.
  * [9] R. Mao and X. Li, “Bridging towers of multi-task learning with a gating mechanism for aspect-based sentiment analysis and sequential metaphor identification,” _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 35, no. 15, pp. 13 534–13 542, 2021.
  * [10] K. He, Y. Huang, R. Mao, T. Gong, C. Li, and E. Cambria, “Virtual prompt pre-training for prototype-based few-shot relation extraction,” _Expert Systems with Applications_ , vol. 213, p. 118927, 2023.
  * [11] K. He, R. Mao, T. Gong, E. Cambria, and C. Li, “Jcbie: a joint continual learning neural network for biomedical information extraction,” _BMC bioinformatics_ , vol. 23, no. 1, pp. 1–20, 2022.
  * [12] Y. Huang, K. He, Y. Wang, X. Zhang, T. Gong, R. Mao, and C. Li, “Copner: Contrastive learning with prompt guiding for few-shot named entity recognition,” in _Proceedings of the 29th International conference on computational linguistics_ , 2022, pp. 2515–2527.
  * [13] S. Ranathunga, E.-S. A. Lee, M. Prifti Skenduli, R. Shekhar, M. Alam, and R. Kaur, “Neural machine translation for low-resource languages: A survey,” _ACM Computing Surveys_ , vol. 55, no. 11, pp. 1–37, 2023.
  * [14] S. Agrawal, C. Zhou, M. Lewis, L. Zettlemoyer, and M. Ghazvininejad, “In-context examples selection for machine translation,” _arXiv preprint arXiv:2212.02437_ , 2022.
  * [15] R. Mao, G. Chen, X. Zhang, F. Guerin, and E. Cambria, “GPTEval: A survey on assessments of ChatGPT and GPT-4,” _arXiv preprint arXiv:2308.12488_ , 2023.
  * [16] K. Singhal, T. Tu, J. Gottweis, R. Sayres, E. Wulczyn, L. Hou, K. Clark, S. Pfohl, H. Cole-Lewis, D. Neal _et al._ , “Towards expert-level medical question answering with large language models,” _arXiv preprint arXiv:2305.09617_ , 2023.
  * [17] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell _et al._ , “Language models are few-shot learners,” _Advances in neural information processing systems_ , vol. 33, pp. 1877–1901, 2020.
  * [18] R. Schaeffer, B. Miranda, and S. Koyejo, “Are emergent abilities of large language models a mirage?” _arXiv preprint arXiv:2304.15004_ , 2023.
  * [19] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling laws for neural language models,” _arXiv preprint arXiv:2001.08361_ , 2020.
  * [20] M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer, “Deep contextualized word representations,” in _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_. New Orleans, Louisiana: Association for Computational Linguistics, Jun. 2018, pp. 2227–2237. [Online]. Available: https://aclanthology.org/N18-1202
  * [21] J. D. M.-W. C. Kenton and L. K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in _Proceedings of NAACL-HLT_ , 2019, pp. 4171–4186.
  * [22] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” 2019.
  * [23] K. He, N. Hong, S. Lapalme-Remis, Y. Lan, M. Huang, C. Li, and L. Yao, “Understanding the patient perspective of epilepsy treatment through text mining of online patient support groups,” _Epilepsy & Behavior_, vol. 94, pp. 65–71, 2019.
  * [24] Y. Li, X. Ma, X. Zhou, P. Cheng, K. He, and C. Li, “Knowledge enhanced lstm for coreference resolution on biomedical texts,” _Bioinformatics_ , vol. 37, no. 17, pp. 2699–2705, 2021.
  * [25] K. He, L. Yao, J. Zhang, Y. Li, and C. Li, “Construction of genealogical knowledge graphs from obituaries: Multitask neural network extraction system,” _Journal of Medical Internet Research_ , vol. 23, no. 8, p. e25670, 2021.
  * [26] B. Mao, C. Jia, Y. Huang, K. He, J. Wu, T. Gong, and C. Li, “Uncertainty-guided mutual consistency training for semi-supervised biomedical relation extraction,” in _2022 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)_. IEEE, 2022, pp. 2318–2325.
  * [27] J. Wu, K. He, R. Mao, C. Li, and E. Cambria, “Megacare: Knowledge-guided multi-view hypergraph predictive framework for healthcare,” _Information Fusion_ , p. 101939, 2023.
  * [28] C. Li, X. Xu, G. Zhou, K. He, T. Qi, W. Zhang, F. Tian, Q. Zheng, J. Hu _et al._ , “Implementation of national health informatization in china: survey about the status quo,” _JMIR medical informatics_ , vol. 7, no. 1, p. e12238, 2019.
  * [29] A. J. Thirunavukarasu, D. S. J. Ting, K. Elangovan, L. Gutierrez, T. F. Tan, and D. S. W. Ting, “Large language models in medicine,” _Nature medicine_ , pp. 1–11, 2023.
  * [30] K. S. Kalyan, A. Rajasekharan, and S. Sangeetha, “Ammu: a survey of transformer-based biomedical pretrained language models,” _Journal of biomedical informatics_ , vol. 126, p. 103982, 2022.
  * [31] M. Moor, O. Banerjee, Z. S. H. Abad, H. M. Krumholz, J. Leskovec, E. J. Topol, and P. Rajpurkar, “Foundation models for generalist medical artificial intelligence,” _Nature_ , vol. 616, no. 7956, pp. 259–265, 2023.
  * [32] K. He, J. Wu, X. Ma, C. Zhang, M. Huang, C. Li, and L. Yao, “Extracting kinship from obituary to enhance electronic health records for genetic research,” in _Proceedings of the Fourth social media mining for health applications (# SMM4H) workshop & shared task_, 2019, pp. 1–10.
  * [33] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou _et al._ , “Chain-of-thought prompting elicits reasoning in large language models,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 24 824–24 837, 2022.
  * [34] Y. Hu, I. Ameer, X. Zuo, X. Peng, Y. Zhou, Z. Li, Y. Li, J. Li, X. Jiang, and H. Xu, “Zero-shot clinical entity recognition using chatgpt,” _arXiv preprint arXiv:2303.16416_ , 2023.
  * [35] V. A. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro, “Reducing activation recomputation in large transformer models,” _Proceedings of Machine Learning and Systems_ , vol. 5, 2023.
  * [36] S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer, “Opt: Open pre-trained transformer language models,” 2022.
  * [37] J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre, “Training compute-optimal large language models,” 2022.
  * [38] R. Taylor, M. Kardas, G. Cucurull, T. Scialom, A. Hartshorn, E. Saravia, A. Poulton, V. Kerkez, and R. Stojnic, “Galactica: A large language model for science,” 2022.
  * [39] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, “Llama: Open and efficient foundation language models,” 2023.
  * [40] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann _et al._ , “Palm: Scaling language modeling with pathways,” _arXiv preprint arXiv:2204.02311_ , 2022.
  * [41] H. Yang, X.-Y. Liu, and C. D. Wang, “Fingpt: Open-source financial large language models,” 2023.
  * [42] S. Milano, J. A. McGrane, and S. Leonelli, “Large language models challenge the future of higher education,” _Nature Machine Intelligence_ , vol. 5, no. 4, pp. 333–334, 2023.
  * [43] A. Arora and A. Arora, “The promise of large language models in health care,” _The Lancet_ , vol. 401, no. 10377, p. 641, 2023.
  * [44] H. Zhang, J. Chen, F. Jiang, F. Yu, Z. Chen, J. Li, G. Chen, X. Wu, Z. Zhang, Q. Xiao _et al._ , “Huatuogpt, towards taming language model to be a doctor,” _arXiv preprint arXiv:2305.15075_ , 2023.
  * [45] S. Chang, C. Baian, L. Fangyu, F. Zihao, S. Ehsan, and N. Collier, “Visual med-alpaca: A parameter-efficient biomedical llm with visual capabilities,” 2023. [Online]. Available: https://cambridgeltl.github.io/visual-med-alpaca/
  * [46] J. A. Omiye, H. Gui, S. J. Rezaei, J. Zou, and R. Daneshjou, “Large language models in medicine: the potentials and pitfalls,” 2023.
  * [47] W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong _et al._ , “A survey of large language models,” _arXiv preprint arXiv:2303.18223_ , 2023.
  * [48] B. Min, H. Ross, E. Sulem, A. P. B. Veyseh, T. H. Nguyen, O. Sainz, E. Agirre, I. Heintz, and D. Roth, “Recent advances in natural language processing via large pre-trained language models: A survey,” _ACM Computing Surveys_ , 2021.
  * [49] P. Lavanya and E. Sasikala, “Deep learning techniques on text classification using natural language processing (nlp) in social healthcare network: A comprehensive survey,” in _2021 3rd international conference on signal processing and communication (ICPSC)_. IEEE, 2021, pp. 603–609.
  * [50] N. Fei, Z. Lu, Y. Gao, G. Yang, Y. Huo, J. Wen, H. Lu, R. Song, X. Gao, T. Xiang _et al._ , “Towards artificial general intelligence via a multimodal foundation model,” _Nature Communications_ , vol. 13, no. 1, p. 3094, 2022.
  * [51] W. Chen, Z. Li, H. Fang, Q. Yao, C. Zhong, J. Hao, Q. Zhang, X. Huang, J. Peng, and Z. Wei, “A benchmark for automatic medical consultation system: frameworks, tasks and datasets,” _Bioinformatics_ , vol. 39, no. 1, p. btac817, 2023.
  * [52] X. Shi, Z. Liu, C. Wang, H. Leng, K. Xue, X. Zhang, and S. Zhang, “Midmed: Towards mixed-type dialogues for medical consultation,” _arXiv preprint arXiv:2306.02923_ , 2023.
  * [53] J. H. Moon, H. Lee, W. Shin, Y.-H. Kim, and E. Choi, “Multi-modal understanding and generation for medical images and text via vision-language pre-training,” _IEEE Journal of Biomedical and Health Informatics_ , vol. 26, no. 12, pp. 6070–6080, 2022.
  * [54] G. Yang, Q. Ye, and J. Xia, “Unbox the black-box for the medical explainable ai via multi-modal and multi-centre data fusion: A mini-review, two showcases and beyond,” _Information Fusion_ , vol. 77, pp. 29–52, 2022.
  * [55] D. Demner-Fushman, K. W. Fung, P. Do, R. D. Boyce, and T. R. Goodwin, “Overview of the tac 2018 drug-drug interaction extraction from drug labels track.” in _TAC_ , 2018.
  * [56] Y. Deng, X. Xu, Y. Qiu, J. Xia, W. Zhang, and S. Liu, “A multimodal deep learning framework for predicting drug–drug interaction events,” _Bioinformatics_ , vol. 36, no. 15, pp. 4316–4322, 2020.
  * [57] B.-W. Zhao, L. Hu, Z.-H. You, L. Wang, and X.-R. Su, “Hingrl: predicting drug–disease associations with graph representation learning on heterogeneous information networks,” _Briefings in bioinformatics_ , vol. 23, no. 1, p. bbab515, 2022.
  * [58] M. Krallinger, O. Rabal, A. Lourenco, J. Oyarzabal, and A. Valencia, “Information retrieval and text mining technologies for chemistry,” _Chemical reviews_ , vol. 117, no. 12, pp. 7673–7761, 2017.
  * [59] K. He, B. Mao, X. Zhou, Y. Li, T. Gong, C. Li, and J. Wu, “Knowledge enhanced coreference resolution via gated attention,” in _2022 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)_. IEEE, 2022, pp. 2287–2293.
  * [60] A. Nesterov and D. Umerenkov, “Distantly supervised end-to-end medical entity extraction from electronic health records with human-level quality,” _arXiv preprint arXiv:2201.10463_ , 2022.
  * [61] R. Mythili, N. Parthiban, and V. Kavitha, “Construction of heterogeneous medical knowledge graph from electronic health records,” _Journal of Discrete Mathematical Sciences and Cryptography_ , vol. 25, no. 4, pp. 921–930, 2022.
  * [62] L. Xiong, Y. Guo, Y. Chen, and S. Liang, “How can entities improve the quality of medical dialogue generation?” in _2023 2nd International Conference on Big Data, Information and Computer Network (BDICN)_. IEEE, 2023, pp. 225–229.
  * [63] X. Lin, X. He, Q. Chen, H. Tou, Z. Wei, and T. Chen, “Enhancing dialogue symptom diagnosis with global attention and symptom graph,” in _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , 2019, pp. 5033–5042.
  * [64] D. S. Wishart, C. Knox, A. C. Guo, S. Shrivastava, M. Hassanali, P. Stothard, Z. Chang, and J. Woolsey, “Drugbank: a comprehensive resource for in silico drug discovery and exploration,” _Nucleic acids research_ , vol. 34, no. suppl_1, pp. D668–D672, 2006.
  * [65] D. A. Lindberg, B. L. Humphreys, and A. T. McCray, “The unified medical language system,” _Yearbook of medical informatics_ , vol. 2, no. 01, pp. 41–51, 1993.
  * [66] G. Wang, A. Badal, X. Jia, J. S. Maltz, K. Mueller, K. J. Myers, C. Niu, M. Vannier, P. Yan, Z. Yu _et al._ , “Development of metaverse for intelligent healthcare,” _Nature Machine Intelligence_ , vol. 4, no. 11, pp. 922–929, 2022.
  * [67] X. Yu, W. Hu, S. Lu, X. Sun, and Z. Yuan, “Biobert based named entity recognition in electronic medical record,” in _2019 10th international conference on information technology in medicine and education (ITME)_. IEEE, 2019, pp. 49–52.
  * [68] M. Chen, F. Du, G. Lan, and V. S. Lobanov, “Using pre-trained transformer deep learning models to identify named entities and syntactic relations for clinical protocol analysis.” in _AAAI Spring Symposium: Combining Machine Learning with Knowledge Engineering (1)_ , 2020, pp. 1–8.
  * [69] Z. Miftahutdinov, I. Alimova, and E. Tutubalina, “On biomedical named entity recognition: experiments in interlingual transfer for clinical and social media texts,” in _European Conference on Information Retrieval_. Springer, 2020, pp. 281–288.
  * [70] Q. Wei, Z. Ji, Y. Si, J. Du, J. Wang, F. Tiryaki, S. Wu, C. Tao, K. Roberts, and H. Xu, “Relation extraction from clinical narratives using pre-trained language models,” in _AMIA annual symposium proceedings_ , vol. 2019. American Medical Informatics Association, 2019, p. 1236.
  * [71] A. Dunn, J. Dagdelen, N. Walker, S. Lee, A. S. Rosen, G. Ceder, K. Persson, and A. Jain, “Structured information extraction from complex scientific text with fine-tuned large language models,” _arXiv preprint arXiv:2212.05238_ , 2022.
  * [72] M. Agrawal, S. Hegselmann, H. Lang, Y. Kim, and D. Sontag, “Large language models are few-shot clinical information extractors,” in _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , 2022, pp. 1998–2022.
  * [73] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray _et al._ , “Training language models to follow instructions with human feedback,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 27 730–27 744, 2022.
  * [74] B. Nye, J. J. Li, R. Patel, Y. Yang, I. J. Marshall, A. Nenkova, and B. C. Wallace, “A corpus with multi-level annotations of patients, interventions and outcomes to support language processing for medical literature,” in _Proceedings of the conference. Association for Computational Linguistics. Meeting_ , vol. 2018. NIH Public Access, 2018, p. 197.
  * [75] S. Moon, S. Pakhomov, N. Liu, J. O. Ryan, and G. B. Melton, “A sense inventory for clinical abbreviations and acronyms created using clinical notes and medical dictionary resources,” _Journal of the American Medical Informatics Association_ , vol. 21, no. 2, pp. 299–307, 2014.
  * [76] S. K. Prabhakar and D.-O. Won, “Medical text classification using hybrid deep learning models with multihead attention,” _Computational intelligence and neuroscience_ , vol. 2021, 2021.
  * [77] S. Baker, A. Korhonen, and S. Pyysalo, “Cancer hallmark text classification using convolutional neural networks,” 2017.
  * [78] M. A. Al-Garadi, Y.-C. Yang, H. Cai, Y. Ruan, K. O’Connor, G.-H. Graciela, J. Perrone, and A. Sarker, “Text classification models for the automatic detection of nonmedical prescription medication use from social media,” _BMC medical informatics and decision making_ , vol. 21, no. 1, pp. 1–13, 2021.
  * [79] X. Sun, X. Li, J. Li, F. Wu, S. Guo, T. Zhang, and G. Wang, “Text classification via large language models,” _arXiv preprint arXiv:2305.08377_ , 2023.
  * [80] H. Wang, C. Xu, and J. McAuley, “Automatic multi-label prompting: Simple and interpretable few-shot classification,” in _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 2022, pp. 5483–5492.
  * [81] T. Schick, H. Schmid, and H. Schütze, “Automatically identifying words that can serve as labels for few-shot text classification,” _arXiv preprint arXiv:2010.13641_ , 2020.
  * [82] M. Rastegar-Mojarad, S. Liu, Y. Wang, N. Afzal, L. Wang, F. Shen, S. Fu, and H. Liu, “Biocreative/ohnlp challenge 2018,” in _ACM-BCB 2018 - Proceedings of the 2018 ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics_ , ser. ACM-BCB 2018 - Proceedings of the 2018 ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics. Association for Computing Machinery, Inc, Aug. 2018, p. 575.
  * [83] T. Botsis, G. Hartvigsen, F. Chen, and C. Weng, “Secondary use of ehr: data quality issues and informatics opportunities,” _Summit on translational bioinformatics_ , vol. 2010, p. 1, 2010.
  * [84] D. Mahajan, A. Poddar, J. J. Liang, Y.-T. Lin, J. M. Prager, P. Suryanarayanan, P. Raghavan, C.-H. Tsou _et al._ , “Identification of semantically similar sentences in clinical notes: Iterative intermediate training using multi-task learning,” _JMIR medical informatics_ , vol. 8, no. 11, p. e22508, 2020.
  * [85] Y. Wang, F. Liu, K. Verspoor, and T. Baldwin, “Evaluating the utility of model configurations and data augmentation on clinical semantic textual similarity,” in _Proceedings of the 19th SIGBioMed workshop on biomedical language processing_ , 2020, pp. 105–111.
  * [86] X. Yang, X. He, H. Zhang, Y. Ma, J. Bian, Y. Wu _et al._ , “Measurement of semantic textual similarity in clinical texts: comparison of transformer-based models,” _JMIR medical informatics_ , vol. 8, no. 11, p. e19735, 2020.
  * [87] X. Yang, A. Chen, N. PourNejatian, H. C. Shin, K. E. Smith, C. Parisien, C. Compas, C. Martin, A. B. Costa, M. G. Flores _et al._ , “A large language model for electronic health records,” _NPJ Digital Medicine_ , vol. 5, no. 1, p. 194, 2022.
  * [88] S. Fox and M. Duggan, “Health online 2013,” 2012.
  * [89] S. Nerella, S. Bandyopadhyay, J. Zhang, M. Contreras, S. Siegel, A. Bumin, B. Silva, J. Sena, B. Shickel, A. Bihorac _et al._ , “Transformers in healthcare: A survey,” _arXiv preprint arXiv:2307.00067_ , 2023.
  * [90] G. Pergola, E. Kochkina, L. Gui, M. Liakata, and Y. He, “Boosting low-resource biomedical QA via entity-aware masking strategies,” in _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_. Online: Association for Computational Linguistics, Apr. 2021, pp. 1977–1985. [Online]. Available: https://aclanthology.org/2021.eacl-main.169
  * [91] J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. H. So, and J. Kang, “Biobert: a pre-trained biomedical language representation model for biomedical text mining,” _Bioinformatics_ , vol. 36, no. 4, pp. 1234–1240, 2020.
  * [92] Z. Chen, G. Li, and X. Wan, “Align, reason and learn: Enhancing medical vision-and-language pre-training with knowledge,” in _Proceedings of the 30th ACM International Conference on Multimedia_ , 2022, pp. 5152–5161.
  * [93] Q. Liu, R. Mao, X. Geng, and E. Cambria, “Semantic matching in machine reading comprehension: An empirical study,” _Information Processing & Management_, vol. 60, no. 2, p. 103145, 2023.
  * [94] K. He, R. Mao, Y. Huang, T. Gong, C. Li, and E. Cambria, “Template-free prompting for few-shot named entity recognition via semantic-enhanced contrastive learning,” in _IEEE Transactions on Neural Networks and Learning Systems_ , 2023.
  * [95] T. Gao, J. Fang, H. Liu, Z. Liu, C. Liu, P. Liu, Y. Bao, and W. Yan, “Lego-absa: A prompt-based task assemblable unified generative framework for multi-task aspect-based sentiment analysis,” in _Proceedings of the 29th international conference on computational linguistics_ , 2022, pp. 7002–7012.
  * [96] K. He, R. Mao, T. Gong, C. Li, and E. Cambria, “Meta-based self-training and re-weighting for aspect-based sentiment analysis,” _IEEE Transactions on Affective Computing_ , 2022.
  * [97] C. Li, F. Gao, J. Bu, L. Xu, X. Chen, Y. Gu, Z. Shao, Q. Zheng, N. Zhang, Y. Wang _et al._ , “Sentiprompt: Sentiment knowledge enhanced prompt-tuning for aspect-based sentiment analysis,” _arXiv preprint arXiv:2109.08306_ , 2021.
  * [98] M. M. Amin, R. Mao, E. Cambria, and B. W. Schuller, “A wide evaluation of ChatGPT on affective computing tasks,” _arXiv preprint arXiv:2308.13911_ , 2023.
  * [99] K. Singhal, S. Azizi, T. Tu, S. S. Mahdavi, J. Wei, H. W. Chung, N. Scales, A. Tanwani, H. Cole-Lewis, S. Pfohl _et al._ , “Large language models encode clinical knowledge,” _arXiv preprint arXiv:2212.13138_ , 2022.
  * [100] A. Pal, L. K. Umapathi, and M. Sankarasubbu, “Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering,” in _Conference on Health, Inference, and Learning_. PMLR, 2022, pp. 248–260.
  * [101] Q. Jin, B. Dhingra, Z. Liu, W. W. Cohen, and X. Lu, “Pubmedqa: A dataset for biomedical research question answering,” _arXiv preprint arXiv:1909.06146_ , 2019.
  * [102] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring massive multitask language understanding,” _arXiv preprint arXiv:2009.03300_ , 2020.
  * [103] A. Hamidi and K. Roberts, “Evaluation of ai chatbots for patient-specific ehr questions,” _arXiv preprint arXiv:2306.02549_ , 2023.
  * [104] Q. Guo, S. Cao, and Z. Yi, “A medical question answering system using large language models and knowledge graphs,” _International Journal of Intelligent Systems_ , vol. 37, no. 11, pp. 8548–8564, 2022.
  * [105] T. Kowatsch, M. Nißen, C.-H. I. Shih, D. Rüegger, D. Volland, A. Filler, F. Künzler, F. Barata, S. Hung, D. Büchter _et al._ , “Text-based healthcare chatbots supporting patient and health professional teams: preliminary results of a randomized controlled trial on childhood obesity,” _Persuasive Embodied Agents for Behavior Change (PEACH2017)_ , 2017.
  * [106] B. Chaix, J.-E. Bibault, A. Pienkowski, G. Delamon, A. Guillemassé, P. Nectoux, B. Brouard _et al._ , “When chatbots meet patients: one-year prospective study of conversations between patients with breast cancer and a chatbot,” _JMIR cancer_ , vol. 5, no. 1, p. e12856, 2019.
  * [107] S. Ji, T. Zhang, K. Yang, S. Ananiadou, E. Cambria, and J. Tiedemann, “Domain-specific continued pretraining of language models for capturing long context in mental health,” _arXiv preprint arXiv:2304.10447_ , 2023.
  * [108] L. Reis, C. Maier, J. Mattke, and T. Weitzel, “Chatbots in healthcare: Status quo, application scenarios for physicians and patients and future directions,” 2020.
  * [109] J. Ni, T. Young, V. Pandelea, F. Xue, and E. Cambria, “Recent advances in deep learning based dialogue systems: A systematic survey,” _Artificial intelligence review_ , vol. 56, no. 4, pp. 3055–3155, 2023.
  * [110] A. A. Abd-Alrazaq, M. Alajlani, N. Ali, K. Denecke, B. M. Bewick, and M. Househ, “Perceptions and opinions of patients about mental health chatbots: scoping review,” _Journal of medical Internet research_ , vol. 23, no. 1, p. e17828, 2021.
  * [111] S. Ji, T. Zhang, L. Ansari, J. Fu, P. Tiwari, and E. Cambria, “Mentalbert: Publicly available pretrained language models for mental healthcare,” 2021.
  * [112] R. L. Pande, M. Morris, A. Peters, C. M. Spettell, R. Feifer, and W. Gillis, “Leveraging remote behavioral health interventions to improve medical outcomes and reduce costs,” _Am J Manag Care_ , vol. 21, no. 2, pp. e141–e151, 2015.
  * [113] D. Milward and M. Beveridge, “Ontology-based dialogue systems,” in _Proc. 3rd Workshop on Knowledge and reasoning in practical dialogue systems (IJCAI03)_ , 2003, pp. 9–18.
  * [114] L. Xu, Q. Zhou, K. Gong, X. Liang, J. Tang, and L. Lin, “End-to-end knowledge-routed relational dialogue system for automatic diagnosis,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 33, no. 01, 2019, pp. 7346–7353.
  * [115] W. Qin, Z. Chen, L. Wang, Y. Lan, W. Ren, and R. Hong, “Read, diagnose and chat: Towards explainable and interactive llms-augmented depression detection in social media,” _arXiv preprint arXiv:2305.05138_ , 2023.
  * [116] L. Yunxiang, L. Zihan, Z. Kai, D. Ruilong, and Z. You, “Chatdoctor: A medical chat model fine-tuned on llama model using medical domain knowledge,” _arXiv preprint arXiv:2303.14070_ , 2023.
  * [117] B. Jing, P. Xie, and E. Xing, “On the automatic generation of medical imaging reports,” in _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2018, pp. 2577–2586.
  * [118] Y. Xue, T. Xu, L. Rodney Long, Z. Xue, S. Antani, G. R. Thoma, and X. Huang, “Multimodal recurrent model with attention for automated radiology report generation,” in _Medical Image Computing and Computer Assisted Intervention–MICCAI 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part I_. Springer, 2018, pp. 457–466.
  * [119] J. Chen, H. Guo, K. Yi, B. Li, and M. Elhoseiny, “Visualgpt: Data-efficient adaptation of pretrained language models for image captioning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2022, pp. 18 030–18 040.
  * [120] S. Wang, Z. Zhao, X. Ouyang, Q. Wang, and D. Shen, “Chatcad: Interactive computer-aided diagnosis on medical image using large language models,” _arXiv preprint arXiv:2302.07257_ , 2023.
  * [121] Z. Chen, Y. Song, T.-H. Chang, and X. Wan, “Generating radiology reports via memory-driven transformer,” in _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 2020, pp. 1439–1449.
  * [122] A. Nicolson, J. Dowling, and B. Koopman, “Improving chest x-ray report generation by leveraging warm-starting,” _arXiv preprint arXiv:2201.09405_ , 2022.
  * [123] Z. Zhao, S. Wang, J. Gu, Y. Zhu, L. Mei, Z. Zhuang, Z. Cui, Q. Wang, and D. Shen, “Chatcad+: Towards a universal and reliable interactive cad using llms,” _arXiv preprint arXiv:2305.15964_ , 2023.
  * [124] Z. Gao, B. Hong, X. Zhang, Y. Li, C. Jia, J. Wu, C. Wang, D. Meng, and C. Li, “Instance-based vision transformer for subtyping of papillary renal cell carcinoma in histopathological image,” in _Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part VIII 24_. Springer, 2021, pp. 299–308.
  * [125] Z. Gao, A. Mao, K. Wu, Y. Li, L. Zhao, X. Zhang, J. Wu, L. Yu, C. Xing, T. Gong _et al._ , “Childhood leukemia classification via information bottleneck enhanced hierarchical multi-instance learning,” _IEEE Transactions on Medical Imaging_ , 2023.
  * [126] J. Shi, L. Tang, Y. Li, X. Zhang, Z. Gao, Y. Zheng, C. Wang, T. Gong, and C. Li, “A structure-aware hierarchical graph-based multiple instance learning framework for pt staging in histopathological image,” _IEEE Transactions on Medical Imaging_ , 2023.
  * [127] S. Liu, Z. Zhu, Q. Qu, and C. You, “Robust training under label noise by over-parameterization,” in _International Conference on Machine Learning_. PMLR, 2022, pp. 14 153–14 172.
  * [128] C. Zhou, P. Liu, P. Xu, S. Iyer, J. Sun, Y. Mao, X. Ma, A. Efrat, P. Yu, L. Yu _et al._ , “Lima: Less is more for alignment,” _arXiv preprint arXiv:2305.11206_ , 2023.
  * [129] X. Li, P. Yu, C. Zhou, T. Schick, L. Zettlemoyer, O. Levy, J. Weston, and M. Lewis, “Self-alignment with instruction backtranslation,” 2023.
  * [130] Y. Peng, S. Yan, and Z. Lu, “Transfer learning in biomedical natural language processing: an evaluation of bert and elmo on ten benchmarking datasets,” _arXiv preprint arXiv:1906.05474_ , 2019.
  * [131] Z. Kraljevic, A. Shek, D. Bean, R. Bendayan, J. Teo, and R. Dobson, “Medgpt: Medical concept prediction from clinical narratives,” _arXiv preprint arXiv:2107.03134_ , 2021.
  * [132] S. Sharma and R. Daniel Jr, “Bioflair: Pretrained pooled contextualized embeddings for biomedical sequence labeling tasks,” _arXiv preprint arXiv:1908.05760_ , 2019.
  * [133] I. B. Ozyurt, “On the effectiveness of small, discriminatively pre-trained language representation models for biomedical text mining,” in _Proceedings of the First Workshop on Scholarly Document Processing_ , 2020, pp. 104–112.
  * [134] Y.-P. Chen, Y.-Y. Chen, J.-J. Lin, C.-H. Huang, F. Lai _et al._ , “Modified bidirectional encoder representations from transformers extractive summarization model for hospital information systems based on character-level tokens (alphabert): development and performance evaluation,” _JMIR medical informatics_ , vol. 8, no. 4, p. e17787, 2020.
  * [135] L. Akhtyamova, “Named entity recognition in spanish biomedical literature: Short review and bert model,” in _2020 26th Conference of Open Innovations Association (FRUCT)_. IEEE, 2020, pp. 1–7.
  * [136] N. Poerner, U. Waltinger, and H. Schütze, “Inexpensive domain adaptation of pretrained language models: Case studies on biomedical ner and covid-19 qa,” _arXiv preprint arXiv:2004.03354_ , 2020.
  * [137] Y. Li, S. Rao, J. R. A. Solares, A. Hassaine, R. Ramakrishnan, D. Canoy, Y. Zhu, K. Rahimi, and G. Salimi-Khorshidi, “Behrt: transformer for electronic health records,” _Scientific reports_ , vol. 10, no. 1, p. 7155, 2020.
  * [138] S. Gururangan, A. Marasović, S. Swayamdipta, K. Lo, I. Beltagy, D. Downey, and N. A. Smith, “Don’t stop pretraining: Adapt language models to domains and tasks,” _arXiv preprint arXiv:2004.10964_ , 2020.
  * [139] X. Meng, C. H. Ganoe, R. T. Sieberg, Y. Y. Cheung, and S. Hassanpour, “Self-supervised contextual language representation of radiology reports to improve the identification of communication urgency,” _AMIA Summits on Translational Science Proceedings_ , vol. 2020, p. 413, 2020.
  * [140] M. Müller, M. Salathé, and P. E. Kummervold, “Covid-twitter-bert: A natural language processing model to analyse covid-19 content on twitter,” _Frontiers in Artificial Intelligence_ , vol. 6, p. 1023281, 2023.
  * [141] J. Copara, J. Knafou, N. Naderi, C. Moro, P. Ruch, and D. Teodoro, “Contextualized French language models for biomedical named entity recognition,” in _Actes de la 6e conférence conjointe Journées d’Études sur la Parole (JEP, 33e édition), Traitement Automatique des Langues Naturelles (TALN, 27e édition), Rencontre des Étudiants Chercheurs en Informatique pour le Traitement Automatique des Langues (RÉCITAL, 22e édition). Atelier DÉfi Fouille de Textes_. Nancy, France: ATALA et AFCP, 6 2020, pp. 36–48. [Online]. Available: https://aclanthology.org/2020.jeptalnrecital-deft.4
  * [142] K. K. Bressem, L. C. Adams, R. A. Gaudin, D. Tröltzsch, B. Hamm, M. R. Makowski, C.-Y. Schüle, J. L. Vahldiek, and S. M. Niehues, “Highly accurate classification of chest radiographic reports using a deep learning natural language model pre-trained on 3.8 million text reports,” _Bioinformatics_ , vol. 36, no. 21, pp. 5255–5261, 2020.
  * [143] Y. Kawazoe, D. Shibata, E. Shinohara, E. Aramaki, and K. Ohe, “A clinical specific bert developed with huge size of japanese clinical narrative,” _medRxiv_ , pp. 2020–07, 2020.
  * [144] N. Zhang, Q. Jia, K. Yin, L. Dong, F. Gao, and N. Hua, “Conceptualized representation learning for chinese biomedical text mining,” _arXiv preprint arXiv:2008.10813_ , 2020.
  * [145] U. Naseem, M. Khushi, V. Reddy, S. Rajendran, I. Razzak, and J. Kim, “Bioalbert: A simple and effective pre-trained language model for biomedical named entity recognition,” in _2021 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 2021, pp. 1–7.
  * [146] H.-C. Shin, Y. Zhang, E. Bakhturina, R. Puri, M. Patwary, M. Shoeybi, and R. Mani, “Biomegatron: Larger biomedical domain language model,” 2020.
  * [147] K. Huang, J. Altosaar, and R. Ranganath, “Clinicalbert: Modeling clinical notes and predicting hospital readmission,” _arXiv preprint arXiv:1904.05342_ , 2019.
  * [148] K. Huang, A. Singh, S. Chen, E. Moseley, C.-Y. Deng, N. George, and C. Lindvall, “Clinical xlnet: Modeling sequential clinical notes and predicting prolonged mechanical ventilation,” in _Proceedings of the 3rd Clinical Natural Language Processing Workshop_ , 2020, pp. 94–100.
  * [149] P. Lewis, M. Ott, J. Du, and V. Stoyanov, “Pretrained language models for biomedical and clinical tasks: understanding and extending the state-of-the-art,” in _Proceedings of the 3rd Clinical Natural Language Processing Workshop_ , 2020, pp. 146–157.
  * [150] E. T. R. Schneider, J. V. A. de Souza, J. Knafou, L. E. S. e. Oliveira, J. Copara, Y. B. Gumiel, L. F. A. d. Oliveira, E. C. Paraiso, D. Teodoro, and C. M. C. M. Barra, “BioBERTpt - a Portuguese neural language model for clinical named entity recognition,” in _Proceedings of the 3rd Clinical Natural Language Processing Workshop_. Online: Association for Computational Linguistics, Nov. 2020, pp. 65–72. [Online]. Available: https://aclanthology.org/2020.clinicalnlp-1.7
  * [151] X. Yang, J. Bian, W. R. Hogan, and Y. Wu, “Clinical concept extraction using transformers,” _Journal of the American Medical Informatics Association_ , vol. 27, no. 12, pp. 1935–1942, 2020.
  * [152] B. Hao, H. Zhu, and I. C. Paschalidis, “Enhancing clinical bert embedding using a biomedical knowledge base,” in _28th International Conference on Computational Linguistics (COLING 2020)_ , 2020.
  * [153] J. Wang, G. Zhang, W. Wang, K. Zhang, and Y. Sheng, “Cloud-based intelligent self-diagnosis and department recommendation service using chinese medical bert,” _Journal of Cloud Computing_ , vol. 10, pp. 1–12, 2021.
  * [154] Y. Gu, R. Tinn, H. Cheng, M. Lucas, N. Usuyama, X. Liu, T. Naumann, J. Gao, and H. Poon, “Domain-specific language model pretraining for biomedical natural language processing,” _ACM Transactions on Computing for Healthcare (HEALTH)_ , vol. 3, no. 1, pp. 1–23, 2021.
  * [155] S. Wada, T. Takeda, S. Manabe, S. Konishi, J. Kamohara, and Y. Matsumura, “Pre-training technique to localize medical bert and enhance biomedical bert,” _arXiv preprint arXiv:2005.07202_ , 2020.
  * [156] Y. Meng, W. Speier, M. K. Ong, and C. W. Arnold, “Bidirectional representation learning from transformers using multimodal electronic health record data to predict depression,” _IEEE Journal of Biomedical and Health Informatics_ , vol. 25, no. 8, pp. 3121–3129, 2021.
  * [157] W. Antoun, F. Baly, and H. Hajj, “Arabert: Transformer-based model for arabic language understanding,” in _Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection_ , 2020, pp. 9–15.
  * [158] N. Boudjellal, H. Zhang, A. Khan, A. Ahmad, R. Naseem, J. Shang, and L. Dai, “Abioner: a bert-based model for arabic biomedical named-entity recognition,” _Complexity_ , vol. 2021, pp. 1–6, 2021.
  * [159] G. Miolo, G. Mantoan, and C. Orsenigo, “Electramed: a new pre-trained language representation model for biomedical nlp,” _arXiv preprint arXiv:2104.09585_ , 2021.
  * [160] Z. Yuan, Y. Liu, C. Tan, S. Huang, and F. Huang, “Improving biomedical pretrained language models with knowledge,” _arXiv preprint arXiv:2104.10344_ , 2021.
  * [161] N. Taghizadeh, E. Doostmohammadi, E. Seifossadat, H. R. Rabiee, and M. S. Tahaei, “Sina-bert: a pre-trained language model for analysis of medical texts in persian,” _arXiv preprint arXiv:2104.07613_ , 2021.
  * [162] L. Rasmy, Y. Xiang, Z. Xie, C. Tao, and D. Zhi, “Med-bert: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction,” _NPJ digital medicine_ , vol. 4, no. 1, p. 86, 2021.
  * [163] G. López-García, J. M. Jerez, N. Ribelles, E. Alba, and F. J. Veredas, “Transformers for clinical coding in spanish,” _IEEE Access_ , vol. 9, pp. 72 387–72 397, 2021.
  * [164] L. N. Phan, J. T. Anibal, H. Tran, S. Chanana, E. Bahadroglu, A. Peltekian, and G. Altan-Bonnet, “Scifive: a text-to-text transformer model for biomedical literature,” _arXiv preprint arXiv:2106.03598_ , 2021.
  * [165] K. r. Kanakarajan, B. Kundumani, and M. Sankarasubbu, “BioELECTRA:pretrained biomedical text encoder using discriminators,” in _Proceedings of the 20th Workshop on Biomedical Language Processing_. Online: Association for Computational Linguistics, Jun. 2021, pp. 143–154. [Online]. Available: https://aclanthology.org/2021.bionlp-1.16
  * [166] Z. Yuan, Z. Zhao, H. Sun, J. Li, F. Wang, and S. Yu, “Coder: Knowledge-infused cross-lingual medical term embedding for term normalization,” _Journal of biomedical informatics_ , vol. 126, p. 103983, 2022.
  * [167] M. Yasunaga, J. Leskovec, and P. Liang, “Linkbert: Pretraining language models with document links,” _arXiv preprint arXiv:2203.15827_ , 2022.
  * [168] U. Naseem, A. G. Dunn, M. Khushi, and J. Kim, “Benchmarking for biomedical natural language processing tasks with a domain specific albert,” _BMC bioinformatics_ , vol. 23, no. 1, pp. 1–15, 2022.
  * [169] H. Yuan, Z. Yuan, R. Gan, J. Zhang, Y. Xie, and S. Yu, “Biobart: Pretraining and evaluation of a biomedical generative language model,” _BioNLP 2022@ ACL 2022_ , p. 97, 2022.
  * [170] F. Liu, E. Shareghi, Z. Meng, M. Basaldella, and N. Collier, “Self-alignment pretraining for biomedical entity representations,” _arXiv preprint arXiv:2010.11784_ , 2020.
  * [171] X. Zhang, C. Wu, Y. Zhang, Y. Wang, and W. Xie, “Knowledge-enhanced visual-language pre-training on chest radiology images,” 2023.
  * [172] Y. Choi, C. Y.-I. Chiu, and D. Sontag, “Learning low-dimensional representations of medical concepts,” _AMIA Summits on Translational Science Proceedings_ , vol. 2016, p. 41, 2016.
  * [173] Z. Zeng, C. Xiao, Y. Yao, R. Xie, Z. Liu, F. Lin, L. Lin, and M. Sun, “Knowledge transfer via pre-training for recommendation: A review and prospect,” _Frontiers in big Data_ , vol. 4, p. 602071, 2021.
  * [174] X. Qiu, T. Sun, Y. Xu, Y. Shao, N. Dai, and X. Huang, “Pre-trained models for natural language processing: A survey,” _Science China Technological Sciences_ , vol. 63, no. 10, pp. 1872–1897, 2020.
  * [175] F. Shamshad, S. Khan, S. W. Zamir, M. H. Khan, M. Hayat, F. S. Khan, and H. Fu, “Transformers in medical imaging: A survey,” _Medical Image Analysis_ , p. 102802, 2023.
  * [176] H. Bao, L. Dong, F. Wei, W. Wang, N. Yang, X. Liu, Y. Wang, J. Gao, S. Piao, M. Zhou _et al._ , “Unilmv2: Pseudo-masked language models for unified language model pre-training,” in _International conference on machine learning_. PMLR, 2020, pp. 642–652.
  * [177] L. Dong, N. Yang, W. Wang, F. Wei, X. Liu, Y. Wang, J. Gao, M. Zhou, and H.-W. Hon, “Unified language model pre-training for natural language understanding and generation,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [178] Y. Tay, M. Dehghani, V. Q. Tran, X. Garcia, D. Bahri, T. Schuster, H. S. Zheng, N. Houlsby, and D. Metzler, “Unifying language learning paradigms,” _arXiv preprint arXiv:2205.05131_ , 2022.
  * [179] Y. Sun, S. Wang, Y. Li, S. Feng, H. Tian, H. Wu, and H. Wang, “Ernie 2.0: A continual pre-training framework for language understanding,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 34, no. 05, 2020, pp. 8968–8975.
  * [180] Y. Sun, S. Wang, S. Feng, S. Ding, C. Pang, J. Shang, J. Liu, X. Chen, Y. Zhao, Y. Lu _et al._ , “Ernie 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation,” _arXiv preprint arXiv:2107.02137_ , 2021.
  * [181] X. Yang, N. PourNejatian, H. C. Shin, K. E. Smith, C. Parisien, C. Compas, C. Martin, M. G. Flores, Y. Zhang, T. Magoc _et al._ , “Gatortron: A large language model for clinical natural language processing,” _medRxiv_ , pp. 2022–02, 2022.
  * [182] V. Liévin, C. E. Hother, and O. Winther, “Can large language models reason about medical questions?” _arXiv preprint arXiv:2207.08143_ , 2022.
  * [183] H. Nori, N. King, S. M. McKinney, D. Carignan, and E. Horvitz, “Capabilities of gpt-4 on medical challenge problems,” _arXiv preprint arXiv:2303.13375_ , 2023.
  * [184] Z. Liu, X. Yu, L. Zhang, Z. Wu, C. Cao, H. Dai, L. Zhao, W. Liu, D. Shen, Q. Li _et al._ , “Deid-gpt: Zero-shot medical text de-identification by gpt-4,” _arXiv preprint arXiv:2303.11032_ , 2023.
  * [185] H. Xiong, S. Wang, Y. Zhu, Z. Zhao, Y. Liu, Q. Wang, and D. Shen, “Doctorglm: Fine-tuning your chinese doctor is not a herculean task,” _arXiv preprint arXiv:2304.01097_ , 2023.
  * [186] T. Han, L. C. Adams, J.-M. Papaioannou, P. Grundmann, T. Oberhauser, A. Löser, D. Truhn, and K. K. Bressem, “Medalpaca–an open-source collection of medical conversational ai models and training data,” _arXiv preprint arXiv:2304.08247_ , 2023.
  * [187] H. Wang, C. Liu, N. Xi, Z. Qiang, S. Zhao, B. Qin, and T. Liu, “Huatuo: Tuning llama model with chinese medical knowledge,” 2023.
  * [188] C. Wu, X. Zhang, Y. Zhang, Y. Wang, and W. Xie, “Pmc-llama: Further finetuning llama on medical papers,” _arXiv preprint arXiv:2304.14454_ , 2023.
  * [189] C. Yirong, W. Zhenyu, X. Xiaofen, X. Zhipei, F. Kai, L. Sihang, W. Junhong, and X. Xiangmin, “Bianque-1.0: Improving the ”question” ability of medical chat model through finetuning with hybrid instructions and multi-turn doctor qa datasets,” 2023. [Online]. Available: https://github.com/scutcyr/BianQue
  * [190] C. Peng, X. Yang, A. Chen, K. E. Smith, N. PourNejatian, A. B. Costa, C. Martin, M. G. Flores, Y. Zhang, T. Magoc _et al._ , “A study of generative large language model for medical research and healthcare,” _arXiv preprint arXiv:2305.13523_ , 2023.
  * [191] G. Wang, G. Yang, Z. Du, L. Fan, and X. Li, “Clinicalgpt: Large language models finetuned with diverse medical data and comprehensive evaluation,” _arXiv preprint arXiv:2306.09968_ , 2023.
  * [192] J. Zhou, X. Chen, and X. Gao, “Path to medical agi: Unify domain-specific medical llms with the lowest cost,” _arXiv preprint arXiv:2306.10765_ , 2023.
  * [193] C. Li, C. Wong, S. Zhang, N. Usuyama, H. Liu, J. Yang, T. Naumann, H. Poon, and J. Gao, “Llava-med: Training a large language-and-vision assistant for biomedicine in one day,” _arXiv preprint arXiv:2306.00890_ , 2023.
  * [194] W. Gao, Z. Deng, Z. Niu, F. Rong, C. Chen, Z. Gong, W. Zhang, D. Xiao, F. Li, Z. Cao _et al._ , “Ophglm: Training an ophthalmology large language-and-vision assistant based on instructions and dialogue,” _arXiv preprint arXiv:2306.12174_ , 2023.
  * [195] C. Yirong, X. Xiaofen, W. Zhenyu, and X. Xiangmin, “Soulchat: The ”empathy” ability of the large model is improved by mixing and fine-tuning the data set of long text consultation instructions and multiple rounds of empathy dialogue,” 6 2023. [Online]. Available: https://github.com/scutcyr/SoulChat
  * [196] M. Moor, Q. Huang, S. Wu, M. Yasunaga, C. Zakka, Y. Dalmia, E. P. Reis, P. Rajpurkar, and J. Leskovec, “Med-flamingo: a multimodal medical few-shot learner,” _arXiv preprint arXiv:2307.15189_ , 2023.
  * [197] Ö. Uzuner, B. R. South, S. Shen, and S. L. DuVall, “2010 i2b2/va challenge on concepts, assertions, and relations in clinical text,” _Journal of the American Medical Informatics Association_ , vol. 18, no. 5, pp. 552–556, 2011.
  * [198] W. Sun, A. Rumshisky, and O. Uzuner, “Evaluating temporal relations in clinical text: 2012 i2b2 challenge,” _Journal of the American Medical Informatics Association_ , vol. 20, no. 5, pp. 806–813, 2013.
  * [199] X. Yang, J. Bian, R. Fang, R. I. Bjarnadottir, W. R. Hogan, and Y. Wu, “Identifying relations of medications with adverse drug events using recurrent convolutional neural networks and gradient boosting,” _Journal of the American Medical Informatics Association_ , vol. 27, no. 1, pp. 65–72, 2020.
  * [200] Y. Wang, S. Fu, F. Shen, S. Henry, O. Uzuner, H. Liu _et al._ , “The 2019 n2c2/ohnlp track on clinical semantic textual similarity: overview,” _JMIR medical informatics_ , vol. 8, no. 11, p. e23375, 2020.
  * [201] C. Shivade, “Mednli-a natural language inference dataset for the clinical domain (version 1.0. 0). physionet,” 2019.
  * [202] A. Pampari, P. Raghavan, J. Liang, and J. Peng, “emrqa: A large corpus for question answering on electronic medical records,” _arXiv preprint arXiv:1809.00732_ , 2018.
  * [203] S. Smith, M. Patwary, B. Norick, P. LeGresley, S. Rajbhandari, J. Casper, Z. Liu, S. Prabhumoye, G. Zerveas, V. Korthikanti _et al._ , “Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model,” _arXiv preprint arXiv:2201.11990_ , 2022.
  * [204] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman _et al._ , “Evaluating large language models trained on code,” _arXiv preprint arXiv:2107.03374_ , 2021.
  * [205] D. Jin, E. Pan, N. Oufattole, W.-H. Weng, H. Fang, and P. Szolovits, “What disease does this patient have? a large-scale open domain question answering dataset from medical exams,” _Applied Sciences_ , vol. 11, no. 14, p. 6421, 2021.
  * [206] D. Hendrycks and K. Gimpel, “Gaussian error linear units (gelus),” _arXiv preprint arXiv:1606.08415_ , 2016.
  * [207] O. Press, N. A. Smith, and M. Lewis, “Train short, test long: Attention with linear biases enables input length extrapolation,” _arXiv preprint arXiv:2108.12409_ , 2021.
  * [208] A. Nentidis, G. Katsimpras, E. Vandorou, A. Krithara, L. Gasco, M. Krallinger, and G. Paliouras, “Overview of bioasq 2021: The ninth bioasq challenge on large-scale biomedical semantic indexing and question answering,” in _Experimental IR Meets Multilinguality, Multimodality, and Interaction: 12th International Conference of the CLEF Association, CLEF 2021, Virtual Event, September 21–24, 2021, Proceedings 12_. Springer, 2021, pp. 239–263.
  * [209] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , 2002, pp. 311–318.
  * [210] A. B. Abacha, E. Agichtein, Y. Pinter, and D. Demner-Fushman, “Overview of the medical question answering task at trec 2017 liveqa.” in _TREC_ , 2017, pp. 1–12.
  * [211] A. B. Abacha, Y. Mrabet, M. Sharp, T. R. Goodwin, S. E. Shooshan, and D. Demner-Fushman, “Bridging the gap between consumers’ medication questions and trusted answers.” in _MedInfo_ , 2019, pp. 25–29.
  * [212] P. Barham, A. Chowdhery, J. Dean, S. Ghemawat, S. Hand, D. Hurt, M. Isard, H. Lim, R. Pang, S. Roy _et al._ , “Pathways: Asynchronous distributed dataflow for ml,” _Proceedings of Machine Learning and Systems_ , vol. 4, pp. 430–449, 2022.
  * [213] H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, E. Li, X. Wang, M. Dehghani, S. Brahma _et al._ , “Scaling instruction-finetuned language models,” _arXiv preprint arXiv:2210.11416_ , 2022.
  * [214] R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto, “Stanford alpaca: An instruction-following llama model,” https://github.com/tatsu-lab/stanford_alpaca, 2023.
  * [215] Z. Du, Y. Qian, X. Liu, M. Ding, J. Qiu, Z. Yang, and J. Tang, “Glm: General language model pretraining with autoregressive blank infilling,” in _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2022, pp. 320–335.
  * [216] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” _arXiv preprint arXiv:2106.09685_ , 2021.
  * [217] T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer, “Llm. int8 (): 8-bit matrix multiplication for transformers at scale,” _arXiv preprint arXiv:2208.07339_ , 2022.
  * [218] T. Dettmers, M. Lewis, S. Shleifer, and L. Zettlemoyer, “8-bit optimizers via block-wise quantization,” _arXiv preprint arXiv:2110.02861_ , 2021.
  * [219] K. Lo, L. L. Wang, M. Neumann, R. Kinney, and D. S. Weld, “S2orc: The semantic scholar open research corpus,” in _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 2020, pp. 4969–4983.
  * [220] J. Fries, L. Weber, N. Seelam, G. Altay, D. Datta, S. Garda, S. Kang, R. Su, W. Kusa, S. Cahyawijaya _et al._ , “Bigbio: a framework for data-centric biomedical natural language processing,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 25 792–25 806, 2022.
  * [221] L. X. Xuanwei Zhang and K. Zhao, “Chatyuan: A large language model for dialogue in chinese and english,” Dec. 2022. [Online]. Available: https://github.com/clue-ai/ChatYuan
  * [222] S. Chen, Z. Ju, X. Dong, H. Fang, S. Wang, Y. Yang, J. Zeng, R. Zhang, R. Zhang, M. Zhou, P. Zhu, and P. Xie, “Meddialog: a large-scale medical dialogue dataset,” _arXiv preprint arXiv:2004.03329_ , 2020.
  * [223] W. Chen, Z. Li, H. Fang, Q. Yao, C. Zhong, J. Hao, Q. Zhang, X. Huang, J. Peng, and Z. Wei, “A Benchmark for Automatic Medical Consultation System: Frameworks, Tasks and Datasets,” _Bioinformatics_ , 12 2022, btac817. [Online]. Available: https://doi.org/10.1093/bioinformatics/btac817
  * [224] N. Zhang, M. Chen, Z. Bi, X. Liang, L. Li, X. Shang, K. Yin, C. Tan, J. Xu, F. Huang, L. Si, Y. Ni, G. Xie, Z. Sui, B. Chang, H. Zong, Z. Yuan, L. Li, J. Yan, H. Zan, K. Zhang, B. Tang, and Q. Chen, “CBLUE: A Chinese biomedical language understanding evaluation benchmark,” in _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Dublin, Ireland: Association for Computational Linguistics, May 2022, pp. 7888–7915. [Online]. Available: https://aclanthology.org/2022.acl-long.544
  * [225] W. Chen, Z. Li, H. Fang, Q. Yao, C. Zhong, J. Hao, Q. Zhang, X. Huang, J. Peng, and Z. Wei, “A Benchmark for Automatic Medical Consultation System: Frameworks, Tasks and Datasets,” _Bioinformatics_ , 12 2022, btac817.
  * [226] S. Zhang, X. Zhang, H. Wang, L. Guo, and S. Liu, “Multi-scale attentive interaction networks for chinese medical question answer selection,” _IEEE Access_ , vol. 6, pp. 74 061–74 071, 2018.
  * [227] R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen _et al._ , “Palm 2 technical report,” _arXiv preprint arXiv:2305.10403_ , 2023.
  * [228] L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima _et al._ , “The pile: An 800gb dataset of diverse text for language modeling,” _arXiv preprint arXiv:2101.00027_ , 2020.
  * [229] M. Herrero-Zazo, I. Segura-Bedmar, P. Martínez, and T. Declerck, “The ddi corpus: An annotated corpus with pharmacological substances and drug–drug interactions,” _Journal of biomedical informatics_ , vol. 46, no. 5, pp. 914–920, 2013.
  * [230] J. Li, Y. Sun, R. J. Johnson, D. Sciaky, C.-H. Wei, R. Leaman, A. P. Davis, C. J. Mattingly, T. C. Wiegers, and Z. Lu, “Biocreative v cdr task corpus: a resource for chemical disease relation extraction,” _Database_ , vol. 2016, 2016.
  * [231] Y. Hou, Y. Xia, L. Wu, S. Xie, Y. Fan, J. Zhu, T. Qin, and T.-Y. Liu, “Discovering drug–target interaction knowledge from biomedical literature,” _Bioinformatics_ , vol. 38, no. 22, pp. 5100–5107, 2022.
  * [232] S. Zhang, X. Zhang, H. Wang, L. Guo, and S. Liu, “Multi-scale attentive interaction networks for chinese medical question answer selection,” _IEEE Access_ , vol. 6, pp. 74 061–74 071, 2018.
  * [233] J. He, M. Fu, and M. Tu, “Applying deep matching networks to chinese medical question answering: a study and a dataset,” _BMC medical informatics and decision making_ , vol. 19, no. 2, pp. 91–100, 2019.
  * [234] J. Li, X. Wang, X. Wu, Z. Zhang, X. Xu, J. Fu, P. Tiwari, X. Wan, and B. Wang, “Huatuo-26m, a large-scale chinese medical qa dataset,” _arXiv preprint arXiv:2305.01526_ , 2023.
  * [235] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” _The Journal of Machine Learning Research_ , vol. 21, no. 1, pp. 5485–5551, 2020.
  * [236] X. He, S. Chen, Z. Ju, X. Dong, H. Fang, S. Wang, Y. Yang, J. Zeng, R. Zhang, R. Zhang _et al._ , “Meddialog: Two large-scale medical dialogue datasets,” _arXiv preprint arXiv:2004.03329_ , 2020.
  * [237] H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” _arXiv preprint arXiv:2304.08485_ , 2023.
  * [238] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark _et al._ , “Learning transferable visual models from natural language supervision,” in _International conference on machine learning_. PMLR, 2021, pp. 8748–8763.
  * [239] S. Zhang, Y. Xu, N. Usuyama, J. Bagga, R. Tinn, S. Preston, R. Rao, M. Wei, N. Valluri, C. Wong _et al._ , “Large-scale domain-specific pretraining for biomedical vision-language processing,” _arXiv preprint arXiv:2303.00915_ , 2023.