# RareBench: Can LLMs Serve as Rare Diseases Specialists?

Xuanzhong Chen  Department of Computer Science and Technology & Institute of
Artificial Intelligence & BNRist, Tsinghua UniversityBeijingChina
<EMAIL_ADDRESS>,  Xiaohao Mao  Department of Computer Science and
Technology & Institute of Artificial Intelligence & BNRist, Tsinghua
UniversityBeijingChina<EMAIL_ADDRESS>,  Qihan Guo  Department of
Computer Science and Technology & Institute of Artificial Intelligence &
BNRist, Tsinghua UniversityBeijingChina<EMAIL_ADDRESS>,  Lun
Wang  Department of Internal Medicine, Peking Union Medical College Hospital,
Chinese Academy of Medical Sciences & Peking Union Medical CollegeBeijingChina
<EMAIL_ADDRESS>,  Shuyang Zhang  Department of Cardiology, Peking Union
Medical College Hospital, Chinese Academy of Medical Sciences & Peking Union
Medical CollegeBeijingChina<EMAIL_ADDRESS>and  Ting Chen
Department of Computer Science and Technology & Institute of Artificial
Intelligence & BNRist, Tsinghua UniversityBeijingChina
<EMAIL_ADDRESS>
(2018)

###### Abstract.

Generalist Large Language Models (LLMs), such as GPT-4, have shown
considerable promise in various domains, including medical diagnosis. Rare
diseases, affecting approximately 300 million people worldwide, often have
unsatisfactory clinical diagnosis rates primarily due to a lack of experienced
physicians and the complexity of differentiating among many rare diseases. In
this context, recent news such as ”ChatGPT correctly diagnosed a 4-year-old’s
rare disease after 17 doctors failed” underscore LLMs’ potential, yet
underexplored, role in clinically diagnosing rare diseases. To bridge this
research gap, we introduce RareBench, a pioneering benchmark designed to
systematically evaluate the capabilities of LLMs on 4 critical dimensions
within the realm of rare diseases. Meanwhile, we have compiled the largest
open-source dataset on rare disease patients, establishing a benchmark for
future studies in this domain. To facilitate differential diagnosis of rare
diseases, we develop a dynamic few-shot prompt methodology, leveraging a
comprehensive rare disease knowledge graph synthesized from multiple knowledge
bases, significantly enhancing LLMs’ diagnostic performance. Moreover, we
present an exhaustive comparative study of GPT-4’s diagnostic capabilities
against those of specialist physicians. Our experimental findings underscore
the promising potential of integrating LLMs into the clinical diagnostic
process for rare diseases. This paves the way for exciting possibilities in
future advancements in this field.

benchmark for LLMs, rare disease diagnosis, evaluation

††copyright: acmlicensed††journalyear: 2018††doi: XXXXXXX.XXXXXXX††conference:
Make sure to enter the correct conference title from your rights confirmation
emai; June 03–05, 2018; Woodstock, NY††isbn: 978-1-4503-XXXX-X/18/06††ccs:
Applied computing Health informatics

##  1\. Introduction

Large Language Models (LLMs) like ChatGPT (OpenAI, 2022a) have obtained
widespread attention for their exceptional human-like language understanding
and generation capabilities. As a result, applying LLMs in medicine is
emerging as a promising research direction in artificial intelligence and
clinical medicine. Several studies have been taken to explore how LLMs can
assist doctors in various medical and clinical tasks, including medical
diagnosis (Kwon et al., 2023; Wang et al., 2023), clinical report generation
(Yang et al., 2022; Singhal et al., 2023), and medicine education (Kung et
al., 2023). However, there is currently a lack of research investigating the
capabilities and limitations of LLMs in the context of rare diseases.

Rare diseases collectively refer to a broad category of diseases, typically
defined by their low prevalence in the population. Over 7,000 types of rare
diseases are currently recognized (Haendel et al., 2020), with approximately
80% being genetic in origin. Patients with rare diseases often face a high
probability of misdiagnosis or underdiagnosis (Marwaha et al., 2022), and the
average time before receiving a confirmative diagnosis extends over several
years (Evans, 2023). The difficulty in diagnosis is largely attributed to the
lack of prior exposure to rare diseases among physicians, hindering the
accurate recognition of rare diseases and their associated phenotypes (such as
symptoms). Additionally, there is significant phenotypic overlap among
different rare diseases, as well as between rare diseases and common diseases,
which further increases the difficulty of disease identification and
diagnosis. Clinical diagnosis of rare diseases typically involves two primary
steps. Initially, physicians collect clinical information from patients,
including epidemiological information, symptoms, signs, past medical history,
and family history, etc., to formulate an initial diagnosis. Next, specialized
tests such as laboratory tests or imaging examinations will be conducted to
further facilitate diagnosis and differential diagnosis. Additionally, due to
the frequent involvement of numerous organs and systems in rare diseases,
consulting specialists from different fields during the diagnostic process can
help achieve a more comprehensive insight and final diagnosis.

There have been many prior works to improve the diagnosis of rare diseases,
including standardizing disease phenotype terminology into a hierarchical
structure in the Human Phenotype Ontology (HPO) (Köhler et al., 2021, 2017,
2009) and building knowledge bases for rare diseases such as the Online
Mendelian Inheritance in Man (OMIM) (Hamosh et al., 2005), Orphanet (Aymé,
2003), and the Compendium of China’s Rare Diseases (CCRD) (He et al., 2019).
These efforts result in a clear and structured representation of rare
diseases: a disease or a patient can be represented by a set of associated
phenotypes. From a machine learning perspective, computational methods can be
developed to classify or rank diseases based on phenotype information. These
computational methods can be classified into two main categories. The first
category treats the diagnosis of rare diseases as a ranking problem. In these
approaches, a patient or a disease can be represented as phenotype vectors.
Diseases are subsequently ranked by computing their semantic similarities with
the patient. Methods falling into this category include PhenoMizer (Köhler et
al., 2009), RDAD (Jia et al., 2018), RDD (Pinol et al., 2017), and LIRICAL
(Robinson et al., 2020). However, these methods are constrained by their
underlying assumptions, the lack of good-quality cases for training and
testing, and the incomplete phenotypic information on many rare diseases in
knowledge bases, often leading to relatively poorer diagnostic performance.
The second category treats the diagnosis of rare diseases as an extreme multi-
class classification task. Due to the scarcity of real-world data and the vast
number of rare diseases to classify, this becomes a typical few-shot
classification problem. Additionally, because of a lack of large-scale public
rare disease patient datasets, most computational methods were only tested on
simulated rare disease patient cases or small disease datasets with few
diseases involved. Therefore, the clinical diagnostic capability of these
methods remains unclear.

The prerequisite for diagnosing rare diseases using computational methods is
to extract standardized and essential phenotypes/symptoms from electronic
health records (EHRs) of clinical cases. To map clinical texts into
standardized phenotypes, various natural language processing (NLP) methods
have been developed, including EHR-Phenolyzer (Son et al., 2018), ClinPhen
(Deisseroth et al., 2019), PhenoTagger (Luo et al., 2021), and PhenoBERT (Feng
et al., 2022). In a recent study, PhenoBCBERT and PhenoGPT (Yang et al., 2023)
models were introduced to identify clinical phenotypes in clinical texts from
pediatric patients. However, there are substantial variations in clinical
texts in how physicians may record patient phenotypes, incorporating distinct
details and terminology. Moreover, the current count of human phenotypes
surpasses 17,000 (Köhler et al., 2021). All these factors present a
significant challenge in precisely mapping or deducing phenotypes from
clinical texts.

Our work leverages LLMs to conduct comprehensive evaluations in the
challenging field of rare diseases. Figure 1 displays the overview of
evaluation results for the four tasks of RareBench. Specifically, Task 4
(Differential Diagnosis among Universal Rare Diseases) also includes the
average diagnostic performance of 50 specialist physicians on 75 high-quality
case records from the PUMCH dataset, along with the 95% confidence interval.
The main contributions are: 1) Dataset and Benchmarking: We develop a diverse,
multi-center and specifically tailored dataset for rare diseases. Alongside
this, we introduce RareBench, a comprehensive benchmarking framework for
evaluating LLMs’ capabilities in real-world complex clinical scenarios like
phenotype extraction and differential diagnosis. 2) Advanced Knowledge
Integration Prompt: We integrate rich knowledge sources to create an
exhaustive knowledge graph for rare diseases. Utilizing a disease-phenotype
graph and the hierarchical structure of the phenotype graph, we devise a novel
random walk algorithm capitalizing on phenotype Information Content (IC)
values to implement dynamic few-shot prompting strategies. This advancement
significantly boosts the performance of LLMs excluding GPT-4 in differential
diagnosis, even surpassing GPT-4. 3) Human-versus-LLMs Comparative Studies: We
demonstrate GPT-4 on par with senior doctors across five distinct specialties
in the differential diagnosis of rare diseases through comparative analysis.
The experiments show that GPT-4’s diagnostic capabilities in rare diseases are
now commensurate with those of experienced specialist physicians.

Figure 1. RareBench’s overview of evaluation results.

##  2\. Related Work

Figure 2. RareBench is the first pioneering benchmark to evaluate LLMs as rare
disease specialists on 4 distinct tasks.

##### Medical Benchmarks for LLMs.

Prominent medical question-and-answer data benchmarks, such as MedQA (Jin et
al., 2021), PubMedQA (Jin et al., 2019), MedMCQA (Pal et al., 2022),
MultiMedQA (Singhal et al., 2023), and CMExam (Liu et al., 2023b), primarily
derive from medical examinations, typically featuring multiple-choice formats.
MedBench (Cai et al., 2023) introduces a large-scale Chinese benchmark for
evaluating LLMs in clinical knowledge and diagnostics. Additionally, DDXPlus
(Fansi Tchango et al., 2022) provides a detailed medical diagnostic dataset
covering symptoms and diagnoses. Our RareBench extends this landscape by
focusing on complex clinical scenarios specific to rare diseases.

##### LLMs’ Medical Capability.

The evolution of General Medical Artificial Intelligence (GMAI) is reshaping
healthcare by automating learning processes and incorporating domain knowledge
to reduce the clinical workload (Moor et al., 2023). GPT-4 (Achiam et al.,
2023), a notable example in this field, has demonstrated exceptional skills in
medical questions (Nori et al., 2023a) and rivals or surpasses state-of-the-
art models in tasks such as radiology text interpretation (Liu et al., 2023a).
Medprompt (Nori et al., 2023b) further enhances this capability through
specialized prompting techniques, enabling foundational models like GPT-4 to
outperform dedicated healthcare models. Besides GPT-4, models like AMIE (Tu et
al., 2024) show superior performance in specific medical tasks, even exceeding
the diagnostic abilities of general primary care physicians in some cases.
These LLMs not only assist in differential diagnosis but also engage in
clinical reasoning, thereby improving diagnostic accuracy (McDuff et al.,
2023; Kwon et al., 2023). Moreover, fine-tuned LLMs can efficiently extract
valuable data from clinical notes, significantly boosting patient care quality
(Vaid et al., 2023). Despite these advancements, challenges in accuracy,
interpretability, and safety persist in the medical application of LLMs,
underscoring the need for continuous refinement (Thirunavukarasu et al., 2023;
Li et al., 2023). Notably, the potential of LLMs in rare disease contexts is
yet to be fully explored, and our research aims to fill this gap.

##### Diagnosis of Rare Disease.

The initial step in clinical diagnosis involves extracting standardized
phenotypes from a patient’s electronic health record (EHR). To translate
clinical texts into standardized Human Phenotype Ontology (HPO) terms, various
natural language processing (NLP) methods have been developed, such as EHR-
Phenolyzer (Son et al., 2018), Doc2hpo (Liu et al., 2019), ClinPhen
(Deisseroth et al., 2019), PhenoTagger (Luo et al., 2021), and PhenoBERT (Feng
et al., 2022). In a recent study (Yang et al., 2023), PhenoBCBERT and PhenoGPT
models were introduced to discern clinical phenotypes in textual clinical
records of pediatric patients. These models also demonstrated the capability
to identify critical phenotypes that the HPO may not document.

For disease diagnosis, current computational methods can be broadly classified
into two categories from a machine-learning perspective. The first category
includes Phen2Disease (Zhai et al., 2023), LIRICAL (Robinson et al., 2020),
Xrare (Li et al., 2019), Phen2Gene (Zhao et al., 2020), EHR-Phenolyzer (Son et
al., 2018). The second category comprises many statistical or machine
learning-based methods, including Phenomizer (Köhler et al., 2009), RDAD (Jia
et al., 2018), RDD (Pinol et al., 2017), and MinIC (Peng et al., 2016).

##  3\. Composition of RareBench

This section presents 4 critical tasks of the RareBench framework with the
most extensive collection of rare disease patient datasets currently
accessible. These tasks include 1) phenotype extraction from EHRs, 2)
screening for specific rare diseases, 3) comparative analysis of common and
rare diseases, and 4) differential diagnosis among universal rare diseases.
Figure 2 demonstrates the process of employing LLMs as rare disease
specialists to complete the four tasks in the RareBench framework using an EHR
or a set of phenotypes of a patient with ALS (Amyotrophic Lateral Sclerosis)
from PUMCH. We also describe our prompting techniques for effectively
deploying LLMs as rare disease specialists. Please refer to the Appendix due
to the page limit for further dataset details and prompt examples.

Table 1. Rare disease datasets of RareBench’s four tasks. Dataset | Type | #.Diseases | #.Cases | Source
---|---|---|---|---
Task 1 | EHRs | 34 | 87 | PUMCH
Task 2 | EHRs | 3 | 33 | PUMCH
Task 3 | EHRs | 77 | 527 | PUMCH
Task 4 | EHRs/symptoms | 421 | 2,185 | Public & PUMCH

###  3.1. Tasks of the RareBench Framework

####  3.1.1. Task 1: Phenotype Extraction from Electronic Health Records

Task 1 involves deriving phenotypes from EHRs for diagnosis. We design 3 sub-
tasks: a) Phenotype Extraction: Extracting precise phenotypes from EHRs. b)
General Entity Extraction: Extracting general entities from EHRs. c) General
Entity Standardization: Standardizing general entities into phenotypes.
General entity extraction and standardization can be considered a two-step
decomposition of phenotype extraction, allowing for a more detailed evaluation
of LLMs’ capabilities of phenotype extraction.

####  3.1.2. Task 2: Screening for Specific Rare Diseases

Task 2 aims to evaluate the capability of LLMs in identifying risk factors or
symptoms associated with specific rare diseases. It utilizes patients’ medical
histories and auxiliary examinations to discover potential rare diseases and
facilitate further diagnosis. For this study, three rare diseases are
selected: ALS (Amyotrophic Lateral Sclerosis), PNH (Paroxysmal Nocturnal
Hemoglobinuria), and MSA (Multiple System Atrophy).

####  3.1.3. Task 3: Comparison Analysis of Common and Rare Diseases

Task 3 aims to validate whether LLMs can differentiate between patients with
common and rare diseases that exhibit similar phenotypes/symptoms. From the
PUMCH dataset, we select 527 electronic health records containing 60 cases
with 13 rare diseases and 467 cases with 64 common diseases, respectively. The
task of LLMs is to predict the top ten most likely diseases from the mentioned
pool of 77 diseases based on a patient’s electronic health record.

####  3.1.4. Task 4: Differential Diagnosis among Universal Rare Diseases

Task 4 is centered on a systematic differential diagnosis across the full
spectrum of known rare diseases to identify the most probable rare disease.
Unlike task 3, this task does not limit the range of rare diseases. Instead,
it leverages LLMs to provide the top ten most likely diagnoses based solely on
the patient’s phenotypes/symptoms. Task 4 involves 2,185 patient cases
encompassing 421 rare diseases, collected from both Public and PUMCH datasets.
Its primary objective is to evaluate LLMs’ capability within the complexities
of real-world clinical scenarios. As the most pivotal and forward-thinking
task of RareBench, it plays a crucial role in accessing the potential of LLMs
in handling intricate medical data.

###  3.2. Rare Disease Patients’ Dataset

This study categorizes datasets into two main groups: publicly available
datasets and the Peking Union Medical College Hospital (PUMCH) datasets. All
of these data are utilized to perform one of the four tasks, as illustrated in
Table 1. The publicly available datasets provide phenotypes/symptoms alongside
confirmed rare diagnoses in OMIM/Orphanet disease codes. Consequently, they
can only be employed to assess the performance of differential diagnosis among
universal rare diseases (Task 4). In total, public datasets include 1,122
cases spanning over 362 rare diseases. Further descriptions can be found in
Appendix A.1.

Electronic health records from PUMCH serve as valuable resources for both
phenotype extraction and three diagnostic tasks. The PUMCH dataset comprises a
total of 1,650 cases, consisting of 1,183 rare disease cases and 467 common
disease cases. Among them, we select specific datasets for each of the four
tasks. Specifically, we first choose 87 EHRs involving 34 rare diseases
annotated with physician-identified phenotypes to perform task 1. Next, we
select 33 cases with complete auxiliary examination and medical history
information to execute task 2. Then, from the remaining medical records
(excluding those for tasks 1 and 2), we select 60 cases with rare diseases and
467 cases with common diseases exhibiting phenotypes similar to those of the
corresponding rare diseases for task 3. Finally, for task 4, we test all the
remaining rare disease records (excluding those for tasks 1 and 2), along with
all the aforementioned public cases, totaling 2,185 cases. It’s important to
note that the patient’s personally identifiable information has been removed
completely. Additionally, doctors from PUMCH monitored all cases before
uploading text information, ensuring the absence of any potential personal
information leaks. Moreover, we apply reasonable filtering criteria to
identify and remove cases of low quality that may be caused by recording
errors or missing information, such as those with uncertain or imprecise
diagnoses and those lacking sufficient relevant information, i.e., fewer than
three phenotypes.

###  3.3. Framework Setup

####  3.3.1. Evaluated Models

We select eight models in our evaluation framework, including API-based and
open-source LLMs, as detailed in Table 2. Specifically, we choose 5 API-based
models, which exhibit superior performance as a result of substantial
investment and advanced development. On the other hand, due to limitations in
computational resources, our selection of open-source models is confined to 3
representatives, each with a model size of fewer than 10 billion parameters.

Table 2. Eight LLMs evaluated as rare disease specialists. Model | #Size | Form | Version
---|---|---|---
GPT-4 (Achiam et al., 2023) | N/A | API | 1106-preview
GPT-3.5-Turbo (OpenAI, 2022a) | N/A | API | 1106
Gemini Pro (Team et al., 2023) | N/A | API | -
GLM4 (Zeng et al., 2022; Du et al., 2022) | N/A | API | -
GLM3-Turbo (Zeng et al., 2022; Du et al., 2022) | N/A | API | -
Mistral-7B (Jiang et al., 2023) | 7B | Open Source | instruct-v0.1
Llama2-7B (Touvron et al., 2023) | 7B | Open Source | chat
ChatGLM3-6B (Zeng et al., 2022; Du et al., 2022) | 6B | Open Source | -

####  3.3.2. Basic Prompt Design

For the evaluation of 8 LLMs, we primarily utilize the most fundamental zero-
shot prompt. We assign the role of a rare diseases specialist to the LLMs by
incorporating ”You are a specialist in the field of rare diseases.” as the
system prompt/initial statement. Additional details on the configuration of
LLMs’ hyper-parameters are available in Appendix A.2.

####  3.3.3. More Prompting Strategies Exploration on GPT-4

We further explore diverse prompting strategies with GPT-4, including Chain-
of-Thought (CoT) (Wei et al., 2022) and random few-shot methods. In the CoT
settings, the zero-shot prompt is supplemented with ”Let us think step by
step.” to foster a sequential thought process, a technique validated in
various general tasks. For random few shots, we provide LLMs with $m$ random
complete input-output examples as prompts, where the choice of $m$ depends on
the specific task.

###  3.4. Knowledge Integration Dynamic Few-shot

Beyond the prompts above, we construct a rare disease domain knowledge graph
by integrating multiple public knowledge bases. This serves as the foundation
for our implemented Information Content (IC) based random walk algorithm. The
phenotype embeddings generated via this algorithm have been pivotal in
developing a dynamic few-shot prompt, depicted in Figure 3. This innovative
approach markedly improves the capabilities of LLMs in differential diagnoses
among universal rare diseases (Task 4).

####  3.4.1. Rare Disease Knowledge Graph Construction

Previous methods (Köhler et al., 2009; Jia et al., 2018; Pinol et al., 2017;
Peng et al., 2016) for the differential diagnosis of rare diseases primarily
rely on similarity calculations between diseases and phenotypes in knowledge
bases. It is feasible to construct a knowledge graph wherein both rare
diseases and phenotypes are represented as nodes, connected by their
interrelations as edges. The HPO hierarchy organizes phenotypes into a
directed acyclic graph where each edge connects a broadly defined phenotype
(parent) to a more specific phenotype (child). For disease-phenotype
information, we integrate data from 4 rare-disease-related knowledge bases:
the Human Phenotype Ontology (HPO) (Köhler et al., 2021, 2017, 2009), Online
Mendelian Inheritance in Man (OMIM) (Hamosh et al., 2005), Orphanet (Aymé,
2003), and the Compendium of China’s Rare Diseases (CCRD) (He et al., 2019).
This integration notably enhances the annotation of associations between rare
diseases and phenotypes. The statistical information of the integrated
knowledge graph is presented in Table 3, with detailed descriptions of each
knowledge base available in Appendix A.4.

Table 3. Key statistics of the integrated rare disease knowledge graph encompassing nodes (P and RD) and edges (P-P and P-RD). The asterisk (”$*$”) indicates the consolidation of rare diseases from various knowledge bases into 9,260 unique entities. Type | Phenotype (P) | Rare Disease (RD) | P-P | P-RD | P-RD | P-RD
---|---|---|---|---|---|---
(Src.) | (HPO) |  ($*$) | (HPO) | (OMIM) | (Orphanet) | (CCRD)
Num. | 17232 | 9260 | 21505 | 54169 | 98031 | 4223

####  3.4.2. Random Walk Based on Information Content

The concept of Information Content (IC) (Cover, 1999) is similar to Inverse
Document Frequency (IDF) utilized in natural language processing. IC is
employed as an index of a phenotype’s specificity to a particular disease.
Notably, a phenotype’s proximity to the root node in the HPO hierarchy, or a
higher frequency of its association with multiple diseases, results in a lower
IC, reflecting lesser significance. On the other hand, a phenotype that is
highly specific to a rare disease has a high IC. The essence of IC lies in its
inverse correlation with the prevalence of a phenotype – the more common a
phenotype, the lower its IC. Let $T$ be the complete set of phenotype terms.
For a given term $t\in T$, the computation of $IC(t)$ is formulated as
follows:

$IC(t)=-log\frac{n(t)}{N},$

where $n(t)$ represents the count of diseases annotated by the HPO phenotype
$t$ and its descendant phenotypes, and $N$ signifies the total number of
annotated diseases. In our integrated rare disease knowledge graph, the value
of $N$ equals 9,260.

Node2vec (Grover and Leskovec, 2016), a typical shallow embedding method, aims
to simulate random walks (Perozzi et al., 2014) across a graph through a
flexible and parameterized walk strategy. These unsupervised walks generate
sequences of nodes, which are employed to create node embeddings by using the
methodologies developed for Word2vec (Mikolov et al., 2013). In Node2vec, the
search bias parameter $\alpha$ controls the search strategy in generating
random walk sequences of nodes, effectively modulating the preference for
breadth-first search (BFS) and depth-first search (DFS) strategies in
exploring neighboring nodes. However, the standard application of Node2vec to
our phenotype-disease knowledge graph in the rare disease domain falls short
of adequately capturing each phenotype’s distinct influence on the
differential diagnosis of diseases. We innovatively integrated IC values into
the Node2vec framework to address this limitation, formulating an IC value-
based random walk algorithm. This enhancement is designed to enrich the
interactions between phenotypes and rare diseases. Under this new scheme, when
a random walk progresses to a phenotype node $t_{1}$ and the subsequent node
is another phenotype $t_{2}$, the walk search bias from $t_{1}$ to $t_{2}$ is
determined as $\alpha=IC(t_{2})$. Conversely, if the following node is a rare
disease $d_{1}$, the search bias from $t_{1}$ to $d_{1}$ is set by
$\alpha=IC(t_{1})$. This modification ensures that phenotype nodes with higher
IC values receive increased focus during the random walk, amplifying
associations with the related rare disease nodes. For a detailed comparative
analysis between Node2vec and the IC value-based random walk, please refer to
Appendix A.4.

Figure 3. Integrated rare disease knowledge graph from 4 databases, with
implemented IC value-based random walk algorithm and dynamic few-shot prompt
schematic.

####  3.4.3. Dynamic Few-shot Prompting Strategy

Employing the IC value-based random walk algorithm, we have developed a
function $f:T\rightarrow\mathbb{R}^{d}$ to project phenotype terms into a
$d$-dimensional vector space, where we have chosen $d=256$. These embeddings
are then utilized to represent patients with rare diseases. A patient $p$ with
a rare disease $d$ is characterized by a set of phenotype terms $t_{x_{i}}$,
expressed as $p=\\{t_{x_{1}},t_{x_{2}},...,t_{x_{n}}\\}$. The embedding of the
patient $p$ is computed as follows:

$Embedding(p)=\frac{1}{\sum_{i=1}^{n}IC(t_{x_{i}})}\sum_{i=1}^{n}IC(t_{x_{i}})\cdot
f(t_{x_{i}}).$

Table 4. Comprehensive results of the RareBench’s 4 tasks. In task 1, ”PPE”, ”GEE”, and ”GES” respectively stand for precise phenotype extraction, general entity extraction, and general entity standardization. Bold numbers indicate the best results, while underlined numbers signify the second-best results. LLMs | Task 1: Phenotype | Task 2: Screening | Task 3: Comparative | Task 4: Differential Diagnosis
---|---|---|---|---
Extraction from EHRs | for Specific RDs | Analysis of Common & RDs | among Universal RDs on 2,185 cases
F1-score (%) ($\uparrow$) | Recall (%) ($\uparrow$) | Top-k Recall (%) ($\uparrow$) | MR ($\downarrow$) | Top-k Recall (%) ($\uparrow$) | MR ($\downarrow$)
PPE | GEE | GES | ALS | PNH | MSA | k=1 | k=3 | k=10 | k=1 | k=3 | k=10
GPT-4 0-shot | 24.5 | 64.9 | 38.7 | 62.5 | 57.1 | 61.1 | 46.1 | 59.6 | 72.1 | 3.0 | 32.3 | 45.4 | 58.9 | 5.0
3-shot | 26.0 | 61.9 | 42.0 | - | - | - | - | - | - | - | 30.4 | 43.8 | 57.9 | 5.0
CoT | 25.1 | 65.9 | 39.7 | 62.5 | 57.1 | 66.7 | 47.4 | 62.0 | 75.0 | 2.0 | 33.2 | 46.4 | 59.5 | 4.0
GPT-3.5-Turbo | 17.2 | 48.4 | 32.3 | 37.5 | 42.9 | 44.4 | 33.2 | 45.4 | 58.1 | 5.0 | 21.1 | 34.2 | 48.2 | >10
Gemini Pro | 10.1 | 50.8 | 34.3 | 25.0 | 28.6 | 33.3 | 24.3 | 32.6 | 44.8 | >10 | 14.6 | 22.7 | 33.0 | >10
GLM4 | 15.9 | 56.8 | 24.0 | 50.0 | 42.9 | 50.0 | 31.3 | 42.5 | 57.1 | 8.0 | 19.1 | 30.4 | 45.5 | >10
GLM3-Turbo | 12.9 | 53.5 | 31.4 | 12.5 | 14.3 | 22.2 | 25.8 | 37.8 | 51.2 | >10 | 12.4 | 21.0 | 33.2 | >10
Mistral-7B | 3.3 | 26.4 | 8.8 | 0.0 | 14.3 | 11.1 | 13.7 | 19.2 | 29.0 | >10 | 7.2 | 12.5 | 18.9 | >10
Llama2-7B | 0.0 | 0.0 | 0.0 | 0.0 | 14.3 | 5.6 | - | - | - | - | 7.4 | 11.0 | 14.8 | >10
ChatGLM3-6B | 10.3 | 48.3 | 14.6 | 12.5 | 0.0 | 5.6 | 9.7 | 16.1 | 22.8 | >10 | 5.0 | 7.2 | 10.7 | >10

In MedPrompt (Nori et al., 2023b), the dynamic few-shot method selects
training examples most similar to the specific input. However, it relies on a
general-purpose text embedding model such as text-embedding-ada-002 (OpenAI,
2022b), which can only measure the relatedness of phenotype text, without
delving into the deeper associations among phenotypes. Therefore, we utilize
the phenotype embeddings generated from our IC value-based random walk
algorithm to represent rare disease patients. We then select the top $m$ most
similar examples from the rare disease patient datasets. Data with the highest
cosine similarity serve as prompts. From LLMs’ perspective, such a retrieval
augmented generation (RAG) (Lewis et al., 2020) strategy enables LLMs to be
more effectively tailored to the rare disease domain, thereby conforming more
closely to differential diagnosis tasks and minimizing hallucinations through
enriched knowledge. From a clinical perspective, the diagnostic process of
specialist physicians relies on medical knowledge and past experiences of
those previously diagnosed patients. Consequently, a curated set of relevant
examples grants LLMs a distilled version of ”clinical experience”.

##  4\. Evaluation Results of RareBench

This section presents the comprehensive results of RareBench in Table 4,
including the evaluation of GPT-4 across three different settings: zero-shot,
few-shot, and Chain of Thought (CoT) with the zero-shot performance of the
other 7 LLMs.

###  4.1. Task 1: Phenotype Extraction from EHR

####  4.1.1. Metric

Task 1 is evaluated using precision, recall, and F1-score. Accuracy requires
exact matches with the reference for precise phenotype extraction and general
entity standardization. For general entity extraction, predictions are correct
if they convey the same meaning as the reference.

####  4.1.2. Results

Although GPT-4 achieves the best among all LLMs, the results show that all
LLMs perform poorly in precise phenotype extraction. The general entity
extraction and standardization results show that LLMs perform well in entity
extraction but have weak capabilities in standardizing entities into
phenotypes. This suggests that the main reason for the poor performance of
LLMs in phenotype extraction is their weak standardization ability.

Beyond Chain-of-Thought (CoT) and random few-shot, our experimentation
includes two additional methods: a) Word-to-phenotype matching, where each
word in the output is aligned with the semantically nearest word in the HPO
phenotype list. The semantic distance is measured by vectorizing words using
OpenAI’s text-embedding-ada-002 (OpenAI, 2022b) model and then assessing
cosine similarity; 2) Expanded semantic matching, where we associate each
output with the semantically closest $n$ words in the HPO phenotype list, then
integrate these matches as a reference output range before re-querying GPT-4.
In our tests, $n$ is set at 20. The F1-score of these two experiments are
0.306 and 0.322, respectively.

###  4.2. Task 2: Screening for Specific Rare Diseases

####  4.2.1. Metric

Screening for three specific rare diseases (ALS, PNH, MSA) is measured using
recall.

####  4.2.2. Results

For this task, GPT-4 again achieves the best performance on all three
diseases, with recalls exceeding 0.55 in both zero-shot and Chain of Thought
(CoT) settings, respectively. Meanwhile, we did not conduct few-shot learning
on GPT-4 due to the limited number of similar cases, as we were concerned that
it might influence or bias the subsequent diagnostic results. Other API-based
LLMs have recalls of less than or equal to 0.50 on all three diseases. Open-
source LLMs have top-1 recalls of less than 0.20 on all three diseases. The
use of the CoT approach by GPT-4 results in a slight performance improvement.
Overall, LLMs demonstrate the potential for screening three rare diseases
using patient information, such as medical history, auxiliary examinations,
and laboratory tests.

###  4.3. Task 3: Comparison Analysis of Common & Rare Diseases

####  4.3.1. Metric

Task 3 employs two key metrics: top-k recall (hit@k, where k=1, 3, 10) and
median rank (MR). Top-k recall evaluates the diagnostic accuracy, deeming a
diagnosis correct if the actual disease appears within the top-k predictions.
Median rank represents the median position of accurate diagnoses within
predictions across all cases.

####  4.3.2. Results

In this task, GPT-4 achieves a top-1 recall of 0.461 under 0-shot settings
(with top-3 and top-10 recalls being 0.596 and 0.721, respectively) and a
median rank of 3.0. In comparison, under the Chain of Thought (CoT) setting,
GPT-4 achieves the best performance on all metrics. The CoT approach
contributes to a moderate improvement in GPT-4’s performance. Additionally, we
did not perform few-shot learning on GPT-4 due to the limited number of cases,
as we were concerned that it might influence or bias the subsequent diagnostic
results. Furthermore, the GPT3.5-Turbo achieves a top-1 recall of 0.332, a
top-3 recall of 0.454, a top-10 recall of 0.581, and a median rank 5.0,
achieving the second-best performance after the GPT-4 method. The third-best
performance is exhibited by GLM4, with a top-1 recall of 0.313, a top-3 recall
of 0.425, a top-10 recall of 0.571, and a median rank 8.0. The other two API-
based models have top-1 recalls less than 0.25, with median ranks greater than
10.0. All the open-source models yield top-1 recalls less than 0.15, and all
have median ranks greater than 10.0. Additionally, we did not present the
results of Llama2-7B because we found its performance on lengthy Chinese EHR
texts too poor to output normal results.

###  4.4. Task 4: Differential Diagnosis among Universal Rare Diseases

####  4.4.1. Metric

Task 4 employs the same metric as Task 3.

####  4.4.2. Results

We yield the following notable findings in our extensive dataset comprising
2,185 patients with 421 distinct rare diseases. For differential diagnosis of
universal rare diseases, GPT-4 achieves a top-1 recall of 0.323 under 0-shot
settings (with top-3 and top-10 recalls being 0.454 and 0.589, respectively)
and a median rank of 5.0. Interestingly, GPT-4’s performance slightly
decreases under random 3-shot, while adopting the Chain of Thought (CoT)
approach leads to a performance improvement with a median rank of 4.0. In
contrast, the top-1 recalls for the other 4 API-based LLMs are around 0.2,
with all their median ranks exceeding 10.0; the top-1 recalls for all 3 open-
source LLMs with fewer than 10 billion parameters are less than 0.1. In
conclusion, GPT-4 demonstrated promising differential diagnosis results in a
dataset of 2,185 rare disease patients, encompassing 421 different rare
diseases. For a comparison of GPT-4 with human physicians, see section 5.2.

##  5\. Analysis and Discussion

This section provides a detailed explanation of the performance enhancement
for LLMs in Task 4 (differential diagnosis among universal rare diseases)
using the dynamic few-shot prompt method based on the knowledge graph.
Additionally, it includes a comparison and thorough analysis and discussion of
the diagnostic capabilities in rare diseases between doctors from PUMCH and
LLMs, using a high-quality subset of the PUMCH dataset.

###  5.1. Knowledge Integration Dynamic Few-shot

Figure 4. Performance of six LLMs in rare disease differential diagnosis under
zero-shot, random 3-shot, and dynamic 3-shot prompts (using GPT-4 zero-shot as
a baseline).

In our research, the IC value-based random walk algorithm we developed is
employed to produce embeddings for Human Phenotype Ontology (HPO) phenotype
nodes within our comprehensively integrated rare disease knowledge graph.
These embeddings are subsequently utilized in dynamic few-shot settings. Our
experimental evaluation, involving 2,185 rare disease patients and conducted
on six LLMs, including GLM4 (Du et al., 2022; Zeng et al., 2022), is
benchmarked against the zero-shot performance of GPT-4. The results are
illustrated in Figure 4, with key findings summarized below:

  * •

A holistic analysis reveals that dynamic 3-shot settings significantly enhance
the performance of the six LLMs compared to their zero-shot capabilities. On
average, there is a notable 108% increase in top-1 recall and substantial
improvements of 80% and 58% in top-3 and top-10 recalls, respectively.

  * •

With dynamic 3-shot, GLM4 outperforms GPT-4’s 0-shot level slightly in top-1
recall. In addition, models with less than 10 billion parameters, like
Llama2-7b, achieve 0-shot performance on par with more advanced models, such
as Gemini Pro and GLM4.

  * •

Our ablation study demonstrates an average enhancement of 23% in top-1 recall
across the six LLMs under random 3-shot settings, corresponding increases of
20% and 21% in top-3 and top-10 recalls, respectively. This highlights the
effectiveness of our knowledge graph-based dynamic few-shot approach in
boosting the capabilities of LLMs, particularly those far behind GPT-4, in the
context of rare disease differential diagnosis.

###  5.2. Human versus LLMs experiment on a subset of Task 4

####  5.2.1. Selection of Test case and Clinical Specialist

We selected a subset from the PUMCH dataset to compare the diagnostic
performance of physicians with various LLMs. The selection criteria included a
wide range of diseases across human organ systems, and multiple cases were
included for each disease to assess the difficulty level in diagnosis.
Ultimately, this test set comprised 75 cases spanning 16 diseases across 5
hospital departments, including Cardiology, Hematology, Nephrology, Neurology,
and Pediatrics. To ensure the integrity of the diagnostic process, any
information within the EHRs that could potentially provide clues about the
underlying diseases was removed.

A total of 50 physicians were employed from 23 Class-A tertiary hospitals in
China. Each of the 5 departments had ten physicians. To minimize diagnostic
result errors, we assigned each case to be diagnosed by 4 physicians. In the
diagnostic process, clinical specialists rely initially on their knowledge and
experience to diagnose cases. For situations where the diagnosis is uncertain,
clinical specialists can use external assistance such as books and web tools.

####  5.2.2. Results

We initially investigated two input approaches in our comparative experiments.
One involved a feature-based input, where the patient’s personal phenotype
information was provided to inquire about LLMs’ diagnostic results. The other
approach involved entering the patient’s EHR text information with all
personally identifiable information removed. We compared the diagnostic
outcomes of two input forms on GPT-4 using the test set from the
aforementioned 75 PUMCH cases. In contrast, specialist physicians rely solely
on EHR text for diagnosis, as shown in Table 5. When utilizing phenotype
input, GPT-4 achieves a top-1 recall of 0.520, a top-3 recall of 0.747, a
top-10 recall of 0.827, and a median rank of 1.0. Conversely, EHR text input
with GPT-4 results in a top-1 recall of 0.453, a top-3 recall of 0.693, a
top-10 recall of 0.800, and a median rank of 2.0. Using extracted phenotypes
significantly reduces the number of input tokens for LLMs and marginally
outperforms the direct use of medical record text in differential diagnosis.
This enhancement stems from the extracted phenotypes encapsulating crucial
information about the patient’s symptoms while eliminating irrelevant details,
rendering this input format more economical and efficient for LLMs. In
contrast, physicians’ diagnostic performance enhancement when aided by
external assistance. The top-1 recall increases from 0.407 to 0.447, the top-3
recall rises from 0.468 to 0.511, and the top-10 recall increases from 0.481
to 0.524.

Table 5. The differential diagnosis performance of GPT-4 with two different input forms and 50 physicians with and without assistance on 75 PUMCH cases. | hit@1 | hit@3 | hit@10 |  MR ($\downarrow$)
---|---|---|---|---
GPT-4 (Phenotypes) | 0.520 | 0.747 | 0.827 | 1.0
GPT-4 (EHR text) | 0.453 | 0.693 | 0.800 | 2.0
Physicians w/o assistance | 0.407 | 0.468 | 0.481 | -
Physicians w/ assistance | 0.447 | 0.511 | 0.524 | -

We also provide a more detailed presentation of GPT-4’s performance across
diseases related to different medical departments. As shown in Figure 5, GPT-4
achieved the best diagnostic results across all 5 departments, surpassing the
level of specialist physicians. In contrast, the diagnostic performance of the
other two LLMs was inferior to that of specialist physicians. Moreover, there
were significant differences in diagnostic results among different
departments. For example, all methods achieved high recall rates in
pediatrics, possibly due to the comprehensive information available in
pediatric cases. Conversely, all methods performed poorly in the cases from
the Department of Cardiology, which was speculated to be associated with the
inherent features of the diseases themselves. Heart diseases often exhibit
similarities and overlaps in symptoms and signs, such as chest pain, chest
tightness, palpitations, edema, etc. The differential diagnosis primarily
relies on the results of objective auxiliary examinations, including
laboratory tests, imaging examinations, and electrophysiological assessments.
For instance, in the case of cardiomyopathy, the imaging patterns of
echocardiography and cardiac magnetic resonance imaging (MRI) play a crucial
role in the diagnostic process. This suggests a future direction that involves
utilizing the large multimodal models (LMMs) to incorporate more comprehensive
clinical information for diagnosis.

Figure 5. Comparison of top-1 and top-3 recall in rare disease differential
diagnosis across five departments between specialist physicians with/without
assistance and three LLMs.

##  6\. Conclusion

This paper delivers RareBench, an innovative benchmark framework aimed at
systematically assessing Large Language Models’ (LLMs) capability in rare
diseases. RareBench combines public datasets with the comprehensive dataset
from PUMCH to create the most extensive collection of rare disease patient
data available. Leveraging a rare disease knowledge graph and an Information-
Content-based random walk algorithm, we introduce a dynamic few-shot prompt,
significantly enhancing LLMs’ diagnostic capabilities. Notably, our
comparative analysis with specialist physicians demonstrates GPT-4’s superior
accuracy in some specific rare disease diagnoses. We expect RareBench to
catalyze further advancements and applications of LLMs in tackling the
complexities of clinical diagnosis, especially for rare diseases.

##  7\. Ethical Consideration

This study was approved by the Ethics Committees at Peking Union Medical
College Hospital, Peking Union Medical College, and the Chinese Academy of
Medical Sciences (approval number S-K2051). It is worth noting that all cases
were monitored by doctors from Peking Union Medical College Hospital prior to
uploading text information, ensuring the absence of any potential personal
information leaks. Xuanzhong Chen, Xiaohao Mao, Qihan Guo, and Ting Chen
affirm that the manuscript is an honest, accurate, and transparent account of
the study being reported; that no important aspects of the study have been
omitted; and that any discrepancies from the study as originally planned (and,
if relevant, registered) have been explained.

## References

  * (1)
  * Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023\.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_ (2023).
  * Aymé (2003) Ségolène Aymé. 2003.  Orphanet, an information site on rare diseases.  _Soins; la revue de référence infirmière_ 672 (2003), 46–47.
  * Cai et al. (2023) Yan Cai, Linlin Wang, Ye Wang, Gerard de Melo, Ya Zhang, Yanfeng Wang, and Liang He. 2023.  MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models.  arXiv:2312.12806 [cs.CL]
  * Cover (1999) Thomas M Cover. 1999.  _Elements of information theory_.  John Wiley & Sons.
  * Deisseroth et al. (2019) Cole A Deisseroth, Johannes Birgmeier, Ethan E Bodle, Jennefer N Kohler, Dena R Matalon, Yelena Nazarenko, Casie A Genetti, Catherine A Brownstein, Klaus Schmitz-Abe, Kelly Schoch, et al. 2019\.  ClinPhen extracts and prioritizes patient phenotypes directly from medical records to expedite genetic disease diagnosis.  _Genetics in Medicine_ 21, 7 (2019), 1585–1593.
  * Du et al. (2022) Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022.  GLM: General Language Model Pretraining with Autoregressive Blank Infilling. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. 320–335.
  * Evans (2023) William RH Evans. 2023.  Dare to think rare. Diagnostic delay and rare diseases.  (2023).
  * Fansi Tchango et al. (2022) Arsene Fansi Tchango, Rishab Goel, Zhi Wen, Julien Martel, and Joumana Ghosn. 2022.  Ddxplus: A new dataset for automatic medical diagnosis.  _Advances in Neural Information Processing Systems_ 35 (2022), 31306–31318.
  * Feng et al. (2022) Yuhao Feng, Lei Qi, and Weidong Tian. 2022.  PhenoBERT: a combined deep learning method for automated recognition of human phenotype ontology.  _IEEE/ACM Transactions on Computational Biology and Bioinformatics_ 20, 2 (2022), 1269–1277.
  * Grover and Leskovec (2016) Aditya Grover and Jure Leskovec. 2016.  node2vec: Scalable feature learning for networks. In _Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining_. 855–864.
  * Haendel et al. (2020) Melissa Haendel, Nicole Vasilevsky, Deepak Unni, Cristian Bologa, Nomi Harris, Heidi Rehm, Ada Hamosh, Gareth Baynam, Tudor Groza, Julie McMurry, et al. 2020\.  How many rare diseases are there?  _Nature reviews drug discovery_ 19, 2 (2020), 77–78.
  * Hamosh et al. (2005) Ada Hamosh, Alan F Scott, Joanna S Amberger, Carol A Bocchini, and Victor A McKusick. 2005.  Online Mendelian Inheritance in Man (OMIM), a knowledgebase of human genes and genetic disorders.  _Nucleic acids research_ 33, suppl_1 (2005), D514–D517.
  * He et al. (2019) Jiangjiang He, Mi Tang, Xueyan Zhang, Duo Chen, Qi Kang, Yan Yang, Jiahao Hu, Chunlin Jin, and Peipei Song. 2019.  Incidence and prevalence of 121 rare diseases in China: Current status and challenges.  _Intractable & rare diseases research_ 8, 2 (2019), 89–97.
  * Jia et al. (2018) Jinmeng Jia, Ruiyuan Wang, Zhongxin An, Yongli Guo, Xi Ni, and Tieliu Shi. 2018.  RDAD: a machine learning system to support phenotype-based rare disease diagnosis.  _Frontiers in genetics_ 9 (2018), 587.
  * Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023\.  Mistral 7B.  _arXiv preprint arXiv:2310.06825_ (2023).
  * Jin et al. (2021) Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021.  What disease does this patient have? a large-scale open domain question answering dataset from medical exams.  _Applied Sciences_ 11, 14 (2021), 6421.
  * Jin et al. (2019) Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019.  PubMedQA: A Dataset for Biomedical Research Question Answering. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. 2567–2577.
  * Köhler et al. (2021) Sebastian Köhler, Michael Gargano, Nicolas Matentzoglu, Leigh C Carmody, David Lewis-Smith, Nicole A Vasilevsky, Daniel Danis, Ganna Balagura, Gareth Baynam, Amy M Brower, et al. 2021\.  The human phenotype ontology in 2021.  _Nucleic acids research_ 49, D1 (2021), D1207–D1217.
  * Köhler et al. (2009) Sebastian Köhler, Marcel H Schulz, Peter Krawitz, Sebastian Bauer, Sandra Dölken, Claus E Ott, Christine Mundlos, Denise Horn, Stefan Mundlos, and Peter N Robinson. 2009.  Clinical diagnostics in human genetics with semantic similarity searches in ontologies.  _The American Journal of Human Genetics_ 85, 4 (2009), 457–464.
  * Köhler et al. (2017) Sebastian Köhler, Nicole A Vasilevsky, Mark Engelstad, Erin Foster, Julie McMurry, Ségolène Aymé, Gareth Baynam, Susan M Bello, Cornelius F Boerkoel, Kym M Boycott, et al. 2017\.  The human phenotype ontology in 2017.  _Nucleic acids research_ 45, D1 (2017), D865–D876.
  * Kung et al. (2023) Tiffany H Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepaño, Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, et al. 2023\.  Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models.  _PLoS digital health_ 2, 2 (2023), e0000198.
  * Kwon et al. (2023) Taeyoon Kwon, Kai Tzu-iunn Ong, Dongjin Kang, Seungjun Moon, Jeong Ryong Lee, Dosik Hwang, Yongsik Sim, Beomseok Sohn, Dongha Lee, and Jinyoung Yeo. 2023.  Large Language Models are Clinical Reasoners: Reasoning-Aware Diagnosis Framework with Prompt-Generated Rationales.  _arXiv preprint arXiv:2312.07399_ (2023).
  * Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020\.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  _Advances in Neural Information Processing Systems_ 33 (2020), 9459–9474.
  * Li et al. (2023) Hanzhou Li, John T Moon, Saptarshi Purkayastha, Leo Anthony Celi, Hari Trivedi, and Judy W Gichoya. 2023.  Ethics of large language models in medicine and medical research.  _The Lancet Digital Health_ 5, 6 (2023), e333–e335.
  * Li et al. (2019) Qigang Li, Keyan Zhao, Carlos D Bustamante, Xin Ma, and Wing H Wong. 2019.  Xrare: a machine learning method jointly modeling phenotypes and genetic evidence for rare disease diagnosis.  _Genetics in Medicine_ 21, 9 (2019), 2126–2134.
  * Liu et al. (2019) Cong Liu, Fabricio Sampaio Peres Kury, Ziran Li, Casey Ta, Kai Wang, and Chunhua Weng. 2019.  Doc2Hpo: a web application for efficient and accurate HPO concept curation.  _Nucleic acids research_ 47, W1 (2019), W566–W570.
  * Liu et al. (2023b) Junling Liu, Peilin Zhou, Yining Hua, Dading Chong, Zhongyu Tian, Andrew Liu, Helin Wang, Chenyu You, Zhenhua Guo, Lei Zhu, et al. 2023b.  Benchmarking Large Language Models on CMExam–A Comprehensive Chinese Medical Exam Dataset.  _arXiv preprint arXiv:2306.03030_ (2023).
  * Liu et al. (2023a) Qianchu Liu, Stephanie Hyland, Shruthi Bannur, Kenza Bouzid, Daniel Castro, Maria Wetscherek, Robert Tinn, Harshita Sharma, Fernando Pérez-García, Anton Schwaighofer, et al. 2023a.  Exploring the Boundaries of GPT-4 in Radiology. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_. 14414–14445.
  * Luo et al. (2021) Ling Luo, Shankai Yan, Po-Ting Lai, Daniel Veltri, Andrew Oler, Sandhya Xirasagar, Rajarshi Ghosh, Morgan Similuk, Peter N Robinson, and Zhiyong Lu. 2021.  PhenoTagger: a hybrid method for phenotype concept recognition using human phenotype ontology.  _Bioinformatics_ 37, 13 (2021), 1884–1890.
  * Marwaha et al. (2022) Shruti Marwaha, Joshua W Knowles, and Euan A Ashley. 2022.  A guide for the diagnosis of rare and undiagnosed disease: beyond the exome.  _Genome medicine_ 14, 1 (2022), 1–22.
  * McDuff et al. (2023) Daniel McDuff, Mike Schaekermann, Tao Tu, Anil Palepu, Amy Wang, Jake Garrison, Karan Singhal, Yash Sharma, Shekoofeh Azizi, Kavita Kulkarni, et al. 2023\.  Towards accurate differential diagnosis with large language models.  _arXiv preprint arXiv:2312.00164_ (2023).
  * Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013.  Distributed representations of words and phrases and their compositionality.  _Advances in neural information processing systems_ 26 (2013).
  * Moor et al. (2023) Michael Moor, Oishi Banerjee, Zahra Shakeri Hossein Abad, Harlan M Krumholz, Jure Leskovec, Eric J Topol, and Pranav Rajpurkar. 2023.  Foundation models for generalist medical artificial intelligence.  _Nature_ 616, 7956 (2023), 259–265.
  * Nori et al. (2023a) Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. 2023a.  Capabilities of gpt-4 on medical challenge problems.  _arXiv preprint arXiv:2303.13375_ (2023).
  * Nori et al. (2023b) Harsha Nori, Yin Tat Lee, Sheng Zhang, Dean Carignan, Richard Edgar, Nicolo Fusi, Nicholas King, Jonathan Larson, Yuanzhi Li, Weishung Liu, et al. 2023b.  Can generalist foundation models outcompete special-purpose tuning? case study in medicine.  _arXiv preprint arXiv:2311.16452_ (2023).
  * OpenAI (2022a) OpenAI. 2022a.  Introducing ChatGPT.  https://openai.com/blog/chatgpt
  * OpenAI (2022b) OpenAI. 2022b.  New and improved embedding model.  https://openai.com/blog/new-and-improved-embedding-model
  * Pal et al. (2022) Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022.  Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In _Conference on Health, Inference, and Learning_. PMLR, 248–260.
  * Peng et al. (2016) Jiajie Peng, Hansheng Xue, Yukai Shao, Xuequn Shang, Yadong Wang, and Jin Chen. 2016.  Measuring phenotype semantic similarity using human phenotype ontology. In _2016 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)_. IEEE, 763–766.
  * Perozzi et al. (2014) Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. 2014.  Deepwalk: Online learning of social representations. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_. 701–710.
  * Philippakis et al. (2015) Anthony A Philippakis, Danielle R Azzariti, Sergi Beltran, Anthony J Brookes, Catherine A Brownstein, Michael Brudno, Han G Brunner, Orion J Buske, Knox Carey, Cassie Doll, et al. 2015\.  The Matchmaker Exchange: a platform for rare disease gene discovery.  _Human mutation_ 36, 10 (2015), 915–921.
  * Pinol et al. (2017) Marc Pinol, Rui Alves, Ivan Teixido, Jordi Mateo, Francesc Solsona, and Ester Vilaprinyó. 2017.  Rare disease discovery: An optimized disease ranking system.  _IEEE Transactions on Industrial Informatics_ 13, 3 (2017), 1184–1192.
  * Robinson et al. (2020) Peter N Robinson, Vida Ravanmehr, Julius OB Jacobsen, Daniel Danis, Xingmin Aaron Zhang, Leigh C Carmody, Michael A Gargano, Courtney L Thaxton, Guy Karlebach, Justin Reese, et al. 2020\.  Interpretable clinical genomics with a likelihood ratio paradigm.  _The American Journal of Human Genetics_ 107, 3 (2020), 403–417.
  * Ronicke et al. (2019) Simon Ronicke, Martin C Hirsch, Ewelina Türk, Katharina Larionov, Daphne Tientcheu, and Annette D Wagner. 2019.  Can a decision support system accelerate rare disease diagnosis? Evaluating the potential impact of Ada DX in a retrospective study.  _Orphanet journal of rare diseases_ 14 (2019), 1–12.
  * Singhal et al. (2023) Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. 2023\.  Large language models encode clinical knowledge.  _Nature_ 620, 7972 (2023), 172–180.
  * Son et al. (2018) Jung Hoon Son, Gangcai Xie, Chi Yuan, Lyudmila Ena, Ziran Li, Andrew Goldstein, Lulin Huang, Liwei Wang, Feichen Shen, Hongfang Liu, et al. 2018\.  Deep phenotyping on electronic health records facilitates genetic diagnosis by clinical exomes.  _The American Journal of Human Genetics_ 103, 1 (2018), 58–73.
  * Team et al. (2023) Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023\.  Gemini: a family of highly capable multimodal models.  _arXiv preprint arXiv:2312.11805_ (2023).
  * Thirunavukarasu et al. (2023) Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. 2023.  Large language models in medicine.  _Nature medicine_ 29, 8 (2023), 1930–1940.
  * Töpel et al. (2010) Thoralf Töpel, Dagmar Scheible, Friedrich Trefz, and Ralf Hofestädt. 2010.  RAMEDIS: a comprehensive information system for variations and corresponding phenotypes of rare metabolic diseases.  _Human mutation_ 31, 1 (2010), E1081–E1088.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023\.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ (2023).
  * Tu et al. (2024) Tao Tu, Anil Palepu, Mike Schaekermann, Khaled Saab, Jan Freyberg, Ryutaro Tanno, Amy Wang, Brenna Li, Mohamed Amin, Nenad Tomasev, et al. 2024\.  Towards Conversational Diagnostic AI.  _arXiv preprint arXiv:2401.05654_ (2024).
  * Vaid et al. (2023) Akhil Vaid, Isotta Landi, Girish Nadkarni, and Ismail Nabeel. 2023.  Using fine-tuned large language models to parse clinical notes in musculoskeletal pain disorders.  _The Lancet Digital Health_ 5, 12 (2023), e855–e858.
  * Wang et al. (2023) Sheng Wang, Zihao Zhao, Xi Ouyang, Qian Wang, and Dinggang Shen. 2023.  Chatcad: Interactive computer-aided diagnosis on medical image using large language models.  _arXiv preprint arXiv:2302.07257_ (2023).
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022\.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in Neural Information Processing Systems_ 35 (2022), 24824–24837.
  * Yang et al. (2023) Jingye Yang, Cong Liu, Wendy Deng, Da Wu, Chunhua Weng, Yunyun Zhou, and Kai Wang. 2023.  Enhancing phenotype recognition in clinical notes using large language models: PhenoBCBERT and PhenoGPT.  _Patterns_ (2023).
  * Yang et al. (2022) Xi Yang, Aokun Chen, Nima PourNejatian, Hoo Chang Shin, Kaleb E Smith, Christopher Parisien, Colin Compas, Cheryl Martin, Anthony B Costa, Mona G Flores, et al. 2022\.  A large language model for electronic health records.  _NPJ Digital Medicine_ 5, 1 (2022), 194.
  * Zeng et al. (2022) Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022\.  Glm-130b: An open bilingual pre-trained model.  _arXiv preprint arXiv:2210.02414_ (2022).
  * Zhai et al. (2023) Weiqi Zhai, Xiaodi Huang, Nan Shen, and Shanfeng Zhu. 2023.  Phen2Disease: a phenotype-driven model for disease and gene prioritization by bidirectional maximum matching semantic similarities.  _Briefings in Bioinformatics_ (2023), bbad172.
  * Zhao et al. (2020) Mengge Zhao, James M Havrilla, Li Fang, Ying Chen, Jacqueline Peng, Cong Liu, Chao Wu, Mahdi Sarmady, Pablo Botas, Julián Isla, et al. 2020\.  Phen2Gene: rapid phenotype-driven gene prioritization for rare diseases.  _NAR genomics and Bioinformatics_ 2, 2 (2020), lqaa032.

##  Appendix A Appendix

###  A.1. Dataset Details

Four publicly available datasets are used in this study: MME(Philippakis et
al., 2015), LIRICAL(Robinson et al., 2020), HMS(Ronicke et al., 2019), and
RAMEDIS(Töpel et al., 2010). These datasets are sourced from published
articles or existing datasets. Quality control measures were implemented to
filter out cases with insufficient information. Following screening
procedures, the MME111https://github.com/ga4gh/mme-apis/tree/master/testing
dataset comprises 40 cases across 17 diseases, the LIRICAL222https://
zenodo.org/record/3905420 dataset encompasses 370 cases spanning 252 diseases,
the HMS dataset encompasses 88 cases covering 39 diseases, and the RAMEDIS
dataset consists of 624 cases spanning 63 diseases.

###  A.2. Framework Settings

The implementation details of the models listed in Table 2 are as follows:
Both GPT-4 and GPT-3.5 are accessed through OpenAI’s API
333https://platform.openai.com/docs/introduction. The models used are
”gpt-4-1106-preview” and ”gpt-3.5-turbo-1106” respectively. For the two
models, the seed parameter is set to 42, while all other parameters are left
at their default settings. Both GLM4 and GLM3 are accessed through Zhipu AI’s
API 444https://open.bigmodel.cn/. The models used are ”glm-4” and
”glm-3-turbo” respectively. In the model parameter settings, temperature is
set to 0.15 and top_p to 0.7; all other parameters are maintained at their
default values. Gemini is accessed through DeepMind’s API
555https://deepmind.google/technologies/gemini/#introduction. The model used
is ”gemini-pro”, and the parameters use default settings. Mistral-7B,
Llama2-7B, and ChatGLM3-6B respectively use
huggingface’s666https://huggingface.co/ ”mistralai/Mistral-7B-Instruct-v0.1”,
”meta-llama/Llama-2-7b-chat” and ”THUDM/chatglm3-6b” models. For Mistral-7b
and Llama2-7b, the parameter settings are ”max_new_tokens=1000,
skip_special_tokens=True, clean_up_tokenization_spaces=False”; for
ChatGLM3-6b, all parameters are set to their default values. All three models
are run on a single NVIDIA A100 80GB graphics card.

###  A.3. Prompt Examples

####  A.3.1. Task 1: Phenotype Extraction from Electronic Health Records

Sub-task 1: Precise Phenotype Extraction

[Task]  Extract disease phenotypes from the following medical record text.

[Requirement]

  1. (1)

Output format: One phenotype per line, the format is [index]. [English name],
such as ”1. Cough”, ”2. Pain”.

  2. (2)

Answer according to the standard English terminology in the HPO database
(https://hpo.jax.org/app/). Do not use colloquialisms, and try to be as
concise as possible. It is not a restatement of the original words, but a
refinement of the phenotype.

  3. (3)

Extract all phenotypes appearing in the text without omitting any. output as
much as possible.

  4. (4)

For symptoms that are denied in the text, such as ”no chest pain,” and ”no
cough,” do not extract the corresponding phenotypes.

[Medical Record Text]

The patient developed shortness of breath accompanied by chest tightness and
facial edema about 3 months ago after physical exertion, with the eyelid edema
being the most severe. The shortness of breath was most severe after physical
activity and improved after rest. He was treated at a local hospital, and on
December 1, 2013, the cardiac function showed by color echocardiography:
tricuspid valve disease. Then he went to our hospital. Ultrasound showed
myocardial disease, right atrium and right ventricle enlargement, moderate to
severe tricuspid valve insufficiency, severe reduction in left and right
ventricular systolic function, aortic valve degeneration, and a small amount
of pericardial effusion. The patient was admitted to our hospital for
outpatient consideration of tricuspid valve disease and is now admitted to our
department for further diagnosis and treatment. Since the onset of the
disease, the patient’s energy, diet, and sleep have been acceptable, and his
bowel movements have been normal. There is no significant change in weight
from before. No dry mouth, dry eyes, mouth ulcers, joint swelling and pain,
rash, etc.

Sub-task 2: General Entity Extraction

[Task]  For the medical record text provided below, mark the text that
represents the disease entity.

[Requirement]

  1. (1)

Output format: One disease entity per line. The format is [Index]. [Original
text]. For example, ”1. Fever”, ”2. Body temperature 39℃”.

  2. (2)

Output in the order in which it appears in the text; do not omit anything, and
output as much as possible.

  3. (3)

For symptoms that are denied in the text, such as ”no chest pain,” and ”no
cough,” do not extract the corresponding entities.

[Medical Record Text]

$\cdots$

Sub-task3: General Entity Standardization

[Task]  Standardize the following medical entities to Human Phenotype Ontology
phenotypes.

[Requirement]

  1. (1)

Answer according to the standard English terminology in the HPO database
(https://hpo.jax.org/app/). Do not be colloquial, and try to be as formal,
standardized, and concise as possible.

  2. (2)

Output format: Each line contains one medical entity and its corresponding HPO
phenotype. The format is [index]. [Entity name]: [English name of phenotype].
For example, ”1. Body temperature 39℃: Fever”, ”2. Excessive urine output:
Polyuria”.

[Entity list]

Chest tightness

ST segment elevation

Activity tolerance gradually decreases

ventricular fibrillation

Aspen syndrome

sore throat

J-point elevation greater than 0.2mv and saddle-like elevation

Shortness of breath

pneumonia

cough

####  A.3.2. Task 2: Screening for Specific Rare Diseases

Zero-shot prompt  As an expert in the field of rare diseases, you are tasked
with diagnosing a real clinical case. Please carefully review the patient’s
basic medical history, specialized examinations, and auxiliary tests to
determine whether the patient may be suffering from [SCREENING DISEASE].

CoT prompt  [Zero-shot prompt], Let us think step by step.

####  A.3.3. Task 3: Comparison Analysis of Common and Rare Diseases

Zero-shot prompt  Please, as a rare disease specialist, answer the following
questions. [EHRs] is the patient’s admitted record, including chief complaint,
present medical history, etc. [77 CANDIDATE DISEASES] are the types of
diseases that the patient may suffer from. The top 10 diagnosed diseases are
selected from highest to lowest probability.

CoT prompt  [Zero-shot prompt], Let us think step by step.

####  A.3.4. Task 4: Differential Diagnosis among Universal Rare Diseases

Here, we present the specific configurations for zero-shot, Chain of Thought
(CoT), and (dynamic) few-shot settings.

System prompt  You are a specialist in the field of rare diseases. You will be
provided and asked about a complicated clinical case; read it carefully and
then provide a diverse and comprehensive differential diagnosis.

Zero-shot prompt  This rare disease patient suffers from symptoms:
[PATIENT_PHENOTYPE]. Enumerate the top 10 most likely diagnoses. Be precise,
listing one diagnosis per line, and try to cover many unique possibilities (at
least 10). The top 10 diagnoses are:

CoT prompt  [Zero-shot prompt]. Let us think step by step.

(Dynamic) Few-shot prompt  Let me give you [K] examples first: The [i] patient
has a rare disease [EXAMPLE_DIAGNOSIS], and his/her [PHENOTYPE or EHR] is as
follows: [EXAMPLE_PHENOTYPE or EXAMPLE_EHR]. Next is the patient case you need
to diagnose: [Zero-shot prompt].

###  A.4. Knowledge Integration Dynamic Few-shot

####  A.4.1. Knowledge Bases Details

The Human Phenotype Ontology (HPO) 777https://hpo.jax.org/app/ provides a
standardized vocabulary for phenotypic abnormalities encountered in human
disease. These phenotype terms form a directed acyclic graph (DAG) and are
connected by directed ”IS_A” edges (denoting subclass relationships).
Additionally, the HPO team annotates relationships between phenotypes and
diseases using information from medical literature and other sources. The
files used in this study are ”hp.obo” and ”phenotype.hpoa”, with the version
being hp/releases/2023-06-06.

Orphanet 888https://www.orpha.net/consor/cgi-bin/index.php, funded by the
French Ministry of Health, is a non-profit online resource and information
platform focusing on rare diseases. It offers information on over 6,000 rare
diseases, written by medical experts and researchers and subjected to rigorous
quality review. This information includes etiology, symptoms, diagnostic
methods, treatment options, and prognosis. Our work primarily utilizes the
annotation information on rare diseases and genes from this platform.

Online Mendelian Inheritance in Man (OMIM) 999https://www.omim.org/ is a
comprehensive database of genes and genetic diseases, collecting and
organizing extensive information about human genetic disorders. It includes
descriptions of over 20,000 genetic diseases, covering genetic etiology,
clinical manifestation, inheritance patterns, molecular mechanisms, and
related literature.

The National Rare Disease Registry System of China (NRDRS)
101010https://www.nrdrs.org.cn/xhrareweb/homeIndex, overseen by Peking Union
Medical College Hospital, is a national-level online registry platform for
rare diseases, aimed at establishing unified rare disease registration
standards and norms. The first list of the Compendium of China’s Rare Disease
(CCRD) compiles detailed information on 144 rare diseases. The annotations of
relationships between diseases and phenotypes are manually extracted. The
version used in our study is 2019-11.

####  A.4.2. Comparison Analysis of Node2vec and IC value-based random walk

In our integrated rare disease knowledge graph, we implement an IC value-based
random walk algorithm to obtain embeddings for phenotype and disease nodes
within the graph. While Node2Vec utilizes two parameters, $p$ and $q$, to
control the walking probability, we dynamically adjust this based on the IC
values of different phenotypes. The settings for other parameters are as
follows:

’embedding_dim’: 256,

’walk_length’: 45,

’context_size’: 35,

’walks_per_node’: 40,

’num_negative_samples’: 1,

’sparse’: True,

’loader_batch_size’: 256,

’loader_shuffle’: True,

’loader_num_workers’: 4,

’learning_rate’: 0.01,

’epoch_nums’: 36

Similarly, we implement the IC value-based random walk algorithm on a single
NVIDIA A100 80GB graphics card. The embeddings of phenotype nodes obtained
through this process are used in the dynamic few-shot prompt. To validate the
effectiveness of the IC value-based random walk algorithm, we compare it with
the original version of Node2vec. We use the learned phenotype node embeddings
to represent rare disease patients, directly measuring their cosine similarity
with disease node embeddings. This approach enables a straightforward method
for rare disease differential diagnosis, with results as shown in Table 6. It
is evident that the IC value-based random walk algorithm effectively learns
the representational relationship between phenotypes and rare diseases,
indirectly demonstrating its efficacy for use in dynamic few-shot scenarios.

Table 6. Comparison of Results: IC Value-Based Random Walk Algorithm vs. Node2v ec. Metrics | hit@1 | hit@3 | hit@10 | Median Rank
---|---|---|---|---
Node2vec | 0.3 | 1.2 | 3.0 | 128.0
IC-based RW | 25.1 | 43.2 | 63.4 | 5.0

###  A.5. More Experiments

####  A.5.1. Qualitative Analysis of LLMs’ Entity Standardization Capabilities

For the task of general entity standardization, Table 7 shows some output
examples from GPT-4, which demonstrated the best performance. HPO code
(HP:xxxxxxx) represents the code of phenotype in the HPO database, which
serves as a unique identifier for the phenotype. It is observable that GPT-4’s
manner of describing phenotypes differs somewhat from the reference.
Furthermore, GPT-4 tends to paraphrase entities with similar meanings rather
than extracting disease symptoms based on the entities, resulting in a level
of summarization that falls short of the reference. Besides, the accuracy of
the HPO codes produced by GPT-4 is relatively low. This indicates that LLMs,
exemplified by GPT-4, have problems with non-standard terminology and poor
lexical accuracy when addressing questions related to rare diseases. To apply
LLMs effectively in the domain of rare diseases, it is necessary to make
normative adjustments to their output terminology.

Table 7. GPT-4’s Output Examples of Entity Standardization. Entity: Activity
tolerance gradually decreases
---
GPT-4 output: Reduced exercise tolerance (HP:0003546)
Reference: Hypokinesia (HP: 0002375)
Entity: Complete M protein (+)
GPT-4 output: Monoclonal immunoglobulin M spike (HP:0032144)
Reference: Abnormality of the lymphatic system (HP: 0100763)
Entity: Right ventricular enlargement
GPT-4 output: Ventricular hypertrophy (HP:0001714)
Reference: Cardiomegaly (HP: 0001640)

####  A.5.2. GPT4’s Capability of Rare Disease Differential Diagnosis with
various prompts

Figure 6. GPT4’s performance on 7 datasets in detail.

In Figure 6, we provide a detailed presentation of GPT-4’s performance on a
total of 2,185 rare disease patients from seven datasets sourced from Public
and PUMCH. The figure illustrates the top-k recall (where k=1,3,10) of GPT-4
under various settings: zero-shot, Chain of Thought (CoT), random 3-shot, and
dynamic few-shot. Overall, GPT-4’s average top-1 recall on the 2,185 rare
disease patients under zero-shot, CoT, random 3-shot, and dynamic 3-shot
settings are 0.323, 0.332, 0.304, and 0.337, respectively. Notably, in the MME
dataset, the dynamic 3-shot approach significantly enhances GPT-4’s capability
in rare disease differential diagnosis, while in the other six datasets, the
improvement is not as pronounced.