# PMC-Patients: A Large-scale Dataset of Patient Summaries and Relations for
Benchmarking Retrieval-based Clinical Decision Support Systems

Zhengyun Zhao111Equal contributions. Qiao Jin222Equal contributions. Fangyuan
Chen  Tuorui Peng  Sheng Yu

###### Abstract

Objective: Retrieval-based Clinical Decision Support (ReCDS) can aid clinical
workflow by providing relevant literature and similar patients for a given
patient. However, the development of ReCDS systems has been severely
obstructed by the lack of diverse patient collections and publicly available
large-scale patient-level annotation datasets. In this paper, we aim to define
and benchmark two ReCDS tasks: Patient-to-Article Retrieval (ReCDS-PAR) and
Patient-to-Patient Retrieval (ReCDS-PPR) using a novel dataset called PMC-
Patients.

Methods: We extract patient summaries from PubMed Central articles using
simple heuristics and utilize the PubMed citation graph to define patient-
article relevance and patient-patient similarity. We also implement and
evaluate several ReCDS systems on the PMC-Patients benchmarks, including
sparse retrievers, dense retrievers, and nearest neighbor retrievers. We
conduct several case studies to show the clinical utility of PMC-Patients.

Results: PMC-Patients contains 167k patient summaries with 3.1M patient-
article relevance annotations and 293k patient-patient similarity annotations,
which is the largest-scale resource for ReCDS and also one of the largest
patient collections. Human evaluation and analysis show that PMC-Patients is a
diverse dataset with high-quality annotations. The evaluation of various ReCDS
systems shows that the PMC-Patients benchmark is challenging and calls for
further research: the best baseline retriever achieves only 16% P@10, 63% R@1k
on ReCDS-PAR, and 6% P@10, 80% R@1k on ReCDS-PPR.

Conclusion: We present PMC-Patients, a large-scale, diverse, and publicly
available patient summary dataset with the largest-scale patient-level
relation annotations. Based on PMC-Patients, we formally define two benchmark
tasks for ReCDS systems and evaluate various existing retrieval methods. PMC-
Patients can largely facilitate methodology research on ReCDS systems and
shows real-world clinical utility. We release all code and data at
https://github.com/pmc-patients/pmc-patients to benefit the community.

###### keywords:

Dataset collection , case report , clinical decision support , patient
similarity , information retrieval

††journal: The Journal of Biomedical Informatics

[label1]organization=Center for Statistical Science, Tsinghua
University,addressline=, city=Beijing, postcode=100084, state=, country=China

[label2]organization=School of Medicine, Tsinghua University,addressline=,
city=Beijing, postcode=100084, state=, country=China

[label3]organization=Department of Physics, Tsinghua University,addressline=,
city=Beijing, postcode=100084, state=, country=China

We introduce PMC-Patients, a first-of-its-kind dataset consisting of 167k
patient summaries extracted from case reports in PubMed Central with high-
quality annotations.

Based on PMC-Patients, we establish two large-scale benchmarks to evaluate
retrieval-based clinical decision support (ReCDS) systems: a patient-to-
article retrieval task with 3.1M relevant articles and a patient-to-patient
retrieval task with 293k similar patients.

We systematically evaluate various baseline methods on the proposed ReCDS
benchmarks and conduct several case studies to demonstrate the clinical
utility.

##  1 Introduction

Clinicians often rely on Evidence-Based Medicine (EBM) to combine clinical
experience with high-quality scientific research to make decisions for
patients [1]. However, finding relevant research can be challenging since the
number of scientific publications is growing exponentially, leaving many
clinical questions unanswered [2]. To address this issue, there has been
increasing research interest in utilizing Natural Language Processing (NLP)
and Information Retrieval (IR) techniques to retrieve relevant articles or
similar patients for assisting patient management [3, 4, 5, 6, 7]. In this
article, we introduce the term “Retrieval-based Clinical Decision Support”
(ReCDS) to describe these tasks. ReCDS can provide clinical assistance for a
given patient by retrieving and analyzing relevant articles or similar
patients to determine the most likely diagnosis and the most effective
treatment plan.

ReCDS with relevant articles is grounded in EBM, where the target articles to
retrieve are up-to-date clinical guidelines or high-quality evidence such as
systematic reviews. Therefore, the majority of ReCDS studies have focused on
retrieving relevant research articles [8, 9, 10], which are primarily
facilitated by the Clinical Decision Support (CDS) Track [11, 12, 3] held
annually from 2014 to 2016 at the Text REtrieval Conference (TREC). Each year,
the TREC CDS Track releases 30 “medical case narratives”, which serve as
idealized representations of actual medical records, including patient
information such as past medical histories and current symptoms. Participants
are asked to return relevant PubMed Central (PMC) articles for each patient
with regard to a given aspect (diagnosis, test, or treatment). Although
sufficient article relevance can be annotated for each patient under the TREC
pooling evaluation setting [13], the size and diversity of the test patient
set in TREC CDS are very limited. Consequently, the generalizability of system
performance to uncovered medical conditions may be constrained.

ReCDS with similar patients, on the other hand, is much under-explored. In
brief, “similar patients with similar features have similar outcomes” [14].
Retrieving the medical records of similar patients can provide valuable
guidance, especially for patients with uncommon conditions such as rare
diseases that lack clinical consensus. Nevertheless, there are various
challenges in conducting this type of research. Unlike scientific articles,
there is currently no publicly available collection of “reference patients” to
retrieve from. Moreover, defining “patient similarity” is non-trial [14] and
large-scale annotation is prohibitively expensive. As a result, there are only
a few studies on similar patient retrieval [15, 16], all of which use private
datasets and annotations.

The aforementioned issues make it clear that a standardized benchmark for
evaluating ReCDS systems is greatly needed. Ideally, such a benchmark should
contain: (1) a diverse set of patient summaries, which serve as both the query
patient set and the reference patient collection; (2) abundant annotations of
the patient summaries with relevant articles and similar patients. Due to
privacy concerns, only a few clinical note datasets from Electronic Health
Records (EHRs) are publicly available. One notable large-scale public EHR
dataset is MIMIC [17, 18]. However, it only contains ICU patients without any
relational annotations, making it unsuitable for evaluating ReCDS systems.

In this article, we aim to benchmark the ReCDS task with PMC-Patients, a novel
dataset collected from the case reports in PMC and the citation graph of
PubMed. Case reports denote a class of medical publication that typically
consists of: (1) a case summary that describes the patient’s admission,
treatment, progress, discharge, and follow-up situations; (2) a literature
review that discusses similar cases and relevant articles, which are cited and
recorded in the PubMed citation graph. To build PMC-Patients, we first extract
167k patient summaries from case reports published in PMC using simple
heuristics. For these patient summaries, we then annotate 3.1M relevant
articles and 293k similar patients using the PubMed citation graph. PMC-
Patients is one of the largest patient summary collections, with the largest
scale of relation annotations for benchmarking ReCDS. Besides, the patients in
our dataset show a much higher level of diversity in terms of demographics and
medical conditions than existing patient collections. Our manual evaluation
shows that both patient summaries and relation annotations in PMC-Patients are
of high quality.

Based on PMC-Patients, we formally define two ReCDS tasks: Patient-to-Article
Retrieval (ReCDS-PAR) and Patient-to-Patient Retrieval (ReCDS-PPR). We
systematically evaluate the performance of various feature-based and learning-
based ReCDS systems, and the experimental results show that both ReCDS-PAR and
ReCDS-PPR are challenging tasks and we call for further improvements. We also
present highly-relevant case studies to demonstrate the potential application
and significance of our retrieval tasks in three typical clinical scenarios.

Figure 1 and Figure 2 show an overview of our dataset and ReCDS benchmark,
respectively. In summary, the key contributions of this study are three-fold:

  1. 1.

We introduce PMC-Patients333Available at https://github.com/pmc-patients/pmc-
patients, a first-of-its-kind dataset consisting of 167k patient summaries
extracted from case reports. We systematically characterize PMC-Patients, and
show that it is a diverse dataset with high-quality annotations.

  2. 2.

Based on PMC-Patients, we formally define two tasks and provide the largest-
scale resources to benchmark Retrieval-based Clinical Decision Support (ReCDS)
systems: Patient-to-Article Retrieval (ReCDS-PAR) and Patient-to-Patient
Retrieval (ReCDS-PPR). ReCDS-PAR contains 3.1M relevant patient-article pairs,
and ReCDS-PPR contains 293k similar patient-patient pairs.

  3. 3.

We systematically evaluate various ReCDS systems on the PMC-Patients
benchmark. We also conduct several case studies to demonstrate the clinical
utility of PMC-Patients.

Figure 1: Overview of the PMC-Patients dataset architecture. Patient summaries
are extracted by identifying certain sections in PMC articles. The cited
articles and patients are considered relevant and similar, respectively.
Patients from the same report are also considered similar. Figure 2: Overview
of the PMC-Patients ReCDS benchmark. Given a query patient, there are two
tasks: 1. Patient-to-article retrieval requires returning relevant articles
from PubMed; 2. Patient-to-patient retrieval requires returning similar
patients from PMC-Patients.

##  2 Material and methods

To collect the PMC-Patients dataset, we utilize the full-text literature
resources in PubMed Central (PMC)444https://www.ncbi.nlm.nih.gov/pmc/ and the
citation relationships in PubMed555https://pubmed.ncbi.nlm.nih.gov/, which
will be described in Section 2.1. Based on PMC-Patients, we formally define
two ReCDS benchmarks: Patient-to-Article Retrieval (ReCDS-PAR) and Patient-to-
Patient Retrieval (ReCDS-PPR). We will present the tasks in Section 2.2, and
introduce the baseline methods in Section 2.3.

###  2.1 PMC-Patients Dataset

We only use PMC articles with at least CC BY-NC-SA license (about $3.2$M) to
build the redistributable PMC-Patients dataset. The collection pipeline can be
summarized in three steps (the implementation details and graphical
illustration of the pipeline can be found in A):

  1. 1.

We identify potential patient summaries in each article section using
extraction triggers, which are a set of regular expressions searching for
specific patterns of patient summaries, such as “Case report” and “Patient
representation” in the section title.

  2. 2.

For sections identified in (a), we extract patient summary candidates using
several extractors. Extractors operate at the paragraph level, so a candidate
patient summary always consists of one or several complete paragraphs.
Besides, we also extract the candidates’ demographics (ages and genders) using
regular expressions.

  3. 3.

We apply various filters to each candidate patient summary extracted in (b) to
exclude candidates that are too short, non-English, or without patient
demographics.

For each extracted patient summary in PMC-Patients, we use the citation graph
of PubMed666Extracted from the PubMed baseline updated until July 2022
(https://ftp.ncbi.nlm.nih.gov/pubmed/baseline/). to automatically annotate (1)
relevant articles in PubMed and (2) similar patients in PMC-Patients.

Annotating relevant articles: We assume that if a PubMed article cites or is
cited by a patient-containing article, the article is relevant to the patient.
Formally, we denote a patient as $p$, and the article that contains $p$ as
$a(p)$. We define any PubMed article $a^{\prime}$ relevant to the patient $p$,
denoted as $Rel(p,a^{\prime})=1$, if:
$a^{\prime}\xrightarrow{\text{cites}}a(p)$, or
$a(p)\xrightarrow{\text{cites}}a^{\prime}$, or $a(p)=a^{\prime}$ (to ensure
label completeness).

Annotating similar patients: We annotate similar patients based on relevant
articles. For each patient in PMC-Patients, if its relevant articles contain
other patients in the dataset, we will label them as similar patients.
Formally, we define any two patients $p_{x}$ and $p_{y}$ in PMC-Patients
similar, denoted as $Sim(p_{x},p_{y})=1$, if:
$a(p_{x})\xrightarrow{\text{cites}}a(p_{y})$, or
$a(p_{y})\xrightarrow{\text{cites}}a(p_{x})$, or $a(p_{x})=a(p_{y})$.

###  2.2 PMC-Patients ReCDS Benchmarks

The PMC-Patients dataset contains $167$k patient summaries, annotated with
$3.1$M relevant articles and $293$k similar patients. Based on the dataset, we
define two benchmarking tasks for ReCDS: Patient-to-Article Retrieval (ReCDS-
PAR) and Patient-to-Patient Retrieval (ReCDS-PPR). Both are modeled as
information retrieval tasks where the input is a patient summary
$p\in\mathcal{P}$, where $\mathcal{P}$ denotes the PMC-Patients dataset. For
ReCDS-PAR, the objective is to retrieve PubMed articles relevant to the input
patient from the corpus $\mathcal{A}$. Instead of using the entire $33.4$M
articles in PubMed, we restrict the retrieval corpus to contain only articles
relevant to at least one patient. Formally, $\mathcal{A}=\\{a\ |\ \exists
p\in\mathcal{P},Rel(p,a)=1\\}$ and contains 1.4M articles, which is a more
feasible setting. For ReCDS-PPR, the objective is to retrieve patients similar
to the input patient from PMC-Patients. The benchmark statistics are shown in
Table 1.

Split |  Source Articles | ReCDS-PAR | ReCDS-PPR
---|---|---|---
Query Patients |  Relevant Articles | A/P |  Query Patients |  Similar Patients | P/P
train | 131k | 155.2k | 2.9M | 18.64 | 94.6k | 257.4k | 2.72
dev | 5k | 5.9k | 107.5k | 18.14 | 2.9k | 6.4k | 2.22
test | 5k | 6.0k | 114.1k | 19.1 | 2.8k | 7.5k | 2.66
Corpus | 1.4M candidate articles | 155.2k candidate patients
Table 1: Statistics of the ReCDS-PAR and ReCDS-PPR benchmarks. A/P: Average
number of relevant articles per query patient. P/P: Average number of similar
patients per query patient.

We split the train/dev/test on the article level. Specifically, we randomly
select two subsets of articles (5k in each) from which PMC-Patients is
extracted and include the corresponding patients in the dev and test dataset
as query patients. Patient summaries extracted from other articles are
included as the training query patients and also used as the retrieval corpus
${\mathcal{P}}$.

We evaluate retrieval models on both benchmarks with Mean Reciprocal Rank
(MRR), Precision at 10 (P@10), normalized Discounted Cumulative Gain at 10
(nDCG@10), and Recall at 1k (R@1k).

###  2.3 Baseline models

We implement three types of baseline retrieval models for both ReCDS-PAR and
ReCDS-PPR: sparse retriever, dense retriever, and nearest neighbor retriever.

Sparse retriever: We implement a BM25 retriever [19] with
Elasticsearch777https://www.elastic.co/elasticsearch. The parameters of the
BM25 algorithm are set as default values in Elasticsearch
($b=0.75,k_{1}=1.2$). For ReCDS-PAR, we index the title and abstract of a
PubMed article as separate fields and the weights given to the two fields when
retrieving are empirically set as $3:1$.

Dense retriever: Dense retrievers represent the patients and articles in a low
dimensional space using BERT-based encoders and perform retrieval based on
maximum inner-product search. Concretely, we denote the encoder as $f$, and
$\mathbf{e}_{d}=f(d)$ refers to the low-dimensional embedding generated by the
encoder for a given passage $d$. Then for a query patient $q$ and an article
$a$ in our retrieval corpus $\mathcal{A}$, the relevance score between them is
defined as the inner product of their embeddings:
$s_{\text{dense}}(q,a)=\mathbf{e}_{q}\cdot\mathbf{e}_{a}$. The similarity
score $s_{\text{dense}}(q,p)$ between $q$ and a patient $p\in\mathcal{P}$ is
defined similarly.

We first try direct transferring of Sentence-BERT [20] and Contriever [21],
two widely-used dense retrievers pre-trained on MS MARCO [22], a general
domain retrieval dataset of large scale. Then we train our own dense
retrievers by fine-tuning pre-trained encoders on the PMC-Patients dataset. To
be specific, for a given query patient $q_{i}$, a similar patient / relevant
article $p_{i}^{+}$, and a set of dissimilar patients / irrelevant articles
$p_{i,1}^{-},p_{i,2}^{-},\dots,p_{i,n}^{-}$ from the training data, we use the
negative log-likelihood of the positive passage as the loss function:

$L(q_{i},p_{i}^{+},p_{i,1}^{-},p_{i,2}^{-},\dots,p_{i,n}^{-})=-\log\frac{e^{s_{\text{dense}}(q_{i},p_{i}^{+})}}{e^{s_{\text{dense}}(q_{i},p_{i}^{+})}+\sum_{j=1}^{n}e^{s_{\text{dense}}(q_{i},p_{i,j}^{-})}}$

We train the dense retrievers with in-batch negatives [23], where
$p_{i,j}^{-}\in\\{p_{k}^{+}\ |\ k\neq i\\}$.

We train several different encoders, all of which are Transformer encoders
[24] initialized by domain-specific BERT [25], including PubMedBERT [26],
Clinical BERT [27], BioLinkBERT [28], and SPECTER[29]. For the ReCDS-PPR task,
only one encoder is used, while for the ReCDS-PAR task, we train two
independent encoders to encode patients and articles separately, due to their
structural differences.

Nearest Neighbor (NN) retriever: We assume that if two patients are similar,
then their respective relevant article and similar patient sets should have a
high overlap degree, based on which we implement the following NN retriever
similar to [30]. For each patient in the training queries $p\in\mathcal{P}$,
we define its relevant article set as
$\mathcal{R}(p)=\\{a|a\in\mathcal{A},Rel(p,a)=1\\}$. For each query patient
$q$, we first retrieve top $K$ similar training patients
$p_{1},p_{2},\dots,p_{k}\in\mathcal{P}$ as its nearest neighbors using
BM25888We also try using fine-tuned dense retrievers which give suboptimal
performance.. We take the union of their relevant articles as the candidate
set:

$\mathcal{C}(q)=\mathcal{R}(p_{1})\cup\mathcal{R}(p_{2})\cup\cdots\cup\mathcal{R}(p_{K})$

Then the candidate articles $c_{i}\in\mathcal{C}(q)$ are ranked by relevance
scores $s_{\text{NN}}(q,c_{i})$ defined as:

$s_{\text{NN}}(q,c_{i})=\sum_{k=1}^{K}s_{\text{BM25}}(q,p_{k})I\\{c_{i}\in\mathcal{R}(p_{k})\\}$

NN retriever for ReCDS-PPR is implemented similarly.

##  3 Results

In this section, we will first analyze the characteristics of the PMC-Patients
dataset in Section 3.1, including basic statistics and patient diversity. We
then show the dataset is of high quality in terms of the summary extraction
and the relation annotation in Section 3.2. In the end, we present the
performance of baseline methods on the ReCDS-PAR and ReCDS-PPR benchmarks in
Section 3.3.

###  3.1 PMC-Patients Dataset

Scale: Table 2 shows the basic statistics of patient summaries in PMC-
Patients, in comparison to MIMIC, the largest publicly available clinical
notes dataset, and TREC CDS, a widely-used dataset for ReCDS. For MIMIC, we
report the statistics of discharge summaries of both MIMIC-III and MIMIC-IV.
For TREC CDS, we combine the data released in three years’ CDS tracks
(2014-2016) and use the “description” fields. PMC-Patients contains $167$k
patient summaries extracted from $141$k PMC articles, making it the largest
patient summary dataset in terms of the number of patients, and the second
largest in terms of the number of notes. Besides, PMC-Patients has $3.1$M
patient-article relevance annotations, which is over $27\times$ the size of
TREC CDS ($113$k in total). PMC-Patients also provides the first large-scale
patient-similarity annotations, consisting of $293$k similar patient pairs.

Dataset | Count |  Average Length (words) | Relations
---|---|---|---
Patients | Notes |  Relevant Articles |  Similar Patients
PMC-Patients (ours) | 167k | 167k | 410 | 3.1M | 293k
MIMIC-III (d.s.) | 41k | 60k | 1,282 | – | –
MIMIC-IV (d.s.) | 146k | 332k | 1,480 | – | –
TREC CDS (all) | 90 | 90 | 92 | 113k | –
Table 2:  Statistics of PMC-Patients, in comparison to MIMIC (d.s.: discharge
summaries), and TREC CDS (2014-2016).

Length: On average, PMC-Patients summaries are much longer than TREC
descriptions ($410$ v.s. $92$ words), but shorter than MIMIC discharge
summaries ($410$ v.s. over 1k words). Figure 3 (a) presents the length
distributions of PMC-Patients, TREC CDS descriptions, and MIMIC-IV discharge
summaries.

Figure 3: (a): Length distributions of PMC-Patients compared to MIMIC-IV
discharge summaries and TREC CDS descriptions (x-axis truncated). (b): Patient
age distributions of PMC-Patients compared to MIMIC-IV. *Exact ages of
patients older than 89 years old are obscured in MIMIC and thus taken as 90 in
the figure. (c): Relative frequency of top 30 ICD codes in MIMIC-IV (left) and
MeSH Diseases terms in PMC-Patients (right). The colors are associated with
relative frequency, and the color bar attached to the figure illustrates this.

Demographics: The age distributions of PMC-Patients and MIMIC-IV are presented
in Figure 3 (b). There are too few patients to observe the age distribution in
TREC CDS, so we don’t include it in the figure. On average, patients in PMC-
Patients are younger than MIMIC-IV ($43.4$ v.s. $58.7$ years old), and patient
ages are more evenly distributed ($6.39$ v.s. $6.09$ Shannon bits). PMC-
Patients covers pediatric patients while MIMIC-IV does not. The gender
distribution in both datasets is balanced. PMC-Patients consists of $52.5$%
male and $47.5$% female, while MIMIC-IV consists of $48.7$% male and $51.3$%
female.

Medical conditions: We also analyze the medical conditions associated with the
patients. For PMC-Patients, we use the MeSH Diseases terms of the articles,
and for MIMIC, we use the ICD codes999There are both ICD-9 and ICD-10 codes in
MIMIC-IV.. The most frequent medical conditions are shown in Figure 3 (c). In
PMC-Patients, the majority of frequent conditions are related to cancer, with
the exception of COVID-19 as the second most frequent condition. In MIMIC-IV,
severe non-cancer diseases (e.g. hypertension) have the highest relative
frequencies, and their absolute values are much higher than those of the most
frequent conditions in PMC-Patients. For example, hypertension and lung
neoplasms are the most frequent condition in MIMIC and PMC-Patients,
respectively. Over 60% of MIMIC patients have hypertension, while less than 4%
of patients in PMC-Patients have lung neoplasms. In addition, PMC-Patients
covers $4,031/4,933$ (81.7%) MeSH Diseases terms, relatively more than the
$8,955/14,666$ (61.1%) ICD-9 codes and $16,464/95,109$ (17.3%) ICD-10 codes
covered by MIMIC-IV.

###  3.2 Dataset Quality Evaluation

####  3.2.1 Patient summary extraction

In this section, we evaluate the quality of the automatically extracted
patient summaries and demographics in PMC-Patients. The evaluation is
performed on a random sample of $500$ articles from the benchmark test set.
Two senior M.D. candidates are employed to label the patient note spans at the
paragraph level and the patient demographics. Agreed annotations are directly
considered as ground truth, while disagreed annotations are discussed until a
final agreement is reached.

Table 3 shows the extraction quality of PMC-Patients and the two human experts
against the ground truth. A total of $604$ patients are extracted by human
experts. The patient note spans extracted in PMC-Patients are of high quality
with a larger than 90% strict F1 score. The extracted demographics are close
to 100% correct. Besides, two annotators exhibit a high level of agreement,
with most disagreements being minor differences regarding the boundary of a
note span.

Quality | Note Span | Age | Gender
---|---|---|---
PMC-Patients | 91.24 | 99.77 | 100.0
Expert A | 97.34 | 100.0 | 100.0
Expert B | 97.28 | 99.91 | 99.49
Table 3: Extraction quality of the PMC-Patients dataset and two experts
against the ground truth. Note span recognition is evaluated by F1 score. Age
recognition is evaluated by min(annotated_age, true_age)/max(annotated_age,
true_age). Gender recognition is evaluated by accuracy. All numbers are
percentages.

####  3.2.2 Patient-level relation annotation

To evaluate the quality of patient-level relation annotations in PMC-Patients,
we retrieve top $5$ relevant articles and top $5$ similar patients using BM25
for each patient extracted by the human experts in the previous section ($604$
patients from $500$ articles), resulting in over 3k patient-article and 3k
patient-patient pairs for human annotation. To annotate patient-article
relevance, we follow the guidelines of the TREC CDS tracks [11, 12, 3], where
we annotate the type of clinical question that can be answered by an article
about a patient, including diagnosis, test, and treatment. To annotate
patient-patient similarity, we follow the recommendations from [14], where we
annotate whether two patients are similar in multiple dimensions: features,
outcomes, exposure, and others. To assess the binary relational annotations in
PMC-Patients against the multi-dimensional human annotations, we simply
convert the latter into an integer score by counting the number of relevant or
similar aspects. For example, if two patients are annotated as similar in
terms of “features” and “outcomes”, we will give it a score of $2$.

Figure 5 shows the distributions of the human scores (x-axis) grouped by the
relation annotations in PMC-Patients (Irrelevant v.s. Relevant and Dissimilar
v.s. Similar). T-test shows that patient-article and patient-patient pairs
with PMC-Patients annotations have significantly higher human scores than
those without ($p<0.01$ for both cases). Besides, almost all positive pairs
are considered relevant/similar by a human expert, indicating PMC-Patients
automatic relational annotations achieve quite high precision.

###  3.3 ReCDS Benchmark Results

The performance of various baseline methods on the test set of two ReCDS tasks
is shown in Table 4. Surprisingly, BM25 remains a strong baseline that
achieves the best performance on MRR for both tasks and also performs the best
on nDCG@10 for ReCDS-PPR. This indicates the importance of matching the exact
words in the case reports for retrieving similar patients or relevant
articles. Sentence-BERT and Contriever, two dense retrievers trained on the
general domain MS MARCO dataset, do not generalize well to our ReCDS tasks.
Their performance on all metrics is much worse than the BM25 baseline, which
is consistent with previous studies [31, 32] that dense retrievers may fail to
perform zero-shot retrieval in specific domains such as biomedicine.

On the other hand, dense retrievers fine-tuned on PMC-Patients show
significant performance improvements over general domain retrievers,
indicating the importance of domain-specific fine-tuning. Although BM25
performs better on MRR, fine-tuned retrievers achieve the highest P@10 on both
tasks and the highest nDCG@10 on ReCDS-PPR. They also have much higher recall
than BM25 which suffers from vocabulary mismatch, showing that semantic
matching is indispensable to retrieve more relevant articles or similar
patients. Fine-tuned Clinical BERT performs the worst among other domain-
specific BERTs. This is probably due to the pre-training corpus and tasks of
these encoders: PubMedBERT and BioLinkBERT are pre-trained on PubMed; SPECTER
and BioLinkBERT incorporate citation graph in pre-training; while Clinical
BERT is trained on MIMIC, whose language distribution is quite different from
PubMed, and never learns citation relationships. However, the metrics of the
best baseline method are still quite low, highlighting the challenge of the
PMC-Patients ReCDS benchmark.

NN retriever generally performs worse than BM25 and dense retrievers,
indicating that measuring patient-article relevance based on citation graph
distance may not be suitable for the task.

Method | ReCDS-PAR | ReCDS-PPR
---|---|---
MRR | Prec | nDCG | Recall | MRR | Prec | nDCG | Recall
BM25 | 48.22 | 9.97 | 15.28 | 30.64 | 22.86 | 4.67 | 18.29 | 69.66
Dense retriever (MS MARCO) |  |  |  |  |  |  |  |
SentenceBERT | 10.58 | 2.71 | 3.53 | 13.52 | 5.28 | 1.17 | 3.88 | 37.55
Contriever | 15.03 | 3.41 | 4.62 | 16.74 | 10.50 | 2.24 | 8.01 | 52.64
Dense retriever (PMC-Patients) |  |  |  |  |  |  |  |
PubMedBERT | 42.96 | 16.08 | 19.51 | 63.40 | 19.37 | 5.05 | 16.30 | 79.35
Clinical BERT | 24.94 | 8.56 | 10.20 | 48.93 | 10.24 | 2.62 | 7.82 | 67.43
BioLinkBERT | 40.89 | 15.33 | 18.47 | 62.44 | 21.20 | 5.59 | 18.06 | 80.49
SPECTER | 46.41 | 15.59 | 19.70 | 57.98 | 15.08 | 3.79 | 12.27 | 73.01
NN retriever | 18.76 | 5.93 | 7.03 | 26.55 | 6.30 | 2.40 | 4.83 | 59.49

Table 4: PAR and PPR performances of baseline retrievers (in percentage).
Numbers in bold indicate the best results in each column. Precision (Prec) and
nDCG are calculated at 10, and recall is calculated at 1,000.

##  4 Related work

###  4.1 Patient summary dataset and case reports

Traditionally, patient summary datasets are collected from clinical notes in
EHRs, such as MIMIC, MTSamples101010www.mtsamples.com, the THYME project [33],
the n2c2111111https://n2c2.dbmi.hms.harvard.edu/ (originally named
i2b2121212https://www.i2b2.org/NLP/DataSets/Main.php) project, and the OHNLP
Challenges [34, 35]. However, except for MIMIC, these datasets are limited by
size and diversity, typically containing only several hundred to a few
thousand clinical note pieces and focusing on specific diseases.

More recently, clinical case reports have been utilized to construct datasets,
but most of the existing works focus on specific tasks such as named entity
recognition [36, 37], abbreviation resolution [38], and semantic similarity
[39]. They only use case reports as a source of clinical texts, and the
resulting datasets are task-oriented, rather than a patient summary dataset.
Only MACCR [40], CAS [41, 42], and the E3C project [43] present patient
summary datasets extracted from case reports. Among them, MACCR focuses on
curating structured metadata of clinical case reports instead of using free-
text patient summaries. CAS and the E3C project mainly focus on European
languages such as French and Spanish rather than English, with the dataset
scales still limited to several thousand. In contrast, PMC-Patients is much
larger, more diverse, and contains patient-level relation annotations.

###  4.2 Retrieval-based clinical decision support

Due to the lack of an adequate patient summary dataset and the prohibitive
costs of manual annotations, there is currently no large-scale ReCDS benchmark
dataset available. Most existing methodology researches on ReCDS-PAR use TREC
CDS and TREC Precision Medicine (PM) [44, 45, 46, 47]. TREC CDS focuses on
retrieving relevant PMC articles for given patient summaries curated by human
experts or excerpted from MIMIC with specific intents (e.g. finding
treatment/diagnosis). TREC PM focuses on retrieving relevant literature from
PubMed or MEDLINE131313https://www.medline.com/ and eligible clinical trials
that can provide precision medicine-related evidence for a cancer patient,
given the patient’s cancer type, genetic variants, basic demographics, and
other potential factors. However, each year, only 30-50 patient summaries are
released and annotated with patient-article relevance, which also severely
limits the patient diversity in these datasets. Furthermore, TREC PM only
contains cancer patients. In contrast, PMC-Patients has a much larger
collection of patient summaries (167k) that cover a wider range of medical
conditions, and the largest scale of patient-article relevance annotations
(3.1M).

To the best of our knowledge, there is no publicly available similar patient
retrieval dataset. PMC-Patients bypasses the difficulty and expense of
patient-level annotations using the PubMed citation graph and construct the
first large-scale ReCDS-PPR dataset of $293$k patient-patient similarity
annotations.

##  5 Discussion

###  5.1 Clinical significance

ReCDS provides valuable insights for healthcare providers in diagnosis,
testing, and treatment of a queried patient, particularly in medically grey
zones where high-level evidence is scarce, personalized management for
multiple active comorbidities, and off-label use of novel therapeutics. We
here present three case studies in the following section to demonstrate how
PMC-Patients can benefit clinicians in different ways. Specifically, we focus
on retrieval of similar patients since this is much less explored than
relevant article retrieval. Table 5 shows the three cases under different
scenarios with query patient summaries, examples of similar patients retrieved
from PMC-Patients, and demonstrations of the clinical significance. The
detailed inputs and outputs for performing case studies are shown in C.

Input summary |  Retrieval output example |  Description and significance
---|---|---
Patient: idiopathic thrombocytopenia, glomerulonephritis, and hearing impairment. Scenario: diagnosis |  Case Report: Pathogenic MYH9 c.5797delC Mutation in a Patient With Apparent Thrombocytopenia and Nephropathy. (patient_uid: 8355614-1) |  Identifying highly-likely combination of associated manifestation and underlying etiology for rare disease like field-experts
Patient: history of atrial fibrillation and deep vein thrombosis, signs of cholangitis. Scenario: test |  Hemorrhagic cholecystitis causing hemobilia and common bile duct obstruction. (patient_uid: 6463387-1) |  Highlighting related active issues for patients with multiple comorbidities thus overcoming cognitive blind-spot
Patient: melanoma, initially responsive to BRAF inhibitor but later progressed despite treated with PD-1 inhibitor. Scenario: treatment |  Response to Ipilimumab/Nivolumab Rechallenge and BRAF Inhibitor/MEK Inhibitor Rechallenge in a Patient with Advanced Metastatic Melanoma Previously Treated with BRAF Targeted Therapy and Immunotherapy. (patient_uid: 7334770-1) |  Out-of-textbook treatment for disease failing standard-of-care, thereby advancing implementation of off-label therapeutics

Table 5: Case studies on three patients under different scenarios. For each
query patient, we present an example of the retrieved similar patients from
PMC-Patients, with corresponding description and significance of assistance in
query-patient management.

The first case involves a diagnostic dilemma of early-onset idiopathic
thrombocytopenia, with co-occurred, seemingly unrelated conditions of renal
disease, hearing loss, and suspicious family history. The top retrieved
patient shows MYH9 mutation [48], which is the exact etiology of this case.
MYH9-related thrombocytopenia is extremely rare (1:20,000-25,000) [49] and is
thus challenging to diagnose for non-experts. Other retrieval results also
show other possible diagnoses including Alport syndrome [50] and anti-basement
membrane disease [51]. Its capability to recognize associated features from
multiple manifestations and proposing insightful diagnoses is therefore
greatly useful, especially in rare diseases.

The second case presents a female patient with a history of atrial
fibrillation and deep venous thrombosis who shows acute hepatobiliary
symptoms. ReCDS retrieves highly relevant cases, covering most common
conditions including cholecystitis [52], bile leak [53], and Mirizzi syndrome
[54]. Impressively, ReCDS is able to bring up potentially dangerous bleeding
complications (hemobilia), via suspecting anticoagulation use from her cardiac
and thrombotic comorbidities [55] This requires further monitoring and
testing, thus standing as important reminder in busy clinics where non-major
medical problems can be easily ignored.

The third case asks an open question for treatment of metastatic melanoma
failing standard care, pursuing answers in precision medicine similarly as the
TREC PM 2020 track [56]. The retrieved cases include attempts of
ipilimumab/nivolumab rechallenge, BRAFi and MEKi rechallenge [57], and single
agent PD-1 inhibitor [58], each of which providing sound evidence with
detailed clinical course background for an oncologist’s reference.
Additionally, the approach itself favors effective treatment combinations
(paradoxically thanks to positive report bias), and thus dynamically
encourages evidence accumulation towards more promising directions,
facilitating future clinical trial designs.

In conclusion, ReCDS can benefit clinicians in various ways, by recognizing
rare diseases, overcoming testing blind spots, and advancing treatment
evidence. With its potential to improve quality of medical care, ReCDS is
especially valuable for clinicians in this era of precision medicine and
personalized health.

###  5.2 Limitations and future work

Our experiments demonstrate that there is still much room for improvement on
the PMC-Patients ReCDS benchmark. We outline some potential directions for
further research: 1\. Many patient summaries in PMC-Patients have token counts
that far exceed BERT’s 512 token limit, and truncation is applied in our
baselines, which suffer from inevitable information loss. Therefore, retrieval
performance may be further enhanced by using efficient transformers [59] such
as Big bird [60] and Longformer [61]. 2\. Reranking using pre-trained encoders
based on cross attentions, including cross-encoders [62], poly-encoders [63],
and ColBERT [64] may significantly improve retrieval performance. 3\. Our
experiments indicate that both lexical and semantic features are crucial for
ReCDS. Previous research has explored combining sparse retrieval and dense
retrieval in the general domain [65, 66], which may also be useful for the
PMC-Patients benchmark.

ReCDS has also been shown helpful in various clinical tasks including question
answering [67], and patient outcome prediction [68], where retrieved relevant
articles serve as additional evidence for the model to refer to. More
recently, with the huge success achieved by Large Language Models (LLMs), many
studies have explored further augmenting LLMs with retrieval evidence [69, 70,
71, 72]. PMC-Patients can serve both as a benchmark for training and
evaluating retrieval systems and as an evidence collection for improving
clinical tasks and augmenting clinical LLMs such as ChatDoctor [73].

##  6 Conclusion

In this paper, we present PMC-Patients, a large-scale, diverse, and publicly
available patient summary dataset with patient-article relevance and patient-
patient similarity annotations. Based on PMC-Patients, we formally define two
tasks and provide the largest-scale dataset to benchmark ReCDS: Patient-to-
Article Retrieval (ReCDS-PAR) and Patient-to-Patient Retrieval (ReCDS-PPR). We
evaluate various ReCDS systems on PMC-Patients ReCDS benchmarks and show that
both tasks are quite challenging, calling for further research. We also
conduct several case studies on our proposed ReCDS benchmark to show the
clinical utility of our dataset.

##  Appendix A Patient summary extraction modules

Figure 4 shows the overall collection pipeline of the PMC-Patients dataset.
The detailed implementations of the patient summary extraction modules
mentioned in Section 2.1 are described below.

Figure 4: Collection pipeline of PMC-Patients. Patient summaries are
identified by extraction triggers, extracted by extractors, and pass various
filters. Patient-level relations are annotated using citation relationships in
PubMed.

###  A.1 Extraction triggers

Extraction triggers are a set of regular expressions to identify whether there
are no, one, or multiple potential patient summaries in a given section,
basically consisting of two successive triggers:

section_title_trigger: Searches in the section title for certain phrases that
indicate the presence of patient summaries, such as “Case Report” and “Patient
Representation”.

multi_patients_trigger: Searches for certain patterns in the first sentence of
each paragraph and the titles of the subsections to identify whether multiple
patients are presented, such as “The second patient” and “Case 1”.

###  A.2 Extractors

Extractors perform at the paragraph level, i.e. an extracted patient summary
always consists of one or several complete paragraphs and no split within a
paragraph is performed. Depending on whether multi_patients_trigger is
triggered, different extractors are used:

single_patient_extractor: Extracts all paragraphs in the section as one
patient summary, if multi_patients_trigger is not triggered.

multi_patients_extractor: Extracts paragraphs between successive triggering
parts (the last one is taken till the end of the section) as multiple patient
summaries, if multi_patients_trigger is triggered.

###  A.3 Filters

We remove noisy candidates with three filters:

length_filter: Excludes candidates with less than $10$ words.

language_filter: Excludes candidates with more than 3% non-English characters.

demographic_filter: Identifies the age and gender of a patient using regular
expressions and excludes candidates missing either demographic characteristic.

The regular expression rules and parameters in the modules above are generated
empirically by manually reading and summarizing hundreds of case reports and
then refined on a test set of $100$ articles, which are independent of the
$500$ articles used for human evaluation in Section 3.2.

##  Appendix B Human evaluation of PMC-Patients relation annotations

Figure 5: Distributions of the human-annotated relevance (left) and similarity
(right) scores grouped by PMC-Patients automatic annotations.

##  Appendix C Top 5 retrieval results in case studies

We show the detailed inputs and retrieved top 5 similar patients and relevant
articles for the three case studies in Table 5 with the titles of the relevant
articles or source articles from which similar patients are extracted.
“Scenario” is not part of the input since we do not distinguish different
scenarios when training the retrievers.

###  C.1 Patient A

Query patient: A 10-year-old girl presents with cytopenia, hematuria and
proteinuria indicating glomerulitis and decreased hearing. Multiple family
members also had hearing impairments.

Scenario: Diagnosis.

Similar patients:

  1. 1.

patient_uid: 8355614-1.

Case Report: Pathogenic MYH9 c.5797delC Mutation in a Patient With Apparent
Thrombocytopenia and Nephropathy.

  2. 2.

patient_uid: 792615-1.

Atypical anti-glomerular basement membrane disease.

  3. 3.

patient_uid: 7434753-1.

Athogenic evaluation of synonymous COL4A5 variants in X-linked Alport syndrome
using a minigene assay.

  4. 4.

patient_uid: 7752573-1.

A case of hypoparathyroidism, deafness, and renal dysplasia (HDR) syndrome
with a novel frameshift variant in GATA3, p.W10Cfs40, lacks kidney
malformation.

  5. 5.

patient_uid: 8101677-1.

A Case of Hearing Impairment with Renal Dysfunction.

Relevant articles:

  1. 1.

PMID: 22567374.

Hearing loss in children with osteogenesis imperfecta.

  2. 2.

PMID: 26331839.

Audiometric Characteristics of a Dutch DFNA10 Family With Mid-Frequency
Hearing Impairment.

  3. 3.

PMID: 16585279.

Universal newborn hearing screening and postnatal hearing loss.

  4. 4.

PMID: 18672655.

Hearing rehabilitation in a patient with sudden sensorineural hearing loss in
the only hearing ear.

  5. 5.

PMID: 16490878.

Idiopathic sudden sensorineural hearing loss in the only hearing ear: patient
characteristics and hearing outcome.

###  C.2 Patient B

Query patient: A 94 year old female with hx recent PE/DVT, atrial
fibrillation, CAD presents with fever and abdominal pain. An abdominal CT
demonstrates a distended gallbladder with gallstones and biliary obstruction
with several CBD stones.

Scenario: Test.

Similar patients:

  1. 1.

patient_uid: 3286734-1.

The Successful Treatment of Chronic Cholecystitis with SpyGlass
Cholangioscopy-Assisted Gallbladder Drainage and Irrigation through Self-
Expandable Metal Stents.

  2. 2.

patient_uid: 6463387-1.

Hemorrhagic cholecystitis causing hemobilia and common bile duct obstruction.

  3. 3.

patient_uid: 4938132-1.

Mirizzi syndrome with an unusual aberrant hepatic duct fistula: a case report.

  4. 4.

patient_uid: 7292700-1.

Biloma: A Rare Manifestation of Spontaneous Bile Leak.

  5. 5.

patient_uid: 7879265-1.

Biliary Peritonitis Caused by Spontaneous Bile Duct Rupture in the Left
Triangular Ligament of the Liver after Endoscopic Sphincterotomy for
Choledocholithiasis.

Relevant articles:

  1. 1.

PMID: 24688200.

Fatal abdominal hemorrhage associated with gallbladder perforation due to
large gallstones.

  2. 2.

PMID: 26265970.

Choledochal Cyst Mimicking Gallbladder with Stones in a Six-Year-Old with
Right-sided Abdominal Pain.

  3. 3.

PMID: 31938556.

Not everything in the gallbladder is gallstones: an unusual case of biliary
ascariasis.

  4. 4.

PMID: 22794521.

Spontaneous cholecystocutaneous fistula as a rare complication of gallstones.

  5. 5.

PMID: 22028722.

Diagnosis and treatment of multiseptate gallbladder with recurrent abdominal
pain.

###  C.3 Patient C

Query patient: A 57-year-old man with stage IIIC melanoma was treated with
vemurafenib for 8 years with complete response until the disease progressed
with brain, lung and liver metastases. After stereotactic radiotherapy, he
received nivolumab but progressed again in 2 months later in lung and liver
metastases, showing hepatic failure and obstructive jaundice, with LDH value
was superior to two-times.

Scenario: Treatment.

Similar patients:

  1. 1.

patient_uid: 7662249-1.

Trametinib Induces the Stabilization of a Dual GNAQ p.Gly48Leu- and FGFR4
p.Cys172Gly-Mutated Uveal Melanoma. The Role of Molecular Modelling in
Personalized Oncology.

  2. 2.

patient_uid: 7334770-1.

Response to Ipilimumab/Nivolumab Rechallenge and BRAF Inhibitor/MEK Inhibitor
Rechallenge in a Patient with Advanced Metastatic Melanoma Previously Treated
with BRAF Targeted Therapy and Immunotherapy.

  3. 3.

patient_uid: 6180387-2.

Immune-related adverse events with immune checkpoint inhibitors affecting the
skeleton: a seminal case series.

  4. 4.

patient_uid: 5557522-3.

Response to single agent PD-1 inhibitor after progression on previous
PD-1/PD-L1 inhibitors: a case series.

  5. 5.

patient_uid: 5109681-1.

Recurrent pleural effusions and cardiac tamponade as possible manifestations
of pseudoprogression associated with nivolumab therapy- a report of two cases.

Relevant articles:

  1. 1.

PMID: 30449777.

Bicytopenia in Primary Lung Melanoma Treated with Nivolumab.

  2. 2.

PMID: 23579338.

Vemurafenib and radiation therapy in melanoma brain metastases.

  3. 3.

PMID: 33859852.

Fulminant Hepatic Failure after Chemosaturation with Percutaneous Hepatic
Perfusion and Nivolumab in a Patient with Metastatic Uveal Melanoma.

  4. 4.

PMID: 29434164.

Nivolumab Induces Sustained Liver Injury in a Patient with Malignant Melanoma.

  5. 5.

PMID: 26805247.

Multidisciplinary Therapy for Advanced Gastric Cancer with Liver and Brain
Metastases.

## References

  * [1] D. L. Sackett, Evidence-based medicine, in: Seminars in perinatology, Vol. 21, Elsevier, 1997, pp. 3–5.
  * [2] J. W. Ely, J. A. Osheroff, M. L. Chambliss, M. H. Ebell, M. E. Rosenbaum, Answering physicians’ clinical questions: obstacles and potential solutions, Journal of the American Medical Informatics Association 12 (2) (2005) 217–224.
  * [3] K. Roberts, D. Demner-Fushman, E. M. Voorhees, W. R. Hersh, Overview of the trec 2016 clinical decision support track, in: The Twenty-fifth Text REtrieval Conference (TREC) Proceedings, 2016.
  * [4] M. Pan, Y. Zhang, Q. Zhu, B. Sun, T. He, X. Jiang, An adaptive term proximity based rocchio’s model for clinical decision support retrieval, BMC Medical Informatics and Decision Making 19 (2019).
  * [5] B. Park, M. Afzal, J. Hussain, A. Abbas, S. Lee, Automatic identification of high impact relevant articles to support clinical decision making using attention-based deep learning, Electronics (2020).
  * [6] Z. Zhang, An improved bm25 algorithm for clinical decision support in precision medicine based on co-word analysis and cuckoo search, BMC Medical Informatics and Decision Making 21 (2021).
  * [7] Z. Zhang, X. Lin, S. Wu, A hybrid algorithm for clinical decision support in precision medicine based on machine learning, BMC Bioinformatics 24 (2023).
  * [8] H. Gurulingappa, L. Toldo, C. Schepers, A. Bauer, G. Megaro, Semi-supervised information retrieval system for clinical decision support, in: TREC, 2016.
  * [9] J. Sankhavara, Biomedical document retrieval for clinical decision support system, in: Annual Meeting of the Association for Computational Linguistics, 2018\.
  * [10] M. Shi, T.-H. Pan, H.-H. Huang, H.-H. Chen, Hybrid re-ranking for biomedical information retrieval at the trec 2021 clinical trials track, 2022.
  * [11] M. S. Simpson, E. M. Voorhees, W. Hersh, Overview of the trec 2014 clinical decision support track, Tech. rep. (2014).
  * [12] K. Roberts, M. S. Simpson, E. M. Voorhees, W. R. Hersh, Overview of the trec 2015 clinical decision support track, in: The Twenty-fourth Text REtrieval Conference (TREC) Proceedings, 2015.
  * [13] C. Buckley, E. M. Voorhees, Retrieval evaluation with incomplete information, in: M. Sanderson, K. Järvelin, J. Allan, P. Bruza (Eds.), SIGIR 2004: Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Sheffield, UK, July 25-29, 2004, ACM, 2004, pp. 25–32.  doi:10.1145/1008992.1009000.
URL https://doi.org/10.1145/1008992.1009000

  * [14] N. D. Seligson, J. L. Warner, W. S. Dalton, D. Martin, R. S. Miller, D. Patt, K. L. Kehl, M. B. Palchuk, G. Alterovitz, L. K. Wiley, et al., Recommendations for patient similarity classes: results of the amia 2019 workshop on defining patient similarity, Journal of the American Medical Informatics Association 27 (11) (2020) 1808–1812.
  * [15] L. Plaza, A. Díaz, Retrieval of similar electronic health records using umls concept graphs, in: International Conference on Application of Natural Language to Information Systems, Springer, 2010, pp. 296–303.
  * [16] C. W. Arnold, S. M. El-Saden, A. A. Bui, R. Taira, Clinical case-based retrieval using latent topic analysis, in: AMIA annual symposium proceedings, Vol. 2010, American Medical Informatics Association, 2010, p. 26.
  * [17] A. E. Johnson, T. J. Pollard, L. Shen, H. L. Li-Wei, M. Feng, M. Ghassemi, B. Moody, P. Szolovits, L. A. Celi, R. G. Mark, Mimic-iii, a freely accessible critical care database, Scientific data 3 (1) (2016) 1–9.
  * [18] A. Johnson, L. Bulgarelli, T. Pollard, S. Horng, L. A. Celi, R. Mark, Mimic-iv (2023).  doi:10.13026/6MM1-EK67.
URL https://physionet.org/content/mimiciv/2.2/

  * [19] S. E. Robertson, H. Zaragoza, The probabilistic relevance framework: Bm25 and beyond, Found. Trends Inf. Retr. 3 (2009) 333–389.
  * [20] N. Reimers, I. Gurevych, Sentence-bert: Sentence embeddings using siamese bert-networks, ArXiv abs/1908.10084 (2019).
  * [21] G. Izacard, M. Caron, L. Hosseini, S. Riedel, P. Bojanowski, A. Joulin, E. Grave, Unsupervised dense information retrieval with contrastive learning, 2021\.
  * [22] D. F. Campos, T. Nguyen, M. Rosenberg, X. Song, J. Gao, S. Tiwary, R. Majumder, L. Deng, B. Mitra, Ms marco: A human generated machine reading comprehension dataset, ArXiv abs/1611.09268 (2016).
  * [23] V. Karpukhin, B. Oğuz, S. Min, P. Lewis, L. Y. Wu, S. Edunov, D. Chen, W. tau Yih, Dense passage retrieval for open-domain question answering, ArXiv abs/2004.04906 (2020).
  * [24] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, I. Polosukhin, Attention is all you need, in: I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, R. Garnett (Eds.), Advances in Neural Information Processing Systems, Vol. 30, Curran Associates, Inc., 2017.
URL
https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-
Paper.pdf

  * [25] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding, in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Association for Computational Linguistics, Minneapolis, Minnesota, 2019, pp. 4171–4186.  doi:10.18653/v1/N19-1423.
URL https://aclanthology.org/N19-1423

  * [26] Y. Gu, R. Tinn, H. Cheng, M. R. Lucas, N. Usuyama, X. Liu, T. Naumann, J. Gao, H. Poon, Domain-specific language model pretraining for biomedical natural language processing, ACM Transactions on Computing for Healthcare (HEALTH) 3 (2022) 1 – 23.
  * [27] E. Alsentzer, J. Murphy, W. Boag, W.-H. Weng, D. Jindi, T. Naumann, M. McDermott, Publicly available clinical bert embeddings, in: Proceedings of the 2nd Clinical Natural Language Processing Workshop, 2019, pp. 72–78.
  * [28] M. Yasunaga, J. Leskovec, P. Liang, Linkbert: Pretraining language models with document links, in: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022, pp. 8003–8016.
  * [29] A. Cohan, S. Feldman, I. Beltagy, D. Downey, D. S. Weld, Specter: Document-level representation learning using citation-informed transformers, in: Annual Meeting of the Association for Computational Linguistics, 2020.
  * [30] Q. Jin, A. Shin, Z. Lu, Lader: Log-augmented dense retrieval for biomedical literature search (2023).  arXiv:2304.04590.
  * [31] N. Thakur, N. Reimers, A. Ruckl’e, A. Srivastava, I. Gurevych, Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models, ArXiv abs/2104.08663 (2021).
  * [32] Y. Kim, Applications and future of dense retrieval in industry, Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (2022).
  * [33] W. F. Styler, S. Bethard, S. Finan, M. Palmer, S. Pradhan, P. C. De Groen, B. Erickson, T. Miller, C. Lin, G. Savova, et al., Temporal annotation in the clinical domain, Transactions of the association for computational linguistics 2 (2014) 143–154.
  * [34] Y. Wang, N. Afzal, S. Liu, M. Rastegar-Mojarad, L. Wang, F. Shen, S. Fu, H. Liu, Overview of the biocreative/ohnlp challenge 2018 task 2: clinical semantic textual similarity, Proceedings of the BioCreative/OHNLP Challenge 2018 (2018).
  * [35] Y. Wang, S. Fu, F. Shen, S. Henry, O. Uzuner, H. Liu, et al., The 2019 n2c2/ohnlp track on clinical semantic textual similarity: overview, JMIR Medical Informatics 8 (11) (2020) e23375.
  * [36] S. Schulz, J. Ševa, S. Rodriguez, M. Ostendorff, G. Rehm, Named entities in medical case reports: Corpus and experiments, arXiv preprint arXiv:2003.13032 (2020).
  * [37] A. Gonzalez-Agirre, M. Marimon, A. Intxaurrondo, O. Rabal, M. Villegas, M. Krallinger, Pharmaconer: Pharmacological substances, compounds and proteins named entity recognition track, in: Proceedings of The 5th Workshop on BioNLP Open Shared Tasks, 2019, pp. 1–10.
  * [38] A. Intxaurrondo, M. Marimón, A. González-Agirre, J. A. Lopez-Martin, H. Rodriguez, J. Santamaria, M. Villegas, M. Krallinger, Finding mentions of abbreviations and their definitions in spanish clinical cases: The barr2 shared task evaluation results., IberEval@ SEPLN 2150 (2018) 280–289.
  * [39] N. R. Smalheiser, M. Luo, S. Addepalli, X. Cui, A manual corpus of annotated main findings of clinical case reports, Database 2019 (2019).
  * [40] J. H. Caufield, Y. Zhou, A. O. Garlid, S. P. Setty, D. A. Liem, Q. Cao, J. M. Lee, S. Murali, S. Spendlove, W. Wang, et al., A reference set of curated biomedical data and metadata from clinical case reports, Scientific data 5 (1) (2018) 1–18.
  * [41] N. Grabar, C. Dalloux, V. Claveau, Cas: corpus of clinical cases in french, Journal of Biomedical Semantics 11 (1) (2020) 1–10.
  * [42] C. Grouin, N. Grabar, V. Claveau, T. Hamon, Clinical case reports for nlp, in: BioNLP 2019-18th ACL Workshop on Biomedical Natural Language Processing, ACL, 2019, pp. 273–282.
  * [43] B. Magnini, B. Altuna, A. Lavelli, M. Speranza, R. Zanoli, The e3c project: European clinical case corpus, Language 1 (L2) (2021) L3.
  * [44] K. Roberts, D. Demner-Fushman, E. M. Voorhees, W. R. Hersh, S. Bedrick, A. J. Lazar, S. Pant, Overview of the trec 2017 precision medicine track, in: The Twenty-Sixth Text REtrieval Conference (TREC) Proceedings, Vol. 26, 2017.
  * [45] K. Roberts, D. Demner-Fushman, E. M. Voorhees, W. R. Hersh, S. Bedrick, A. J. Lazar, Overview of the trec 2018 precision medicine track, in: The Twenty-Seventh Text REtrieval Conference (TREC) Proceedings, 2018.
  * [46] K. Roberts, D. Demner-Fushman, E. M. Voorhees, W. R. Hersh, S. Bedrick, A. J. Lazar, S. Pant, F. Meric-Bernstam, Overview of the trec 2019 precision medicine track, in: The Twenty-Eighth Text REtrieval Conference (TREC) Proceedings, 2019.
  * [47] K. Roberts, D. Demner-Fushman, E. M. Voorhees, S. Bedrick, W. R. Hersh, Overview of the trec 2020 precision medicine track, in: The Twenty-Ninth Text REtrieval Conference (TREC) Proceedings, 2020.
  * [48] P. Ren, H. Chen, Y. Wang, C. Wang, S. Feng, H. Jiang, J. Chen, Case report: Pathogenic myh9 c. 5797delc mutation in a patient with apparent thrombocytopenia and nephropathy, Frontiers in Genetics 12 (2021) 705832.
  * [49] R. Fernandez-Prado, S. M. Carriazo-Julio, R. Torra, A. Ortiz, M. V. Perez-Gomez, Myh9-related disease: it does exist, may be more frequent than you think and requires specific therapy, Clinical kidney journal 12 (4) (2019) 488–493.
  * [50] T. Horinouchi, T. Yamamura, S. Minamikawa, C. Nagano, N. Sakakibara, K. Nakanishi, Y. Shima, N. Morisada, S. Ishiko, Y. Aoto, et al., Pathogenic evaluation of synonymous col4a5 variants in x-linked alport syndrome using a minigene assay, Molecular genetics & genomic medicine 8 (8) (2020) e1342.
  * [51] M. L. Troxell, D. C. Houghton, Atypical anti-glomerular basement membrane disease, Clinical Kidney Journal 9 (2) (2016) 211–221.
  * [52] E. Gutkin, S. A. Hussain, S. H. Kim, The successful treatment of chronic cholecystitis with spyglass cholangioscopy-assisted gallbladder drainage and irrigation through self-expandable metal stents, Gut and liver 6 (1) (2012) 136\.
  * [53] T. Fukui, T. Chochi, T. Maeda, C. Lee, Y. Wada, M. Ohashi, J. Tashiro, M. Arai, M. Kurata, T. Yoshida, et al., Biliary peritonitis caused by spontaneous bile duct rupture in the left triangular ligament of the liver after endoscopic sphincterotomy for choledocholithiasis, Case Reports in Gastroenterology 15 (1) (2021) 53–61.
  * [54] M. Wang, Y. Xing, Q. Gao, Z. Lv, J. Yuan, Mirizzi syndrome with an unusual aberrant hepatic duct fistula: a case report, International Medical Case Reports Journal (2016) 173–177.
  * [55] A. Sweeny, N. A. Smith, J. A. Serfin, Hemorrhagic cholecystitis causing hemobilia and common bile duct obstruction, Journal of Surgical Case Reports 2019 (3) (2019) rjz081.
  * [56] K. Roberts, D. Demner-Fushman, E. M. Voorhees, S. Bedrick, W. R. Hersh, Overview of the TREC 2020 precision medicine track 1266 (2020).
URL https://trec.nist.gov/pubs/trec29/papers/OVERVIEW.PM.pdf

  * [57] C. N. Myrdal, S. Sundararajan, Response to ipilimumab/nivolumab rechallenge and braf inhibitor/mek inhibitor rechallenge in a patient with advanced metastatic melanoma previously treated with braf targeted therapy and immunotherapy, Case Reports in Oncological Medicine 2020 (2020).
  * [58] D. J. Martini, A.-K. A. Lalani, D. Bossé, J. A. Steinharter, L. C. Harshman, F. S. Hodi, P. A. Ott, T. K. Choueiri, Response to single agent pd-1 inhibitor after progression on previous pd-1/pd-l1 inhibitors: a case series, Journal for ImmunoTherapy of Cancer 5 (1) (2017) 1–5.
  * [59] Y. Tay, M. Dehghani, D. Bahri, D. Metzler, Efficient transformers: A survey, arXiv preprint arXiv:2009.06732 (2020).
  * [60] M. Zaheer, G. Guruganesh, K. A. Dubey, J. Ainslie, C. Alberti, S. Ontanon, P. Pham, A. Ravula, Q. Wang, L. Yang, et al., Big bird: Transformers for longer sequences, Advances in Neural Information Processing Systems 33 (2020) 17283–17297.
  * [61] I. Beltagy, M. E. Peters, A. Cohan, Longformer: The long-document transformer, arXiv preprint arXiv:2004.05150 (2020).
  * [62] R. Nogueira, W. Yang, K. Cho, J. J. Lin, Multi-stage document ranking with bert, ArXiv abs/1910.14424 (2019).
  * [63] S. Humeau, K. Shuster, M.-A. Lachaux, J. Weston, Poly-encoders: Architectures and pre-training strategies for fast and accurate multi-sentence scoring, in: International Conference on Learning Representations, 2020.
  * [64] O. Khattab, M. A. Zaharia, Colbert: Efficient and effective passage search via contextualized late interaction over bert, Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (2020).
  * [65] X. Chen, K. Lakhotia, B. Oğuz, A. Gupta, P. Lewis, S. Peshterliev, Y. Mehdad, S. Gupta, W. tau Yih, Salient phrase aware dense retrieval: Can a dense retriever imitate a sparse one?, in: Conference on Empirical Methods in Natural Language Processing, 2021.
  * [66] S.-C. Lin, J. J. Lin, Densifying sparse representations for passage retrieval by representational slicing, ArXiv abs/2112.04666 (2021).
  * [67] G. Frisoni, M. Mizutani, G. Moro, L. Valgimigli, Bioreader: a retrieval-enhanced text-to-text transformer for biomedical literature, in: Conference on Empirical Methods in Natural Language Processing, 2022.
  * [68] A. Naik, S. Parasa, S. Feldman, L. L. Wang, T. Hope, Literature-augmented clinical outcome prediction, ArXiv abs/2111.08374 (2021).
  * [69] K. Guu, K. Lee, Z. Tung, P. Pasupat, M.-W. Chang, Realm: Retrieval-augmented language model pre-training, ArXiv abs/2002.08909 (2020).
  * [70] D. S. Sachan, S. Reddy, W. Hamilton, C. Dyer, D. Yogatama, End-to-end training of multi-document reader and retriever for open-domain question answering, ArXiv abs/2106.05346 (2021).
  * [71] G. Izacard, P. Lewis, M. Lomeli, L. Hosseini, F. Petroni, T. Schick, J. A. Yu, A. Joulin, S. Riedel, E. Grave, Few-shot learning with retrieval augmented language models, ArXiv abs/2208.03299 (2022).
  * [72] S. Ge, C. Xiong, C. Rosset, A. Overwijk, J. Han, P. Bennett, Augmenting zero-shot dense retrievers with plug-in mixture-of-memories, ArXiv abs/2302.03754 (2023).
  * [73] L. Yunxiang, L. Zihan, Z. Kai, D. Ruilong, Z. You, Chatdoctor: A medical chat model fine-tuned on llama model using medical domain knowledge, 2023.