# BEHRT: Transformer for Electronic Health Records

Yikuan Li
Deep Medicine
Oxford Martin School
University of Oxford
Oxford, UK
<EMAIL_ADDRESS>
&Shishir Rao
Deep Medicine
Oxford Martin School
University of Oxford
Oxford, UK
<EMAIL_ADDRESS>
Jose Roberto Ayala Solares
Deep Medicine
Oxford Martin School
University of Oxford
&Abdelaali Hassaine
Deep Medicine
Oxford Martin School
University of Oxford
&Dexter Canoy
Deep Medicine
Oxford Martin School
University of Oxford
&Yajie Zhu
Deep Medicine
Oxford Martin School
University of Oxford
&Kazem Rahimi
Deep Medicine
Oxford Martin School
University of Oxford
&Gholamreza Salimi-Khorshidi
Deep Medicine
Oxford Martin School
University of Oxford
Shishir Rao and Yikuan Li have equally contributed to this work as first
authors. Shishir Rao is Corresponding Author for this work.

###### Abstract

Today, despite decades of developments in medicine and the growing interest in
precision healthcare, vast majority of diagnoses happen once patients begin to
show noticeable signs of illness. Early indication and detection of diseases,
however, can provide patients and carers with the chance of early
intervention, better disease management, and efficient allocation of
healthcare resources. The latest developments in machine learning (more
specifically, deep learning) provides a great opportunity to address this
unmet need. In this study, we introduce BEHRT: A deep neural sequence
transduction model for EHR (electronic health records), capable of multitask
prediction and disease trajectory mapping. When trained and evaluated on the
data from nearly 1.6 million individuals, BEHRT shows a striking absolute
improvement of 8.0-10.8%, in terms of Average Precision Score, compared to the
existing state-of-the-art deep EHR models (in terms of average precision, when
predicting for the onset of 301 conditions). In addition to its superior
prediction power, BEHRT provides a personalised view of disease trajectories
through its attention mechanism; its flexible architecture enables it to
incorporate multiple heterogeneous concepts (e.g., diagnosis, medication,
measurements, and more) to improve the accuracy of its predictions; and its
(pre-)training results in disease and patient representations that can help us
get a step closer to interpretable predictions.

##  1 Introduction

The field of precision healthcare aims to improve the provision of care
through precise and personalised prediction, prevention, and intervention.

In recent years, advances in deep learning (DL) - a subfield of machine
learning (ML) - has led to great progress towards personalised predictions in
cardiovascular medicine, radiology, neurology, dermatology, ophthalmology, and
pathology, just to name a few. For instance, [1] introduced a DL model that
can predict the risk of lung cancer from a patient’s tomography images with a
striking 94.4% accuracy; [2] showed that DL can predict a range of
cardiovascular risk factors from just a retinal fundus photograph; and the
list continues (more examples can be found in [3] and [4]). A key contributing
factor to this success, in addition to the developments in DL algorithms, was
the massive influx of large multimodal biomedical data, including but not
limited to, mega cohorts such as UK Biobank [5], and routinely-collected
health data such as electronic health records (EHR) [6].

In the recent years, the adoption of EHR systems has greatly increased;
percent of hospitals in the US and UK that have adopted EHR systems now
exceeds 84% and 94%, respectively [7, 8]. As a result, EHR systems of a
national (and/or a large) medical organisation now are likely to capture data
from millions of individuals over many years (or decades, sometimes). Each
individual’s EHR can link data from many sources (e.g., doctor visits and
hospital episodes) and hence contain “concepts” such as diagnoses,
interventions, lab tests, clinical narratives, and more. Each instance of a
concept can mean a single or multiple data points; just a single
hospitalisation alone, for instance, can generate thousands of data points for
an individual, whereas a diagnosis can be a single data point (i.e., an ICD
code). This makes large-scale EHR a uniquely rich source of insight and an
unrivalled data for training data-hungry ML models.

In traditional research on EHR (including the ones using ML), individuals are
represented to models as a vector of attributes, or "features" [9]. This
approach relies on experts’ ability to define the appropriate features, and
design the model’s structure (i.e., answering questions such as “what are the
key features for this prediction?” or, “which features should have
interactions with one another?”). Recent developments in deep learning,
however, provided us with models that can learn useful representations (e.g.,
of individuals, concepts, or an entire record) from raw or minimally-processed
data, with minimal need for expert guidance. This happens through a sequence
of layers, each employing a large number of simple linear and nonlinear
operations to map their corresponding inputs to a representation; the progress
from layer to layer, is expected to result in a final representation in which
the data points form distinguishable patterns.

As one of the earliest works on applying deep learning to EHR, Liang et al
[10] showed that deep neural networks can outperform SVM and decision tree
paired with manual feature engineering, over a number prediction tasks on a
number of different datasets. In another early work in this space, Tran et al
[11] proposed the use of restricted Boltzmann machines (RBM) for learning a
distributed representation of EHR, which was shown to outperform the manual
feature extraction, when predicting the risk of suicide from individuals’ EHR.
In a similar approach, Miotto et al [12] employed a stack of denoising
autoencoders (SDA) instead of RBM, and showed that it outperforms many popular
feature extraction and feature transformation approaches (e.g., PCA, ICA and
Gaussian mixture models) for providing classifiers with useful features to
predict the onset of a number of diseases from EHR.

These early works on the application of DL to EHR did not take into account
the subtleties of EHR data (e.g., the irregularity of the inter-visit
intervals, and the temporal order or events, to name a few). In attempt to
address this, Nguyen et al [13] introduced a convolutional neural network
(CNN) model called Deepr (Deep record) for predicting the probability of
readmission; they treated one’s medical history as a sequence of concepts
(e.g., diagnosis and medication) and inserted a special word between each pair
of consecutive visits to denote the time difference between them. In another
similar attempt, Choi et al. [14] introduced a shallow recurrent neural
network (RNN) model to predict the diagnoses and medications that are likely
to occur in the subsequent visit. Both these works employed some form of
embedding to map the non-numeric medical concepts to an algebraic space in
which the sequence models can operate.

One of the improvements that was next introduced to the DL models of EHR aimed
to enable them to capture the long-term dependencies among events (e.g., key
diagnoses such as diabetes can stay a risk factor over a person’s life, even
decades after their first occurrence; certain surgeries may prohibit certain
future interventions). Pham et al [15] introduced a Long Short-Term Memory
(LSTM) architecture with attention mechanism, called DeepCare, which
outperformed standard ML techniques, LSTM, and plain RNN in tasks such as
prediction of the onset of diabetes. In a similar development, Choi et al [16]
proposed a model based on reverse-time attention mechanism to consolidate past
influential visits using an end-to-end RNN model named RETAIN for the
prediction of heart failure. RETAIN outperformed most of the models at the
time of its publication and provided a decent baseline for the medical deep
learning research [17].

In this study, given the success of deep sequence models and attention
mechanisms in the past DL research for EHR, we aim to build on some of the
latest developments in deep learning and natural language processing (NLP) –
more specifically, Transformer architecture [18] – while taking into account
various EHR-specific challenges, and provide improved accuracy for the
prediction of future diagnoses. We named our model BEHRT (i.e., BERT for EHR),
due to architectural similarities that it has with (and our original
inspirations that came from) BERT [18]; one of the most powerful Transformer-
based architectures in NLP.

##  2 Materials and Methods

In this section, after providing an introduction to our EHR data, we will
introduce the earlier works that inspired BEHRT, as well as the novel features
that this new architecture contributes to the field.

###  2.1 Data

In this study, we used CPRD (Clinical Practice Research Datalink) [19, 17]; it
contains longitudinal primary care data from a network of 674 general
physician (GP) practices in the UK, which is linked to secondary care (i.e.,
hospital episode statistics, or HES) and other health and administrative
databases (e.g., office for national statistics’ death registration). Around 1
in 10 GP practices (and nearly 7% of the population) in the UK contribute data
to CPRD; it covers 35 million patients, among whom nearly 10 million are
currently registered patients [19]. CPRD is broadly representative of the
population by age, sex, and ethnicity. It has been extensively validated and
is considered as the most comprehensive longitudinal primary care database
[20], with several large-scale epidemiological reports [21, 22, 19] adding to
its credibility.

HES, on the other hand, contains data on hospitalisations, outpatient visits,
accident and emergency for all admissions to National Health Service (NHS)
hospitals in England [23]. Approximately 75% of the CPRD GP practices in
England (58% of all UK CPRD GP practices) participate in patient-level record
linkage with HES, which is performed by the Health and Social Care Information
Centre [24]. In this study, we only considered the data from GP practices that
consented to (and hence have) record linkage with HES. The importance of
primary care at the centre of the national health system in the UK, the
additional linkages, and all the aforementioned properties, make CPRD one of
the most suitable EHR datasets in the world for data-driven clinical/medical
discovery and machine learning.

###  2.2 Preprocessing of CPRD

We start our preprocessing with 8 million patients; we only included patients
that are eligible for linkage to HES and meet CPRD’s quality standard (i.e.,
using the flags and time windows that CPRD provides to indicate the quality of
one’s EHR). Furthermore, to only keep the records that have enough history to
be useful for prediction, we only kept individuals who have at least 5 visits
in their EHR. At the end of this process, we are left with $P=1.6$ million
patients to train and evaluate BEHRT on. More details on the steps we took and
the number of patients after each one of them can be seen in Fig 1.

Figure 1: Linkage and filtering of CPRD data. This flow lists all the key
steps of our data cleaning and linkage procedure. At each step, the number of
patients that are included is shown.

In CPRD, diseases are classified using Med Code (which can be simply mapped to
NHS’s standard Read Code [25]) and ICD-10 [26] schemes, for primary and
hospital care, respectively. In the ICD-10 universe, one can define diseases
at the desired level of granularity that is appropriate for the analysis of
interest by simply choosing the level of hierarchy they want to operate at;
for instance, operating ICD-10 chapter level will lead to 22 diseases, while
operation at full ICD-10 code level will lead to 10,000 diseases. With Med
Code, however, such a hierarchy does not exist and hence one needs to carry
out exhaustive disease review, in order to define the diseases of interest for
a given analysis. To alleviate this extra data-processing burden, we first
mapped Med Codes to Read Codes (using the mapping provided by CPRD) and
excluded the procedure codes. After that, we mapped both ICD-10 codes (at
level 4) and Read Codes to Caliber codes [27], which is an expert checked
mapping dictionary from University College London. Eventually, this resulted
in a total of $G=301$ codes for diagnoses. We denote the list of all these
diseases as $D=\\{d_{i}\\}_{i=1}^{G}$, where $d_{i}$ denotes the $i$th disease
code.

For each patient $p\in\\{1,2,\ldots,P\\}$ the medical history consists of a
sequence of visits to GP and hospitals; each visit can contain concepts such
as diagnosis, medications, measurements and more. In this study, however, we
are only considering the diagnoses; we denote each patient’s EHR as
$V_{p}=\\{\mathbf{v}_{p}^{1},\mathbf{v}_{p}^{2},\mathbf{v}_{p}^{3},\ldots,\mathbf{v}_{p}^{n_{p}}\\}$,
where $n_{p}$ denotes the number of visits in patient $p$’s EHR, and
$v_{p}^{j}$ contains the diagnoses in the $j$th visit, which can be viewed as
a list of $m_{p}^{j}$ diagnoses (i.e.,
$\mathbf{v}_{p}^{j}=\\{d_{1},\ldots,d_{m_{p}^{j}}\\}$). In order to prepare
the data for BEHRT, we order the visits (hence diseases) temporally, and
introduce a term to denote the start of medical history (i.e., $CLS$), and the
space between visits (i.e., $SEP$), which results in a new sequence,
$V_{p}=\\{CLS,\mathbf{v}_{p}^{1},SEP,\mathbf{v}_{p}^{2},SEP,\ldots,\mathbf{v}_{p}^{n_{p}},SEP\\}$,
that from now on will be how we see/denote each patient’s EHR as. This process
is illustrated in Figure 2.

Figure 2: Preparation of CPRD data for BEHRT. An example patient’s EHR
sequence can be seen in (a), which consists of 8 visits. In each visit, the
record can consist of concepts such as diagnoses, medications and
measurements; all these values are artificial and for illustration purposes.
For this study, we are only interested in age and diagnoses. Therefore, as
shown in at the bottom of the figure, we have taken only the diagnosis and age
subset of the record to form the necessary sequences. This resulting sequence
is what we represent every patient’s EHR as to our modelling process. Note
that, the visits shown in purple boxes are not going to be represented to the
model, due to them lacking diagnoses.

###  2.3 BEHRT: A Transformer-based Model for EHR

In this study, we aim to use a given patient’s past EHR to predict his/her
future diagnoses (if any), as a multi-task classification problem; this will
result in a single predictive model that scales across a range of diseases (as
opposed to needing to train one predictive model per disease). Modelling of
EHR sequences requires dealing with four key challenges [15]: (C.1) complex
and nonlinear interactions among past, present and future concepts; (C.2)
long-term dependencies among concepts (e.g., diseases occurring early in the
history of a patient effecting events far later in future); (C.3) difficulties
of representing multiple heterogeneous concepts of variable sizes and forms to
the model; and (C.4) the irregular intervals between consecutive visits.

Similarities between sequences in EHR and natural language led to successful
use of techniques such as BoW [12], Skip-gram [14], RNN [16], and attention
[15, 28] (a la their NLP usage) for learning complex EHR representations in
the past. In this study, we get our inspiration from the striking success that
Transformers [29], and more specifically, a Transformer-based architecture
known as BERT [18]. By depicting diagnoses as words, the content of in each
visit as a sentence, and a patient’s entire medical history as a document, we
facilitate the the use of multi-headed self-attention, positional encoding,
and masked language models (MLM), for EHR - under a new architecture we call
BEHRT.

We refer readers to the original papers [29, 18] for an exhaustive background
description for both Transformer and BERT. Figure 3B illustrates BEHRT’s
architecture, which is designed to pre-train deep bidirectional
representations of medical concepts by jointly conditioning on both left and
right contexts in all layers. The pre-trained representations can be simply
employed for a wide range of downstream tasks, e.g., prediction of the next
diseases, and disease phenomapping. Such bidirectional contextual awareness of
BEHRT’s representations is a big advantage when dealing with EHR, where due to
variabilities in practice of care and/or simply due to random events, the
order at which certain diseases happen can be reversed, or the time interval
between two diagnoses can be shorter or longer than actually recorded.

Figure 3: BEHRT architecture. Using the artificial data shown in Figure 2, (a)
shows how BEHRT sees one’s EHR. In addition to diagnosis and age, BEHRT also
employs an encoding for event’s positions (shown as POSITION) to help it
understand each event’s place in the sequence, and a visit separator (shown as
SEGMENT) to help it understand the start and end of a visit. The combination
of all these results is a final embedding shown at the bottom of (a), which
will be the latent contextual representation of one’s EHR at a given visit.
Using this pre-trained embedding, BEHRT’s deep Transformer-based model – shown
in (b) – learns to predict the diseases in one’s future.

BEHRT, has many structural advantages over many of the previous methods for
modelling EHR data. Firstly, we use feedforward neural networks to model time
sensitive EHR data by examining the various forms of sequential order within
the data (e.g. age, order of visits) instead of using traditional state-of-
the-art RNN and CNN that were explored in the past [13, 16]. Recurrent neural
networks are known to be notoriously hard to train, due to their exploding and
vanishing gradient problems [30]; these issues hamper these networks’ ability
to learn (particularly, when dealing with long sequential data). On the other
hand, convolutional neural networks only capture limited amount of information
with convolutional kernels in the lower layers, and need to expand their
receptive field though increasing the number of layers in a hierarchical
architecture. BEHRT’s feedforward structure alleviates the exploding and
vanishing gradient problems and capture information by considering the full
sequence at the same time; a more efficient training through learning the data
in parallel rather than in sequence (unlike the RNN).

The embedding layer in BEHRT, as shown in Figure 3, learns the evolution of
one’s EHR through a combination of four embeddings: disease, "position", age,
and "segment". This combination enables BEHRT to define a representation that
can capture one’s EHR in as much detail as possible. Disease codes are of
course important in informing the model of the future state of one’s health.
That is, there many common disease trajectories and multi-morbidity patterns
[31] that knowing one’s past diseases can improve the accuracy of the
prediction. Positional encodings are either trainable or pre-determined
encodings for a position to determine relative position of words in a sequence
of words. Pre-determined encodings are used in this paper to avoid weak
learning of positional embedding caused by imbalanced distribution of medical
sequence length. Telling the network of a disease’s position, enables it to
capture the positional interactions among diseases. Given the feed-forward
architecture of our network, positional encodings plays a key role in filling
the gap resulting from the lack of a recurrent structure that was the most
common/successful approach for learning from sequences. For BEHRT, we followed
the same position encoding rule proposed by [29].

Age and segment are two embeddings that did not exist in the original BERT
implementation for NLP and is unique to BEHRT; an attempt to empower it for
dealing with the challenges we mentioned earlier. The segment embedding can be
either A or B; the purpose of this is to use two trainable vectors to provide
extra information for visit separation. Age, of course, is known to be a key
risk factor in epidemiology. By embedding age and linking it to each
visit/diagnosis, not only we provide the model with a sense of time (i.e., the
time between events, as well as a universal notion of time for when things
happened that is comparable across patients).

Through a unique combination of the four aforementioned embeddings, we not
only provide the model with disease sequences, but also give it with a precise
sense of timing of events, and data about the delivery of care. In other
words, the model has the ability to learn from the diagnosis history, the age
at which diagnoses happened and the pattern at which the patient was visited.
All these, when combined, can provide a picture of one’s health that
traditionally we might have sought to paint through additional features
extracted from EHR data. Of course, we do not advocate for not using the full
richness of the EHR, however, the complexity of our architecture when paired
with simple subset of EHR can still provide an accurate prediction of one’s
future health. Plus, BEHRT’s flexible architecture enables the use of
additional concepts, e.g., by simply adding a fifth or sixth (or more)
embedding to the existing four.

###  2.4 Pre-training BEHRT using MLM

In EHR, just like language, it is intuitively reasonable to believe that a
deep bidirectional model is more powerful than either a left-to-right model or
the shallow concatenation of a left-to-right and a right-to-left model.
Therefore, we pre-trained BEHRT using the same approach as the original BERT
paper [18], using MLM. That is, we chose 15% of the disease words at random,
and modified them according to the following probabilities:

  1. 1.

80% of the times replace them word with [MASK]

  2. 2.

10% of the times replace them with a random disease word

  3. 3.

10% of the times do nothing and keep them unchanged.

Under this setting, BEHRT does not know which of the disease words are masked,
so it stores a contextual representation of all of the disease words.
Additionally, given the small prevalence of change (i.e., only for 15% of all
disease words) will not hamper the model’s ability to understand the EHR
language. Lastly, the replacement of the disease words acts as injected noise
into the model; it will distract the model from learning the true left and
right context, and instead forces the model to fight through the noise and
continue learning the overall disease trajectories. The pre-training MLM task
was evaluated using precision score [32], which calculates the ratio of true
positive over the number of predicted positive samples (precision calculated
at a threshold of 0.5). The average is calculated over all labels and over all
patients.

###  2.5 Disease Prediction

In order to provide a comprehensive evaluation of BEHRT, we assess its
learning for three predictive tasks: prediction of concepts in the next visit
(T1), prediction of the occurrence of diseases in the next 6 months (T2), and
prediction of the occurrence of diseases in the next 12 months (T3). In order
to train our model and assess the goodness of its predictions across these
tasks, we first randomly allocated the patients into three groups of train,
validation and test (each containing 80%, 10% and 10%, or patients,
respectively). To define the training examples (i.e., input-output pairs) for
T1, we randomly choose an index $j$ ($3<j<n_{p}$) for each patient and form
$\mathbf{x}_{p}=\\{\mathbf{v}_{p}^{1},\ldots,\mathbf{v}_{p}^{j}\\}$ and
$\mathbf{y}_{p}=\mathbf{w}_{j+1}$, as input and output, respectively, where
$\mathbf{w}_{j+1}$ is a multi-hot vector of length $G$, with 1 for diseases
that exist in $\mathbf{v}_{p}^{j+1}$. Note that, for each patient, we have
only input-output pair.

For both T2 and T3, the formation of input and label are slightly modified.
First, patients that do not have 6 or 12 months (for T2 and T3, respectively)
worth of EHR (with or without a visit) after $\mathbf{v}_{p}^{4}$ will not be
included in these analyses. Second, $j$ is chosen randomly from $(3,n_{*})$,
where $n_{*}$ denotes the highest index after which there is 6 or 12 months
(for T2 and T3, respectively) worth of EHR (with or without a visit). Lastly,
$\mathbf{y}_{p}=\mathbf{w}_{6m}$ and $\mathbf{y}_{p}=\mathbf{w}_{12m}$ are
multi-hot vectors of length $G$, with 1 for concepts/diseases that exist in
the next 6 and 12 months, respectively. As a result of this final filtering of
patients, we had 699K, 391K, and 342K patients for T1, T2, and T3,
respectively.

We denote the model’s prediction for patient $p$ in the aforementioned tasks
as $\mathbf{y}_{p}^{*}$, where the $i$th entry is the model’s prediction of
that person having $d_{i}$. The evaluation metrics we used to compare
$\mathbf{y}$s and $\mathbf{y}^{*}$s, are area under the ROC curve (AUROC) [33]
and average precision score (APS) [34]; the latter is a weighted mean of
precision and recall numbers achieved at different thresholds. We calculated
the APS and AUROC for each patient first, and then averaged the resulting APS
and AUROC scores across all patients; this average is the key metric, when
comparing BEHRT with other state-of-the-art architectures in the field. The
methods we used here for APS and AUROC are described in [33, 34].

##  3 Results

To begin with, we used Bayesian Optimisation [35] to find the optimal
hyperparameters for the MLM pre-training. The main hyperparameters here are
the number of layers, the number of attention heads, hidden size, and
intermediate size. Intermediate size is the size of the neural network layer
titled "intermediate layer". This process resulted in an optimal architecture
with 6 layers, 12 attention heads, intermediate layer size of 512, and hidden
size of 288; model’s performance in the MLM task described in Section 2.4 was
0.6597 in precision score. Further details can be found in Appendix A.

In order for BEHRT’s numerical processes to be applicable to EHR, we first
need to map the non-numeric concepts such as diagnoses to a vector space
(i.e., disease embedding). Therefore, we start the results by showing the
performance of our pre-training process in embedding the diseases, where we
mapped each of the $G$ diseases into a 288-dimensional vector. Note that, for
evaluating an embedding technique – even in NLP where the literature has a
longer history and hence is more mature – there is not a single gold standard
metric [36]. In this study, our assessment is based on two techniques: visual
investigation (i.e., in comparison with medical knowledge), and evaluation in
a prediction task. For the former, we used t-SNE [37] to reduce the
dimensionality of the disease vectors to two – results are shown in Figure 4.
Based on the resulting patterns in lower dimension, we can see that diseases
that are known to co-occur and/or belong to the same clinical groups, are
grouped together.

Figure 4: Disease Embedding Analysis. In this image we see a graph of disease
embeddings projected in two dimensions where distance represents closeness of
contextual association. Most associations are accepted by medical experts and
maintain the gender-based divisions in illnesses, among other things. We zoom
in and profile four clusters in this plot – shown in subfigures A-D.

A reassuring pattern that can be seen in Figure 4 is the natural
stratification of gender-specific diseases. For instance, diseases that are
unique to women (e.g., Endometriosis, Dysmenorrhea, Menorrhagia, …) are quite
distant from those that are unique to men (e.g., Erectile Dysfunction, Primary
Malignancy of Prostate, …). Such patterns seem to suggest that our disease
embedding built an understanding of the context in which diagnoses happen, and
hence infer factors such as gender that it is not explicitly fed.

Furthermore, the colour in Figure 4 represent the original Caliber disease
chapters (see the legends in the main subplot). As can be seen, natural
clusters are formed that in most cases consist of disease of the same chapter
(i.e., the same colour). Some of these clusters, however, are correlated but
not identical to these chapters; for instance, many Eye and Adnexa diseases
are amongst nervous system diseases and many nervous system disease are also
among many musculoskeletal diseases. Overall, this map can be seen as
diseases’ correspondence to each other based on 1.6 million people’s EHR.
Overall, it seems to be safe to say that this embedding seems to make sense
and hence it passes the visual evaluation test.

Another interesting property of BEHRT is its self-attention mechanism; it
gives our model the ability to find the relationships among events that go
beyond temporal/sequence adjacency. This self-attention mechanism is able to
unearth deeper and more complex relationships between a disease in one visit
and other surrounding diagnoses. We visualise this self-attention using the
approach introduced in [38]; we plot each medical history against itself, and
we see how each disease relates to others around it in each patient. The
results from a couple of example patients are shown in Figure 5. Note that,
since BEHRT is bidirectional, the self-attention mechanism captures non-
temporal/non-directional relationships among diseases.

Figure 5: Disease Self-Attention Analysis. This figure shows the EHR history
(shown chronologically, going downwards) of two patients A and B, each
presented as two identical columns for the convenience of association
analyses. The left side of the column represents the disease of interest and
the right column indicates the corresponding associations to the highlighted
disease on the left. The intensity of the blue on the right column represents
the strength of the attention score – the stronger the intensity, the stronger
the association and hence the stronger the attention score. The attention
scores are specifically retrieved from the attention component of the last
layer of BEHRT network.

For patient A for example, the self-attention mechanism has shown strong
connections between Rheumatoid Arthritis and Enthesopathies and synovial
disorders (far in the future of the patient). This is a great example of where
attention can go beyond recent events and find long-range dependencies among
diseases. Note that, as described earlier and illustrated in Figure 3, the
sequence we model is a combination of four embedding (disease, plus age,
segment and position) that go through layers of transformations to form a
latent space abstraction. While in Figure 5 we labeled the cells with disease
names, a more precise labelling will be diseases in their context (e.g., at a
given age).

BEHRT after the MLM pre-training can be considered a universal EHR feature
extractor that with small additional training can be employed for a range of
downstream tasks. In this work, the downstream task of choice is the multi-
disease prediction problem that we described in Section 2.5. To train a
predictor, we feed the output from BEHRT to a single feed-forward classifier
layer and train it three separate time for each of the three tasks (T1-T3)
described in Section 2.5. The evaluation of the model’s performance is shown
in Table 1, which demonstrates BEHRT’s superior predictive power compared to
two of the most successful approaches in the literature (i.e., RETAIN [16] and
DeepR [12]). We used Bayesian Optimisation to find the optimal hyperparameters
for RETAIN and Deepr before assessing them in terms of APS and AUROC scores.
More details on their hyperparameter search and optimisation can be found in
AppendixA.

Table 1: Next Visit Prediction Task Model Name |  Next Visit (APS | AUROC) |  Next 6M (APS | AUROC) |  Next 12M (APS | AUROC)
---|---|---|---
BEHRT |  0.462 | 0.954 |  0.525 | 0.958 |  0.506 | 0.955
DeepR |  0.360 | 0.942 |  0.393 | 0.943 |  0.393 | 0.943
RETAIN |  0.382 | 0.921 |  0.417 | 0.927 |  0.413 | 0.928

Besides comparing the APS, which provides an average view across all patients,
all diseases and all thresholds, we are also interested in analysing the
model’s performance for predicting for each disease. To do so for a given
disease $d_{i}$, we only considered the $i$th location in $\mathbf{y}_{p}$ and
$\mathbf{y}_{p}^{*}$ vectors and calculated AUROC and APS scores, as well as
occurrence ratio for comparison. The results for T2 (or, next 6-months
prediction task) is shown in Figure 6. For visual convenience, we did not
include rare diseases with prevalence of less than 1% in our data. The result
shows that the model is able to make predictions with relatively high
precision and recall for diseases such as Epilepsy (0.016), Primary Malignancy
Prostate (0.011), Polymyalgia Rheumatica (0.013), Hypo or hyperthyroidism
(0.047) and Depression (0.0768). A numerical summary of this analysis can be
found in be found in Appendix B. Furthermore, a comparison of the general
APS/AUROC trends across the three models - BEHRT, RETAIN, and Deepr can be
found in Appendix C.

Figure 6: Disease-wise precision analysis. Each circle in these graphs
represents a disease, who and color and size are their caliber chapter and
prevalence, respectively. Also, in these plots, we show APS and AUROC on the
x- and y-axis, respectively. Therefore, the further right and higher a
disease, the better BEHRT’s job at predicting its occurrence in the next 6
months. Subplot A, illustrated the full results, and subplots B and C
illustrate the best and worth sections of the plot, in terms of BEHRT’s
performance.

##  4 Conclusions and Future Works

In this paper, we introduced a novel deep neural model for EHR called BEHRT,
which can be pre-trained on a large dataset and then with small fine tuning
result in a striking performance in a wide range of downstream tasks. We
demonstrated this property of the model by training and testing it on CPRD -
one of the largest linked primary care EHR systems. Based on our results,
BEHRT outperformed all the best deep EHR models in the literature over a range
of diseases (i.e., in a multi-label diagnoses prediction in near future), by
$\sim$8% (absolute improvement) in any given task.

BEHRT offers a flexible architecture that is capable of capturing more
modalities of EHR data. In this paper, we designed and tested a BEHRT model
that relied on 4 key embeddings - diseases, age, segment and position. Through
this mix, the model will not only have the ability to learn about the past
diseases and their relationships with one another (and hence their effect on
the next likely disease), but also gain insights about the underlying
generating process of EHR; we can refer to this as practice of care. In other
words, the model will engineer features (i.e., complex representations) that
are capable of capturing concepts such as "this patient had diseases X and Y
at young ages, and suddenly, the frequency of visits increased and new
diagnoses appeared, which can lead to high chance of the next disease being
Z". In future works, one can add more to the four concepts we employed and
bring medication, tests and interventions to the model with minimum
architectural changes - only a vector addition in Figure 3.

Our primary objective in this study was to provide the field with an accurate
predictive models for the prediction of next diseases. However, BEHRT provides
multiple byproducts that can be useful on their own and/or as the foundation
preprocessing for the future works. For instance, the disease embeddings
resulting from BEHRT can provide a great insight to how various diseases are
related to each other - it goes beyond simple disease co-occurrence and rather
learns to score closeness of diseases based on their trajectories in a big
population of patients. Furthermore, the disease correspondence that results
from BEHRT’s attention mechanism has been shown to be a useful tool for
illustrating the disease trajectories for multi-morbid patients; not only it
shows how diseases co-occur, but also it shows the influence of certain
disease in the past on future diseases of interest. These correspondences are
not strictly temporal but rather contextual. As a future work, we aim to
provide these attention-visualisation tools to medical researchers to help
them better understand the contextual meaning of a diagnoses in the midst of
other diagnoses of patients. Through this tool, medical researchers can even
craft medical history timelines based on certain diseases or patterns and in a
way, query our BEHRT model and visualiser to perhaps uncover novel disease
contexts.

For the future, we wish to make improvements to our model and use ensembles of
BEHRTs and variations for better predictive power. Furthermore, we also plan
to bring in more medical features such as treatment records and more
demographic information (region, ethnicity, etc). Also, as discussed before,
since Caliber consolidates specific codes into 301 disease codes and fails to
map many other codes, we wish to look into more stable mappings that preserve
the comprehensiveness of CPRD and reduce noise in our dataset. In addition to
this, we wish to also embark on a deep dive of a single disease, perhaps heart
failure or hypertension, to use BEHRT’s diagnostic power for specific disease
prediction.

##  5 Acknowledgements

This research was funded by the Oxford Martin School (OMS) and supported by
the National Institute for Health Research (NIHR) Oxford Biomedical Research
Centre (BRC). The views expressed are those of the authors and not necessarily
those of the OMS, the UK National Health Service (NHS), the NIHR or the
Department of Health and Social Care. This work uses data provided by patients
and collected by the NHS as part of their care and support and would not have
been possible without access to this data. The NIHR recognises and values the
role of patient data, securely accessed and stored, both in underpinning and
leading to improvements in research and care. We also thank Wayne Dorrington
for his work in creating figures for this paper (Figure 2 and Figure 3).

## References

  * [1] Diego Ardila, Atilla P Kiraly, Sujeeth Bharadwaj, Bokyung Choi, Joshua J Reicher, Lily Peng, Daniel Tse, Mozziyar Etemadi, Wenxing Ye, Greg Corrado, and David P Naidich.  End-to-end lung cancer screening with three-dimensional deep learning on low-dose chest computed tomography.  Nature Medicine, 25(June), 2019.
  * [2] Ryan Poplin, Avinash V Varadarajan, Katy Blumer, Yun Liu, Michael V McConnell, Greg S Corrado, Lily Peng, and Dale R Webster.  Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning.  Nature Biomedical Engineering, 2(3):158–164, 2018.
  * [3] Eric J Topol.  human and artificial intelligence.  Nature Medicine, 25(January), 2019.
  * [4] Andre Esteva, Alexandre Robicquet, Bharath Ramsundar, Volodymyr Kuleshov, Mark DePristo, Katherine Chou, Claire Cui, Greg Corrado, Sebastian Thrun, and Jeff Dean.  A guide to deep learning in healthcare.  Nature medicine, 25(1):24–29, 2019.
  * [5] Cathie Sudlow, John Gallacher, Naomi Allen, Valerie Beral, Paul Burton, John Danesh, Paul Downey, Paul Elliott, Jane Green, Martin Landray, Bette Liu, Paul Matthews, Giok Ong, Jill Pell, Alan Silman, Alan Young, Tim Sprosen, Tim Peakman, and Rory Collins.  UK Biobank: An Open Access Resource for Identifying the Causes of a Wide Range of Complex Diseases of Middle and Old Age.  PLOS Medicine, 12(3):e1001779, 2015.
  * [6] Benjamin Shickel, Patrick Tighe, Azra Bihorac, and Parisa Rashidi.  Deep EHR: A Survey of Recent Advances in Deep Learning Techniques for Electronic Health Record (EHR) Analysis.  IEEE Journal of Biomedical and Health Informatics, 22(5):1589–1604, 2018.
  * [7] O N C Annual Meeting.  Electronic Public Health Reporting.  None, 2018.  Available at: https://www.healthit.gov/sites/default/files/2018-12/ElectronicPublicHealthReporting.pdf.
  * [8] Sonal Parasrampuria and Jawanna Henry.  Hospitals’ Use of Electronic Health Records Data, 2015-2017.  ONC Data Brief, No. 46, 2019.
  * [9] Fatemeh Rahimian, Gholamreza Salimi-Khorshidi, Amir H Payberah, Jenny Tran, Roberto Ayala Solares, Francesca Raimondi, Milad Nazarzadeh, Dexter Canoy, and Kazem Rahimi.  Predicting the risk of emergency admission with machine learning: Development and validation using linked electronic health records.  PLoS Medicine, 15(11):1–18, 2018.
  * [10] Znaonui Liang, Gang Zhang, Jimmy Xiangji Huang, and Qmming Vivian Hu.  Deep learning for healthcare decision making with EMRs.  Proceedings - 2014 IEEE International Conference on Bioinformatics and Biomedicine, IEEE BIBM 2014, pages 556–559, 2014.
  * [11] Truyen Tran, Tu Dinh Nguyen, Dinh Phung, and Svetha Venkatesh.  Learning vector representation of medical objects via EMR-driven nonnegative restricted Boltzmann machines (eNRBM).  Journal of Biomedical Informatics, 2015.
  * [12] Riccardo Miotto, Li Li, Brian A Kidd, and Joel T Dudley.  Deep Patient: An Unsupervised Representation to Predict the Future of Patients from the Electronic Health Records.  Scientific Reports, 6(May):1–10, 2016.
  * [13] Phuoc Nguyen, Truyen Tran, Nilmini Wickramasinghe, and Svetha Venkatesh.  Deepr: A Convolutional Net for Medical Records.  IEEE Journal of Biomedical and Health Informatics, 21(1):22–30, may 2017.
  * [14] Edward Choi, Mohammad Taha Bahadori, Andy Schuetz, Walter F Stewart, and Jimeng Sun.  Doctor AI: Predicting Clinical Events via Recurrent Neural Networks.  JMLR workshop and conference proceedings, 56:301–318, 2016.
  * [15] Trang Pham, Truyen Tran, Dinh Phung, and Svetha Venkatesh.  DeepCare: A deep dynamic memory model for predictive medicine.  Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 9652 LNAI(i):30–41, 2016.
  * [16] Edward Choi, Mohammad Taha Bahadori, Joshua A. Kulas, Andy Schuetz, Walter F. Stewart, and Jimeng Sun.  RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism.  arxiv, 2016.
  * [17] Jose Roberto Ayala Solares, Francesca Elisa Diletta Raimondi, Yajie Zhu, Fatemeh Rahimian, Dexter Canoy, Jenny Tran, Ana Catarina Pinho Gomes, Amir Payberah, Mariagrazia Zottoli, Milad Nazarzadeh, Nathalie Conrad, Kazem Rahimi, and Gholamreza Salimi-Khorshidi.  Deep Learning for Electronic Health Records: A Comparative Review of Multiple Deep Neural Architectures.  preprint, page 57, 2019.
  * [18] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  arxiv, 2018.
  * [19] Emily Herrett, Arlene M Gallagher, Krishnan Bhaskaran, Harriet Forbes, Rohini Mathur, Tjeerd Van Staa, and Liam Smeeth.  Data Resource Profile: Clinical Practice Research Datalink (CPRD).  International Journal of Epidemiology, 44(3):827–836, 2015.
  * [20] T Walley and A Mantgani.  The uk general practice research database.  The Lancet, 350(9084):1097 – 1099, 1997.
  * [21] Connor A Emdin, Simon G Anderson, Thomas Callender, Nathalie Conrad, Gholamreza Salimi-Khorshidi, Hamid Mohseni, Mark Woodward, and Kazem Rahimi.  Usual blood pressure, peripheral arterial disease, and vascular risk: Cohort study of 4.2 million adults.  BMJ (Online), 2015.
  * [22] Connor A. Emdin, Simon G. Anderson, Gholamreza Salimi-Khorshidi, Mark Woodward, Stephen MacMahon, Terrence Dwyer, and Kazem Rahimi.  Usual blood pressure, atrial fibrillation and vascular risk: Evidence from 4.3 million adults.  International Journal of Epidemiology, 2017.
  * [23] F. Lee, H. R.S. Patel, and M. Emberton.  The ’top 10’ urological procedures: A study of hospital episodes statistics 1998-99.  BJU International, 2002.
  * [24] Hamid Mohseni, Amit Kiran, Reza Khorshidi, and Kazem Rahimi.  Influenza vaccination and risk of hospitalization in patients with heart failure: A self-controlled case series study.  European Heart Journal, 2017.
  * [25] NHS.  Read Codes.  Available at: https://digital.nhs.uk/services/terminology-and-classifications/read-codes.
  * [26] WHO.  ICD-10 online versions.  Available at https://icd.who.int/browse10/2016/e.
  * [27] Valerie Kuan, Spiros Denaxas, Arturo Gonzalez-izquierdo, Kenan Direk, Osman Bhatti, Shanaz Husain, Shailen Sutaria, Melanie Hingorani, Dorothea Nitsch, Constantinos A Parisinos, R Thomas Lumbers, Rohini Mathur, Reecha Sofat, Juan P Casas, Ian C K Wong, and Harry Hemingway.  Articles A chronological map of 308 physical and mental health conditions from 4 million individuals in the English National Health Service.  The Lancet Digital Health, 1(2):e63–e77, 2019.
  * [28] Kyunghyun Cho.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation.  arxiv, 2013.
  * [29] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention Is All You Need.  arXiv:1706.03762 [cs], apr 2017.
  * [30] Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio.  On the difficulty of training Recurrent Neural Networks.  arxiv, 2012.
  * [31] The Academy of Medical Sciences.  Multimorbidity: a priority for global health research.  The Academy of Medical Sciences, pages 1–127, 2018.
  * [32] David M W Powers.  Evaluation : From Precision , Recall and F-Factor to ROC , Informedness , Markedness & Correlation.  arxiv, 2007.
  * [33] Tom Fawcett.  An introduction to ROC analysis.  Pattern Recognition Letters, 2006.
  * [34] Mu Zhu.  Recall, precision and average precision.  Department of Statistics and Actuarial Science, …, 2004.
  * [35] Ryan Snoek, Jasper; Larochelle, Hugo; Adams.  Practical Bayesian Optimization of Machine Learning Algorithms.  NIPS, 2(12):e540, 2017.
  * [36] Bin Wang, Student Member, Angela Wang, Fenxiao Chen, Student Member, Yuncheng Wang, and C Jay Kuo.  Evaluating Word Embedding Models : Methods and Experimental Results.  arxiv, pages 1–13, 2019.
  * [37] Laurens Van Der Maaten and Geoffrey Hinton.  Visualizing Data using t-SNE.  JMLR, 9:2579–2605, 2008.
  * [38] Jesse Vig.  Visualizing Attention in Transformer-Based Language Representation Models.  arxiv, pages 2–7, 2019.

##  Appendix A Hyperparameter Tuning

We show the hyperparameter tuning results here in the following section. In
Table 2, we show the results of the MLM training hyperparameter tuning
process. We performed Bayesian Optimization to retrieve optimal parameters for
the model.
In the following sections, we also perform hyperparameter searches for the
Deepr (Table 3) and RETAIN (Table 4) models to ensure proper comparison of
model performance between BEHRT and the aforementioned models.

Table 2: MLM Hyperparameter Tuning Iteration |  Hidden Size |  Layers |  Attention Heads |  Intermediate Size |  Precision
---|---|---|---|---|---
1 |  216 |  3 |  6 |  256 |  0.6191
2 |  288 |  9 |  12 |  512 |  0.6399
3 |  216 |  3 |  12 |  512 |  0.6175
4 |  432 |  3 |  18 |  512 |  0.6397
5 |  288 |  6 |  6 |  784 |  0.6380
6 |  216 |  6 |  18 |  512 |  0.6262
7 |  288 |  3 |  18 |  512 |  0.6292
8 |  432 |  3 |  6 |  784 |  0.6426
9 |  288 |  6 |  12 |  512 |  0.6356
10 |  288 |  3 |  12 |  256 |  0.6283
11 |  432 |  9 |  18 |  512 |  0.6466
12 |  576 |  9 |  6 |  1024 |  0.6538
13 |  432 |  3 |  18 |  1024 |  0.6411
14 |  432 |  9 |  6 |  1024 |  0.6508
15 |  576 |  6 |  6 |  256 |  0.6503
16 |  576 |  6 |  12 |  256 |  0.6510
17 |  360 |  9 |  18 |  512 |  0.6404
18 |  576 |  9 |  6 |  512 |  0.6513
19 |  288 |  6 |  6 |  512 |  0.6363
20 |  288 |  3 |  6 |  512 |  0.6297
21 |  288 |  6 |  12 |  512 |  0.6597
22 |  576 |  3 |  12 |  512 |  0.6487
23 |  360 |  6 |  6 |  784 |  0.6412
24 |  432 |  9 |  6 |  512 |  0.6497
25 |  360 |  6 |  12 |  512 |  0.6423
Table 3: Deepr Best Model - Subsequent Visit Prediction Task Iteration |  Filters |  Kernel Size |  FC I |  FC II |  FC III |  Dropout I |  Dropout II |  Dropout III |  Learning Rate |  Average Precision
---|---|---|---|---|---|---|---|---|---|---
1 |  37 |  7 |  10 |  46 |  28 |  0.4139 |  0.4997 |  0.3718 |  0.0004 |  0.2599
2 |  24 |  5 |  28 |  19 |  13 |  0.4936 |  0.1722 |  0.4643 |  0.0062 |  0.2319
3 |  17 |  7 |  16 |  48 |  40 |  0.2250 |  0.3945 |  0.3264 |  0.0310 |  0.1815
4 |  33 |  7 |  6 |  10 |  25 |  0.1602 |  0.4476 |  0.3544 |  0.0026 |  0.2640
5 |  32 |  7 |  4 |  30 |  32 |  0.3714 |  0.3382 |  0.3573 |  0.0353 |  0.2005
6 |  13 |  4 |  50 |  17 |  47 |  0.3424 |  0.1830 |  0.3056 |  0.0008 |  0.3274
7 |  12 |  3 |  3 |  6 |  43 |  0.4201 |  0.2387 |  0.3884 |  0.0123 |  0.2112
8 |  30 |  7 |  16 |  26 |  41 |  0.2055 |  0.3343 |  0.1541 |  0.0011 |  0.3256
9 |  35 |  5 |  50 |  24 |  25 |  0.1567 |  0.4056 |  0.1329 |  0.0004 |  0.3433
10 |  41 |  7 |  9 |  35 |  19 |  0.4885 |  0.3548 |  0.3080 |  0.0004 |  0.2343
11 |  4 |  4 |  33 |  12 |  39 |  0.1811 |  0.1251 |  0.1066 |  0.0010 |  0.3051
12 |  48 |  4 |  36 |  45 |  37 |  0.2143 |  0.3486 |  0.1222 |  0.0015 |  0.3504
13 |  36 |  3 |  39 |  10 |  48 |  0.1992 |  0.4164 |  0.1183 |  0.0050 |  0.3291
14 |  45 |  4 |  44 |  40 |  26 |  0.2329 |  0.2900 |  0.1280 |  0.0903 |  0.0042
15 |  40 |  4 |  35 |  48 |  11 |  0.1600 |  0.4704 |  0.1211 |  0.0019 |  0.3125
16 |  49 |  3 |  47 |  41 |  40 |  0.1002 |  0.2541 |  0.1284 |  0.0019 |  0.3588
17 |  47 |  3 |  50 |  47 |  12 |  0.2519 |  0.2125 |  0.1668 |  0.0005 |  0.2988
18 |  47 |  3 |  37 |  35 |  50 |  0.1059 |  0.4225 |  0.1120 |  0.0034 |  0.3487
19 |  47 |  4 |  48 |  34 |  39 |  0.2177 |  0.3607 |  0.1301 |  0.0016 |  0.3567
20 |  46 |  3 |  46 |  45 |  43 |  0.2007 |  0.1609 |  0.1196 |  0.0044 |  0.3356
Table 4: RETAIN Best Model - Subsequent Visit Prediction Task Iteration |  Embedding Size |  Recurrent Size |  Dropout Embedding |  Dropout Context |  L2 |  Average Precision
---|---|---|---|---|---|---
1 |  142 |  90 |  0.3846 |  0.0224 |  0.0891 |  0.1822
2 |  124 |  43 |  0.3922 |  0.0382 |  0.0003 |  0.1815
3 |  173 |  90 |  0.3929 |  0.2238 |  0.0014 |  0.3479
4 |  145 |  91 |  0.4117 |  0.0404 |  0.0102 |  0.2049
5 |  153 |  92 |  0.4569 |  0.0642 |  0.0116 |  0.2049
6 |  120 |  37 |  0.4335 |  0.4017 |  0.0728 |  0.1815
7 |  180 |  102 |  0.3567 |  0.3307 |  0.0039 |  0.2469
8 |  195 |  38 |  0.3805 |  0.2711 |  0.0010 |  0.3740
9 |  165 |  119 |  0.4969 |  0.1964 |  0.0047 |  0.2292
10 |  174 |  92 |  0.3862 |  0.2078 |  0.0019 |  0.3329
11 |  145 |  110 |  0.3928 |  0.1926 |  0.0891 |  0.1813
12 |  195 |  83 |  0.4418 |  0.4787 |  0.0011 |  0.3543
13 |  187 |  110 |  0.3456 |  0.2083 |  0.0123 |  0.2122
14 |  144 |  80 |  0.3717 |  0.0932 |  0.0032 |  0.3038
15 |  193 |  68 |  0.4528 |  0.3950 |  0.0022 |  0.3280
16 |  198 |  45 |  0.4344 |  0.3324 |  0.0442 |  0.1828
17 |  145 |  64 |  0.4213 |  0.1950 |  0.0626 |  0.1813
18 |  171 |  116 |  0.4166 |  0.4950 |  0.0028 |  0.3197
19 |  186 |  38 |  0.4062 |  0.4916 |  0.0011 |  0.3309
20 |  136 |  54 |  0.3503 |  0.1678 |  0.0067 |  0.2162

##  Appendix B Disease-wise Model Performance

Here we show Disease-wide BEHRT performance in terms of AUROC and APS. We have
displayed codes with an occurrence ratio of 0.01 at the least. And detailed
below is the description of the Caliber code and the chapter along with the
APS/AUROC.

Table 5: Diseases Prediction Performance  Caliber |  APS |  AUROC |  Description |  Ratio |  Caliber Chapter
---|---|---|---|---|---
92 |  0.066765 |  0.723828 |  Gastritis and duodenitis |  0.011198 |  Diseases of the digestive system
66 |  0.108185 |  0.797633 |  Diaphragmatic hernia |  0.011490 |  Diseases of the digestive system
100 |  0.118093 |  0.742646 |  Hearing loss |  0.021964 |  Diseases of the ear and mastoid process
273 |  0.132567 |  0.773249 |  Spondylosis |  0.013459 |  Diseases of the musculoskeletal system and connective tissue
189 |  0.142594 |  0.844596 |  Pleural effusion |  0.010229 |  Diseases of the respiratory system
172 |  0.162186 |  0.798654 |  Other anaemias |  0.023303 |  Diseases of the blood and blood-forming organs and certain disorders involving the immune mechanism
29 |  0.163355 |  0.822707 |  Bacterial Diseases (excl TB) |  0.023979 |  Certain infectious and parasitic diseases
130 |  0.167457 |  0.804545 |  Iron deficiency anaemia |  0.020780 |  Diseases of the blood and blood-forming organs and certain disorders involving the immune mechanism
295 |  0.169674 |  0.836610 |  Urinary Tract Infections |  0.022534 |  Diseases of the genitourinary system
69 |  0.170852 |  0.811759 |  Diverticular disease of intestine (acute and chronic) |  0.015966 |  Diseases of the digestive system
8 |  0.177170 |  0.810068 |  Allergic and chronic rhinitis |  0.023241 |  Diseases of the respiratory system
170 |  0.181182 |  0.847938 |  Osteoporosis |  0.013982 |  Diseases of the musculoskeletal system and connective tissue
71 |  0.184000 |  0.790655 |  Dyslipidaemia |  0.026010 |  Endocrine, nutritional and metabolic diseases
168 |  0.184629 |  0.799592 |  Oesophagitis and oesophageal ulcer |  0.022426 |  Diseases of the digestive system
217 |  0.203218 |  0.855593 |  Primary Malignancy Other Skin and subcutaneous tissue |  0.012013 |  Neoplasms
93 |  0.206455 |  0.776367 |  Gastro-oesophageal reflux disease |  0.026271 |  Diseases of the digestive system
290 |  0.208524 |  0.814481 |  Type 1 Diabetes Mellitus, Type 2 Diabetes Mellitus, and Diabetes Mellitus – other or not specified |  0.021226 |  Endocrine, nutritional and metabolic diseases
3 |  0.219637 |  0.869366 |  Actinic keratosis |  0.012490 |  Diseases of the skin and subcutaneous tissue
131 |  0.220017 |  0.874319 |  Irritable bowel syndrome |  0.011182 |  Diseases of the digestive system
294 |  0.223863 |  0.824114 |  Urinary Incontinence |  0.020057 |  Diseases of the genitourinary system
169 |  0.234444 |  0.785766 |  Osteoarthritis (excl spine) |  0.043714 |  Diseases of the musculoskeletal system and connective tissue
176 |  0.245906 |  0.834062 |  Other or unspecified infectious organisms |  0.030963 |  Diseases of the respiratory system
95 |  0.249208 |  0.894444 |  Glaucoma |  0.011367 |  Diseases of the eye and adnexa
109 |  0.250573 |  0.885547 |  Hyperplasia of prostate |  0.020842 |  Diseases of the genitourinary system
184 |  0.264687 |  0.879325 |  Peripheral arterial disease |  0.010951 |  Diseases of the circulatory system
79 |  0.265863 |  0.762939 |  Enthesopathies & synovial disorders |  0.047036 |  Diseases of the musculoskeletal system and connective tissue
81 |  0.267187 |  0.905812 |  Erectile dysfunction |  0.017873 |  Mental and behavioural disorders
140 |  0.268504 |  0.867094 |  Lower Respiratory Tract Infections |  0.023518 |  Certain infectious and parasitic diseases
63 |  0.271027 |  0.753816 |  Dermatitis (atopc/contact/other/unspecified) |  0.049051 |  Diseases of the skin and subcutaneous tissue
142 |  0.292598 |  0.893802 |  Macular degeneration |  0.010752 |  Diseases of the eye and adnexa
274 |  0.296798 |  0.889236 |  Stable Angina |  0.032039 |  Diseases of the circulatory system
57 |  0.301041 |  0.900088 |  Coronary heart disease not otherwise specified |  0.035177 |  Diseases of the circulatory system
275 |  0.307238 |  0.911618 |  Stroke Not otherwise specified (NOS) |  0.023395 |  Diseases of the nervous system
45 |  0.319433 |  0.863447 |  Cataract |  0.042099 |  Diseases of the eye and adnexa
1 |  0.319972 |  0.845171 |  Abdominal Hernia |  0.019180 |  Diseases of the digestive system
44 |  0.325143 |  0.843480 |  Carpal tunnel syndrome |  0.012013 |  Diseases of the nervous system
101 |  0.334902 |  0.912117 |  Heart failure |  0.024918 |  Diseases of the circulatory system
164 |  0.335131 |  0.879670 |  Obesity |  0.017442 |  Endocrine, nutritional and metabolic diseases
22 |  0.348523 |  0.885055 |  Asthma |  0.026133 |  Diseases of the respiratory system
97 |  0.349361 |  0.882694 |  Gout |  0.018058 |  Diseases of the musculoskeletal system and connective tissue
65 |  0.350132 |  0.942604 |  Diabetic ophthalmic complications |  0.018919 |  Endocrine, nutritional and metabolic diseases
147 |  0.368465 |  0.911541 |  Migraine |  0.012028 |  Diseases of the nervous system
229 |  0.398842 |  0.904686 |  Psoriasis |  0.011751 |  Diseases of the musculoskeletal system and connective tissue
17 |  0.410899 |  0.858498 |  Anxiety disorders |  0.041914 |  Mental and behavioural disorders
146 |  0.433645 |  0.969406 |  Menorrhagia and polymenorrhoea |  0.015504 |  Diseases of the genitourinary system
113 |  0.489456 |  0.905032 |  Hypo or hyperthyroidism |  0.047897 |  Endocrine, nutritional and metabolic diseases
302 |  0.491672 |  0.855823 |  Vitamin B12 deficiency anaemia |  0.014489 |  Diseases of the blood and blood-forming organs and certain disorders involving the immune mechanism
51 |  0.501496 |  0.923082 |  Chronic obstructive pulmonary disease (COPD) |  0.036869 |  Diseases of the respiratory system
23 |  0.514881 |  0.901268 |  Atrial Fibrillation and flutter |  0.077629 |  Diseases of the circulatory system
110 |  0.531597 |  0.819527 |  Hypertension |  0.200618 |  Diseases of the circulatory system
61 |  0.542223 |  0.950442 |  Dementia |  0.024656 |  Mental and behavioural disorders
62 |  0.553561 |  0.877904 |  Depression |  0.076876 |  Mental and behavioural disorders
85 |  0.573880 |  0.934049 |  Female genital prolapse |  0.015781 |  Diseases of the genitourinary system
220 |  0.575574 |  0.964776 |  Primary Malignancy Prostate |  0.011844 |  Neoplasms
6 |  0.583305 |  0.952656 |  Alcohol Problems |  0.014535 |  Mental and behavioural disorders
194 |  0.647243 |  0.955062 |  Polymyalgia Rheumatica |  0.013213 |  Diseases of the musculoskeletal system and connective tissue
80 |  0.648763 |  0.977907 |  Epilepsy |  0.016104 |  Diseases of the nervous system

##  Appendix C Comparison of APS/AUROC across Models

In Figure 7, we see the general trend of the three models. BEHRT’s predictions
remain in the upper right quadrant of the graph (for the most part) denoting
higher APS/AUROC than the other two models.

Figure 7: Disease-wise precision comparison for BEHRT, Deepr and RETAIN, all
models trained on same dataset and trained on the same task (next 6 months),
each point represents the precision/AUROC of one disease on corresponding
model.


\textit{Bose Condensates and Fermi Gases at Zero Temperature.} Phys. Rev. Lett. \textbf{80}, 1804-1807 (1998,3), https://link.aps.org/doi/10.1103/PhysRevLett.80.1804

[52]
H. Narnhofer,
G. L. Sewell.
\textit{Vlasov hydrodynamics of a quantum mechanical model. }
Comm. Math. Phys. 79 (1981), no. 1, 9–24.

[53]
P. Pickl.
\textit{A simple derivation of mean field limits for quantum systems. }
Lett. Math. Phys. 97 (2011), no. 2, 151–164.

[54]
M. Porta,
S. Rademacher,
C. Saffirio,
B. Schlein.
\textit{Mean Field Evolution of Fermions with Coulomb Interaction. }
J Stat Phys 166, 1345–1364 (2017).

[55]
I. Rodnianski,
B. Schlein.
\textit{Quantum fluctuations and rate of convergence towards mean field dynamics. }
Comm. Math. Phys. 291 (2009), no. 1, 31–61.

[56] %This paper makes a degenerate BEC in a sea of fermions. The fermionic part is not degenerate.
F. Schreck,
L. Khaykovich,
K. Corwin,
G. Ferrari,
T. Bourdel,
J. Cubizolles,
C. Salomon.
\textit{Quasipure Bose-Einstein Condensate Immersed in a Fermi Sea.} {Phys. Rev. Lett.}. \textbf{87}, 080403 (2001,8), https://link.aps.org/doi/10.1103/PhysRevLett.87.080403

[57]
J.P. Solovej.
\textit{Many Body Quantum Mechanics. }
Lecture Notes. Summer (2007), http://www.

[58]
H. Spohn.
\textit{			Kinetic equations from Hamiltonian dynamics: Markovian limits. }
Rev. Mod- ern Phys. 52 (1980), no. 3, 569–615.

[59]
H. Spohn.
\textit{On the Vlasov hierarchy, }
Math. Methods Appl. Sci. 3 (1981), no. 4, 445–455.

% NEW REFERENCES BELOW

\end{thebibliography}

\end{document}


* Haehnelt et al. (1998) Haehnelt, M. G., Natarajan, P., & Rees, M. J. 1998, MNRAS, 300, 817, doi: 10.1046/j.1365-8711.1998.01951.x
  * Hagiwara et al. (2016) Hagiwara, Y., Horiuchi, S., Doi, A., Miyoshi, M., & Edwards, P. G. 2016, ApJ, 827, 69, doi: 10.3847/0004-637X/827/1/69
  * Hagiwara et al. (2021) Hagiwara, Y., Horiuchi, S., Imanishi, M., & Edwards, P. G. 2021, ApJ, 923, 251, doi: 10.3847/1538-4357/ac3089
  * Hagiwara et al. (2013) Hagiwara, Y., Miyoshi, M., Doi, A., & Horiuchi, S. 2013, ApJ, 768, L38, doi: 10.1088/2041-8205/768/2/L38
  * Hamilton et al. (2012) Hamilton, L., Scowcroft, B., Ayers, M., et al. 2012, Blue Ribbon Commission on America’s Nuclear Future (BRC), Washington, DC
  * Harlow (2016) Harlow, D. 2016, Reviews of Modern Physics, 88, 015002, doi: 10.1103/RevModPhys.88.015002
  * Heesen (2018) Heesen, R. 2018, Journal of Philosophy, 115, 661, doi: 10.5840/jphil20181151239
  * Herrnstein (1997) Herrnstein, J. R. 1997, PhD thesis, Harvard University, Massachusetts
  * Hinshaw et al. (2009) Hinshaw, G., Weiland, J. L., Hill, R. S., et al. 2009, ApJS, 180, 225, doi: 10.1088/0067-0049/180/2/225
  * Ho et al. (2019) Ho, A. Y. Q., Phinney, E. S., Ravi, V., et al. 2019, ApJ, 871, 73, doi: 10.3847/1538-4357/aaf473
  * Hoak et al. (2022) Hoak, D., Barrett, J., Crew, G., & Pfeiffer, V. 2022, Galaxies, 10, 119, doi: 10.3390/galaxies10060119
  * Hu et al. (2000) Hu, W., Barkana, R., & Gruzinov, A. 2000, Phys. Rev. Lett., 85, 1158, doi: 10.1103/PhysRevLett.85.1158
  * Humphreys et al. (2005) Humphreys, E. M. L., Greenhill, L. J., Reid, M. J., et al. 2005, ApJ, 634, L133, doi: 10.1086/498890
  * Humphreys et al. (2016) Humphreys, E. M. L., Vlemmings, W. H. T., Impellizzeri, C. M. V., et al. 2016, A&A, 592, L13, doi: 10.1051/0004-6361/201629168
  * IceCube Collaboration et al. (2018a) IceCube Collaboration, Aartsen, M. G., Ackermann, M., et al. 2018a, Science, 361, 147, doi: 10.1126/science.aat2890
  * IceCube Collaboration et al. (2018b) —. 2018b, Science, 361, eaat1378, doi: 10.1126/science.aat1378
  * Inoue et al. (2014) Inoue, M., Algaba-Marcos, J. C., Asada, K., et al. 2014, Radio Science, 49, 564, doi: 10.1002/2014RS005450
  * Issaoun et al. (2019) Issaoun, S., Johnson, M. D., Blackburn, L., et al. 2019, ApJ, 871, 30, doi: 10.3847/1538-4357/aaf732
  * Issaoun et al. (2022) Issaoun, S., Wielgus, M., Jorstad, S., et al. 2022, ApJ, 934, 145, doi: 10.3847/1538-4357/ac7a40
  * Issaoun et al. (2023) Issaoun, S., Pesce, D. W., Roelofs, F., et al. 2023, Galaxies, 11, doi: 10.3390/galaxies11010028
  * Janssen et al. (2022) Janssen, M., Radcliffe, J. F., & Wagner, J. 2022, Universe, 8, 527, doi: 10.3390/universe8100527
  * Janssen et al. (2019) Janssen, M., Goddi, C., van Bemmel, I. M., et al. 2019, A&A, 626, A75, doi: 10.1051/0004-6361/201935181
  * Janssen et al. (2021) Janssen, M., Falcke, H., Kadler, M., et al. 2021, Nature Astronomy, 5, 1017, doi: 10.1038/s41550-021-01417-w
  * Jaroszynski & Kurpiewski (1997) Jaroszynski, M., & Kurpiewski, A. 1997, A&A, 326, 419, doi: 10.48550/arXiv.astro-ph/9705044
  * Jebeile (2020) Jebeile, J. 2020, Social Epistemology, 34, 453, doi: 10.1080/02691728.2020.1779380
  * Jia et al. (2023) Jia, H., Ripperda, B., Quataert, E., et al. 2023, arXiv e-prints, arXiv:2301.09014, doi: 10.48550/arXiv.2301.09014
  * Jiang et al. (2021) Jiang, W., Shen, Z., Martí-Vidal, I., et al. 2021, ApJ, 922, L16, doi: 10.3847/2041-8213/ac375c
  * Jiang et al. (2023) Jiang, W., Zhao, G.-Y., Shen, Z.-Q., et al. 2023, Galaxies, 11, doi: 10.3390/galaxies11010003
  * Jiménez-Rosales et al. (2021) Jiménez-Rosales, A., Dexter, J., Ressler, S. M., et al. 2021, MNRAS, 503, 4563, doi: 10.1093/mnras/stab784
  * Johannsen & Psaltis (2010) Johannsen, T., & Psaltis, D. 2010, ApJ, 718, 446, doi: 10.1088/0004-637X/718/1/446
  * Johnson (2016) Johnson, M. D. 2016, ApJ, 833, 74, doi: 10.3847/1538-4357/833/1/74
  * Johnson et al. (2017) Johnson, M. D., Bouman, K. L., Blackburn, L., et al. 2017, ApJ, 850, 172, doi: 10.3847/1538-4357/aa97dd
  * Johnson et al. (2018) Johnson, M. D., Narayan, R., Psaltis, D., et al. 2018, ApJ, 865, 104, doi: 10.3847/1538-4357/aadcff
  * Johnson et al. (2020) Johnson, M. D., Lupsasca, A., Strominger, A., et al. 2020, Science Advances, 6, eaaz1310, doi: 10.1126/sciadv.aaz1310
  * Jorstad et al. (2023) Jorstad, S., Wielgus, M., Lico, R., et al. 2023, ApJ, 943, 170, doi: 10.3847/1538-4357/acaea8
  * Kahanamoku et al. (2020) Kahanamoku, S., Alegado, R., Kagawa-Viviani, A., et al. 2020, arXiv preprint arXiv:2001.00970
  * Kardashev (1986) Kardashev, N. S. 1986, AZh, 63, 845
  * Kauffmann et al. (2023) Kauffmann, J., Rajagopalan, G., Akiyama, K., et al. 2023, Galaxies, 11, doi: 10.3390/galaxies11010009
  * Kelly & Merloni (2012) Kelly, B. C., & Merloni, A. 2012, Advances in Astronomy, 2012, 970858, doi: 10.1155/2012/970858
  * Kessler (2012) Kessler, E. A. 2012, Picturing the cosmos: Hubble Space Telescope images and the astronomical sublime (University of Minnesota Press)
  * Kim & Fish (2023) Kim, D.-J., & Fish, V. 2023, Galaxies, 11, doi: 10.3390/galaxies11010010
  * Kim et al. (2020) Kim, J.-Y., Krichbaum, T. P., Broderick, A. E., et al. 2020, A&A, 640, A69, doi: 10.1051/0004-6361/202037493
  * King et al. (2008) King, A. R., Pringle, J. E., & Hofmann, J. A. 2008, MNRAS, 385, 1621, doi: 10.1111/j.1365-2966.2008.12943.x
  * Kitcher (1990) Kitcher, P. 1990, Journal of Philosophy, 87, 5, doi: 10.2307/2026796
  * Kitcher (1993) —. 1993, The Advancement of Science: Science Without Legend, Objectivity Without Illusions (Oxford, England: Oxford University Press)
  * Knorr Cetina (1999) Knorr Cetina, K. 1999, Epistemic Cultures: How the Sciences Make Knowledge (Harvard University Press).  http://www.jstor.org/stable/j.ctvxw3q7f
  * Kocherlakota et al. (2021) Kocherlakota, P., Rezzolla, L., Falcke, H., et al. 2021, Phys. Rev. D, 103, 104047, doi: 10.1103/PhysRevD.103.104047
  * Körding et al. (2006) Körding, E. G., Jester, S., & Fender, R. 2006, MNRAS, 372, 1366, doi: 10.1111/j.1365-2966.2006.10954.x
  * Kormendy & Ho (2013) Kormendy, J., & Ho, L. C. 2013, ARA&A, 51, 511, doi: 10.1146/annurev-astro-082708-101811
  * Kovalev et al. (2020) Kovalev, Y. Y., Pushkarev, A. B., Nokhrina, E. E., et al. 2020, MNRAS, 495, 3576, doi: 10.1093/mnras/staa1121
  * Krichbaum et al. (1998) Krichbaum, T. P., Graham, D. A., Witzel, A., et al. 1998, A&A, 335, L106
  * Kuletz (1998) Kuletz, V. L. 1998, The Tainted Desert (New York: Routledge)
  * Kulier et al. (2015) Kulier, A., Ostriker, J. P., Natarajan, P., Lackner, C. N., & Cen, R. 2015, ApJ, 799, 178, doi: 10.1088/0004-637X/799/2/178
  * Kuo et al. (2011) Kuo, C. Y., Braatz, J. A., Condon, J. J., et al. 2011, ApJ, 727, 20, doi: 10.1088/0004-637X/727/1/20
  * Lalli et al. (2020a) Lalli, R., Howey, R., & Wintergrün, D. 2020a, Scientometrics, 122, 1129, doi: 10.1007/s11192-019-03327-1
  * Lalli et al. (2020b) —. 2020b, in The Renaissance of General Relativity in Context, ed. A. S. Blum, R. Lalli, & J. Renn, Einstein Studies (Cham: Springer International Publishing), 15–84, doi: 10.1007/978-3-030-50754-1_2
  * Levis et al. (2022) Levis, A., Srinivasan, P. P., Chael, A. A., Ng, R., & Bouman, K. L. 2022, in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 19809–19818, doi: 10.1109/CVPR52688.2022.01922
  * Lico et al. (2023) Lico, R., Jorstad, S. G., Marscher, A. P., et al. 2023, Galaxies, 11, doi: 10.3390/galaxies11010017
  * Light & Moody (2021) Light, R., & Moody, J. 2021, The Oxford Handbook of Social Networks (Oxford University Press), doi: 10.1093/oxfordhb/9780190251765.001.0001
  * Liska et al. (2022) Liska, M. T. P., Chatterjee, K., Issa, D., et al. 2022, ApJS, 263, 26, doi: 10.3847/1538-4365/ac9966
  * Longino (2019) Longino, H. 2019, in The Stanford Encyclopedia of Philosophy, Summer 2019 edn., ed. E. N. Zalta (Metaphysics Research Lab, Stanford University)
  * Lu et al. (2016) Lu, R.-S., Roelofs, F., Fish, V. L., et al. 2016, ApJ, 817, 173, doi: 10.3847/0004-637X/817/2/173
  * Luminet (1979) Luminet, J. P. 1979, A&A, 75, 228
  * Margutti et al. (2019) Margutti, R., Metzger, B. D., Chornock, R., et al. 2019, ApJ, 872, 18, doi: 10.3847/1538-4357/aafa01
  * Markoff et al. (2007) Markoff, S., Bower, G. C., & Falcke, H. 2007, MNRAS, 379, 1519, doi: 10.1111/j.1365-2966.2007.12071.x
  * Marscher et al. (2008) Marscher, A. P., Jorstad, S. G., D’Arcangelo, F. D., et al. 2008, Nature, 452, 966, doi: 10.1038/nature06895
  * Masco (2013) Masco, J. 2013, in The Nuclear Borderlands (Princeton University Press)
  * McNutt et al. (2018) McNutt, M. K., Bradford, M., Drazen, J. M., et al. 2018, Proceedings of the National Academy of Sciences of the United States of America, 115, 2557—2560, doi: 10.1073/pnas.1715374115
  * Mellers et al. (2015) Mellers, B., Stone, E., Murray, T., et al. 2015, Perspectives on Psychological Science, 10, 267
  * Merritt & Milosavljević (2005) Merritt, D., & Milosavljević, M. 2005, Living Reviews in Relativity, 8, 8, doi: 10.12942/lrr-2005-8
  * Merz & Sorgner (2022) Merz, M., & Sorgner, H. 2022, Synthese, 200, 1, doi: 10.1007/s11229-022-03649-3
  * Miller-Jones et al. (2004) Miller-Jones, J. C. A., Blundell, K. M., Rupen, M. P., et al. 2004, ApJ, 600, 368, doi: 10.1086/379706
  * Milosavljević & Merritt (2003) Milosavljević, M., & Merritt, D. 2003, in American Institute of Physics Conference Series, Vol. 686, The Astrophysics of Gravitational Wave Sources, ed. J. M. Centrella, 201–210, doi: 10.1063/1.1629432
  * Mirabel & Rodríguez (1999) Mirabel, I. F., & Rodríguez, L. F. 1999, ARA&A, 37, 409, doi: 10.1146/annurev.astro.37.1.409
  * Miyoshi et al. (1995) Miyoshi, M., Moran, J., Herrnstein, J., et al. 1995, Nature, 373, 127, doi: 10.1038/373127a0
  * Mooley et al. (2018) Mooley, K. P., Deller, A. T., Gottlieb, O., et al. 2018, Nature, 561, 355, doi: 10.1038/s41586-018-0486-3
  * Moriyama et al. (2019) Moriyama, K., Mineshige, S., Honma, M., & Akiyama, K. 2019, ApJ, 887, 227, doi: 10.3847/1538-4357/ab505b
  * Mościbrodzka et al. (2017) Mościbrodzka, M., Dexter, J., Davelaar, J., & Falcke, H. 2017, MNRAS, 468, 2214, doi: 10.1093/mnras/stx587
  * Mościbrodzka & Gammie (2018) Mościbrodzka, M., & Gammie, C. F. 2018, MNRAS, 475, 43, doi: 10.1093/mnras/stx3162
  * Müller & Lobanov (2022) Müller, H., & Lobanov, A. P. 2022, A&A, 666, A137, doi: 10.1051/0004-6361/202243244
  * Narayan et al. (2022) Narayan, R., Chael, A., Chatterjee, K., Ricarte, A., & Curd, B. 2022, MNRAS, 511, 3795, doi: 10.1093/mnras/stac285
  * Narayan et al. (2003) Narayan, R., Igumenshchev, I. V., & Abramowicz, M. A. 2003, PASJ, 55, L69, doi: 10.1093/pasj/55.6.L69
  * Narayan et al. (2019) Narayan, R., Johnson, M. D., & Gammie, C. F. 2019, ApJ, 885, L33, doi: 10.3847/2041-8213/ab518c
  * Narayan et al. (2012) Narayan, R., Sądowski, A., Penna, R. F., & Kulkarni, A. K. 2012, MNRAS, 426, 3241, doi: 10.1111/j.1365-2966.2012.22002.x
  * ngEHT Collaboration (2023) ngEHT Collaboration. 2023, in prep
  * ngEHT Fundamental Physics SWG et al. (2023) ngEHT Fundamental Physics SWG, T., et al. 2023, Fundamental physics opportunities with the next-generation Event Horizon Telescope
  * Nichols (2022) Nichols, T. 2022, PhD thesis, Harvard University
  * Nichols (2023) —. 2023, Unpublished Manuscript
  * Nogrady (2023) Nogrady, B. 2023, Nature, 615, 175, doi: 10.1038/d41586-023-00575-3
  * Okino et al. (2022) Okino, H., Akiyama, K., Asada, K., et al. 2022, ApJ, 940, 65, doi: 10.3847/1538-4357/ac97e5
  * Pacucci & Loeb (2020) Pacucci, F., & Loeb, A. 2020, ApJ, 895, 95, doi: 10.3847/1538-4357/ab886e
  * Padin et al. (1990) Padin, S., Woody, D. P., Hodges, M. W., et al. 1990, ApJ, 360, L11, doi: 10.1086/185800
  * Paine et al. (2020) Paine, J., Darling, J., Graziani, R., & Courtois, H. M. 2020, ApJ, 890, 146, doi: 10.3847/1538-4357/ab6f00
  * Palumbo et al. (2022) Palumbo, D. C. M., Gelles, Z., Tiede, P., et al. 2022, ApJ, 939, 107, doi: 10.3847/1538-4357/ac9ab7
  * Palumbo & Wong (2022) Palumbo, D. C. M., & Wong, G. N. 2022, ApJ, 929, 49, doi: 10.3847/1538-4357/ac59b4
  * Palumbo et al. (2020) Palumbo, D. C. M., Wong, G. N., & Prather, B. S. 2020, ApJ, 894, 156, doi: 10.3847/1538-4357/ab86ac
  * Palumbo (2023) Palumbo, D. P. 2023, Spin signatures of rotating black holes
  * Park et al. (2021) Park, J., Byun, D.-Y., Asada, K., & Yun, Y. 2021, ApJ, 906, 85, doi: 10.3847/1538-4357/abcc6e
  * Peccei & Quinn (1977) Peccei, R. D., & Quinn, H. R. 1977, Phys. Rev. D, 16, 1791, doi: 10.1103/PhysRevD.16.1791
  * Penrose (1969) Penrose, R. 1969, Nuovo Cimento Rivista Serie, 1, 252
  * Pesce (2021) Pesce, D. W. 2021, AJ, 161, 178, doi: 10.3847/1538-3881/abe3f8
  * Pesce et al. (2016) Pesce, D. W., Braatz, J. A., & Impellizzeri, C. M. V. 2016, ApJ, 827, 68, doi: 10.3847/0004-637X/827/1/68
  * Pesce et al. (2022) Pesce, D. W., Palumbo, D. C. M., Ricarte, A., et al. 2022, Galaxies, 10, doi: 10.3390/galaxies10060109
  * Pesce et al. (2020) Pesce, D. W., Braatz, J. A., Reid, M. J., et al. 2020, ApJ, 891, L1, doi: 10.3847/2041-8213/ab75f0
  * Pesce et al. (2021) Pesce, D. W., Palumbo, D. C. M., Narayan, R., et al. 2021, ApJ, 923, 260, doi: 10.3847/1538-4357/ac2eb5
  * Planck Collaboration et al. (2020) Planck Collaboration, Aghanim, N., Akrami, Y., et al. 2020, A&A, 641, A6, doi: 10.1051/0004-6361/201833910
  * Plavin et al. (2020) Plavin, A., Kovalev, Y. Y., Kovalev, Y. A., & Troitsky, S. 2020, ApJ, 894, 101, doi: 10.3847/1538-4357/ab86bd
  * Plavin et al. (2021) Plavin, A. V., Kovalev, Y. Y., Kovalev, Y. A., & Troitsky, S. V. 2021, ApJ, 908, 157, doi: 10.3847/1538-4357/abceb8
  * Plavin et al. (2023) —. 2023, MNRAS submitted, arXiv:2211.09631, doi: 10.48550/arXiv.2211.09631
  * Plavin et al. (2019) Plavin, A. V., Kovalev, Y. Y., Pushkarev, A. B., & Lobanov, A. P. 2019, MNRAS, 485, 1822, doi: 10.1093/mnras/stz504
  * Porth et al. (2019) Porth, O., Chatterjee, K., Narayan, R., et al. 2019, ApJS, 243, 26, doi: 10.3847/1538-4365/ab29fd
  * Preskill et al. (1983) Preskill, J., Wise, M. B., & Wilczek, F. 1983, Physics Letters B, 120, 127, doi: 10.1016/0370-2693(83)90637-8
  * Psaltis et al. (2018) Psaltis, D., Johnson, M., Narayan, R., et al. 2018, arXiv e-prints, arXiv:1805.01242.  https://arxiv.org/abs/1805.01242
  * Psaltis et al. (2020) Psaltis, D., Medeiros, L., Christian, P., et al. 2020, Phys. Rev. Lett., 125, 141104, doi: 10.1103/PhysRevLett.125.141104
  * Pushkarev et al. (2012) Pushkarev, A. B., Hovatta, T., Kovalev, Y. Y., et al. 2012, A&A, 545, A113, doi: 10.1051/0004-6361/201219173
  * Qiu et al. (2023) Qiu, R., Ricarte, A., Narayan, R., et al. 2023, MNRAS, 520, 4867, doi: 10.1093/mnras/stad466
  * Ramakrishnan et al. (2023) Ramakrishnan, V., Nagar, N., Arratia, V., et al. 2023, Galaxies, 11, doi: 10.3390/galaxies11010015
  * Raymond et al. (2021) Raymond, A. W., Palumbo, D., Paine, S. N., et al. 2021, ApJS, 253, 5, doi: 10.3847/1538-3881/abc3c3
  * Redfield (2000) Redfield, P. 2000, Space in the tropics: From convicts to rockets in French Guiana (Univ of California Press)
  * Reid et al. (2013) Reid, M. J., Braatz, J. A., Condon, J. J., et al. 2013, ApJ, 767, 154, doi: 10.1088/0004-637X/767/2/154
  * Reid & Honma (2014) Reid, M. J., & Honma, M. 2014, ARA&A, 52, 339, doi: 10.1146/annurev-astro-081913-040006
  * Rennie et al. (1997) Rennie, D., Yank, V., & Emanuel, L. 1997, JAMA, 278, 579, doi: 10.1001/jama.1997.03550070071041
  * Resnik (1997) Resnik, D. B. 1997, Science and Engineering Ethics, 3, 237, doi: 10.1007/s11948-997-0023-5
  * Reynolds (2021) Reynolds, C. S. 2021, ARA&A, 59, 117, doi: 10.1146/annurev-astro-112420-035022
  * Ricarte et al. (2023) Ricarte, A., Johnson, M. D., Kovalev, Y. Y., Palumbo, D. C. M., & Emami, R. 2023, Galaxies, 11, doi: 10.3390/galaxies11010005
  * Ricarte & Natarajan (2018) Ricarte, A., & Natarajan, P. 2018, MNRAS, 474, 1995, doi: 10.1093/mnras/stx2851
  * Ricarte et al. (2022a) Ricarte, A., Palumbo, D. C. M., Narayan, R., Roelofs, F., & Emami, R. 2022a, ApJ, 941, L12, doi: 10.3847/2041-8213/aca087
  * Ricarte et al. (2020) Ricarte, A., Prather, B. S., Wong, G. N., et al. 2020, MNRAS, 498, 5468, doi: 10.1093/mnras/staa2692
  * Ricarte et al. (2022b) Ricarte, A., Tiede, P., Emami, R., Tamar, A., & Natarajan, P. 2022b, Galaxies, 11, 6, doi: 10.3390/galaxies11010006
  * Richter et al. (2022) Richter, J., Bernstein, M. J., & Farooque, M. 2022, Energy Research & Social Science, 87, 102473
  * Riess et al. (2019) Riess, A. G., Casertano, S., Yuan, W., Macri, L. M., & Scolnic, D. 2019, ApJ, 876, 85, doi: 10.3847/1538-4357/ab1422
  * Rioja & Dodson (2011) Rioja, M., & Dodson, R. 2011, AJ, 141, 114, doi: 10.1088/0004-6256/141/4/114
  * Rioja et al. (2023) Rioja, M. J., Dodson, R., & Asaki, Y. 2023, Galaxies, 11, doi: 10.3390/galaxies11010016
  * Riordan et al. (2015) Riordan, M., Hoddeson, L., & Kolb, A. W. 2015, Tunnel Visions: The Rise and Fall of the Superconducting Super Collider (University of Chicago Press)
  * Ripperda et al. (2022) Ripperda, B., Liska, M., Chatterjee, K., et al. 2022, ApJ, 924, L32, doi: 10.3847/2041-8213/ac46a1
  * Ritson (2021) Ritson, S. 2021, Synthese, 199, 11887
  * Robinson (1975) Robinson, D. C. 1975, Phys. Rev. Lett., 34, 905, doi: 10.1103/PhysRevLett.34.905
  * Roelofs et al. (2023) Roelofs, F., Blackburn, L., Lindahl, G., et al. 2023, Galaxies, 11, doi: 10.3390/galaxies11010012
  * Romero (2020) Romero, G. E. 2020, arXiv e-prints, arXiv:2010.00738.  https://arxiv.org/abs/2010.00738
  * Rowan et al. (2017) Rowan, M. E., Sironi, L., & Narayan, R. 2017, ApJ, 850, 29, doi: 10.3847/1538-4357/aa9380
  * Salazar (2014) Salazar, J. A. 2014, PhD thesis, University of Hawai’i at Manoa
  * Salpeter (1964) Salpeter, E. E. 1964, ApJ, 140, 796, doi: 10.1086/147973
  * Samek et al. (2019) Samek, W., Montavon, G., Vedaldi, A., Hansen, L. K., & Müller, K.-R. 2019, Explainable AI: interpreting, explaining and visualizing deep learning, Vol. 11700 (Springer Nature)
  * Sari et al. (1999) Sari, R., Piran, T., & Halpern, J. P. 1999, ApJ, 519, L17, doi: 10.1086/312109
  * Selina et al. (2018) Selina, R. J., Murphy, E. J., McKinnon, M., et al. 2018, in Astronomical Society of the Pacific Conference Series, Vol. 517, Science with a Next Generation Very Large Array, ed. E. Murphy, 15.  https://arxiv.org/abs/1810.08197
  * Senovilla & Garfinkle (2015) Senovilla, J. M. M., & Garfinkle, D. 2015, Classical and Quantum Gravity, 32, 124008, doi: 10.1088/0264-9381/32/12/124008
  * Shakura & Sunyaev (1973) Shakura, N. I., & Sunyaev, R. A. 1973, A&A, 24, 337
  * Shrum et al. (2001) Shrum, W., Chompalov, I., & Genuth, J. 2001, Social Studies of Science, 31, 681\.  http://www.jstor.org/stable/3183103
  * Singh (2019) Singh, J. A. 2019, BMC Medical Ethics, 20, 1
  * Sironi & Spitkovsky (2014) Sironi, L., & Spitkovsky, A. 2014, ApJ, 783, L21, doi: 10.1088/2041-8205/783/1/L21
  * Smith et al. (1993) Smith, R. W., Hanle, P. A., Kargon, R. H., & Tatarewicz, J. N. 1993, The Space Telescope. A study of NASA, science, technology, and politics.
  * Sokolovsky et al. (2011) Sokolovsky, K. V., Kovalev, Y. Y., Pushkarev, A. B., & Lobanov, A. P. 2011, A&A, 532, A38, doi: 10.1051/0004-6361/201016072
  * Sorgner (2022) Sorgner, H. 2022, Science & Technology Studies
  * Sullivan (2009) Sullivan, W. T. 2009, Cosmic noise : a history of early radio astronomy / Woodruff T. Sullivan, III (Cambridge University Press Cambridge [England] ; New York), xxxii, 542 p. :
  * Sun & Bouman (2020) Sun, H., & Bouman, K. L. 2020, arXiv e-prints, arXiv:2010.14462.  https://arxiv.org/abs/2010.14462
  * Sun et al. (2022) Sun, H., Bouman, K. L., Tiede, P., et al. 2022, ApJ, 932, 99, doi: 10.3847/1538-4357/ac6be9
  * Supernant & Warrick (2014) Supernant, K., & Warrick, G. 2014, Canadian Journal of Archaeology/Journal Canadien d’Archéologie, 563
  * Swanner (2017) Swanner, L. 2017, Historical Studies in the Natural Sciences, 47, 293
  * Swanner (2013) Swanner, L. A. 2013, Mountains of Controversy: Narrative and the Making of Contested Landscapes in Postwar American Astronomy (Harvard University)
  * Takahashi (2004) Takahashi, R. 2004, ApJ, 611, 996, doi: 10.1086/422403
  * Teo (2003) Teo, E. 2003, General Relativity and Gravitation, 35, 1909, doi: 10.1023/A:1026286607562
  * Tetarenko et al. (2017) Tetarenko, A. J., Sivakoff, G. R., Miller-Jones, J. C. A., et al. 2017, MNRAS, 469, 3141, doi: 10.1093/mnras/stx1048
  * Tetarenko et al. (2019) —. 2019, MNRAS, 482, 2950, doi: 10.1093/mnras/sty2853
  * Thompson et al. (2017) Thompson, A. R., Moran, J. M., & Swenson, George W., J. 2017, Interferometry and Synthesis in Radio Astronomy, 3rd Edition (Springer), doi: 10.1007/978-3-319-44431-4
  * Thorne (1974) Thorne, K. S. 1974, ApJ, 191, 507, doi: 10.1086/152991
  * Tiede (2022) Tiede, P. 2022, The Journal of Open Source Software, 7, 4457, doi: 10.21105/joss.04457
  * Tiede et al. (2022) Tiede, P., Johnson, M. D., Pesce, D. W., et al. 2022, Galaxies, 10, doi: 10.3390/galaxies10060111
  * Tiede et al. (2020) Tiede, P., Pu, H.-Y., Broderick, A. E., et al. 2020, ApJ, 892, 132, doi: 10.3847/1538-4357/ab744c
  * Tomblin et al. (2017) Tomblin, D., Pirtle, Z., Farooque, M., et al. 2017, Astropolitics, 15, 141
  * Traweek (1988) Traweek, S. 1988, Beamtimes and Lifetimes: The World of High Energy Physicists (Harvard University Press).  http://www.jstor.org/stable/j.ctv260711b
  * Tseng et al. (2016) Tseng, C.-Y., Asada, K., Nakamura, M., et al. 2016, ApJ, 833, 288, doi: 10.3847/1538-4357/833/2/288
  * Vertesi (2020) Vertesi, J. 2020, Shaping Science: Organizations, Decisions, and Culture on NASA’s Teams (University of Chicago Press), doi: 10.7208/chicago/9780226691114.001.0001
  * Vincent et al. (2021) Vincent, F. H., Wielgus, M., Abramowicz, M. A., et al. 2021, Astron. Astrophys., 646, A37, doi: 10.1051/0004-6361/202037787
  * Vos et al. (2022) Vos, J., Mościbrodzka, M. A., & Wielgus, M. 2022, A&A, 668, A185, doi: 10.1051/0004-6361/202244840
  * Walker et al. (2018) Walker, R. C., Hardee, P. E., Davies, F. B., Ly, C., & Junor, W. 2018, ApJ, 855, 128, doi: 10.3847/1538-4357/aaafcc
  * Watkins (2005) Watkins, J. 2005, Annu. Rev. Anthropol., 34, 429
  * Weinberger et al. (2018) Weinberger, R., Springel, V., Pakmor, R., et al. 2018, MNRAS, 479, 4056, doi: 10.1093/mnras/sty1733
  * Werner et al. (2018) Werner, G. R., Uzdensky, D. A., Begelman, M. C., Cerutti, B., & Nalewajko, K. 2018, MNRAS, 473, 4840, doi: 10.1093/mnras/stx2530
  * Wielgus (2021) Wielgus, M. 2021, Phys. Rev. D, 104, 124058, doi: 10.1103/PhysRevD.104.124058
  * Wielgus et al. (2020) Wielgus, M., Akiyama, K., Blackburn, L., et al. 2020, ApJ, 901, 67, doi: 10.3847/1538-4357/abac0d
  * Wielgus et al. (2022) Wielgus, M., Moscibrodzka, M., Vos, J., et al. 2022, A&A, 665, L6, doi: 10.1051/0004-6361/202244493
  * Winsberg (2018) Winsberg, E. 2018, Philosophy and climate science (Cambridge University Press)
  * Wong (2021) Wong, G. N. 2021, ApJ, 909, 217, doi: 10.3847/1538-4357/abdd2d
  * Wong et al. (2020) Wong, K. C., Suyu, S. H., Chen, G. C. F., et al. 2020, MNRAS, 498, 1420, doi: 10.1093/mnras/stz3094
  * Wray (2006) Wray, K. B. 2006, Studies in History and Philosophy of Science Part A, 37, 505, doi: 10.1016/j.shpsa.2005.07.011
  * Wüthrich (Forthcoming) Wüthrich, A. Forthcoming, Synthese
  * Yates et al. (1997) Yates, J. A., Field, D., & Gray, M. D. 1997, MNRAS, 285, 303, doi: 10.1093/mnras/285.2.303
  * Younsi et al. (2023) Younsi, Z., Psaltis, D., & Özel, F. 2023, ApJ, 942, 47, doi: 10.3847/1538-4357/aca58a
  * Yu et al. (2023a) Yu, W., Lu, R.-S., Shen, Z.-Q., & Weintroub, J. 2023a, Galaxies, 11, doi: 10.3390/galaxies11010007
  * Yu et al. (2023b) Yu, W., Romein, J. W., Dursi, L. J., et al. 2023b, Galaxies, 11, doi: 10.3390/galaxies11010013
  * Yuan & Narayan (2014) Yuan, F., & Narayan, R. 2014, ARA&A, 52, 529, doi: 10.1146/annurev-astro-082812-141003
  * Yusef-Zadeh et al. (2006) Yusef-Zadeh, F., Roberts, D., Wardle, M., Heinke, C. O., & Bower, G. C. 2006, ApJ, 650, 189, doi: 10.1086/506375
  * Zednik (2021) Zednik, C. 2021, Philosophy & technology, 34, 265
  * Zhu et al. (2019) Zhu, Z., Johnson, M. D., & Narayan, R. 2019, ApJ, 870, 6, doi: 10.3847/1538-4357/aaef3d
  * Zollman (2013) Zollman, K. J. 2013, Philosophy Compass, 8, 15, doi: https://doi.org/10.1111/j.1747-9991.2012.00534.x
  * Zollman (2007) Zollman, K. J. S. 2007, Philosophy of Science, 74, 574–587, doi: 10.1086/525605
  * Zollman (2010) —. 2010, Erkenntnis (1975-), 72, 17.  http://www.jstor.org/stable/20642278
  * Šešelja (2022) Šešelja, D. 2022, Philosophy Compass, 17, e12855, doi: https://doi.org/10.1111/phc3.12855