# Deep Reinforcement Learning for Personalized Diagnostic Decision Pathways
Using Electronic Health Records: A Comparative Study on Anemia and Systemic
Lupus Erythematosus

Lillian Muyama  Antoine Neuraz  Adrien Coulet

###### Abstract

Background: Clinical diagnosis is typically reached by following a series of
steps recommended by guidelines authored by colleges of experts. Accordingly,
guidelines play a crucial role in rationalizing clinical decisions but suffer
from limitations as they are built to cover the majority of the population and
fail at covering patients with uncommon conditions. Moreover, their updates
are long and expensive, making them unsuitable for emerging diseases and
practices.
Methods: Inspired by guidelines, we formulate the task of diagnosis as a
sequential decision-making problem and study the use of Deep Reinforcement
Learning (DRL) algorithms to learn the optimal sequence of actions to perform
in order to obtain a correct diagnosis from Electronic Health Records (EHRs).
We apply DRL on synthetic, but realistic EHRs and develop two clinical use
cases: Anemia diagnosis, where the decision pathways follow the schema of a
decision tree; and Systemic Lupus Erythematosus (SLE) diagnosis, which follows
a weighted criteria score. We particularly evaluate the robustness of our
approaches to noisy and missing data since these frequently occur in EHRs.
Results: In both use cases, and in the presence of imperfect data, our best
DRL algorithms exhibit competitive performance when compared to the
traditional classifiers, with the added advantage that they enable the
progressive generation of a pathway to the suggested diagnosis which can both
guide and explain the decision-making process.
Conclusion: DRL offers the opportunity to learn personalized decision pathways
to diagnosis. We illustrate with our two use cases their advantages: they
generate step-by-step pathways that are self-explanatory; and their
correctness is competitive when compared to state-of-the-art approaches.

###### keywords:

Diagnosis , Decision support , Deep Reinforcement Learning , Decision pathways
, Anemia , Lupus

[inst1]organization=Inria Paris,city=Paris, postcode=75012, country=France

[inst2]organization=Centre de Recherche des Cordeliers, Inserm, Université
Paris Cité, Sorbonne Université, city=Paris, postcode=75006, country=France

[inst3]organization=Hôpital Necker, Assistance Publique - Hôpitaux de Paris,
city=Paris, postcode=75015, country=France

##  1 Introduction

For complex diagnosis decisions, clinicians customarily refer to diagnosis
guidelines, which define the successive steps one should take to reach a
diagnosis. These documents are typically established by a college of experts
based on the best available evidence [1], and rationalize and normalize
clinical decisions. The succession of steps they recommend includes
information collection, observations, laboratory test orders, and other exams.
However, clinical guidelines suffer several drawbacks. First, they are
designed to cover the majority of the population and hence may fail to guide
to the right diagnosis in the case of uncommon patients such as elders, or
those with multiple diseases. Second, their establishment is long and
expensive, and updates are usually done after several years [2]. This makes
them unsuitable to fast emerging practices such as those associated with a
recently developed laboratory test or an emerging disease, such as the recent
COVID-19 pandemic. Moreover, as clinical guidelines are expensive and time-
consuming to produce, their development does not scale to the full spectrum of
diseases. More versatile and scalable methods are therefore required to
provide insights when clinical guidelines are incomplete, or not available.

We think that machine learning approaches trained on clinical data may
complement diagnosis guidelines. In particular, we aim at developing
approaches able to guide the decision process in a step-by-step manner, as
described in [3]. We believe that such an approach may reduce the number of
irrelevant tests therefore optimizing healthcare costs, but primarily may
propose more personalized and accurate diagnoses, especially in the case of
patients with uncommon conditions.

The collection of patient-level data in Electronic Health Records (EHRs)
offers great opportunities to gain knowledge about clinical practice [4]. EHRs
encompass structured, semi-structured, and unstructured data about patients’
health such as medications, laboratory test orders and results, diagnoses, as
well as demographic features. Previous works relied on EHRs to train machine
learning (ML) methods to automatically suggest diagnoses for patients such as
in [5, 6, 7]. However, in these studies, supervised ML methods are employed to
predict a unique endpoint, i.e., the diagnosis represented as a class label.
We believe that for data-driven approaches to find adoption in clinical
practice, it is important for a diagnosis not to be limited to an endpoint,
but to be represented as a pathway that follows steps of medical reasoning and
decision-making.

In this work, we propose to leverage EHR data and investigate how it can be
used to train deep reinforcement learning (DRL) models to build explainable
pathways supporting the diagnosis of complex conditions. We develop our
approach on two primary use cases: the diagnosis of anemia and Systemic Lupus
Erythematosus (SLE), commonly referred to as Lupus. Anemia is a clinical
condition defined as a lower-than-normal amount of healthy red blood cells in
the body, which we chose for three reasons: its diagnosis is made mainly based
on a series of laboratory tests that are available in most EHRs; it is a
common diagnosis implying that the associated amount of data may be sufficient
to train RL models; and the differential diagnosis of anemia is frequently
complex to establish, making its guidance useful. SLE is a chronic autoimmune
disease where the immune system attacks the body’s health tissues. We selected
SLE because its diagnosis is complex to establish and because the sequence of
steps recommended to reach the diagnosis follows a schema that differs from
anemia. In anemia, recommendations mainly follow the schema of a decision tree
[8], whereas recommendations for SLE diagnosis are based on a weighted
criteria score, covering an alternate decision schema.

We choose to use RL because it builds a model that can be used sequentially,
passing through various steps named _actions_ and _states_ to reach a final
objective state, which is similar to our objective. We propose here an
adaptation of the framework of RL to construct individualized pathways of
observations in a step-by-step manner, in order to suggest a diagnostic
decision. For example, in the anemia use case, a pathway is a sequence of
laboratory test requests (actions), whose results are then obtained (states)
before either requesting for another test or terminating on a diagnosis. We
make the assumption that the constructed pathways can complement clinical
guidelines to aid practitioners in decision-making during the diagnosis
process.

Our main contributions are: (i) an adaptation of the RL framework to
progressively construct optimal sequences of actions to perform in order to
reach a diagnosis and (ii) an empirical analysis on synthetic EHRs that
illustrates the validity of our approach, identifies the most suitable
algorithms, and evaluates their robustness with regard to varying levels of
missing and noisy data.

The rest of the paper is structured as follows. Section 2 presents related
works. Section 3 details the methods we deploy in this paper, including
subsection 3.5 that explains the construction of our synthetic datasets.
Section 4 presents the results of our empirical analysis, evaluating these
methods in various scenarios. Section 5 discusses our findings and their
limitations. Finally, Section 6 concludes the study and provides directions
for future work.

##  2 Related Work

Previously, studies have used various process mining, machine learning and
statistical methods to extract clinical pathways from medical data. Zhang et
al. proposed Markov Chains for the identification of clinical pathways using
patient visits data [9]. Perer et al. extracted common patterns from data in
order to build pathways [10], while Baker et al. used a Markov model to
extract meaningful clinical events from patient data [11]. Others adopted a
statistical approach to build treatment pathways by using a Latent Dirichlet
Allocation-based method [12, 13, 14]. Machine learning methods such as Long
Short-Term Memory and RL have also been used to construct pathways for optimal
treatment in several studies such as in [15] and [16], respectively. However,
their focus is on building pathways for _treatment_ decision, whereas we aim
at building pathways for _diagnosis_.

Likewise, numerous studies have leveraged Machine Learning (ML) methods for
diagnosis. Given the longitudinal nature of EHRs, a prevalent choice has been
Recurrent Neural Networks (RNNs) [5, 7]. Additionally, RNNs have also been
used in the prediction of future patient outcomes such as in [17]. In [18, 19,
20], novel efforts are made to not only employ ML approaches to provide
clinical diagnoses but also to provide explanatory elements to enable the
interpretation and contextualization of the results of the model. However, our
aim is not only to diagnose the patient with the correct condition but also to
find the optimal sequence of features to observe to reach this diagnosis for
each patient, which can be seen, in turn, as elements of explanation for the
decision. In other terms, we aim to construct personalized diagnosis pathways
that delineate the steps leading to the diagnosis, thereby explaining the
diagnostic process.

Previous works used RL methods for costly feature acquisition in
classification tasks [21, 22]. However, these works did not primarily focus on
diagnosis. Yu et al. aimed to optimize the financial cost of several clinical
processes, including the prediction of Acute Kidney Injury, using RL [23].
While it was shown that their method reduced the cost of diagnosis, the
actions taken at each step were not shown, therefore neglecting the relevant
and explainable dimensions of pathways. Other studies used RL to diagnose
patients by inquiring about the presence of disease-related symptoms from
users [24, 25, 26]. While our approach similarly formulates the diagnosis
process as a sequential decision-making problem and applies deep reinforcement
learning (DRL) techniques, these studies use a symptom self-checking approach,
whereas we aim to use EHR data. We believe that using EHRs is more suitable as
they encompass data collected routinely in clinical practice, including
objective and normalized measurements such as laboratory results therefore
composing a great resource for training machine learning models and obtaining
data-driven insights to aid clinical decision-making.

##  3 Methods

###  3.1 Decision Problem

We consider the clinical diagnosis process as a sequential decision-making
problem and formulate it as a Markov Decision Process (MDP) [27], following
the RL [28] paradigm. Accordingly, we define an _agent_ interacting with an
_environment_ in order to maximize a cumulative reward signal. At each time
step $t$, the agent receives an observation $o$ of the _state_ of the
environment, takes an _action_ , and obtains a _reward_. The goal is to learn
a policy, i.e., a function that maps states to actions, that maximizes the
reward signal. In this study, our agents are taking observations from
synthetic EHRs of a Clinical Data Warehouse (CDW), in order to reach a
diagnosis, which is a final action.

Let $\mathcal{D}$ denote a dataset of size $n\times(m+1)$ composed of $n$
patients, $m$ features, plus one diagnosis. $F$ is the set of the $m$ feature
names; and $C$ the set of possible diagnosis values. An instance $D^{i}$ in
$\mathcal{D}$ is a pair $(X^{i},Y^{i})$ with
$X^{i}=\\{x^{i}_{1},\dots,x^{i}_{j},\dots,x^{i}_{m}\\}$ where $x^{i}_{j}$ is
the value of the feature $j$ for the patient $i$, $m$ is the total number of
features and $Y^{i}\in C$ is the anemia diagnosis of patient $i$. Accordingly,
our MDP is defined by the quadruple $(S,A,T,R)$ as follows:

  * 1.

$S$ is the set of states. At each time step, the agent receives an observation
$o$ of the state $s_{t}$, which is a vector of fixed size $m$ comprising the
values of the features that have already been queried by the agent at time
$t$; features that have not been queried yet are associated with the value -1.
Equation 1 defines the $j^{th}$ element of $o$, where $F^{\prime}$ denotes the
set of features that have already been queried by the agent.

$o_{j}=\begin{cases}\;x_{j},&\text{if $f_{j}\in F^{\prime}$}\\\
-1,&\text{otherwise.}\end{cases}$ (1)

  * 2.

$A$ is the set of possible actions, which is the union of the set of _feature
value acquisition actions_ , $A_{f}$ (or feature actions for short), and the
set of _diagnostic actions_ , $A_{d}$. At each time step, the agent takes an
action $a_{t}\in A$. Actions from $A_{f}$ are taking values from the set of
features $F$. Accordingly a specific value $f_{j}\in F$ will trigger the
action of querying for the value of this feature to the CDW. Actions from
$A_{d}$ are taking values from the set of possible diagnoses $C$. At a time
step $t$, if $a_{t}\in A_{d}$, the episode is terminated. Also, for anemia, if
an episode reaches the number of maximum specified steps without reaching a
diagnosis, it is terminated.

  * 3.

$T$ denotes the transition function that gives the probability of moving from
a state $s_{t}$ to a state $s_{t+1}$ given an agent action $a_{t}\in A$.

  * 4.

$R$ is the reward function. $r_{t+1}$, which can be written as
$R(s_{t},a_{t})$, is the immediate reward when an agent in a state $s_{t}$
takes an action $a_{t}$. In the case of a diagnostic action $a_{t}\in A_{d}$,
if the diagnosis is correct, the reward is +1, otherwise the reward is -1.

$\text{if}\ a_{t}\in A_{d},R(s_{t},a_{t})=\begin{cases}\;\;\,1,&\text{if
$a_{t}=Y^{i}$}\\\ -1,&\text{otherwise}\end{cases}$ (2)

Considering the anemia use case, for a feature action $a_{t}\in A_{f}$, if the
feature has already been queried, the agent receives a penalty of -1 and the
episode is terminated. Otherwise, the agent receives a reward of 0.
Accordingly, the reward function for the feature actions is formalized as
follows:

$\text{if}\ a_{t}\in A_{f},R(s_{t},a_{t})=\begin{cases}-1,&\text{if $a_{t}\in
F^{\prime}$}\\\ \;\;\,0,&\text{otherwise}.\end{cases}$ (3)

Considering the lupus use case, for a feature action $a_{t}\in A_{f}$, if the
feature has already been queried, the agent receives a penalty of -1 and the
episode is terminated. Otherwise, the agent incurs a penalty determined by the
penalty weight associated with the action (See Subsection 3.5). The reward
function is:

$\text{if}\ a_{t}\in A_{f},R(s_{t},a_{t})=\begin{cases}-1,&\text{if $a_{t}\in
F^{\prime}$}\\\ \frac{-1}{\lambda c},&\text{otherwise},\end{cases}$ (4)

where $c$ is the penalty weight of the feature action and
$\lambda\in\mathbb{R}$ is a scaling factor.

###  3.2 DQN and its extensions

Q-learning [29] is an RL algorithm that outputs the best action to take in a
given state based on the expected future reward of taking that action in that
particular state. The expected future reward is named the Q-value of that
state-action pair, noted $Q(s,a)$. At each time step, the agent selects an
action following a policy $\pi$, and the goal is to find the optimal policy
$\pi^{*}$ that maximizes the reward function. During model training, the
Q-values are updated using the Bellman Equation as described in equation 8 .

In our use case, since the state space is large and continuous, we propose to
use a Deep Q-Network (DQN) [30] which uses a neural network to approximate the
Q-value function. In order to improve DQN stability and performance, several
extensions of the DQN algorithm have been developed. Particularly Double DQN
(DDQN), Dueling DQN and Prioritized Experience Replay (PER), which we test in
the subsequent sections of this paper and briefly describe in A.1. The
stability gained from these techniques enhances the performance by making the
process faster and reducing overestimation bias.

###  3.3 State-of-the-Art Classifiers

As part of the study, we compared the performance of the DRL models with four
traditional supervised learning algorithms that are commonly used for
classification tasks, namely Decision Tree (DT), Random Forest (RF), Support
Vector Machine (SVM), and a Feed-Forward Neural Network (FFNN). DT has a
particular status in our study for two reasons. First, in the anemia use case,
the labels of the synthetic dataset were assigned according to a decision
tree. For this reason, the DT approach is expected to perform very well on
that data. Second, DT is the only considered classifier that is self-
explanatory, as the path in the DT that is taken to classify an instance
constitutes a pathway to the diagnosis.

###  3.4 Evaluation Approach and Implementation

80% of the dataset was used to train the model while the remaining 20% was
used as the test set. Additionally, 10% of the training dataset (i.e., 8% of
the dataset) was used for validation. Since one aim of this study is to learn
diagnosis pathways for anemia and lupus, and to evaluate how methods are
robust to noise and missing data, the validation and test sets used in all the
experiments are constant (i.e., without noise or missingness).

Ten runs were conducted for the first experiment comparing DQN approaches and
five runs for the subsequent experiments on the models’ robustness. Each run
was performed with the same datasets but with different seeds, which
influenced the models’ interactions with the environment, as well as
initialisation.

The metrics used to measure the performance of the models are:

  * 1.

Accuracy: The ratio of episodes that terminated with a correct diagnosis.

  * 2.

Mean episode length: The average number of actions performed by the agent per
episode. This metric does not apply to the SOTA classifiers that consider all
the features.

  * 3.

F1 score: The harmonic mean of the precision and recall scores. We report one-
vs-rest and macro-averaging F1 score.

  * 4.

ROC-AUC (Receiver Operating Characteristic - Area Under Curve) score: It
measures the model ability to distinguish between the different classes. We
also used a one-vs-rest approach and macro-averaging.

Additional metrics were used in the lupus use case, as described below:

  * 1.

Average pathway score (APS): The average score of a pathway is computed by
summing the penalty weight of each feature leading up to the diagnosis.
Formally,

$s(p)=1-\frac{\sum_{i=1}^{|p|-1}w_{i}}{\sum_{i=1}^{m}w_{i}}$ (5)

it is the score of a pathway $p$ where $w_{i}$ represents the penalty weight
of the $i^{th}$ feature of $p$, $|p|$ is the length of $p$, and $m$ is the
total number of features.

Equation 6 computes the average pathway score for the entire model:

$\bar{s}=\frac{1}{n}\sum_{i=1}^{n}s(p_{i})$ (6)

where $p_{i}$ is the $i^{th}$ pathway generated by the model and $n$ is the
total number of pathways generated.

  * 2.

wPAHM (weighted Pathway score and Accuracy Harmonic Mean): we define wPAHM as
the weighted harmonic mean of the accuracy and average pathway score of a
model. This metric provides a unique value that reflects both the accuracy and
average pathway score, according to their relative importance in a study
context. Formally,

$\text{wPAHM}=\frac{w_{a}+w_{\bar{s}}}{\frac{w_{a}}{a}+\frac{w_{\bar{s}}}{\bar{s}}}$
(7)

where $w_{a}$ and $w_{\bar{s}}$ denote the weights for the accuracy and
average pathway score, respectively. In our case, we consider the accuracy of
the diagnosis of higher priority than the score of our pathways. We therefore
assigned weights of 0.9 and 0.1 to the accuracy ($w_{a}$) and average pathway
score ($w_{\bar{s}}$), respectively.

We used the OpenAI Gym Python library [31] to implement our environment. For
model training, we created an instance of the environment using the training
data. At the start of each episode, the environment was reset using an
instance drawn randomly from the training data. To build our agents, we used
the stable-baselines package [32], and the values of the hyperparameters are
as shown in Table A.1 in A. They were chosen based on prior knowledge,
existing literature, and experimentation. As for the number of timesteps, for
the _anemia_ use case, this was determined in the first run (using a grid-
search strategy) using the validation data and remained constant for the other
nine runs, while for the _lupus_ use case, all the models were trained using a
constant number of timesteps, and a checkpoint for the model was saved at
constant intervals. Using the validation data, the model checkpoint with the
best performance was selected. The hyperparameters for the SOTA models were
chosen using a grid search strategy. The source code is available at
https://github.com/lilly-muyama/Deep_RL_diagnosis_pathways.

###  3.5 Datasets

To experiment with our method we built two synthetic datasets, one for each
use case (i.e., anemia and lupus). The details of the selection of the
features, their values, the labeling of the diagnostic classes, and the
inclusion of various levels of noise and missingness are detailed in B.

The main difference in the construction of the two datasets is the rationale
used to label synthetic patients with their diagnoses. For anemia, we rely on
the DT depicted in Figure 1, whereas, for lupus, we rely on the weighted
criteria score system defined in [33]. Here, the entry criterion for lupus
diagnosis is the presence of Antinuclear antibodies (ANA) at a titer of $\geq$
1:80 on HEp-2 cells. The rest of the features are divided into several
categories and only the highest criteria weight within a category is counted
towards the patient’s total weighted criteria score. If the patient’s criteria
weight is $\geq$ 10, the patient is diagnosed with Lupus. For example, if a
patient is experiencing both delirium (criteria weight=2) and seizure
(criteria weight=5), only 5 will be added to the patient’s total score since
both features belong to the neuropsychiatric category.

Figure 1: The decision tree used to label our anemia dataset, adapted from [8]
and [34].

More Information regarding the datasets’ characteristics and relevant details
about the use cases such as the lupus features’ penalty weights are presented
in C and D respectively.

##  4 Results

###  4.1 Performance Comparison of DQN and its extensions

In the first round of experiments, we trained DQN, Double DQN, Dueling DQN,
and Dueling Double DQN on our datasets; and also enabled prioritized
experience replay (PER) for each of these models. In addition, we trained a
Proximal Policy Optimization (PPO) algorithm using a similar number of
timesteps for comparison purposes. Model names associated with the suffix -PER
correspond to the mentioned DQN extension together with PER. In order to gauge
the stability of each model, we conducted ten different runs of the same
experiment using different seeds, which results are reported in Table 1.

In both use cases, Dueling DQN-PER provided the best accuracy, while Dueling
DDQN-PER exhibited a similar performance with slightly lower accuracy. In
addition, Dueling DDQN-PER had the lowest standard deviation in the anemia use
case and the second lowest in the lupus use case. Remarkably, it also
demonstrated a significantly shorter mean episode length than the Dueling DQN-
PER in the lupus use case. For these reasons, we used only these two DQN
models for the rest of the experiments. It should be noted that there are a
few slight differences in the methodologies used in the two use-cases as seen
in section 3.

Model | Accuracy | Mean episode length
---|---|---
Anemia |  Lupus |  Anemia |  Lupus
PPO |  25.58 $\pm$ 16.54 |  88.06 ± 7.19 |  1.63 ± 0.85 |  3.81 ± 1.63
DQN |  89.31 $\pm$ 14.68 |  88.57 ± 4.89 |  4.27 ± 0.45 |  14.82 ± 3.17
DDQN |  92.79 $\pm$ 5.37 |  90.97 ± 1.56 |  4.66 ± 0.31 |  11.42 ± 2.68
Dueling DQN |  92.46 $\pm$ 5.31 |  97.08 ± 0.72 |  4.70 ± 0.24 |  22.57 ± 1.05
Dueling DDQN |  77.15 $\pm$ 31.99 |  94.20 ± 2.05 |  4.44 ± 1.02 |  16.28 ± 3.75
DQN-PER |  93.71 $\pm$ 5.51 |  94.17 ± 5.18 |  4.67 ± 0.29 |  20.46 ± 2.52
DDQN-PER |  95.57 $\pm$ 2.01 |  94.26 ± 2.79 |  4.92 ± 0.36 |  13.27 ± 2.64
Dueling DQN-PER |  96.64$\pm$ 1.46 |  98.81± 0.32 |  4.59 ± 0.20 |  22.09 ± 0.85
Dueling DDQN-PER |  96.33$\pm$ 1.32 |  98.39 ± 0.41 |  4.82 ± 0.45 |  14.40 ± 2.49
Table 1: Performance of the RL models. Metrics are average and standard
deviation over 10 runs.

###  4.2 Performance Comparison with SOTA classifiers

The results of these experiments are shown in Table 2. For the _anemia_ use
case, since the diagnosis labels of the synthetic dataset were assigned
following the decision tree in Figure 1, the tree-based agent, which is based
on the same tree achieved a perfect score. This tree was not applicable to the
_lupus_ use case. Also, we built an agent that acts randomly at each timestep
for comparison purposes. Because the SOTA classifiers use a constant set of
features to make a diagnosis, they do not have a mean episode length. The
tree-based algorithms and the FFNN performed better than the DQN models, while
SVM had a similar performance in the _lupus_ use case but had a lower
performance in the _anemia_ use case.

Model | Accuracy | Mean episode length
---|---|---
Anemia |  Lupus |  Anemia |  Lupus
Random Agent |  12.34 $\pm$ 0.33 |  17.69 ± 0.38 |  1.53 ± 0.01 |  4.75 ± 0.02
Tree-based Agent |  100 $\pm$ 0.00 |  N/A |  3.98 ± 0.00 |  N/A
Decision Tree |  99.96 $\pm$ 0.00 |  99.42 ± 0.01 |  N/A |  N/A
Random Forest |  99.90 $\pm$ 0.01 |  99.38 ± 0.02 |  N/A |  N/A
FFNN |  97.97 $\pm$ 0.27 |  99.92± 0.03 |  N/A |  N/A
SVM |  94.89 $\pm$ 0.00 |  99.54 ± 0.00 |  N/A |  N/A
Dueling DQN-PER |  96.64 $\pm$ 1.46 |  98.81 ± 0.32 |  4.59 ± 0.20 |  22.09 ± 0.85
Dueling DDQN-PER |  96.33 $\pm$ 1.32 |  98.39 ± 0.41 |  4.82 ± 0.45 |  14.40 ± 2.49
Table 2: Performance of the DQN and the state-of-the-art classifiers. The
tree-based agent has a perfect score because it acts according to the decision
tree used to build the _anemia_ dataset.

###  4.3 Performance comparison using different metrics

For the _lupus_ use case, since accuracy is not the only desirable metric, we
also selected the models with the best performance according to an alternate
metric, i.e. the wPAHM in Equation 7. The comparison results between the
models with the best accuracy vs wPAHM scores are shown in Table 3.

Model |  Goal metric |  Acc. |  Avg. ep. length |  F1 |  ROC-AUC |  APS |  wPAHM
---|---|---|---|---|---|---|---
Dueling DQN-PER |  Accuracy |  98.81 |  22.09 |  98.82 |  98.82 |  10.24 |  50.98
wPAHM |  95.83 |  17.30 |  95.82 |  95.83 |  30.06 |  77.93
Dueling DDQN-PER |  Accuracy |  98.39 |  14.40 |  98.39 |  98.39 |  43.15 |  86.18
wPAHM |  96.41 |  9.56 |  96.42 |  96.42 |  63.64 |  91.59
Table 3: The mean performance of the best-performing DQN models for the
_lupus_ dataset, based on a chosen goal metric (accuracy or wPAHM); with
$\lambda$=9 in the reward function.

###  4.4 Results with different values of $\lambda$

In the reward function for the feature value acquisition actions in the
_lupus_ use case (shown in Equation 4), the $\lambda$ acts as a scaling
factor. We performed experiments with different values for $\lambda$, and the
mean accuracy and wPAHM scores are illustrated in Figures
LABEL:sub@varying_beta_acc and LABEL:sub@varying_beta_wpahm respectively. As
shown, with the increase in the $\lambda$ value, the accuracy score rises
while the wPAHM score declines.

(a)

(b)

Figure 2: Graphs showing the effect of the $\lambda$ value in the reward
function on LABEL:sub@varying_beta_acc the accuracy and
LABEL:sub@varying_beta_wpahm the wPAHM scores for the dueling DQN-PER model.

Additionally, Figure 3 illustrates the trade-off between the accuracy and the
average pathway score for different values of $\lambda$.

Figure 3: The accuracy-average pathway score tradeoff based on different
values of $\lambda$ in the reward function. The results for the dueling DQN-
PER model are used here.

###  4.5 Varying levels of missing data and noise

Figures LABEL:sub@anem_missingness and Figure LABEL:sub@lupus_missingness
depict the mean accuracy as the level of missing data in the _anemia_ and
_lupus_ training datasets changes respectively. In particular, for lupus,
before model training, the missing values were imputed using KNN imputer [35]
with the number of neighbors set to one. All models exhibited a decline in
performance, with SVM demonstrating the speediest deterioration in both use
cases. The DT and FFNN had the best performance for _anemia_ and _lupus_
respectively.

(a)

(b)

(c)

(d)

Figure 4: Accuracy of approaches with varying levels of missingness, noisiness
and train set size for the _anemia_ dataset. LABEL:sub@anem_missingness shows
the mean accuracy of the models at different missingness levels;
LABEL:sub@anem_noisiness shows the mean accuracy of the models at different
noisiness levels; LABEL:sub@anem_missingness_noisiness shows the mean accuracy
of the models at a constant noisiness level (0.2) and different missingness
levels; LABEL:sub@anem_varying_sizes shows the mean accuracy and the 95%
confidence interval of the models based on the size of the train set. The
y-axis of all the plots starts at 60 to make the performance difference
clearer.

(a)

(b)

(c)

(d)

Figure 5: Accuracy of approaches with varying levels of missingness,
noisiness, and train set size, with the _lupus_ dataset. The graphs show the
mean accuracy of the models at different LABEL:sub@lupus_missingness
missingness levels; LABEL:sub@lupus_noisiness noisiness levels;
LABEL:sub@lupus_missingness_noisiness missingness levels at a constant
noisiness level (0.2). LABEL:sub@lupus_varying_sizes shows the mean accuracy
and the 95% confidence interval of the models as a function of the size of the
train set. The y-axis of all the plots starts at 60 to make the performance
difference clearer.

Figures LABEL:sub@anem_noisiness and LABEL:sub@lupus_noisiness show the mean
accuracy as the level of noise in both training sets increases. For anemia,
SVM showed the sharpest decline, while the tree-based models (DT and RF) still
had the best performance. On the other hand, for lupus, the FFNN had the best
performance, with the DQN models outperforming the other models.

Figures LABEL:sub@anem_missingness_noisiness and
LABEL:sub@lupus_missingness_noisiness depict the varying mean accuracy as the
level of missingness increases but with a fixed level of noise of 0.2. Here,
again, for the _lupus_ use case, the missing values were imputed as described
earlier. The DQN models has a better/steadier performance than all the models
for anemia, while they had a slightly lower performance than the FFNN for
lupus, but had a better performance than the rest of the models. In the lupus
use case, the tree-based models (DT and RF) had the worst performance, while
SVM nad the worst for the anemia use case followed by DT.

###  4.6 Varying the size of train sets

In order to understand whether this method can be applied to the diagnosis of
conditions where data may be scarce, such as with rare diseases, we conducted
experiments with train sets of different sizes. For each size, a fraction of
the train set was randomly selected and removed, leaving the remaining data to
form the new set. Runs were repeated five times for each size of the train
set, whereby a different subset was removed each time. The Dueling DDQN-PER
model was used for these experiments and for each run, the same number of
timesteps was used. Figures LABEL:sub@anem_varying_sizes and
LABEL:sub@lupus_varying_sizes show the mean accuracy for each train set size
for the two use cases. The shaded region represents the 95% confidence
interval. The RF classifier is also depicted in the figure for comparison
purposes and it was chosen for its stability and high performance. As
depicted, the DQN model was able to learn the clinical pathways and diagnose
the patients. However, for all train set sizes, its performance was lower than
the RF for the _anemia_ use case but was higher for the _lupus_ use case.
Additionally, as the size of the train sets decreased, the model’s performance
became less stable.

##  5 Discussion

Our study illustrates that DRL models trained on EHRs are a promising solution
to suggest optimal, individual, and step-by-step pathways to diagnosis. In
comparing our two best DQN models to state-of-the-art classifiers, while the
latter performed extremely well on perfect data, the two DQN models performed
comparably and sometimes better on imperfect data in both use cases (i.e.,
anemia and lupus), showing their robustness. Furthermore, using different
training sizes, the DQN Models showed their capacity to learn diagnostic
decision pathways associated with decent performance, even in the context of
reasonably small-sized training datasets (e.g about 600 patients).

###  5.1 Generated Pathways

Besides quantitative evaluation, the pathways constructed by DQN models
present two advantages. The fact that they are composed of progressive
observations that lead to a diagnostic decision, makes them explainable; one
may understand why a particular diagnosis is reached or why a new exam is
ordered by looking at the sequence of features selected and their associated
values, in a way that is similar to clinician reasoning. We note that the DT
algorithms are also intrinsically explainable, but our results show that they
are less robust to imperfect data.

In addition, the set of pathways generated by a test set can be aggregated to
generate a data structure somewhat similar to diagnosis guidelines. An
illustration of such an aggregation is provided in Figure 6 in the form of a
Sankey diagram. It shows the pathways learned by the model for the diagnosis
of _Anemia of Chronic Disease_ and _Aplastic anemia_ , which are colored
_coral_ and _blue_ respectively. A side-by-side illustration of some of the
learned pathways and a branch of the decision tree used to label the dataset
is also shown in Figure C.3.

Figure 6: The diagnostic decision pathways for _Anemia of Chronic Disease
(ACD)_ and _Aplastic anemia_ as learned by the agent.

Regarding the lupus use case, Figure 7 illustrates the possible aggregations,
showcasing the combination of the three commonest pathways for the _No lupus_
class. These pathways were generated by one of our experiments: the dueling
DDQN model chosen based on the best wPAHM score, with $\lambda$ set to 9. More
pathways are shown in Figures D.1, D.2 and D.3 in D.

Figure 7: The three commonest diagnostic decision pathways for the _No lupus_
class generated by one of the models with the best wPAHM score. The model used
to generate the pathways was the one with the wPAHM score closest to the mean
wPAHM of the ten runs.

Other aggregations of pathways can be visualized and interacted with at
https://lilly-muyama.github.io/. These provide visualization facilities and
additional information about the support of nodes and edges, as well as
descriptive statistics (e.g., mean, max, min values) of the various features.

###  5.2 DQN Performance

Dueling DQN and Double DQN both improve the accuracy and stability of the
standard DQN as observed in Table 1. Additionally, note that for each of the
four algorithms discussed above, combining them with Prioritized Experience
Replay resulted in better results both in terms of mean and SD of the
accuracy. This is explained by the fact that in DQN, Double DQN, and Dueling
DQN, during training, the experiences are sampled uniformly from the replay
buffer. However, PER prioritizes these experiences such that those with a
higher priority are sampled more often. In a task such as ours, this type of
DQN favors transitions with a non-zero reward, which includes the diagnosis
actions, on which the accuracy is primarily based. Sampling these actions more
frequently leads the model to learn better and faster about them.

###  5.3 Quality of diagnoses

For the anemia use case, regarding the comparison with SOTA classifiers, since
we built our dataset using a decision tree, it was expected that the tree-
based classifiers would perform extremely well. The performance of the two DQN
Models was slightly lower than those of the tree-based methods and the neural
network, while SVM had slightly lower performance.

Table 4 displays the classification report of one particular experiment with
the Dueling DQN-PER model, with an accuracy of 97.19, which is the closest
accuracy to the mean. The report shows that most of the anemia classes
exhibited a decent performance. However, while the _No anemia_ class had the
best recall with a perfect score of 1, it also had the lowest precision at
0.92, meaning that several instances were being diagnosed as _No anemia_ ,
whereas they should not, based on their laboratory test results. Upon further
inspection, we noted that all the misdiagnosed instances had values near the
threshold value. In the _No anemia_ misclassifications, all the hemoglobin
levels were close to the 13 and 12 g/dl thresholds for men and women,
respectively (see Figure 1). In particular, the hemoglobin of men had a mean
of 12.98 and a standard devisation of 0.01, while the one of women is 11.93
with SD=0.04. However, it should be noted that from a clinical point of view,
hemoglobin levels below but close to the normal threshold usually do not lead
to an anemia diagnosis. In this regard, the decision tree of a guideline may
lead to more false negatives than our approach. Similarly, looking at
_Hemolytic anemia_ , which had the lowest recall score at 0.94 (along with
_Aplastic anemia_), 35 of its instances were diagnosed with _Vitamin
B12/Folate deficiency anemia_. These instances had a mean MCV value of 99.77
(SD=0.17), thus close to the threshold of 100. The 11 other instances
diagnosed with _Anemia of chronic disease_ had a mean MCV level of 80.20
(SD=0.15), close to the threshold of 80. And the other 30 instances with an
_Inconclusive diagnosis_ all had values near the thresholds, but missing
values for other features in the pathway leading to an inconclusive diagnosis.

Class Name (support) | Precision | Recall | F1-Score
---|---|---|---
Inconclusive | 0.94 | 0.97 | 0.95
diagnosis (1344)
Aplastic | 1.00 | 0.94 | 0.97
anemia (1806)
Hemolytic | 1.00 | 0.94 | 0.97
anemia (1805)
Iron deficiency | 0.98 | 0.98 | 0.98
anemia (1679)
Anemia of chronic | 0.99 | 0.97 | 0.98
disease (1772)
Unspecified | 1.00 | 0.98 | 0.99
anemia (1793)
Vitamin B12/Folate | 0.96 | 0.98 | 0.97
defic. anemia (1801)
No anemia (2000) | 0.92 | 1.00 | 0.96
Table 4: Classification report showing the detailed performance of the Dueling
DQN-PER model. Classes of anemia diagnosis are reported with their respective
support.

Regarding the lupus use case, for a positive SLE diagnosis to be made, the
patient has to have a weighted criteria score of at least 10 (See [33] for
details about how this total is calculated). However, as reflected by the
metrics, there were a few misdiagnosed episodes. For instance, the model whose
pathways are presented in Figures 7 and D.1, had an accuracy and a wPAHM score
of 96.57 and 91.65 respectively. However, on closer examination, it was found
that for this particular model, there were 130 _Lupus_ cases that were
misdiagnosed as _No lupus_. Subsequent analysis showed that this was primarily
due to the pathways being terminated earlier than they should have. That is to
say, some of the features, which were positive and would have led to a
weighted criteria score greater than the threshold (10), were not tested. For
example, the commonest pathway in these misdiagnosed episodes has a length of
15. Although these features are all key (the pathway still has an accuracy of
97.73%), there are cases where the remaining nine features had values that
would have changed the diagnosis.

Moreover, for the patients misdiagnosed as _Lupus_ instead of _No lupus_ , the
model made the diagnosis too early when the weighted criteria score was near
the threshold. For example, for this particular model, the commonest pathway
for these misdiagnosed episodes was “ANA=1 –> Anti-dsDNA antibody=1 –> Lupus”.
The weighted criteria score of this pathway is less than 10 and yet a Lupus
diagnosis was given. While the presence of anti-dsDNA antibody has one of the
highest weights, it is not sufficient for a lupus diagnosis on its own
according to [33]. Additionally, the second commonest pathway for these
misdiagnosed episodes had a length of 23, missing only two features. On
examining the features that were not selected by the model, it was revealed
that one (renal biopsy-proven lupus nephritis) had the lowest penalty weight,
while the other (acute pericarditis) had the lowest prevalence in the dataset.

It should be noted that the above error analysis concerns a model based on the
best wPAHM score. The model with the best accuracy score had longer pathways,
hence fewer misdiagnosed episodes. Some of its pathways are shown in Figures
D.2 and D.3 in D.

###  5.4 Quality of lupus pathways

When the value of $\lambda$ increases, the penalty for selecting the various
actions decreases, hence the pathways become longer. While this leads to a
higher diagnosis accuracy because the agent has more information to make the
diagnosis, it also leads to a reduction in the quality of the diagnosis
pathways, since actions that are not fully necessary may be taken. On the
other hand, if $\lambda$ is too low, this leads to shorter pathways leading to
a diagnosis being given before all the necessary actions have been conducted.
Figure 8 shows the effect of the value of $\lambda$ has on the mean episode
length for the dueling DDQN-PER model.

Figure 8: Mean episode length vs. $\lambda$ for the Dueling DDQN-PER model.

Additionally, when comparing the performance of the Dueling DDQN-PER models
based on a single goal metric, i.e., by looking at the models with the best
accuracy or wPAHM as shown in Table 3, we see that with accuracy, the model
has a lower average pathway score, a lower wPAHM score and a longer mean
episode length (i.e., longer pathways), while with wPAHM as goal metric, we
observe a lower accuracy, but a higher average pathway score and shorter
episode lengths.

###  5.5 DQN Model Robustness

The DQN models show robustness to both noise and missing data, maintaining a
consistent performance across various levels of increasing noise and
missingness in both use cases. We believe that this robustness can be
attributed to the reliance of the DQN on neural networks, which have been
shown to be robust to noise in many scenarios [36]. This is because adding
noise to the training input can act as a form of regularization by preventing
overfitting and leading to better generalization of the neural network.
Likewise, neural networks have also demonstrated their capability to handle
missing data as they are able to learn the underlying representation of the
data even with missing values.

For example, in the anemia use case, we remarked that introducing noise and
missingness led to decreased performance of the _Inconclusive diagnosis_ and
_Iron deficiency anemia_ classes (results not shown). This is first attributed
to the fact that some of the instances with noise or missingness are being
diagnosed with _Inconclusive diagnosis_ instead of their anemia class, which
is to be expected. In the real world, clinicians may need to use their own
judgment in such cases. Second, regarding _Iron Deficiency Anemia_
misclassifications, those are almost all _Anemia of Chronic Disease_ instances
that were falsely attributed to _Iron Deficiency Anemia_ , which are two
classes difficult to discriminate, as illustrated by their location on the
same branch of the decision tree (see Figure 1).

We also note that in the _lupus_ use case, the Feed-Forward neural network has
the best performance in all the various scenarios that were tested, followed
closely by the DQN models, which performed better than all the tree-based
models and SVM in all the scenarios with imperfect data. This can be
attributed to the fact that the anemia dataset was labeled using a decision
tree hence the very good performance of the tree-based models. However, since
this is not the case with the lupus dataset, the neural network-based models
performed better for that use case.

We also observe that while the DT performs very well in both use cases on the
perfect data, its performance decreases when both noise and missing data are
present, becoming significantly lower than that of the other models. This can
be attributed to decision tree sensitivity to noise and missing data. In
addition, decision trees are prone to overfitting, such that a slight change
in the training dataset may lead to a different tree altogether.
Alternatively, Random Forest builds multiple decision trees during their
training process and is therefore less likely to overfit than the DT, making
it more impervious to noise and missing data.

Finally, using the same hyperparameters for the model, the DQN model was able
to interestingly perform consistently as the training set size gradually
decreased. The reason for this is the same hyperparameters were used for the
experiments therefore a smaller dataset just meant that the same experiences
were sampled more often. However, there is still a loss in the information
gained from the data, and therefore the smaller the dataset, the lower the
accuracy and the more varied the results, as shown in Figures
LABEL:sub@anem_varying_sizes and LABEL:sub@lupus_varying_sizes. In the anemia
use case, the tree-based model (RF) performed better than the DQN Model, while
the DQN model performed better than RF in the lupus use case. Again, this can
be attributed to the different way diagnoses are made (i.e., decision tree-
vs. weighted criteria-based approach) in the two use cases.

Since we experimented on two use cases with different diagnosis processes,
i.e. decision tree and weighted criteria system, and we were able to achieve a
good performance with both, we are therefore confident that our approach can
generalize to other diagnoses that follow a similar diagnostic process as that
of anemia or lupus.

###  5.6 Differences in use cases

For exploration and experimentation purposes, we handled missing data
differently in the two use cases. For lupus, we used KNN imputation. However,
this was not considered necessary for the anemia use case with the presence of
the _inconclusive diagnosis_. Both methods ensured a diagnosis was given by
the model even in the presence of missing data and we achieved good results
with both methods making them both relevant to this work.

Additionally, for the reward function, we chose not to include a step penalty
in the reward function of the anemia use case. Despite that, the model was
able to learn the relevant features to test for each anemia class. However,
because the diagnostic decision for lupus is more complex, it was relevant to
add a penalty for each step taken by the agent in order to avoid overlong
pathways.

Furthermore, we only applied penalty weights and the additional performance
metrics (average pathway score, wPAHM score) to the lupus use case and not
anemia because the lupus features were more and extremely diverse. All the
anemia features were laboratory test results derived from a blood sample,
while lupus features comprised physical symptoms such as fever, invasive
procedures such as a renal biopsy, laboratory tests such as low C3, etc.

Finally, we only added a maximum length constraint to the anemia dataset. This
was dependent on the use case. Since the anemia diagnosis is reached via a
decision tree, a maximum length could be interpreted as the maximum depth of
the decision tree. Also, the anemia dataset included extra features not
related to the anemia diagnosis. Therefore, in no case in the anemia use case
is a pathway with all the features included needed to make a diagnosis. In
contrast, for lupus, all the features in the dataset are relevant for the
diagnosis, and in some cases, depending on the feature values, the agent may
need to select all the features in the dataset before reaching a diagnosis.
That said, experiments without the maximum length constraint for the anemia
use case showed that the models were still able to learn the pathways with
similar pathway lengths. Additionally, experiments with the lupus use case
with the maximum length set to the number of features plus one (for the
diagnosis) exhibited similar performance to the experiments without the
maximum length whose results are shown in this work.

It is therefore important to note that our approach was adapted to each use
case, guided by empirical results, to obtain optimal performance. Therefore,
similar adjustments may be required for other use cases to ensure its
effectiveness.

###  5.7 Limitations

One significant limitation of this study is the absence of experiments on a
real-world dataset to evaluate our method. While the results from the
synthetic datasets are very encouraging, and while we endeavor to reproduce
imperfections of EHR data in our study, it is paramount that we test on real-
world data. Additionally, the features that make up the pathways need to be
available in the EHRs, which is not always the case. Furthermore, the
synthetic datasets used are not longitudinal. This initial study with anemia
and lupus aimed to show the viability of the proposed method. Real-world EHR
datasets often have a temporal dimension which will provide more insights into
the progression of the pathways. With the availability of such data, we will
be able to compare our approach with models capable of handling sequential
data, such as Recurrent Neural Networks.

Moreover, training the DRL agents took a significantly longer time and used
considerably more computing resources than the SOTA classifiers as shown in
Tables D.6 and C.3 in the Appendices. Also, the DQN model has many
hyperparameters that would need to be further optimized. However, it should be
noted that after model training, generating a diagnosis pathway for a test
instance is trivial since the policy has already been learned by the model
(see Tables D.6 and C.3 in the Appendices).

Furthermore, there were slight differences in the methodology and performance
evaluation strategy used in the two use cases. While we thought it was
important to illustrate the results with these differences for research
purposes, it should also be noted that they may account for some of the minor
differences in the performance between the two use cases.

##  6 Conclusion

In this work, we adapted the RL framework to propose step-by-step guidance
toward clinical diagnosis from synthetic EHRs. We particularly demonstrated
the capacity of DRL to produce accurate diagnosis in the context of imperfect
data (i.e., with noise, missingness), and variable dataset sizes; and an
explainable diagnosis with the generation of decision pathways.

Our approach with DQN is suitable as it has a comparable performance with the
SOTA classifiers with the added advantage of constructing personalized
pathways for each patient. These pathways have the potential to guide the
diagnosis of a new patient or to be aggregated to summarize possible pathways
for an uncommon patient population that might not be covered by guidelines in
place. In this manner, this approach has the potential to suggest enrichments
for clinical guidelines, learned from practice data. In future work, we aim to
evaluate our approach on real-world EHRs, and to extend the type of considered
input to multimodal data. Additionally, we will also consider the use of Large
Language Model-based methods.

## CRediT authorship contribution statement

Lillian Muyama: Data Curation, Methodology, Software, Investigation,
Validation, Visualization, Writing - original draft, Writing - Review &
Editing. Antoine Neuraz: Conceptualization, Methodology, Validation,
Resources, Writing - Review & Editing, Supervision, Project Administration,
Funding acquisition. Adrien Coulet: Conceptualization, Methodology,
Validation, Resources, Writing - Review & Editing, Supervision, Project
Administration, Funding acquisition.

## Declaration of Competing Interests

The authors declare that they have no known competing financial interests or
personal relationships that could have appeared to influence the work reported
in this paper.

## Acknowledgements

This work is supported by the Inria CORDI-S Ph.D. program, and benefited from
a government grant managed by the Agence Nationale de la Recherche under the
France 2030 program, reference ANR-22-PESN-0007 ShareFAIR, and
ANR-22-PESN-0008 NEUROVASC.

## References

  * [1] M. J. Field, K. N. Lohr, et al., Clinical practice guidelines, Directions for a new program (1990) 1990.
  * [2] E. Steinberg, S. Greenfield, D. M. Wolman, M. Mancher, R. Graham, et al., Clinical practice guidelines we can trust, National Academies Press, 2011.
  * [3] J. Adler-Milstein, J. H. Chen, G. Dhaliwal, Next-Generation Artificial Intelligence for Diagnosis: From Predicting Diagnostic Labels to “Wayfinding”, JAMA 326 (24) (2021) 2467–2468.
  * [4] P. B. Jensen, L. J. Jensen, S. Brunak, Mining electronic health records: towards better research applications and clinical care, Nature Reviews Genetics 13 (6) (2012) 395–405.
  * [5] Z. C. Lipton, D. C. Kale, C. Elkan, R. Wetzel, Learning to diagnose with LSTM recurrent neural networks, arXiv preprint arXiv:1511.03677 (2015).
  * [6] R. Miotto, L. Li, B. A. Kidd, J. T. Dudley, Deep patient: an unsupervised representation to predict the future of patients from the electronic health records, Scientific Reports 6 (1) (2016) 1–10.
  * [7] E. Choi, M. T. Bahadori, A. Schuetz, W. F. Stewart, J. Sun, Doctor AI: Predicting clinical events via recurrent neural networks, in: Machine learning for healthcare conference, PMLR, 2016, pp. 301–318.
  * [8] R. Zaiden, Evaluation of anemia, BMJ Best Practicehttps://bestpractice.bmj.com/topics/en-us/93/diagnosis-approach Accessed: 2022-09-08 (2022).
  * [9] Y. Zhang, R. Padman, N. Patel, Paving the COWpath: Learning and visualizing clinical pathways from electronic health record data, Journal of Biomedical Informatics 58 (2015) 186–197.
  * [10] A. Perer, F. Wang, J. Hu, Mining and exploring care pathways from electronic medical records with visual analytics, Journal of Biomedical Informatics 56 (2015) 369–378.
  * [11] K. Baker, E. Dunwoodie, R. G. Jones, A. Newsham, O. Johnson, C. P. Price, J. Wolstenholme, J. Leal, P. McGinley, C. Twelves, et al., Process mining routinely collected electronic health records to define real-life clinical pathways during chemotherapy, International Journal of Medical Informatics 103 (2017) 32–41.
  * [12] Z. Huang, X. Lu, H. Duan, Latent treatment pattern discovery for clinical processes, Journal of Medical Systems 37 (2) (2013) 1–10.
  * [13] Z. Huang, W. Dong, L. Ji, C. Gan, X. Lu, H. Duan, Discovery of clinical pathway patterns from event logs using probabilistic topic models, Journal of Bomedical Informatics 47 (2014) 39–57.
  * [14] Z. Huang, Z. Ge, W. Dong, K. He, H. Duan, Probabilistic modeling personalized treatment pathways using electronic health records, Journal of Biomedical Informatics 86 (2018) 33–48.
  * [15] X. Lin, Y. Li, Y. Xu, W. Guo, W. He, H. Zhang, L. Cui, C. Miao, Personalized clinical pathway recommendation via attention based pre-training, in: 2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), IEEE, 2021, pp. 980–987.
  * [16] T. Li, Z. Wang, W. Lu, Q. Zhang, D. Li, Electronic health records based reinforcement learning for treatment optimizing, Information Systems 104 (2022) 101878.
  * [17] H. Koshimizu, R. Kojima, K. Kario, Y. Okuno, Prediction of blood pressure variability using deep neural networks, International Journal of Medical Informatics 136 (2020) 104067.
  * [18] G. Obaido, B. Ogbuokiri, T. G. Swart, N. Ayawei, S. M. Kasongo, K. Aruleba, I. D. Mienye, I. Aruleba, W. Chukwu, F. Osaye, et al., An interpretable machine learning approach for Hepatitis B diagnosis, Applied Sciences 12 (21) (2022) 11127.
  * [19] Y. Zoabi, S. Deri-Rozov, N. Shomron, Machine learning-based prediction of COVID-19 diagnosis based on symptoms, npj Digital Medicine 4 (1) (2021) 3.
  * [20] R. Kavya, J. Christopher, S. Panda, Y. B. Lazarus, Machine learning and XAI approaches for allergy diagnosis, Biomedical Signal Processing and Control 69 (2021) 102681.
  * [21] Y. Li, J. Oliva, Active feature acquisition with generative surrogate models, in: International Conference on Machine Learning, PMLR, 2021, pp. 6450–6459.
  * [22] J. Janisch, T. Pevnỳ, V. Lisỳ, Classification with costly features using deep reinforcement learning, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33, 2019, pp. 3959–3966.
  * [23] Z. Yu, Y. Li, J. Kim, K. Huang, Y. Luo, M. Wang, Deep reinforcement learning for cost-effective medical diagnosis, arXiv preprint arXiv:2302.10261 (2023).
  * [24] K.-F. Tang, H.-C. Kao, C.-N. Chou, E. Y. Chang, Inquire and diagnose: Neural symptom checking ensemble using deep reinforcement learning, in: NIPS Workshop on Deep Reinforcement Learning, 2016.
  * [25] Z. Wei, Q. Liu, B. Peng, H. Tou, T. Chen, X.-J. Huang, K.-F. Wong, X. Dai, Task-oriented dialogue system for automatic diagnosis, in: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 2018, pp. 201–207.
  * [26] H.-C. Kao, K.-F. Tang, E. Chang, Context-aware symptom checking for disease diagnosis using hierarchical reinforcement learning, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32, 2018.
  * [27] M. Littman, Markov decision processes, in: N. J. Smelser, P. B. Baltes (Eds.), International Encyclopedia of the Social & Behavioral Sciences, Pergamon, Oxford, 2001, pp. 9240–9242.
  * [28] R. S. Sutton, A. G. Barto, Reinforcement learning: An introduction, MIT press, 2018.
  * [29] C. J. Watkins, P. Dayan, Q-learning, Machine learning 8 (3) (1992) 279–292.
  * [30] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al., Human-level control through deep reinforcement learning, Nature 518 (7540) (2015) 529–533.
  * [31] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, W. Zaremba, OpenAI Gym, arXiv preprint arXiv:1606.01540 (2016).
  * [32] A. Hill, A. Raffin, M. Ernestus, A. Gleave, A. Kanervisto, R. Traore, P. Dhariwal, C. Hesse, O. Klimov, A. Nichol, M. Plappert, A. Radford, J. Schulman, S. Sidor, Y. Wu, Stable Baselines, https://github.com/hill-a/stable-baselines (2018).
  * [33] M. Aringer, K. Costenbader, D. Daikh, R. Brinks, M. Mosca, R. Ramsey-Goldman, J. S. Smolen, D. Wofsy, D. T. Boumpas, D. L. Kamen, et al., 2019 European League Against Rheumatism/American College of Rheumatology classification criteria for Systemic Lupus Erythematosus, Arthritis & Rheumatology 71 (9) (2019) 1400–1412.
  * [34] M. W. Short, J. E. Domagalski, Iron deficiency anemia: evaluation and management, American Family Physician 87 (2) (2013) 98–104.
  * [35] O. Troyanskaya, M. Cantor, G. Sherlock, P. Brown, T. Hastie, R. Tibshirani, D. Botstein, R. B. Altman, Missing value estimation methods for dna microarrays, Bioinformatics 17 (6) (2001) 520–525.
  * [36] I. Goodfellow, Y. Bengio, A. Courville, Deep learning, MIT press, 2016.
  * [37] H. Van Hasselt, A. Guez, D. Silver, Deep reinforcement learning with double Q-learning, in: Proceedings of the AAAI conference on artificial intelligence, Vol. 30, 2016.
  * [38] Z. Wang, T. Schaul, M. Hessel, H. Hasselt, M. Lanctot, N. Freitas, Dueling network architectures for deep reinforcement learning, in: International conference on machine learning, PMLR, 2016, pp. 1995–2003.
  * [39] T. Schaul, J. Quan, I. Antonoglou, D. Silver, Prioritized experience replay, arXiv preprint arXiv:1511.05952 (2015).
  * [40] M. Koperska, Hematocrit/Hemoglobin ratio calculator, https://www.omnicalculator.com/health/hct-hgb, accessed: 2022-11-22 (2023).
  * [41] D. Nedea, Transferrin saturation calculator, https://www.mdapp.co/transferrin-saturation-calculator-444/, accessed: 2022-11-24 (2020).
  * [42] F. Naeim, P. Nagesh Rao, S. X. Song, W. W. Grody, 61 - disorders of red blood cells—anemias, in: F. Naeim, P. Nagesh Rao, S. X. Song, W. W. Grody (Eds.), Atlas of Hematopathology, Academic Press, 2013, pp. 675–704.
  * [43] H. Timlin, A. Syed, U. Haque, B. Adler, G. Law, K. Machireddy, R. Manno, Fevers in adult lupus patients, Cureus 10 (1) (2018).
  * [44] A. Fayyaz, A. Igoe, B. T. Kurien, D. Danda, J. A. James, H. A. Stafford, R. H. Scofield, Haematological manifestations of lupus, Lupus Science & Medicine 2 (1) (2015) e000078.
  * [45] N. Galanopoulos, A. Christoforidou, Z. Bezirgiannidou, Lupus thrombocytopenia: pathogenesis and therapeutic implications, Mediterranean Journal of Rheumatology 28 (1) (2017) 20–26.
  * [46] L. R. Crawford, N. Neparidze, Refractory autoimmune hemolytic anemia in a Systemic Lupus Erythematosus patient: A clinical case report, Clinical Case Reports 10 (3) (2022) e05583.
  * [47] R. B. Nayak, G. S. Bhogale, N. M. Patil, S. S. Chate, Psychosis in patients with Systemic Lupus Erythematosus, Indian Journal of Psychological Medicine 34 (1) (2012) 90–93.
  * [48] K. Desai, M. Miteva, Recent insight on the management of Lupus Erythematosus Alopecia, Clinical, Cosmetic and Investigational Dermatology (2021) 333–347.
  * [49] M. Kudsi, L. D. Nahas, R. Alsawah, A. Hamsho, A. Omar, The prevalence of oral mucosal lesions and related factors in Systemic Lupus Erythematosus patients, Arthritis Research & Therapy 23 (2021) 1–5.
  * [50] C. M. Grönhagen, F. Nyberg, Cutaneous lupus erythematosus: An update, Indian Dermatology Online Journal 5 (1) (2014) 7.
  * [51] T. T. Provost, The relationship between discoid and Systemic Lupus Erythematosus, Archives of Dermatology 130 (10) (1994) 1308–1310.
  * [52] X. Yao, M. Abd Hamid, A. Sundaralingam, A. Evans, R. Karthikappallil, T. Dong, N. M. Rahman, N. I. Kanellakis, Clinical perspective and practices on pleural effusions in chronic systemic inflammatory diseases, Breathe 16 (4) (2020).
  * [53] S. Almousa, H. Wannous, K. Khedr, H. Qasem, Unusual case presentation of Systemic Lupus Erythematosus in a young woman, Rheumato 2 (4) (2022) 93–97.
  * [54] V. K. Narang, J. Bowen, O. Masarweh, S. Burnette, M. Valdez, L. Moosavi, F. Joolhar, T. T. Win, Acute pericarditis leading to a diagnosis of SLE: a case series of 3 patients, Journal of Investigative Medicine High Impact Case Reports 10 (2022) 23247096221077832.
  * [55] F. Ceccarelli, C. Perricone, E. Cipriano, L. Massaro, F. Natalucci, G. Capalbo, I. Leccese, D. Bogdanos, F. R. Spinelli, C. Alessandri, et al., Joint involvement in Systemic Lupus Erythematosus: from pathogenesis to clinical assessment, in: Seminars in Arthritis and Rheumatism, Vol. 47, Elsevier, 2017, pp. 53–64.
  * [56] N. Wiegley, Low-grade proteinuria in patients with Systemic Lupus Erythematosus, Kidney News 14 (10/11) (2022) 63 – 63.
  * [57] W. Hong, Y.-L. Ren, J. Chang, G. Luo, S. Ling-Yun, A systematic review and meta-analysis of prevalence of biopsy-proven lupus nephritis, Archives of Rheumatology 33 (1) (2018) 17.
  * [58] O. Ünlü, S. Zuily, D. Erkan, The clinical significance of antiphospholipid antibodies in Systemic Lupus Erythematosus, European Journal of Rheumatology 3 (2) (2016) 75.
  * [59] B. Dema, N. Charles, Autoantibodies in SLE: specificities, isotypes and receptors, Antibodies 5 (1) (2016) 2.
  * [60] M. Ramos-Casals, M. Campoamor, A. Chamorro, G. Salvador, S. Segura, J. Botero, J. Yagüe, R. Cervera, M. Ingelmo, J. Font, Hypocomplementemia in Systemic Lupus Erythematosus and primary antiphospholipid syndrome: prevalence and clinical significance in 667 patients, Lupus 13 (10) (2004) 777–783.
  * [61] C. Fabrizio, C. Fulvia, P. Carlo, M. Laura, M. Elisa, M. Francesca, S. Francesca Romana, T. Simona, A. Cristiano, V. Guido, et al., Systemic Lupus Erythematosus with and without anti-dsDNA antibodies: analysis from a large monocentric cohort, Mediators of Inflammation 2015 (2015).
  * [62] M. Arroyo-Ávila, Y. Santiago-Casas, G. McGwin, R. S. Cantor, M. Petri, R. Ramsey-Goldman, J. D. Reveille, R. P. Kimberly, G. S. Alarcón, L. M. Vilá, et al., Clinical associations of anti-Smith antibodies in PROFILE: a multi-ethnic lupus cohort, Clinical Rheumatology 34 (2015) 1217–1223.

##  Appendix A Deep-Q Network (DQN)

###  A.1 DQN and its extensions

Q-learning [29] is an RL algorithm that outputs the best action to take in a
given state based on the expected future reward of taking that action in that
particular state. The expected future reward is named the Q-value of that
state-action pair, noted $Q(s,a)$. At each time step, the agent selects an
action following a policy $\pi$, and the goal is to find the optimal policy
$\pi^{*}$ that maximizes the reward function. During model training, the
Q-values are updated using the Bellman Equation as follows:

$Q_{t+1}(s_{t},a_{t})\leftarrow Q_{t}(s_{t},a_{t})+\alpha[r_{t+1}+\gamma\
\underset{a}{\max}\ Q_{t}(s_{t+1},a)-Q_{t}(s_{t},a_{t})]$ (8)

where $\alpha$ is the learning rate and $\gamma$ is the discount factor that
determines the importance of future reward relative to immediate reward.

In our use case, since the problem has a large state space, we propose to use
a Deep Q-Network [30] which uses a neural network to approximate the Q-value
function. A DQN comprises two networks of similar architecture, i.e., the
policy network, which interacts with the environment and learns the optimal
policy, and the target network, which is used to define the target Q-value. At
each time step, the policy network takes a state, $s_{t}$, as its input and
outputs the Q-values for taking the different actions in that state. The
weights of the target network are frozen and updated at a specified interval
by copying the weights of the policy network. The DQN algorithm learns by
minimizing the loss function shown in Equation 9, where $\theta$ and
$\theta^{-}$ represent the weights of the policy and target networks,
respectively. Additionally, at each time step, a record of the model’s
interaction with the environment (known as an experience) is stored in a
memory buffer in the form $(s_{t},a_{t},r_{t+1},s_{t+1})$.

$L(\theta)=\mathop{\mathbb{E}}[(r_{t+1}+\gamma\ \underset{a}{\max}\
Q(s_{t+1},a,\theta^{-})-Q(s_{t},a_{t},\theta))^{2}]$ (9)

In order to improve DQN stability and performance, several extensions of the
DQN algorithm have been developed which we use in this paper and we briefly
describe below:

Double DQN (DDQN) [37]: The action is selected using the policy network, while
the target network estimates the value of that action unlike in the standard
DQN where the same network is used for both tasks. The loss function is thus
modified as follows:

$L(\theta)=\mathop{\mathbb{E}}[(r_{t+1}+\gamma\
Q(s_{t+1},\underset{a}{\arg\max}\
Q(s_{t+1},a,\theta),\theta^{-})-Q(s_{t},a_{t},\theta))^{2}]$ (10)

Dueling DQN [38]: The Q-value function is split into two parts: a value
function $V(s)$ that provides the value for being in that state, and an
advantage function $A(s,a)$ that gives the advantage of the action $a$ in the
state $s$, as compared to the other actions. The two functions are then
combined to get the Q values as shown in Equation 11.

$Q(s,a)=V(s)+(A(s,a)-\frac{1}{|\mathcal{A}|}\underset{a}{\sum}A(s,a))$ (11)

where $|\mathcal{A}|$ is the total number of possible actions in that state.

Prioritized Experience Replay (PER) [39]: Each experience in the buffer is
assigned a priority such that experiences that have a higher priority are
sampled more often during training.

###  A.2 Model Hyperparameters

Table A.1 shows the values of the hyperparameters used for the DQN model in
this study.

Hyperparameter | Value
---|---
Buffer size | 1000000
Learning rate | 0.0001
Target network update frequency | 10000
Learning starts | 50000
Final epsilon value | 0.05
Discount factor | 0.99
Train frequency | 4
Table A.1: DQN hyperparameter values.

##  Appendix B Dataset Construction

To experiment with our method, we built two synthetic datasets, one for each
use case.

###  B.1 Anemia Dataset

####  B.1.1 Feature Inclusion

The first step of our dataset construction was the definition of a set of 17
features, associated or not, with anemia diagnosis. The hemoglobin level is
the primary feature that is considered to determine whether a patient has
anemia, therefore was included. Since the normal level of hemoglobin varies
between men and women, gender was included as well. We also included features
from a decision tree of the standard guidelines for the diagnosis of anemia by
Zaïden [8]: mean corpuscular volume (MCV), ferritin, reticulocyte count,
segmented neutrophils and Total Iron Binding Capacity (TIBC).

Additionally, we included features that are not in the tree, but can be used
to diagnose anemia, according to discussions with a domain expert: hematocrit,
transferrin saturation (TSAT), red blood cells (RBC), serum iron, and folate.
Interestingly for our study, three of these features (hematocrit, TSAT and
RBC) can be derived from other features and are accordingly correlated with
features from our initial selection (e.g., hemoglobin and hematocrit).
Furthermore, we added features that are not relevant to the diagnosis of
anemia to observe their potential impact on the behavior of our model:
creatinine, cholesterol, copper, ethanol, and glucose.

####  B.1.2 Dataset Construction

The second step of the dataset construction is the instantiation of features
with values associated with one of 7 diagnosis classes: _No anemia_ , _Vitamin
B12/Folate deficiency anemia_ , _Unspecified anemia_ , _Anemia of chronic
disease (ACD)_ , _Iron deficiency anemia (IDA)_ , _Hemolytic anemia_ , and
_Aplastic anemia_. Each anemia class dataset was built based on the decision
tree represented in Figure 1, which was manually constructed based on [8, 34].

For each class, the values of the features were generated using a uniform
probability distribution whose parameters (min and max values), were
determined through manual reviewing of the medical literature and thresholds
of the decision tree in Figure 1. Features that are correlated with others
were derived using known equations, that is, hematocrit [40], TSAT [41] and
RBC [42]. 10,000 instances were created for each of the diagnosis classes,
which were in turn combined to create a single dataset of 70,000 instances.
Next, an eighth diagnosis class, _Inconclusive diagnosis_ was created for
cases when the model is unsure about the diagnosis of an instance. The
_Inconclusive diagnosis_ instances were created out of the existing 70,000
instances by randomly selecting and removing 10% of the non-missing values of
each feature, except for hemoglobin, gender and MCV, which are necessary to
the diagnosis of almost all the anemia classes. In C, the class distribution
is illustrated in Figure C.1; the ratio of observed vs. missing values for
each feature is shown in Figure C.2; an example of an instance from the
dataset is provided Table C.1; and the summarized statistics of the dataset
are shown in Table C.2.

####  B.1.3 Simulating Imperfect Data

To compare the robustness of various approaches to imperfect data, we
artificially introduced different levels of noise and missingness to our
training data.

A percentage of the values of each feature, excluding hemoglobin and gender,
were randomly replaced with missing values. Hemoglobin and gender were
excepted because a patient’s hemoglobin level is key to the diagnosis of
anemia and the normal levels of hemoglobin vary between men and women.

Since our original dataset perfectly follows a decision tree, we simulated
noisiness using the following procedure: For each anemia class, except _No
anemia_ and _Inconclusive diagnosis_ , a specified fraction of the values in
each of the features in its branch of the tree in Figure 1 were replaced by
another value generated from a normal distribution, $N(\mu,\sigma^{2})$, where
the mean, $\mu$, is the threshold in a node with that feature in the tree, and
the standard deviation, $\sigma$, was defined by us arbitrarily.

For example, using hemolytic anemia, the features in its tree branch, that is,
the features used to diagnose a patient with it are _hemoglobin_ , _MCV_ and
_reticulocyte count_ as shown in Figure 1. Therefore, for a noise level of
0.2, we replaced 20% of the reticulocyte count values using a normal
distribution function $N(2,0.2)$ where $\mu$ value is picked following the
threshold in the decision tree. Similarly, for MCV, 10% of its feature values
were replaced using a normal distribution $N(80,2)$, while another 10% was
replaced using a normal distribution function $N(100,2)$. We gradually
increased the noise level fraction in order to analyze its effect on the
performance of our model and the pathways it creates. Additionally, for all
the noise levels, 10% of the anemic instances were randomly labeled as No
anemia.

We further created datasets with both noisy and missing data. Using a training
dataset with a fixed noisiness level of 0.2 as described above, we added
missing data at different levels to this dataset using the same procedure used
to create the datasets with missing data.

###  B.2 Lupus Dataset

####  B.2.1 Feature Inclusion

To construct the dataset, we generated features based on the clinical practice
guideline outlined in [33] for the classification of Systemic Lupus
Erythematosus (SLE). In adherence to this guideline, we identified a total of
24 features, namely, antinuclear antibodies (ANA), fever, leukopenia,
thrombocytopenia, autoimmune hemolysis, delirium, psychosis, seizure, non-
scarring alopecia, oral ulcers, cutaneous lupus, pleural effusion, pericardial
effusion, acute pericarditis, joint involvement, proteinuria, renal biopsy-
proven lupus nephritis, anti-cardiolipin antibodies, anti-$\beta$2GP1
antibodies, lupus anticoagulant, low C3, low C4, anti-dsDNA antibody, and
anti-Smith antibody.

This guideline describes a weighted score using these features to determine
whether an individual has SLE or not. Consequently, we used the same criteria
to label our synthetic instances, establishing two classes, i.e., _Lupus_ and
_No lupus_.

####  B.2.2 Dataset Construction

With reference to [33], we consider the presence of ANA at a titer $\geq$ 1:80
on HEP-2 cells as an entry criterion (referred to as the ANA feature for
clarity) for SLE diagnosis, and accordingly we constructed two datasets. In
one, instances were labeled as 1 (True) for the ANA feature, indicating a
positive ANA result; in the second dataset, instances were labeled as 0
(False), indicating a negative ANA result. Specifically, we generated 50,000
instances with a positive ANA and 20,000 instances with a negative ANA.

In both datasets, all features are binary indicating whether a specific
feature is present (1) or absent (0) in a given instance; except for two
features: cutaneous lupus and renal biopsy-proven lupus nephritis that are
categorical, taking on one of four values (0 for False, 1 for subacute
cutaneous lupus, 2 for acute cutaneous lupus, 3 for discoid lupus) for
cutaneous lupus, and one of six values (0 for False, 1 for class I, 2 for
class II, 3 for class III, 4 for class IV, 5 for class V) for renal biopsy-
proven lupus nephritis, respectively.

In the positive ANA dataset, the prevalence of positive features was
determined from literature sources (see Table D.3 in D). Conversely, for the
negative ANA dataset, the prevalence of features was derived considering their
relative importance in lupus diagnosis, as indicated by their weight in the
diagnosis criteria. Features with larger weights were considered less
prevalent in patients, following the logic that their rarity contributes to
their significance in the diagnosis process.

The features were populated using a weighted probability distribution, taking
into account the prevalence levels or weighted criteria as previously
outlined. Subsequently, the two datasets were labeled based on the weighted
criteria specified in [33], then merged to create a unified dataset of 70,000
instances and 24 features, where 35,056 and 34,944 instances belonged to the
No lupus and Lupus classes, respectively. Table D.4 provides an example of an
instance. Table D.5 presents descriptive statistics for the dataset.

####  B.2.3 Simulating Imperfect Data

A percentage of values for each feature, excluding ANA, were randomly replaced
with missing values. ANA was excluded because, as mentioned earlier, it serves
as the entry criterion for SLE diagnosis therefore without this feature, a
patient should not be diagnosed with lupus according to [33].

Additionally, we introduced noisiness into the dataset through the following
method. For each level of noisiness, a specified fraction of instances in the
dataset was randomly selected, and their labels were changed to the opposite
class. For instance, at a noisiness level of 0.2, the diagnoses of 20% of the
instances in the training set were altered; those initially diagnosed with
Lupus were changed to No lupus, and vice versa. Consequently, the training
dataset no longer perfectly adhered to the diagnosis criteria outlined in
[33].

Furthermore, we generated datasets incorporating both noisy and missing data.
Starting with a training dataset featuring a noisiness level fixed at 0.2, we
introduced missing data at various levels using the same procedure employed in
creating datasets with missing values. This approach allowed us to explore the
impact of both noise and missing information on the dataset simultaneously.

##  Appendix C Anemia Use Case

###  C.1 Dataset Description

Figure C.1 shows the number of instances per diagnosis class in the synthetic
dataset described in section B.1.2. Figure C.2 shows the ratio of missing
versus observed values for each feature in the dataset. Additionally, Table
C.1 shows an example of a synthetic patient instance and Table C.2 shows a
summarized statistical description of the features in the dataset.

Figure C.1: Number of patients per anemia class. Figure C.2: Number of present vs missing values for each feature in the _anemia_ dataset. Feature | Value
---|---
Hemoglobin | 9.007012
Ferritin | -
Reticulocyte count | -
Segmented neutrophils | 3.519565
TIBC | 440.499323
MCV | 103.442762
Serum iron | 59.017997
RBC | 2.612173
Gender | Male
Creatinine | 0.650757
Cholestrol | 114.794964
Copper | 112.308159
Ethanol | 25.612786
Folate | 5.969710
Glucose | 116.026042
Hematocrit | 27.021037
TSAT | 13.397977
label | Vitamin B12/Folate deficiency anemia
Table C.1: An instance in the _anemia_ dataset.

Feature | All Classes | No anemia |  Vitamin B12/Folate deficiency anemia
---|---|---|---
Hemoglobin | 10.239 (8.067, 12.102) | 14.570 (13.281, 15.848) | 9.510 (7.771, 11.274)
Ferritin | 209.968 (69.870, 343.337) | 251.436 (128.145, 373.501) | 251.775 (126.676, 374.486)
Reticulocyte count | 2.821 (1.270, 4.342) | 2.981 (1.520, 4.423) | 3.024 (1.526, 4.564)
Segmented neutrophils | 2.930 (0.762, 4.898) | 3.532 (1.824, 5.266) | 3.516 (1.791, 5.22)
TIBC | 334.276 (222.108, 457.942) | 310.821 (206.744, 415.958) | 306.678 (202.380, 410.634)
MCV | 89.998 (78.918, 101.093) | 90.029 (82.631, 97.598) | 102.504 (101.225, 103.794)
Serum iron | 135.030 (77.793, 192.645) | 134.959 (76.979, 192.764) | 136.613 (80.087, 193.071)
RBC | 3.348 (2.641, 3.936) | 4.899 (4.379, 5.356) | 2.784 (2.274, 3.298)
Gender |  |  |
Male | 38268 (54.67%) | 4048 (40.48%) | 5108 (56.73%)
Female | 31732 (45.33%) | 5952 (59.52%) | 3896 (43.27%)
Creatinine | 1.103 (0.651, 1.552) | 1.119 (0.670, 1.566) | 1.102 (0.655, 1.540)
Cholestrol | 74.878 (37.388, 112.244) | 74.037 (36.257, 111.019) | 75.020 (37.099, 112.819)
Copper | 80.095 (55.182, 105.245) | 79.510 (54.965, 104.816) | 80.001 (55.014, 105.150)
Ethanol | 39.887 (19.876, 59.749) | 40.256 (19.769, 60.865) | 39.445 (18.622, 59.590)
Folate | 15.262 (7.832, 22.715) | 15.054 (7.843, 22.174) | 15.400 (8.099, 22.976)
Glucose | 90.039 (65.128, 115.077) | 90.918 (66.287, 115.892) | 90.021 (64.290, 116.194)
Hematocrit | 30.716 (24.201, 36.306) | 43.709 (39.843, 47.544) | 28.530 (23.313, 33.821)
TSAT | 49.601 (23.103, 62.608) | 52.553 (24.626, 67.235) | 53.935 (25.712, 68.534)

Feature | Unspecified anemia |  Anemia of chronic disease | Iron deficiency anemia
---|---|---|---
Hemoglobin | 9.534 (7.769, 11.301) | 9.514 (7.751, 11.254) | 9.539 (7.795, 11.276)
Ferritin | 250.757 (123.567, 375.763) | 268.551 (152.371, 384.475) | 48.654 (22.742, 74.091)
Reticulocyte count | 3.006 (1.546, 4.459) | 2.957 (1.458, 4.457) | 2.975 (1.441, 4.502)
Segmented neutrophils | 0.000 (0.000, 0.000) | 3.580 (1.848, 5.284) | 3.582 (1.854, 5.330)
TIBC | 311.332 (208.894, 414.291) | 301.558 (199.155, 402.335) | 452.223 (458.074, 499.101)
MCV | 102.525 (101.274, 103.774) | 77.472 (76.206, 194.049) | 77.527 (76.299, 78.796)
Serum iron | 134.970 (77.499, 192.216) | 135.313 (77.093, 194.049) | 135.625 (77.915, 193.272)
RBC | 2.790 (2.274, 3.305) | 3.685 (3.008, 4.361) | 3.693 (3.013, 4.371)
Gender |  |  |
Male | 5175 (57.73%) | 5067 (57.20%) | 4765 (56.74%)
Female | 3789 (42.27%) | 3792 (42.8%) | 3633 (43.26%)
Creatinine | 1.096 (0.643, 1.554) | 1.098 (0.643, 1.544) | 1.103 (0.643, 1.558)
Cholestrol | 75.734 (38.838, 113.730) | 74.994 (37.212, 112.230) | 74.700 (37.507, 112.034)
Copper | 80.465 (56.729, 105.069) | 80.081 (54.461, 104.850) | 79.764 (54.906, 104.545)
Ethanol | 39.500 (19.102, 22.784) | 39.784 (19.802, 59.703) | 39.501 (20.018, 58.701)
Folate | 15.284 (7.919, 22.784) | 15.131 (7.769, 22.650) | 15.462 (7.948, 22.976)
Glucose | 89.457(65.322, 113.808) | 90.131 (66.091, 115.144) | 90.153 (65.518, 114.407)
Hematocrit | 28.602 (23.307, 33.902) | 28.541 (23.253, 33.761) | 28.617 (23.386, 33.828)
TSAT | 52.689 (24.323, 67.385) | 54.538 (24.730, 71.416) | 32.675 (17.377, 42.591)

Feature | Hemolytic anemia | Aplastic anemia |  Inconclusive diagnosis
---|---|---|---
Hemoglobin | 9.510 (7.741, 11.262) | 9.518 (7.801, 11.250) | 9.486 (7.764, 11.224)
Ferritin | 251.082 (123.988, 380.350) | 246.611 (119.831, 374.303) | 195.158 (67.362, 319.364)
Reticulocyte count | 4.049 (3.079, 5.007) | 1.005 (0.514, 1.500) | 2.988 (1.437, 4.514)
Segmented neutrophils | 3.553 (1.820, 5.271) | 3.517 (1.789, 5.189) | 3.478 (1.745, 5.214)
TIBC | 310.197 (203.256, 418.237) | 310.744 (205.529, 414.594) | 338.282 (225.384, 461.215)
MCV | 89.944 (84.920, 94.938) | 89.985 (85.003, 95.013) | 88.696 (78.100, 100.800)
Serum iron | 133.068 (76.467, 191.047) | 135.668 (79.215, 192.992) | 133.704 (76.167, 191.807)
RBC | 3.185 (2.580, 3.761) | 3.186 (2.596, 3.747) | 3.257 (2.628, 3.809)
Gender |  |  |
Male | 5166 (57.24%) | 5084 (56.31%) | 3855 (57.36%)
Female | 3859 (42.76%) | 3945 (43.69%) | 2866 (42.64%)
Creatinine | 1.098 (0.656, 1.538) | 1.100 (0.646, 1.551) | 1.108 (0.646, 1.563)
Cholestrol | 74.968 (37.216, 112.418) | 74.990 (37.919, 112.314) | 74.602 (37.416, 111.479)
Copper | 80.348 (55.201, 105.698) | 80.502 (55.372, 106.089) | 80.166 (55.087, 105.649)
Ethanol | 40.220 (20.430, 60.389) | 40.525 (20.545, 60.523) | 39.744 (21.113, 58.165)
Folate | 15.396 (7.854, 22.746) | 15.309 (7.731, 22.812) | 15.045 (7.521, 22.613)
Glucose | 90.140 (65.137, 115.555) | 89.852 (64.575, 114.951) | 89.348 (63.492, 113.911)
Hematocrit | 28.530 (23.223, 33.785) | 28.554 (23.402, 33.751) | 28.458 (23.292, 33.671)
TSAT | 53.800 (24.902, 69.688) | 53.292 (26.181, 69.297) | 48.753 (21.476, 63.018)

Table C.2: Summary descriptive statistics of the _anemia_ dataset showing the
mean and interquartile range (in parentheses) of the features. Gender, which
is a binary variable, is described using the sample number and the percentage.

###  C.2 Sample Learned Pathways

Figures C.3 provides another example of pathways generated with our approach.
It displays a side-by-side illustration of a branch of the decision tree used
to label the dataset and a Sankey diagram showing the pathways learned by the
Dueling DQN-PER model for the diagnosis of solely _No anemia_ , _Unspecified
anemia_ and _Hemolytic anemia_ , which are colored _blue_ , _coral_ and _light
green_ , respectively. The feature actions are represented by the orange
nodes, while the diagnosis actions are the dark green nodes. In addition, the
size of each flow corresponds to its support, i.e., the number of patients
that have it in their pathway.

(a)

(b)

Figure C.3: A side-by-side illustration of (a) a branch of the decision tree
in Figure 1 and (b) the corresponding pathways learned by the model for _No
anemia_ , _Hemolytic anemia_ and _Unspecified anemia_.

###  C.3 Computing Time

Model |  Training time |  Testing time
---|---|---
Decision Tree |  299 ms ± 9.72 ms |  18.6 µs ± 453 ns
Random Forest |  5.85 s ± 8.56 ms |  3.27 ms ± 156 µs
SVM |  25.8 s ± 437 ms |  166 µs ± 8.33 µs
FFNN |  2min 36s ± 35 s |  645 µs ± 28.6 µs
Dueling DQN-PER |  1h 42 min 14s ± 19 min 19s |  722 µs ± 35.1 µs
Dueling DDQN-PER |  2h 12min 45s ± 7min 36s |  759 µs ± 56.1 µs
Table C.3: The computing time to train the model and to generate a diagnosis
(pathway) for a single _anemia_ test instance.

In Table C.3, the time taken to train each model (training time), as well as,
the time taken to generate a diagnosis pathway or a diagnosis (testing time)
for the DQN models and the SOTA models respectively for a single instance in
the test dataset are shown.

##  Appendix D Lupus Use Case

###  D.1 Determining Penalty Weights

Table D.1 shows the penalty weights of the different features that were used
to determine the penalty incurred by the agent on selecting that action as
used in the reward function in 4. The scores were calculated based on the
level of invasiveness of the action, the turnaround time for its observation,
and the financial cost of performing that action. These values were derived
from a review of existing literature and online sources. A breakdown of how
the scores of each criterion were generated is shown in Table D.2.

Feature | Invasiveness | Turnaround time | Financial cost | Weighted score ($c$)
---|---|---|---|---
ANA | 4 | 3 | 4 | 11.5
Fever | 5 | 5 | 5 | 15
Leukopenia | 4 | 4 | 4 | 12
Thrombocytopenia | 4 | 4 | 4 | 12
Autoimmune hemolysis | 4 | 4 | 4 | 12
Delirium | 5 | 5 | 5 | 15
Psychosis | 5 | 5 | 5 | 15
Seizure | 5 | 5 | 5 | 15
Non-scarring alopecia | 5 | 5 | 5 | 15
Oral ulcers | 5 | 5 | 5 | 15
Cutaneous lupus | 2 | 2 | 3 | 6.5
Pleural effusion | 5 | 3 | 3 | 13
Pericardial effusion | 5 | 3 | 3 | 13
Acute pericarditis | 5 | 3 | 3 | 13
Joint involvement | 5 | 3 | 3 | 13
Proteinuria | 5 | 3 | 4 | 13.5
Renal biopsy-proven lupus nephritis | 0 | 2 | 0 | 1
Anti-cardiolipin antibodies | 4 | 1 | 3 | 10
Anti-$\beta$2GP1 antibodies | 4 | 0 | 3 | 9.5
Lupus anti-coagulant | 4 | 0 | 3 | 9.5
Low C3 | 4 | 1 | 4 | 10.5
Low C4 | 4 | 1 | 4 | 10.5
Anti-dsDNA antibody | 4 | 3 | 4 | 11.5
Anti-Smith antibody | 4 | 3 | 4 | 11.5

Table D.1: The penalty weights of the different features. The final score for each feature is a weighted number calculated by $2\times\text{invasiveness}+0.5\times\text{turnaround time}+0.5\times\text{financial cost}$. A breakdown of how the penalty weights for each criterion were generated is shown in Table D.2. Rating | Invasiveness | Turnaround time | Financial cost
---|---|---|---
5 | none | none | none
4 | very low | within 1 day | < USD 50
3 | low | 1 - 3 days | USD 50 - 300
2 | medium | 3 - 7 days | USD 301 - 700
1 | high | 7 - 14 days | USD 701 - 1000
0 | very high | > 2 weeks | > USD 1000
Table D.2: The values used to determine the penalty weights.

###  D.2 Dataset Description

Table D.3 shows the prevalence of the different features in patients with SLE
as well as the literature source from which they were determined. An instance
in the dataset is shown in Table D.4, while a summary statistical description
of the dataset is shown in Table D.5.

Feature name |  Literature source |  Percentage from literature |  Percentage used in dataset creation for SLE positive patients
---|---|---|---
ANA |  [33] |  100% |  100%
Fever |  [43] |  36 - 86% |  18%
Leukopenia |  [44] |  50 - 60% |  25%
Thrombocytopenia |  [45] |  20 - 40% |  10%
Autoimmune hemolysis |  [46] |  3% |  1.5%
Delirium |  [47] |  49.33% |  24.5%
Psychosis |  [47] |  12% |  6%
Seizure |  [47] |  10.67% |  5.5%
Non-scarring alopecia |  [48] |  85% |  42.5%
Oral ulcers |  [49] |  8 - 45% |  4%
Subacute cutaneous lupus |  [50] |  10% |  5%
Acute cutaneous lupus |  [50] |  3% |  1.5%
Discoid lupus |  [51] |  15 - 20% |  7.5%
Pleural effusion |  [52] |  17 - 60% |  8.5%
Pericardial effusion |  [53] |  50% |  25%
Acute pericarditis |  [54] |  1% |  0.5%
Joint involvement |  [55] |  69 - 95% |  34.5%
Proteinuria |  [56] |  60% |  30%
Renal biopsy-proven lupus nephritis |  [57] |  16.9% |  8.5%
Anti-cardiolipin antibodies |  [58] |  17 - 40% |  8.5%
Anti-$\beta$2GP1 antibodies |  [59] |  10 - 35% |  5%
Lupus anticoagulant |  [58] |  11 - 30% |  5.5%
Low C3 |  [60] |  45% |  22.5%
Low C4 |  [60] |  44% |  22%
Anti-dsDNA antibody |  [61] |  70 - 98% |  35%
Anti-Smith antibody |  [62] |  9 - 49% |  4.5%
Table D.3: The prevalence of the various features in SLE-positive patients in the _lupus_ dataset. The dataset utilized half the minimum percentage from the literature source to prevent a situation where nearly every positive ANA patient is also diagnosed with lupus.  Feature | Value
---|---
ANA | 1
Fever | 0
Leukopenia | 0
Thrombocytopenia | 1
Autoimmune hemolysis | 0
Delirium | 1
Psychosis | 0
Seizure | 0
Non-scarring alopecia | 0
Oral ulcers | 0
Cutaneous lupus | 0
Pleural effusion | 0
Pericardial effusion | 0
Acute pericarditis | 0
Joint involvement | 0
Proteinuria | 1
Renal biopsy-proven lupus nephritis | 0
Anti-cardiolipin antibodies | 0
Anti-$\beta$2GP1 antibodies | 0
Lupus anticoagulant | 0
Low C3 | 0
Low C4 | 0
Anti-dsDNA antibody | 0
Anti-Smith antibody | 0
Label | Lupus
Table D.4: An instance in the _Lupus_ dataset.

Feature | All Classes | No lupus | Lupus
---|---|---|---
ANA |  0 - 20000 (28.6%) 1 - 70000(71.4%) |  0 - 20000 (57.1%) 1 - 15056(42.9%) |  0 - 0 (0%) 1 - 34944(100%)
Fever |  0 - 60274 (86.1%) 1 - 9726(13.9%) |  0 - 32363 (92.3%) 1 - 2693 (7.7%) |  0 - 27911 (79.9%) 1 - 7033(20.1%)
Leukopenia |  0 - 56776 (81.1%) 1 - 13224 (18.9%) |  0 - 31929 (91.1%) 1 - 3127 (8.9%) |  0 - 24847 (71.1%) 1 - 10097 (28.9%)
Thrombocytopenia |  0 - 64388 (92.0%) 1 - 5612 (8%) |  0 - 33712 (96.2%) 1 - 1344 (3.8%) |  0 - 30676 (87.8%) 1 - 4268 (12.2%)
Autoimmune hemolysis |  0 - 68600 (98%) 1 - 1400 (2%) |  0 - 34304 (97.9%) 1 - 752 (2.1%) |  0 - 34296 (98.1%) 1 - 648 (1.9%)
Delirium |  0 - 56898 (81.3%) 1 - 13102 (18.7%) |  0 - 31521 (89.9%) 1 - 3535 (10.1%) |  0 - 25377 (72.6%) 1 - (27.4%)
Psychosis |  0 - 66335 (94.8%) 1 - 3665 (5.2%) |  0 - 33840 (96.5%) 1 - 1216 (3.5%) |  0 - 32495 (93.0%) 1 - 2449 (7.0%)
Seizure |  0 - 66762 (95.4%) 1 - 3238 (4.6%) |  0 - 34277 (97.8%) 1 - 779 (2.2%) |  0 - 32485 (93.0%) 1 - (7.0%)
Non-scarring alopecia |  0 - 48040 (68.6%) 1 - 21960 (31.4%) |  0 - 29220 (83.4%) 1 - 5836 (16.6%) |  0 - 18820 (53.9%) 1 - 16124 (46.1%)
Oral ulcers |  0 - 67247 (96.1%) 1 - 2753 (3.9%) |  0 - 33792 (96.4%) 1 - 1264 (3.6%) |  0 - 33455 (95.7%) 1 - 1489 (4.3%)
Cutaneous lupus |  0 - 61370 (87.7%) 1 - 3143 (4.5%) 2 - 1111 (1.6%) 3 - 4376 (6.3%) |  0 - 32408 (92.4%) 1 - 1003 (2.9%) 2 - 447 (1.3%) 3 - 1198 (3.4%) |  0 - 28962 (82.9%) 1 - 2140 (6.1%) 2 - 664 (1.9%) 3 - 3178 (9.1%)
Pleural effusion |  0 - 65212 (93.2%) 1 - 4788 (6.8%) |  0 - 34043 (97.1%) 1 - 1013 (2.9%) |  0 - 31169 (89.2%) 1 - 3775 (10.8%)
Pericardial effusion |  0 - 57098 (81.6%) 1 - 12902 (18.4%) |  0 - 33075 (94.3%) 1 - 1981 (5.7%) |  0 - 24023 (68.7%) 1 - 10921 (31.3%)
Acute pericarditis |  0 - 69328 (99.0%) 1 - 672 (1.0%) |  0 - 34633 (98.8%) 1 - 423 (1.2%) |  0 - 34695 (99.3%) 1 - 249 (0.7%)
Joint involvement |  0 - 52195 (74.6%) 1 - 17805 (25.4%) |  0 - 33248 (94.8%) 1 - 1808 (5.2%) |  0 - 18947 (54.2%) 1 - 15997 (45.8%)
Proteinuria |  0 - 54516 (77.9%) 1 - 15484 (22.1%) |  0 - 32138 (91.7%) 1 - 2918 (8.3%) |  0 - 22378 (64.0%) 1 - 12566 (36.0%)
Renal biopsy-proven lupus nephritis |  0 - 64148 (91.6%) 1 - 1306 (1.9%) 2 - 954 (1.4%) 3 - 1089 (1.6%) 4 - 1829 (2.6%) 5 - 674 (1.0%) |  0 - 33352 (95.1%) 1 - 1126 (3.2%) 2 - 211 (0.6%) 3 - 95 (0.3%) 4 - 83 (0.2%) 5 - 189 (0.5%) |  0 - 30796 (88.1%) 1 - 180 (0.5%) 2 - 743 (2.1%) 3 - 994 (2.8%) 4 - 1746 (5.0%) 5 - 485 (1.4%)
Anti-cardiolipin antibodies |  0 - 64890 (92.7%) 1 - 5110 (7.3%) |  0 - 33395 (95.3%) 1 - 1661 (4.7%) |  0 - 31495 (90.1%) 1 - 3449(9.9%)
Anti-$\beta$2GP1 antibodies |  0 - 66581 (95.1%) 1 - 3419 (4.9%) |  0 - 33703 (96.1%) 1 - 1353 (3.9%) |  0 - 32878 (94.1%) 1 - 2066 (5.9%)
Lupus anticoagulant |  0 - 66424 (94.9%) 1 - 3576 (5.1%) |  0 - 33640 (96.0%) 1 - 1416 (4.0%) |  0 - 32784 (93.8%) 1 - 2160 (6.2%)
Low C3 |  0 - 57962 (82.8%) 1 - 12038 (17.2%) |  0 - 32112 (91.6%) 1 - 2944 (8.4%) |  0 - 25850 (74.0%) 1 - 9094 (26.0%)
Low C4 |  0 - 58221 (83.2%) 1 - 11779 (16.8%) |  0 - 32179 (91.8%) 1 - 2877 (8.2%) |  0 - 26042 (74.5%) 1 - 8902 (25.5%)
Anti-dsDNA antibody |  0 - 52168 (74.5%) 1 - 17832 (25.5%) |  0 - 33107 (94.4%) 1 - 1949 (5.6%) |  0 - 19061 (54.5%) 1 - 15883 (45.5%)
Anti-Smith antibody |  0 - 67399 (96.3%) 1 - 2601 (3.7%) |  0 - 34439 (98.2%) 1 - 617 (1.8%) |  0 - 32960 (94.3%) 1 - 1984 (5.7%)

Table D.5: Summary statistics of the _Lupus_ dataset showing the sample number
and percentage of each value for each feature.

###  D.3 Sample Learned Pathways

Figure D.1 shows the three commonest pathways for the _Lupus_ class generated
by one of our experiments: the dueling DDQN model chosen based on the wPAHM
score.

Figure D.1: The three commonest diagnostic decision pathways for the _Lupus_
class generated by one of the models with the best wPAHM score. The model used
to generate the pathways was the one with a wPAHM score closest to the mean
wPAHM of the ten runs.

Figures D.2 and D.3 show the three commonest pathways for the _No lupus_ and
_Lupus_ classes respectively, generated by one of our experiments: the dueling
DDQN model chosen based on the accuracy score.

Figure D.2: The three commonest diagnostic decision pathways for the _No
lupus_ class generated by one of the models with the best accuracy score. The
model used to generate the pathways was the one with an accuracy score closest
to the mean accuracy of the ten runs. Figure D.3: The three commonest
diagnostic decision pathways for the _Lupus_ class generated by one of the
models with the best accuracy score. The model used to generate the pathways
was the one with an accuracy score closest to the mean accuracy of the ten
runs.

The models used were the ones with a score closest to the mean metric of the
ten runs with $\lambda$ set to 9.

###  D.4 Computing Time

Model |  Training time |  Testing time
---|---|---
Decision Tree |  25.3 ms ± 991 µs |  29.3 µs ± 771 ns
Random Forest |  772 ms ± 46.8 |  3.16 ms ± 98.5 µs
SVM |  3.09 s ± 35.1 ms |  114 µs ± 1.66 µs
FFNN |  3min 21s ± 633 ms |  597 µs ± 22.9 µs
Dueling DQN-PER |  13h 53min 35s ± 12min 10s |  1.79 ms ± 62.7 µs
Dueling DDQN-PER |  15h 16min 38s ± 1h 49min 51s |  1.2 ms ± 12.6 µs
Table D.6: The computing time to train the model and to generate a diagnosis
(pathway) for a single _lupus_ test instance.

In Table D.6, the time taken to train each model (training time), as well as,
the time taken to generate a diagnosis pathway or a diagnosis (testing time)
for the DQN models and the SOTA models respectively for a single instance in
the test dataset are shown.