# Methodology for Interpretable Reinforcement Learning
for Optimizing Mechanical Ventilation

Joo Seung Lee, Malini Mahendra, and Anil Aswani *This material is based upon
work supported by the National Science Foundation under Grant No. DGE-2125913
and Grant No. CMMI-1847666.J.S. Lee and A. Aswani are with the Department of
Industrial Engineering and Operations Research, University of California,
Berkeley, CA 94720, USA<EMAIL_ADDRESS><EMAIL_ADDRESS>M.
Mahendra is with the Department of Pediatrics, Division of Pediatric Critical
Care, UCSF Benioff Children’s Hospital, University of California at San
Francisco, San Francisco, USA and Philip R. Lee Institute for Health Policy
Studies, University of California, San Francisco, CA, USA.
<EMAIL_ADDRESS>
###### Abstract

Mechanical ventilation is a critical life-support intervention that uses a
machine to deliver controlled air and oxygen to a patient’s lungs, assisting
or replacing spontaneous breathing. While several data-driven approaches have
been proposed to optimize ventilator control strategies, they often lack
interpretability and agreement with general domain knowledge. This paper
proposes a methodology for interpretable reinforcement learning (RL) using
decision trees for mechanical ventilation control. Using a causal,
nonparametric model-based off-policy evaluation, we evaluate the policies in
their ability to gain increases in $\text{SpO}_{\text{2}}$ while avoiding
aggressive ventilator settings which are known to cause ventilator induced
lung injuries and other complications. Numerical experiments using MIMIC-III
data on the stays of real patients’ intensive care unit stays demonstrate that
the decision tree policy outperforms the behavior cloning policy and is
comparable to state-of-the-art RL policy. Future work concerns better aligning
the cost function with medical objectives to generate deeper clinical
insights.

##  I INTRODUCTION

Mechanical ventilation is a critical life-support intervention involving the
use of a ventilator, a machine that delivers a controlled mixture of air and
oxygen to the patient’s lungs to facilitate proper gas exchange and to assist
or replace the spontaneous breathing of patients who are unable to maintain
adequate respiratory function. Respiratory failure requiring mechanical
ventilation can result from various medical conditions such as infection,
surgery, or trauma.

Improper administration of mechanical ventilation poses a considerable risk,
potentially leading to ventilator-induced lung injuries and other
complications[1, 2]. Specifically, choice of high tidal volumes is known to
cause inflammation and pulmonary edema [3, 2], and high fraction of inspired
oxygen is known to cause hyperoxia [4], which all have deleterious outcomes in
patients.

###  I-A Reinforcement Learning for Mechanical Ventilation

The application of reinforcement learning (RL) in the context of mechanical
ventilator control holds significant relevance due to the inherently complex
and heterogeneous nature of lung physiology. Respiratory conditions can evolve
dynamically, necessitating continuous monitoring and adjustment of ventilation
strategies. Traditionally, this process has relied heavily on the expertise
and clinical judgment of medical professionals [2]. However, RL techniques
present a promising alternative by leveraging the wealth of data available
from intensive care units and enhancing ventilation control strategies while
minimizing the risk of adverse effects associated with suboptimal ventilation
practices.

RL has emerged as a prominent approach for addressing sequential decision-
making problems in clinical care, drawing increased attention in recent years.
Applications spanning diverse clinical domains, including HIV antiretroviral
therapy [5, 6], sepsis treatment [7, 8], mechanical ventilation [9, 10], and
insulin dosing [11, 12], have demonstrated promising results by outperforming
traditional clinician-driven treatments. Specifically, in the context of
mechanical ventilation, several approaches have been proposed, including the
use of tabular Q-Learning to minimize 90-day mortality rate for patients
undergoing mechanical ventilation [9] and the use of Conservative Q-Learning
with a similar goal of optimizing patient mortality while regularizing the
reward function using APACHE-II score to reflect patient welfare [10].

However, we note that RL faces a fundamental issue of non-interpretability, as
most advanced RL methods are black boxes expressed by deep neural networks.
The lack of transparency hinders the adoption of RL policies in safety-
critical applications such as precision medicine, where clinicians require
explanations and assurances of safety before implementing new treatments [13].
Additionally, off-policy evaluation methods are known for their high variance
and limited ability to provide detailed insights into decision-making
processes [14]. Moreover, our experience suggests that previous approach using
k-means clustering to construct Markov chains perform poorly, as the resulting
clusters tend to be equidistant from each other, leading to suboptimal state
representations [9].

$\hat{s}^{\prime}_{0}=\frac{\sum_{i=1}^{n}K(\|s_{i}-s_{0}\|/h_{s})\left[\prod_{k\in{1,2,3}}K(\|a_{i,k}-a_{0,k}\|/h_{a,k})\right]K(\|z_{i}-z_{0}\|/h_{z})\cdot
s_{i}^{\prime}}{\left(\sum_{i=1}^{n}K(\|s_{i}-s_{0}\|/h_{s})\left[\prod_{k\in{1,2,3}}K(\|a_{i,k}-a_{0,k}\|/h_{a,k})\right]K(\|z_{i}-z_{0}\|/h_{z})\right)+\lambda}$
(1)

###  I-B Contributions and Outline

In this paper, we develop a methodology for interpretable RL to optimize
mechanical ventilation strategy that can be interpreted using a decision tree,
based on Conservative Q-Improvement (CQI) algorithm [15]. We also propose a
causal non-parametric off-policy evaluation method to obtain more in-depth
policy evaluation beyond value functions, such as the overall gain in
$\text{SpO}_{\text{2}}$ or the proportion of the times in which aggressive
ventilation settings were chosen. We define a reward function that seeks
increases in blood oxygen level, which is the purpose of mechanical
ventilation [16], while explicitly discouraging aggressive settings in tidal
volume and fraction of inspired oxygen to avoid ventilator-induced
complications. We compare the performances among a supervised learning method
on clinicians (behavior cloning), state-of-the-art deep RL method
(Conservative Q-learning), and CQI method, using the proposed off-policy
evaluation method which generates counterfactual trajectories of patients’
conditions, to inspect their performances measured by gains in
$\text{SpO}_{\text{2}}$ and the frequency at which the policies take
aggressive actions.

##  II METHODS

###  II-A Data Collection

We used MIMIC-III for data collection, which is a large-scale open-access
database [17]. It contains de-identified health-related data associated with
over 40,000 patients who stayed at critical care units of the Beth Israel
Deaconess Medical Center between 2001 and 2012. The data points include
information such as patient demographics, vital signs, blood lab values,
caregiver notes, and mortality.

Patient stays with mechanical ventilation events were retrieved using
PostgreSQL, and patient health conditions and ventilator settings were
retrieved at 4-hour interval. If several ventilation events were included in a
single ICU stay, the first event that lasted for more than 24 hours was
selected.

The criteria for inclusion for the study are as follows, similar to the
approach taken in [9]: Age is at least 18 at the time of admission; 90-days
mortality is documented; ventilation events last for at least 24 hours; set
tidal volume is documented to ensure that the ventilation events are volume-
controlled and that we exclude pressure-controlled ventilation, which is a
distinct control mechanism.

This resulted in a total of 10739 mechanical ventilation events. A total of 27
features are collected for each patient, which includes lab values,
demographics, and vital signs.

###  II-B Preprocessing Steps

For each patient state feature, outliers were clipped at the 0.3th percentile
and the 99.7th percentile. Mechanical ventilation events lasting less than 24
hours were discarded, and stays extending beyond 72 hours were truncated at
the 18th time step. Missing values were initially imputed by forward and
backward filling for up to 5 consecutive time steps. For any remaining missing
data points, KNN imputation was applied. A stay was discarded if an action
parameter required missing data imputation for more than 50% of its time
steps.

ICD-9 codes associated with each hospital admission were collected to provide
additional context. A ventilation event is flagged as sepsis-related if any of
the code was associated with sepsis. This step helps distinguish ventilation
events linked to respiratory failure from those related to other causes, such
as drug intake or post-surgery recovery.

###  II-C MDP Setup

From MIMIC-III data, we collected the following time-series information for
each patient episode. Each episode is associated with the patient’s age,
sepsis indicator, sex, weight, 90-day mortality, and whether the patient had
been admitted to ICU before. At each time step, we retrieve 18 time-dependent
variables, categorized as follows:

  1. 1.

Demographics (6): Sex, Sepsis, Weight, Age, 90 days mortality, ICU readmission

  2. 2.

Lab Values (3): $\text{PaCO}_{\text{2}}$, $\text{PaO}_{\text{2}}$,
$\text{SpO}_{\text{2}}$

  3. 3.

Vital Signs (11): Systolic BP, Diastolic BP, Mean BP, Glasgow Coma Scale
(GCS), Heart Rate, $\text{PaO}_{\text{2}}\text{FiO}_{\text{2}}\text{ Ratio}$
(P/F Ratio), Temperature, Respiratory Rate, Spontaneous Tidal Volume, Shock
index, Urine output

  4. 4.

Others (5): Total IV fluids administered, Cumulative fluid balance, SIRS,
SOFA, Mean Airway Pressure (MAP)

Out of these 25 features, we selected 10 features111Sepsis, Weight, Age,
Heartrate, Resprate, $\text{SpO}_{\text{2}}$, P/F Ratio, MAP,
$\text{PaCO}_{\text{2}}$, Systolic BP to be observable by the RL agent.

To address confounders in the transition dynamics, we introduce a score
denoted by $z$ available to the agent, recoverable based on the 14 non-
observable features, and on which the transition dynamics depend. This score
$z$ is assumed to identify the causal effects in the transition of a patient’s
state given a treatment, an approach taken in several policy evaluation
methods [18, 19].

We employ propensity scoring method to recover this score. Using the remaining
14 unobservable features as covariates, a logistic regression model is trained
to predict 90-day mortality with the initial patient state as an input. The
logistic regression model’s output $z$ serves as the proxy for the confounders
and stays constant over a patient’s mechanical ventilation event. This mirrors
the observation that unobserved variables in medical settings often exhibit
high persistence [20], such as in the cases of underlying disease, genetic
factors, habits, environmental factors, which may not be documented in
databases like MIMIC-III and not likely to change within a single mechanical
ventilation event. The score $z$ is subsequently employed for propensity score
matching in the implementation of the transition model.

State Space The state space $S\subseteq\mathbb{R}^{11}$ includes the 10
selected variables and the propensity score $z$ inferred from the remaining
variables.

Action Space The ventilator has the three settings, which are:

  * •

Ideal weight adjusted tidal volume ($\text{Vt}_{\text{set}}$)

  * •

Positive end-expiratory pressure (PEEP)

  * •

Fraction of inspired oxygen ($\text{FiO}_{\text{2}}$)

Based on a clustering analysis, we binned the action space into 3 dimensions
of 7, 4, and 7 treatment levels, respectively. Each level represents a
specific range of ventilator settings, as shown in the Fig. 1.

bin | $\text{Vt}_{\text{set}}$ | PEEP | $\text{FiO}_{\text{2}}$
---|---|---|---
1 | [0, 4.02) | [0, 7.04) | [0, 0.36)
2 | [4.02, 5.41) | [7.04, 11.25) | [0.36, 0.45)
3 | [5.41, 6.57) | [11.25, 16.38) | [0.45, 0.55)
4 | [6.57, 7.75) |  [16.38, $\infty$) | [0.55, 0.66)
5 | [7.75, 9.15) |  | [0.66, 0.76)
6 | [9.15, 11.18) |  | [0.76, 0.89)
7 |  [11.18, $\infty$) |  | [0.89, 1]
Figure 1: Action space binning intervals

Reward The primary objective of operating a mechanical ventilator is to
facilitate the delivery of oxygen and remove carbon dioxide from the body
[16]. The effectiveness of oxygen delivery can be assessed by monitoring
$\text{SpO}_{\text{2}}$, which represents blood oxygen levels [21, 22].
Although setting high values of $\text{Vt}_{\text{set}}$ and
$\text{FiO}_{\text{2}}$ is correlated with immediate increases in
$\text{SpO}_{\text{2}}$, such severe ventilator settings can increase the risk
of barotrauma or volutrauma [23, 24]. To discourage the learned RL policy from
selecting excessively large values for $\text{Vt}_{\text{set}}$ and
$\text{FiO}_{\text{2}}$ values, the agent is incurred a penalty in the form of
subtracted rewards when $\text{Vt}_{\text{set}}$ is larger than 10
$\text{mm}\text{H}_{2}\text{O}$ or when $\text{FiO}_{\text{2}}$ is larger than
0.60. Therefore, the final reward function is expressed as:

$\displaystyle r(s_{t},s_{t+1},a_{t})$
$\displaystyle=s_{t+1}(\text{SpO}_{\text{2}})-s_{t}(\text{SpO}_{\text{2}})$
$\displaystyle-$
$\displaystyle\alpha\mathbbm{1}\\{a_{t}(\text{Vt}_{\text{set}})>10\\}-\beta\mathbbm{1}\\{a_{t}(\text{FiO}_{\text{2}})>0.6\\},$

where $\alpha$ and $\beta$ are fixed constants, which are to be determined by
cross-validation. Since the action space has been discretized, the action
indices for $\text{Vt}_{\text{set}}$ and $\text{FiO}_{\text{2}}$ above 6 and 4
respectively, out of 1-7, are considered aggressive, as they each contain $10$
and $0.6$.

Figure 2: Sample trajectory using Nadaraya-Watson estimator transition model
P/F RatioP/F RatioMAPpt$\text{Vt}_{\text{set}}$: 6.01PEEP: 5.0
$\text{FiO}_{\text{2}}$: 1.0$\leq$ 17.9pt$\text{Vt}_{\text{set}}$: 3.51PEEP:
20.0 $\text{FiO}_{\text{2}}$: 1.0$>$ 17.9$\leq
120$MAPpt$\text{Vt}_{\text{set}}$: 6.01PEEP: 10.0 $\text{FiO}_{\text{2}}$:
0.6$\leq$ 15.9pt$\text{Vt}_{\text{set}}$: 4.84PEEP: 12.75
$\text{FiO}_{\text{2}}$: 0.6$>$ 15.9$>120$$\leq$ 160P/F
RatioMAPpt$\text{Vt}_{\text{set}}$: 7.11PEEP: 5.0 $\text{FiO}_{\text{2}}$:
0.5$\leq$ 13.0pt$\text{Vt}_{\text{set}}$: 4.84PEEP: 10.0
$\text{FiO}_{\text{2}}$: 0.5$>$ 13.0$\leq$ 220P/F
Ratiopt$\text{Vt}_{\text{set}}$: 6.01PEEP: 5.0 $\text{FiO}_{\text{2}}$:
0.4$\leq 280$pt$\text{Vt}_{\text{set}}$: 7.11PEEP: 5.0
$\text{FiO}_{\text{2}}$: 0.3$>$ 280$>$ 220$>$ 160 Figure 3: Decision tree
policy with maximum depth of 3, learned from the clinicians’ behavioral data

###  II-D Learning Scheme

Behavior Clone (BC) A feed-forward neural network was used to learn a behavior
clone policy. Out of 196 possible combinations of actions, 184 action triplets
were observed in the data. The neural network was trained with ordinal
regression objective, rather than classification objective, to exploit the
ordinal structure of the action classes.

CQL A noteworthy advancement in the domain of offline RL is the Conservative
Q-Learning (CQL) algorithm [25]. CQL is designed to mitigate the common
challenge of overestimation of Q-values in offline learning by introducing a
regularization mechanism. Notably, it addresses the specific issue of
overestimation in underrepresented state-action pairs within the dataset. The
d3rlpy module was used to train a CQL policy [26].

CQI Conservative Q-Improvement [15] learns a decision tree policy directly. We
implemented CQI, with a slight modification to the original one (e.g. to
implement maximum depth). In contrast to CART algorithms which split a node
based on information gain, CQI algorithm splits a node based on gain of
Q-values. Each branching splits the state space into two half spaces on a
specific feature that leads to the Q-value gain, and the leaf node corresponds
to the action with the highest Q-value in each state.

###  II-E Transition Model

We use Nadaraya-Watson estimator (NWE), a kernel density based nonparametric
regression model, to estimate the state variables of the next time step given
the current state and action variables [27, 28]. Consider a nonlinear model
$Y=g(X)+\epsilon$, where $g(\cdot)$ is an unknown nonlinear function. Suppose
that given $x_{0}$, we would like to only estimate
$g(x_{0})=\mathbb{E}[Y|X=x_{0}]$ using the i.i.d. data
$(X_{i},Y_{i})\in\mathbb{R}^{d}\times\mathbb{R}^{k}$ for $i=1,\ldots,n$. The
Nadaraya-Watson estimator is defined as

$\hat{g}(x_{0})=\frac{\sum_{i=1}^{n}K(\|X_{i}-x_{0}\|/h)\cdot
Y_{i}}{\sum_{i=1}^{n}K(\|X_{i}-x_{0}\|/h+\lambda},$

where $K(\cdot)$ is a kernel function, $\lambda$ is a regularization constant,
and $h$ is a bandwidth parameter.

In the context of patient state transition modeling, we apply separate
bandwidths for the state variables, action variables, and the propensity
score. Therefore, training samples which fall outside of bandwidth for any of
state, action, or propensity score is not counted towards the Nadaraya-Watson
estimate. That is, given the bandwidth parameters $h_{s}$, $h_{a}$, and
$h_{z}$, the current state $s_{0}$, action taken $a_{0}$, propensity score
$z_{0}$, our estimate of the patient state at the next time step
$\hat{s}^{\prime}_{0}$ is given in Equation (1).

###  II-F Generation of Simulated Trajectories

As a data preparation, training dataset is prepared in a form of MDP
transitions, i.e. $\\{(s^{i}_{t},a^{i}_{t},r^{i}_{t},s^{i}_{t+1},z_{i})\\}$,
for the transition at time $t$ for the $i$th episode. Since age, weight, and
sepsis flag are assumed to not vary in a single episode, the target variable
$s_{t+1}$ has the three features removed. The propensity score $z_{i}$ is
computed with the initial state of the episode. Given an evaluation policy
$\pi_{e}$, an action $a\sim\pi_{e}(s_{0})$ is drawn, and the Nadaraya-Watson
estimator transition model predicts the next state and iterates until the
specified time horizon, which is set to equal the actual duration of the
patient episode.

SpO2MAPP/F Ratiopt$\text{Vt}_{\text{set}}$: 3.51PEEP: 5.0
$\text{FiO}_{\text{2}}$: 0.3$\leq$ 220pt$\text{Vt}_{\text{set}}$: 3.51PEEP:
5.0 $\text{FiO}_{\text{2}}$: 0.3$>$ 220$\leq$
6.874Weightpt$\text{Vt}_{\text{set}}$: 7.11PEEP: 10.0 $\text{FiO}_{\text{2}}$:
0.3$\leq$ 120.3pt$\text{Vt}_{\text{set}}$: 6.01PEEP: 20.0
$\text{FiO}_{\text{2}}$: 0.5$>$ 120.3$>$ 6.874$\geq$
95$\text{SpO}_{\text{2}}$Agept$\text{Vt}_{\text{set}}$: 7.11PEEP: 5.0
$\text{FiO}_{\text{2}}$: 0.4$\leq$ 55pt$\text{Vt}_{\text{set}}$: 7.11PEEP: 5.0
$\text{FiO}_{\text{2}}$: 0.5$>$ 55$\geq$
91$\text{SpO}_{\text{2}}$pt$\text{Vt}_{\text{set}}$: 7.11PEEP: 5.0
$\text{FiO}_{\text{2}}$: 0.5$\leq 87$pt$\text{Vt}_{\text{set}}$: 8.33PEEP: 5.0
$\text{FiO}_{\text{2}}$: 0.4$>$ 87$<$ 91$<$ 95 Figure 4: Decision tree policy
trained by CQI algorithm with the maximum depth of 3 and the penalty
coefficients $(\alpha^{*},\beta^{*})$, i.e. CQI∗-D$3$

###  II-G Experimental Setup

The ventilation events were split into training and testing dataset by 80-20.
Grid search was performed to choose 1) the architecture and hyperparameters of
the BC, CQL, and CQI policy, 2) the penalty coefficients $\alpha$ and $\beta$,
and 3) the bandwidth parameters for the Nadaraya-Watson transition model,
using 5-fold cross-validation.

Hyperparameters for the BC policy was chosen on its ability to predict back
the clinician’s actual choice of discretized action. The mean and standard
deviation of the differences in the actions time step-wise were each
$(0.0208,-0.642,0.00498)$ and $(1.09,0.541,0.295)$, respectively for
$\text{Vt}_{\text{set}}$, PEEP, and $\text{FiO}_{\text{2}}$. The logistic
regression model to compute propensity score had a testing ROC-AUC of $0.658$.

The bandwidth parameters that best replicated the trajectories of
$\text{SpO}_{\text{2}}$ under the BC policy for the Nadaraya-Watson transition
model was chosen. The considered bandwidths were $[2.236,2.646,3.0]$,
$[0.5,1.0,1.5]$, $[10^{-1},10^{-2},10^{-3}$, $10^{-4}]$ for $h_{s}$, $h_{a}$
and $\lambda$, respectively, and $2.646$, $1.0$, and $10^{-3}$ were chosen,
which achieved $1.40$ mean absolute error in $\text{SpO}_{\text{2}}$ timestep-
wise. The Nadaraya-Watson model used Epanechnikov function as the kernel.

Figure 5: Distribution of actions across different RL & BC policies

Algorithm | $\overline{\Delta(\text{$\text{SpO}_{\text{2}}$})}$ | % Vt_set | % FiO2
---|---|---|---
BC | 0.5029 | 12.05 | 17.02
CQL(0,0) | 0.5603 | 2.22 | 6.62
CQL∗ | 0.5493 | 7.12 | 17.19
CQI(0,0)-D3 | 1.0345 | 0 | 93.36
CQI∗-D3 | 0.2992 | 0 | 0
CQI(0,0)-D5 | 0.5788 | 44.99 | 57.28
CQI∗-D5 | 0.5519 | 0 | 0

Figure 6: Comparisons of the different RL & BC policies.
$\overline{\Delta(\text{$\text{SpO}_{\text{2}}$})}$ indicates the average
value of the increase in $\text{SpO}_{\text{2}}$ per event.

The penalty coefficients for aggressive actions that had the best tradeoff
between the average increase in $\text{SpO}_{\text{2}}$ and the average
proportion of times with aggressive action, among
$\alpha,\beta\in[\frac{3}{4^{0}},\frac{3}{4^{1}},\frac{3}{4^{2}},\frac{3}{4^{3}},\frac{3}{4^{4}}]$.
The CQL algorithm was employed for this process, with the discount factor of
$\gamma=0.99$, to model the fact that prolonged ventilation events is not
desirable, as it is shown to be associated with significantly higher risk of
complications and increase in ICU and hospital mortality [29].

The selected $\alpha^{*}$ and $\beta^{*}$ were each $0.75$ and $3.0$, with the
results as follows: Average increase in $\text{SpO}_{\text{2}}$, %
Aggressiveness in $\text{Vt}_{\text{set}}$, % Aggressiveness in
$\text{FiO}_{\text{2}}$ were each $0.047,4.87$ and $20.0$.

A sample trajectory rolled out on the initial state drawn from testing dataset
with the chosen hyperparameters is shown in Fig. 2.

##  III Experimental Results

Let BC denote the behavior clone policy, CQL$(0,0)$ and CQL∗ each refer to CQL
policy with $\alpha=\beta=0$ and $\alpha^{*},\beta^{*}$, and CQI$(0,0)$-D$n$
and CQI∗-D$n$ each refer to CQI policy with maximum depth of $n$, each with
$\alpha=\beta=0$ and $\alpha^{*},\beta^{*}$.

###  III-A Distribution of actions and Performance Comparison

We compare the distribution of action choices under different RL and
behavioral policies (see Fig. 5). Policies trained with nonzero penalty
coefficients are observed to be less aggressive (i.e. they pick actions with
$\text{Vt}_{\text{set}}$ $\geq 10$ and $\text{FiO}_{\text{2}}$ $\geq 0.6$)
than the behavior cloning policy or the policies that were trained with
$\alpha^{*}$ and $\beta^{*}$. For the CQI policy with maximum depth of 3, we
observe that it is inclined to taking one particular setting for all of
$\text{Vt}_{\text{set}}$, PEEP, and $\text{FiO}_{\text{2}}$, which is likely
due to the limited number of leaf nodes in a shallow decision tree of depth 3,
which restricts the policy to a small number of distinct actions and
sacrifices flexibility and expressiveness in favor of increased
interpretability.

We also evaluate the performance of the RL agents under different policies
(see Fig. 6). As our goal is to increase $\text{SpO}_{\text{2}}$ in patients
while minimizing the frequency of extreme $\text{Vt}_{\text{set}}$ and
$\text{FiO}_{\text{2}}$ values, instead of the comparing the cumulative reward
using the reward function proposed in Section III.C., we compute the
cumulative increase in $\text{SpO}_{\text{2}}$ and the proportion of timesteps
at which extreme values of $\text{Vt}_{\text{set}}$ and
$\text{FiO}_{\text{2}}$ were chosen.

We observe that the CQL algorithm, even with $\alpha=\beta=0$, achieves higher
increase in $\text{SpO}_{\text{2}}$ while reducing aggressiveness in
$\text{Vt}_{\text{set}}$ and $\text{FiO}_{\text{2}}$ settings. The
CQI$(0,0)$-D$3$ policy achieves the best performance in
$\text{SpO}_{\text{2}}$ gain, but at the expense of being extremely aggressive
with $\text{FiO}_{\text{2}}$ strategy. This is perhaps a result of picking up
on the correlation between increase in $\text{SpO}_{\text{2}}$ and high
$\text{FiO}_{\text{2}}$ on an inflexible model, without any regularization for
high $\text{FiO}_{\text{2}}$. Overall, CQL$(0,0)$, CQL∗, CQI$(0,0)$-D$5$ and
CQI∗ result in comparable performance in $\text{SpO}_{\text{2}}$, despite the
large difference in their aggressiveness. The CQI policies can be observed to
recover the performance loss of having depth 3, just by the addition of 2
extra depth layers.

###  III-B Decision Tree

We next inspect the decision tree learned by the CQI algorithm and the
decision tree of the clinicians. The action values in the leaf nodes from both
trees are represented by the median of each interval that defined the action
discretization in Fig. 1.

The decision trees of the clinician is learned by applying CART algorithm on
the dataset. The decision trees can be seen in Fig. 3 and 4. The decision tree
of the clinicians generally agrees with clinical knowledge: the conditions
corresponding to the right subtree (P/F Ratio $>$ 220) indicate little or no
lung injury and needs very little support from the ventilator. This is shown
with a low $\text{FiO}_{\text{2}}$, low PEEP, and slightly liberalized tidal
volumes (around 7ml/kg). The left half of the tree represents moderate to
severe lung injury,and the second leaf which goes directly to PEEP of 20 when
the MAP is 18 seems aggressive without good reason. This may be capturing the
knowledge that higher PEEPs are in need in patients with high BMIs, a variable
that was not reliably accessible.

CQI∗-D$3$ is in less agreement with the general clinical knowledge: the
patient condition with $\text{SpO}_{\text{2}}$ at least 95 is seen to not have
lung injury, and should be treated with liberalized tidal volumes and be
getting ready to extubate [30]. A limitation is evident in CQI∗-D$3$, as
ventilator settings may not be changed if small changes in
$\text{SpO}_{\text{2}}$ are seen and $\text{SpO}_{\text{2}}$ is above 95%. The
practicality of the decision tree may be blunted by the first branching point,
resulting in one half of the tree unavailing at the first branching point
($\text{SpO}_{\text{2}}$$>95$). Additionally, lowered oxygen saturation goals
is the standard of care in treating acute respiratory distress syndrome, a
common reason for mechanical ventilation [31].

##  IV Conclusion

In conclusion, we present an interpretable RL policy for mechanical
ventilation control and a novel causal non-parametric model-based approach for
evaluating RL policies in an offline setting. By comparing various policies
(behavior cloning, CQI and CQL), we empirically establish the superior
performance of the latter two policies over the behavior clone to the
clinicians as observed in MIMIC-III database in two ways: achieving greater
increases in $\text{SpO}_{\text{2}}$ on average while taking aggressive
actions less frequently.

However, it is crucial to acknowledge the inherent limitations of offline RL
and offline off-policy evaluation methods, which may suffer from issues such
as data bias, distribution shift and lack of exploration. Although the model
may outperform the clinicians for the goals set in this study, the goals set
by the clinician are unknown as the clinical scenario is not included in the
predictors, similarly to previous RL approaches to control ventilators.
Therefore, while the results are promising, additional verification on more
diverse datasets outside of MIMIC-III is necessary before clinical deployment.
Nevertheless, the interpretable decision tree policies learned by CQI have the
potential to provide useful clinical insights in future studies by
highlighting the important features and decision rules. Moving forward,
integrating more domain knowledge and considering safety constraints will be
crucial steps towards developing trustworthy RL systems for safety-critical
healthcare applications like mechanical ventilation control.

## References

  * [1] D. G. Blauvelt, H. S. Inany, J. M. Furlong-Dillard, D. K. Bailly, P. Oishi, M. A. Steurer, and M. Mahendra, “Association of ventilator settings with mortality in pediatric patients treated with extracorporeal life support for respiratory failure,” ASAIO Journal, vol. 68, p. 1536–1543, Mar. 2022.
  * [2] A. S. Slutsky and V. M. Ranieri, “Ventilator-induced lung injury,” New England J. Medicine, vol. 369, no. 22, pp. 2126–2136, 2013.
  * [3] R. Pinheiro de Oliveira, M. P. Hetzel, M. dos Anjos Silva, D. Dallegrave, and G. Friedman, “Mechanical ventilation with high tidal volume induces inflammation in patients without lung disease,” Critical Care, vol. 14, no. 2, p. R39, 2010.
  * [4] S. Rachmale, G. Li, G. Wilson, M. Malinchoc, and O. Gajic, “Practice of excessive FiO2 and effect on pulmonary outcomes in mechanically ventilated patients with acute lung injury,” Respiratory Care, vol. 57, no. 11, pp. 1887–1893, 2012.
  * [5] C. Yu, Y. Dong, J. Liu, and G. Ren, “Incorporating causal factors into reinforcement learning for dynamic treatment regimes in HIV,” BMC Med. Inform. and Decis. Making, vol. 19, p. 60, Apr 2019.
  * [6] S. Parbhoo, J. Bogojeska, M. Zazzi, V. Roth, and F. Doshi-Velez, “Combining kernel and model based learning for HIV therapy selection,” AMIA Jt Summits Transl. Sci. Proc., vol. 2017, pp. 239–248, July 2017.
  * [7] M. Komorowski, L. A. Celi, O. Badawi, A. C. Gordon, and A. A. Faisal, “The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care,” Nature Medicine, vol. 24, pp. 1716–1720, Nov 2018.
  * [8] T. Nanayakkara, G. Clermont, C. J. Langmead, and D. Swigon, “Unifying cardiovascular modelling with deep reinforcement learning for uncertainty aware control of sepsis treatment,” PLOS Digital Health, vol. 1, pp. 1–20, 02 2022.
  * [9] A. Peine et al., “Development and validation of a reinforcement learning algorithm to dynamically optimize mechanical ventilation in critical care,” npj Digital Medicine, vol. 4, no. 1, p. 32, 2021.
  * [10] F. Kondrup et al., “Towards safe mechanical ventilation treatment using deep offline reinforcement learning,” in Proc. AAAI Conf. Artif. Intell., vol. 37, pp. 15696–15702, Sep. 2023.
  * [11] M. Oroojeni Mohammad Javad, S. O. Agboola, K. Jethwani, A. Zeid, and S. Kamarthi, “A reinforcement learning–based method for management of type 1 diabetes: Exploratory study,” JMIR Diabetes, vol. 4, p. e12905, Aug. 2019.
  * [12] H. Emerson, M. Guy, and R. McConville, “Offline reinforcement learning for safer blood glucose control in people with type 1 diabetes,” J. Biomed. Inform., vol. 142, p. 104376, June 2023.
  * [13] C. Yu, J. Liu, S. Nemati, and G. Yin, “Reinforcement learning in healthcare: A survey,” ACM Comput. Surv., vol. 55, no. 1, 2021.
  * [14] O. Gottesman et al., “Evaluating reinforcement learning algorithms in observational health settings,” arXiv:1805.12298, 2018.
  * [15] A. M. Roth, N. Topin, P. Jamshidi, and M. Veloso, “Conservative Q-Improvement: Reinforcement learning for an interpretable decision-tree policy,” arXiv:1907.01180, 2019.
  * [16] A. S. Slutsky, “Mechanical ventilation,” Chest, vol. 104, no. 6, pp. 1833–1859, 1993.
  * [17] A. E. Johnson et al., “MIMIC-III, a freely accessible critical care database,” Scientific Data, vol. 3, no. 1, 2016.
  * [18] M. Oberst and D. Sontag, “Counterfactual off-policy evaluation with Gumbel-max structural causal models,” in Proc. Int. Conf. Mach. Learn., vol. 97, pp. 4881–4890, 2019.
  * [19] A. Bennett and N. Kallus, “Policy evaluation with latent confounders via optimal balance,” in Adv. Neural Inf. Process. Syst., vol. 32, 2019.
  * [20] M. A. Brookhart, T. Stürmer, R. J. Glynn, J. Rassen, and S. Schneeweiss, “Confounding control in healthcare database research: Challenges and potential approaches,” Medical Care, vol. 48, no. 6, p. S114–S120, 2010.
  * [21] G. Tusman, S. H. Bohm, and F. Suarez-Sipmann, “Advanced uses of pulse oximetry for monitoring mechanically ventilated patients,” Anesthesia & Analgesia, vol. 124, no. 1, 2017.
  * [22] L. M. Schnapp and N. H. Cohen, “Pulse oximetry: Uses and abuses,” Chest, vol. 98, no. 5, pp. 1244–1250, 1990.
  * [23] S. W. Salyer, “Chapter 15 - Pulmonary Emergencies,” in Essential Emergency Medicine (S. W. Salyer, ed.), pp. 844–913, Philadelphia: W.B. Saunders, 2007.
  * [24] G. F. Curley, J. G. Laffey, H. Zhang, and A. S. Slutsky, “Biotrauma and ventilator-induced lung injury,” Chest, vol. 150, p. 1109–1117, Nov. 2016.
  * [25] A. Kumar, A. Zhou, G. Tucker, and S. Levine, “Conservative Q-Learning for offline reinforcement learning,” in Adv. Neural Inf. Process. Syst., vol. 33, pp. 1179–1191, 2020.
  * [26] T. Seno and M. Imai, “d3rlpy: An offline deep reinforcement learning library,” J. Mach. Learn. Res., vol. 23, Oct. 2022.
  * [27] E. Nadaraya, “On estimating regression,” Theory of Probability and Its Applications, vol. 9, pp. 141–142, 1964.
  * [28] G. S. Watson, “Smooth regression analysis,” Sankhyā: The Indian J. Statist., Series A (1961-2002), vol. 26, no. 4, pp. 359–372, 1964.
  * [29] S. H. Loss et al., “The reality of patients requiring prolonged mechanical ventilation: a multicenter study,” Rev. Bras. Ter. Intensiva, vol. 27, no. 1, pp. 26–35, 2015.
  * [30] M. Girardis et al., “Effect of conservative vs conventional oxygen therapy on mortality among patients in an intensive care unit: The oxygen-icu randomized clinical trial,” JAMA, vol. 316, p. 1583, Oct. 2016.
  * [31] M. W. Semler et al., “Oxygen-saturation targets for critically ill adults receiving mechanical ventilation,” New England J. Medicine, vol. 387, p. 1759–1769, Nov. 2022.