# A Comprehensive Benchmark for COVID-19 Predictive Modeling Using Electronic
Health Records in Intensive Care: Choosing the Best Model for COVID-19
Prognosis

Junyi Gao  Peking University, Beijing, 100871, China  University of Edinburgh,
Edinburgh, EH8 9AB, UK  Health Data Research UK  Yinghao Zhu  Peking
University, Beijing, 100871, China  Wenqing Wang  Peking University, Beijing,
100871, China  Yasha Wang  Peking University, Beijing, 100871, China  Wen Tang
Peking University Third Hospital, Beijing, 100191, China  Liantao Ma  Peking
University, Beijing, 100871, China  corresponding author: Liantao Ma (
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Objective: The COVID-19 pandemic has posed a heavy burden to the healthcare
system worldwide and caused huge social disruption and economic loss. Many
deep learning models have been proposed to conduct clinical predictive tasks
such as mortality prediction for COVID-19 patients in intensive care units
using Electronic Health Record (EHR) data. Despite their initial success in
certain clinical applications, there is currently a lack of benchmarking
results to achieve a fair comparison so that we can select the optimal model
for clinical use. Furthermore, there is a discrepancy between the formulation
of traditional prediction tasks and real-world clinical practice in intensive
care.

Methods: To fill these gaps, we propose two clinical prediction tasks,
Outcome-specific length-of-stay prediction and Early mortality prediction for
COVID-19 patients in intensive care units. The two tasks are adapted from the
naive length-of-stay and mortality prediction tasks to accommodate the
clinical practice for COVID-19 patients. We propose fair, detailed, open-
source data-preprocessing pipelines and evaluate 17 state-of-the-art
predictive models on two tasks, including 5 machine learning models, 6 basic
deep learning models and 6 deep learning predictive models specifically
designed for EHR data.

Results: We provide benchmarking results using data from two real-world
COVID-19 EHR datasets. Both datasets are publicly available without needing
any inquiry and one dataset can be accessed on request. We provide fair,
reproducible benchmarking results for two tasks.

Conclusions: We deploy all experiment results and models on an online
platform. We also allow clinicians and researchers to upload their data to the
platform and get quick prediction results using our trained models. We hope
our efforts can further facilitate deep learning and machine learning research
for COVID-19 predictive modeling.

Software Repository: https://github.com/yhzhu99/covid-ehr-benchmarks

###### keywords:

electronic health record, benchmark, deep learning, COVID-19, intensive care

##  1 Introduction

The COVID-19 pandemic needs no introduction. As of May 2022, the virus has
caused over 500 million infected cases and over 6 million deaths [1]. Though
research shows that new variants of COVID-19 are less deadly, they are more
spreadable and cause the number of cases still surging globally [2]. Under
current circumstances, achieving early risk prediction and estimating the
disease progression especially for COVID-19 patients in intensive care units
have been an important topic to allocate limited medical resources and relieve
the burdens of our healthcare system.

Electronic health record (EHR) data and intelligent models have been viable
solutions to solve this challenge. Many machine learning and deep learning
models have been proposed to utilize COVID-19 patients’ EHR data to conduct
clinical prediction tasks including severity [3, 4, 5, 6, 7, 8, 9, 10, 11,
12], diagnosis [13, 14], length-of-stay (LOS) [15, 16], etc. There are more
previous general EHR predictive models, which can also be applied to COVID-19
prediction tasks. These works achieve better prediction performances compared
with simple statistical models. However, they still face two challenges in
terms of task practicality and clinical versatility:

  * •

How to adapt traditional prediction tasks for COVID-19 patients in intensive
care units. Existing COVID-19 risk and LOS prediction works simply follow the
same task settings in previous EHR data-mining works. For risk or mortality
prediction models, they only output the predicted risk score at the last
timestep of the available EHR sequences[3, 12]. However, for high-risk
COVID-19 patients (especially the ICU patients), patients’ status at the last
timestep is already highly dangerous and it may be too late to conduct life-
saving treatments even if the model gives the alarm. On the other hand, the
LOS prediction task is often formulated as a regression task[17]. The model
will output the predicted remaining LOS at each timestep. This setting
produces an apparent issue: high-risk patients and low-risk patients may have
the same LOS value since previous benchmarking works defined both deaths and
leaving ICU as the end of the stay. Though the model can learn the non-
linearity and map these statuses in separate locations in the latent embedding
space, it may be difficult for clinicians to tell the difference when applied
in clinical usage. Therefore, both tasks need to be adapted to accommodate the
COVID-19 intensive care settings.

  * •

How to choose the best model among various options of predictive models.
Though existing works achieve great prediction performances on specific
datasets, other researchers or clinicians may find it difficult to apply the
state-of-the-art prediction models to their new data. Should we use deep
learning models or traditional machine learning models? What are the
differences in terms of prediction performances among all these models? This
is mostly because of the lack of fair comparison for existing models. It’s
difficult to select the appropriate model for new data since re-implementing
all existing models on new data is expensive. Furthermore, most COVID-19 EHR
datasets used in existing works are not publicly available due to privacy
issues, which makes it more difficult to reproduce the results and make
further improvements.

Based on the above challenges, in this work, we propose a comprehensive
benchmark for COVID-19 predictive modeling with EHR data in intensive care.
The proposed pipeline of this benchmark is shown in Fig. 1.

Figure 1: The proposed benchmark pipeline. The pipeline has four steps: Data
pre-processing, Model training, Evaluation and Deployment.

Our contributions are three-fold:

  * •

Task and metric designing. We propose two tasks based on the clinical practice
of COVID-19 patients in intensive care: Outcome-specific length-of-stay
prediction and Early mortality prediction. Different from existing LOS
prediction settings, the outcome-specific LOS prediction is formulated as a
multi-target task, which will output both outcome and corresponding LOS so
that clinicians can differentiate the progression statuses between low-risk
and high-risk patients. The early mortality prediction task encourages the
model to raise alarms for high-risk patients as early as possible to prevent
possible treatment delays. We also designed evaluation metrics for both tasks
(i.e., early mortality score and outcome-specific mean absolute error) to
evaluate models’ performances.

  * •

Data preparing and pre-processing pipelines. We set up the data pre-processing
pipelines including cleaning, filtering, missing value imputation and cohort
construction for two real-world COVID-19 EHR datasets, TJH[3] and CDSL [18].
Both datasets are heterogeneous longitudinal EHR datasets in intensive care
units. The features include lab tests, vital signs, diagnosis and static
demographic information. The former dataset is publicly available without
inquiry, and the other dataset can be accessed on request.

  * •

Modeling and benchmarking. We implement and evaluate 17 state-of-the-art
predictive models on two tasks, including 5 machine learning models, 6 basic
deep learning models and 6 deep learning predictive models specifically
designed for multivariate time-series EHR data. We conduct fair and
reproducible comparisons and report detailed benchmarking results to
facilitate further deep learning and machine learning research in this field.

To the best of our knowledge, this is the first benchmark work for patient-
level COVID-19 prediction tasks in intensive care. Our code is already
available online, so that anyone can build the complete benchmarks and
reproduce all results. We provide well-organized data preprocessing and
modeling modules in our code so that they can also be easily applied to
generate customized tasks and results. Our benchmark code is publicly
available at https://github.com/yhzhu99/covid-ehr-benchmarks. We also build an
online platform to deploy benchmark experiment results, including model
performances with all hyper-parameter combinations on both tasks. We deploy
all trained models online, allowing clinicians and researchers to easily use
these models to get prediction results using their own data. The system is
publicly available at http://106.15.42.91:8000/.

##  2 Results

In this section, we report the dataset and task description, baselines,
experiment settings, detailed benchmark model performances, code usage and
platform deployment.

###  2.1 Dataset description and problem formulation

The datasets used in this study and the proposed prediction tasks are as
follows:

####  2.1.1 EHR datasets for COVID-19 patients in intensive care

In this work, we use two COVID-19 EHR datasets to conduct benchmark
experiments. We conduct a detailed search for globally publicly available
COVID-19 EHR datasets. Our criteria is that the datasets either can be
directly downloaded or the access request is simple and does not have area
restrictions. We finally select two datasets for our benchmarking:

######  Dataset 1 (Tongji Hospital COVID-19 Dataset, TJH).

[3] The TJH dataset contains 485 anonymized COVID-19 patients hospitalized in
Tongji Hospital, China, between 10 January 2020 and 24 February 2020. All EHR
are collected during patients’ hospitalization. The dataset contains 73 lab
tests and vital signs, which are all numerical features and 2 demographic
features (i.e., age and gender). This dataset is publicly available at GitHub
111https://github.com/HAIRLAB/Pre_Surv_COVID_19. We also include the download
script in our benchmark code.

######  Dataset 2 (Covid Data Save Lives, CDSL).

[18] The CDSL dataset comes from the HM Hospitales EHR system, which contains
the anonymized records of 4,479 patients, admitted with a diagnosis of COVID
POSITIVE or COVID PENDING, since the beginning of the epidemic to date. This
dataset collects heterogeneous medical features, including detailed
information on diagnoses, treatments, admissions, ICU admissions, diagnostic
imaging tests, laboratory results, discharge or death. This dataset is open to
global researchers and can be accessed on quest
222https://www.hmhospitales.com/coronavirus/covid-data-save-lives/english-
version. The applicant should fill out the request form.

These two datasets have been used in several mortality prediction works [3,
15]. However, we do not find any well-organized data preprocessing pipeline
code and benchmarking results for them. This motivates us to provide
comprehensive benchmark analysis and data-process suit for these two valuable
publicly-available datasets, and further facilitate the related research.

####  2.1.2 Problem Formulation and Evaluation Metrics

In this work, we design two tasks to evaluate and compare the performances of
existing machine learning and deep learning models: Outcome-specific length-
of-stay prediction and Early mortality prediction. These two tasks are derived
from the common LOS and mortality prediction but adapted to the COVID-19
intensive care settings.

######  Problem 1 (Outcome-specific length-of-stay prediction).

We define the outcome-specific length-of-stay prediction as a multi-target
prediction task, which includes a binary outcome classification task and a
length-of-stay regression task. The model will have two outputs at each
timestep: the predicted outcome $\hat{y}_{m}\in\\{0,1\\}$ (i.e., mortality or
survival) and the predicted LOS $\hat{y}_{l}\geq 0$ denoting the remaining
days corresponding to the mortality outcome or the end of ICU stay. Compared
with the traditional LOS prediction task, which only has $\hat{y}_{l}$ as
output, our setting is more applicable in clinical practice. Clinicians can
comprehensively evaluate patients’ progression and differentiate the LOS to
different outcomes. The model learning process will also benefit from this
setting since patients with different health statuses are now explicitly
modeled and can be divided in the latent space.

(a) Illustrations of the OSMAE Metric.

(b) Illustration of the early mortality prediction score.

Figure 2: Illustrations of the proposed OSMAE and ES metrics.

The main prediction target (i.e., length-of-stay) will be evaluated using the
mean absolute error (MAE), rooted mean squared error (RMSE) and mean squared
error (MSE). We further propose a new metric, outcome-specific MAE (OSMAE), to
comprehensively evaluate this multi-target task:

$OSMAE=\left\\{\begin{array}[]{ll}\epsilon(max(E-\hat{y}_{l},0)+max(E-y_{l},0))&,\hat{y}_{m}\neq
y_{m}\\\ \epsilon|\hat{y}_{l}-y_{l}|&,\hat{y}_{m}=y_{m}\end{array}\right.\\\
,\epsilon=\left\\{\begin{array}[]{ll}\frac{t}{E-L+\gamma}&,0<t\leq L-\gamma\\\
1&,t>L-\gamma\end{array}\right.\\\ $

where $t$ denotes the timestep, $L$ denotes the total length-of-stay, $E$
denotes the 95% maximum of total length-of-stay of all patients, $\epsilon$ is
the penalty term, which is a piecewise function. The lower the OSMAE is, the
better the model performance is. Ignoring the penalty term $\epsilon$, for
true positive and true negative predictions, i.e., $\hat{y}_{m}=y_{m}$, OSMAE
is the same as the normal MAE. For false positive and false negative patients,
the OSMAE is much larger than the original MAE as the model predicts the
patient outcome in completely opposite directions. We introduce the penalty
term to prevent the model to receive high OSMAE at early timesteps, since
patients’ status may be uncertain at the early ICU-stay phase. $\gamma$ is the
penalty threshold, which means we expect the model to correctly predict the
targets within the last $\gamma$ days. As shown in Fig. 2(a), we use three
examples (i.e., true positive, false negative and true negative cases) to show
the calculation of OSMAE and the penalty term.

######  Problem 2 (Early mortality prediction).

We define the early mortality prediction as a binary sequential prediction
task. At each timestep of the longitudinal EHR input sequence, the model will
predict the mortality risk $\hat{y}_{r}\in\\{0,1\\}$ of this ICU stay, which
indicates whether the patient will have the mortality outcome at the end of
this ICU stay. A good early prediction model should raise the alarm as soon as
possible during the stay. Existing mortality prediction works input the entire
EHR sequence into the model and make predictions at the last timestep or
predict the mortality risk in a future time window (e.g., 24h or 12h).
However, for COVID-19 patients in ICU, health statuses may have severely
deteriorated and the risks are evident to clinicians at the very last
timestep. It may be too late to conduct live-saving treatments or procedures
with the last-timestep alarm. Therefore, it is significant to evaluate models’
early prediction ability.

We use the area under the receiver operating characteristic (AUROC) and the
area under the precision-recall curve (AUPRC) to evaluate the regular
mortality prediction performance. We also design the early prediction score
(ES) to evaluate models’ early prediction performance. The illustration of ES
is shown in Fig. 2(b). Inspired by previous works [19], the idea is to give
early true positive predictions full score while penalizing late false
negative predictions, which means the model cannot predict patients’ outcomes
correctly even at the last moments. We do not give true negative or false
positive predictions any score or penalty. We normalize the total ES for one
patient as:

$ES_{normalized}=\max{(ES_{total}/ES_{optimal},-1)}$

This is to make sure that the ground-truth (i.e., the highest possible score)
receives a normalized score of 1 and the worst algorithm that only outputs
negative predictions receives a normalized score of -1.

###  2.2 Model training and evaluation

Model training and evaluation is one of the most essential parts of the
benchmark, which ensures the fairness of comparison and reproducibility of the
results. To this end, we adopt two different training and evaluation
strategies for two datasets.

Since the size of TJH dataset is much smaller than CDSL, to reduce the
randomness of the test sample selection process, we use the stratified 10-fold
cross-validation strategy to train and evaluate the models. We randomly split
patients into 10 folds. Then we repeat the training and evaluation process 10
times. At the $i$-th iteration, we select the $i$-th fold as the test set and
split other folds into training and validation sets in a ratio of 8:1. We
provide an illustration example of this process in Figure 3. Finally, we
report the means and standard deviations of the model performances on all test
sets of 10 iterations. This strategy is to ensure we can evaluate models’
generalizability fairly with a small sample size. For the CDSL dataset, the
sample size is much larger. Therefore, the randomness of the test sample
selection process is smaller. We adopt the traditional train-validation-test
split setting in an 8:1:1 ratio.

Figure 3: The K-fold cross-validation strategy. We take 4-fold as an example
in the figure. We use a stratified shuffle split to ensure the proportions of
alive and dead patients on all folds are the same as the total cohort.

All deep learning models are trained on the training set for 100 epochs with
an early stopping strategy. On each epoch, we evaluate models’ performances on
the validation set. After the training procedure, we load the model parameters
with the best performance on the validation set and then evaluate the model on
the test set. For the outcome prediction task, we choose models based on the
AUPRC score. For the length of stay prediction task, we use the MAE score as
the criteria.

###  2.3 Baseline selection and hyper-parameter setting

In this work, we try to incorporate related works as our baseline models as
broad as possible. To provide a comprehensive benchmark comparison between
existing works, we categorize the baseline models into three categories:
machine learning models, basic deep learning models and EHR-specific
predictive models. Machine learning and basic deep learning models are popular
in the general classification and regression tasks. EHR-specific predictive
models are specifically designed for clinical predictive tasks with EHR data.

Machine learning models

  * •

Decision tree (DT) is a non-parametric supervised learning algorithm with a
hierarchical tree structure. DT has been widely applied in many clinical
predictive tasks, such as mortality prediction for peritoneal dialysis
patients[20].

  * •

Random forest (RF) is a supervised ensemble learning method for
classification, regression and other tasks. It constructs a multitude of
decision trees during the training period. RF has been used to predict the
patients’ severity of the COVID-19 case and possible mortality outcome,
recovery or death [21].

  * •

Gradient Boosting Decision Tree (GBDT) is a widely-used algorithm for solving
prediction problems in both classification and regression tasks. GBDT takes
decision trees as the base learner and determines final prediction results
based on a series of DTs’ outputs. GBDT exhibits good performance in
performing COVID-19 mortality prediction tasks with structured clinical data
[22].

  * •

XGBoost (XGB) [23] is a recursive tree-based supervised machine learning
classifier. XGBoost has been used to predict the mortality for COVID-19
infected patients[3].

  * •

CatBoost [24] is also a gradient boost machine learning framework. It builds a
symmetric tree and uses an ordered boosting strategy with a faster training
and prediction speed compared to XGBoost. Catboost has been used to predict
and detect the number of confirmed and death cases of COVID-19 [25].

Basic deep learning models

  * •

Multi-layer perceptron (MLP) is the feed-forward-based neural network. MLP has
been used to predict acute kidney injury[26].

  * •

Recurrent neural network (RNN) [27] is the most popular framework to learn the
abstract embedding of variable-length time series. RNN has been widely used to
predict the risk of the first episode of psychosis or heart failure[28, 29].

  * •

Long-short term memory network (LSTM) [30] is a variant of the Recurrent
Neural Network (RNN), capable of learning long-term dependencies. LSTM has
been used to perform the 90-day all-cause mortality in the intensive care unit
(ICU), based on the concatenated static features and dynamic features[31].

  * •

Gated recurrent units (GRU) [32] embeds the time series as the input to
perform the target prediction. It is a widely applied variant of the Recurrent
Neural Network (RNN), which improves the capability to maintain historical
memories and reduces parameters in the update and reset gates. GRU has been
used to predict several severe complications (mortality, renal failure with a
need for renal replacement therapy, and postoperative bleeding leading to
operative revision) in post-cardiosurgical care in real-time[33].

  * •

Temporal convolutional networks (TCN) [34] is a generic temporal convolutional
network architecture for sequence modeling. TCN has been used to forecast
hospital resource utilization (i.e., the number of hospital beds and
ventilators) for COVID-19 patients [35].

  * •

Transformer [36] in this experiment is the encoder part of the original
Transformer, which comprises the positional encoding module and the self-
attention module. The transformer has been used to perform the mortality risk
analysis for liver transplant recipients [37].

EHR-specific predictive models

  * •

RETAIN [38] is the deep-based Reverse Time Attention model for analyzing EHR
data. It utilizes a two-level neural attention module to attend important
clinical visits and features. RETAIN has been used to predict heart failure by
taking previous diagnoses (categorical variables) as features. In our
benchmark, we modify the input layer of RETAIN with multi-layer perceptron
(MLP) to make it capable of using numerical variables as features.

  * •

StageNet [39] is the Stage-aware neural Network, which extracts disease stage
information from EHR and integrates it into risk prediction. This model
comprises a stage-aware LSTM module that extracts health stage variations
unsupervisedly and a stage-adaptive convolutional module that incorporates
stage-related progression patterns. StageNet has been used to perform the
decompensation prediction for ICU patients in the MIMIC-III dataset and the
mortality prediction of End-Stage Renal Disease (ESRD) patients in Peking
University Third Hospital.

  * •

Dr. Agent [40] augments RNN with 2 policy gradient agents. It learns a dynamic
skip connection to focus on the relevant information over time. Dr. Agent has
been used to perform the in-hospital mortality prediction, acute care
phenotype classification, physiologic decompensation prediction and length of
stay forecasting task on the MIMIC-III (Medical Information Mart for Intensive
Care) dataset.

  * •

AdaCare [41] employs a multi-scale adaptive dilated convolutional module to
capture the long and short-term variations of biomarkers to depict the health
status in multiple time scales. AdaCare has been used to perform
decompensation prediction on the MIMIC-III dataset and the mortality
prediction on the End-Stage Renal Disease (ESRD) dataset.

  * •

ConCare [42] employs multi-channel embedding architecture and self-attention
mechanism to model the relationship of feature sequences and build the health
representation. ConCare has been used to perform the morality prediction on
the MIMIC-III and ESRD datasets.

  * •

GRASP [43] is a generic framework for healthcare models, which leverages the
information extracted from patients with similar conditions to enhance the
cohort representation learning results. GRASP has been used to perform the
sepsis prediction on the cardiology dataset and mortality prediction on the
ESRD dataset.

For machine learning models, we decide their hyper-parameters using the grid-
search strategy on the validation set for each task. The search range and the
final hyper-parameter settings are shown in Appendix. The hyperparameters of
some deep learning models are more complex. We use the recommended setting in
the original work. We also run the grid-search for the dimensions of the main
network components (e.g., the dimension of LSTM or GRU) in the range of [64,
128, 256].

Machine learning models and some deep learning models (e.g., MLP) cannot
handle sequential data as input. When training these models, we conduct visit-
level prediction, using the feature values at the current timestep as input
and predicting the target. Furthermore, since the outcome-specific length-of-
stay prediction is defined as a multi-target prediction task, we adopt two
strategies to train the model. The first strategy is end-to-end multi-task
learning. We train one single model backbone and two MLP prediction heads, one
for outcome prediction and another for length-of-stay prediction. The backbone
and prediction heads are trained in an end-to-end way. The second strategy is
two-stage training. We train two separate models with identical structures and
hyper-parameters to predict outcome and length-of-stay, respectively. The
illustrations of the two training settings are shown in Figure 4. We report
the performances for both settings. It is worth noting that machine learning
models do not support multi-task learning. Therefore, we only report the two-
stage training performances for machine learning models.

Figure 4: Illustrations of the two-stage training and multi-task training
settings

###  2.4 Benchmarking performance analysis

In this section, we report the detailed benchmarking results of all predictive
models on all datasets and tasks.

####  2.4.1 Benchmarking performance of early mortality prediction

The benchmarking performances of early mortality prediction are shown in Table
1. For the TJH dataset, AdaCare achieves the highest AUPRC, AUROC and ES on
the TJH dataset. ConCare achieves the highest accuracy. For the CDSL dataset,
ConCare achieves the highest AUPRC, AUROC and accuracy. StageNet achieves the
highest ES. In machine learning models, GBDT and CatBoost achieve better
performances on both datasets. In basic deep learning models, TCN and RNN
achieve better performances on both datasets.

For the TJH dataset, the performances of all models are high, which indicates
that the task is easier on this dataset. We also find that the performances of
EHR-specific models are generally better than basic deep learning models,
while basic deep learning models perform better than machine learning models.
This is natural since most EHR-specific models can better utilize
characteristics in EHR data such as disease progressions, while machine
learning models cannot utilize temporal relationships in sequential data. We
notice that the small sample size of the TJH dataset has significant effects
on some deep learning models’ performances. For example, StageNet achieves low
performances with high standard deviations on the TJH dataset, while it
achieves good performances on the CDSL dataset. This reveals that the small
test size of the TJH dataset causes the model performs poorly on some testing
folds. Therefore, improving the stability of deep learning models on small EHR
datasets is still challenging.

Table 1: Benchmarking performance of the early mortality prediction task. (Top rows) ML models; (Middle rows) Basic DL models; (Bottom rows) EHR-specific DL models. The reported score is of the form $mean\pm std$. All 4 metrics are multiplied by 100 for readability purposes. Dataset | TJH | CDSL
---|---|---
Metric |  AUPRC ($\uparrow$) |  AUROC ($\uparrow$) |  Accuracy ($\uparrow$) |  ES ($\uparrow$) |  AUPRC ($\uparrow$) |  AUROC ($\uparrow$) |  Accuracy ($\uparrow$) |  ES ($\uparrow$)
RF |  94.59 $\pm$ 3.25 |  92.95 $\pm$ 3.96 |  93.22 $\pm$ 3.90 |  82.91 $\pm$ 7.29 |  58.33 $\pm$ 0.94 |  68.44 $\pm$ 0.37 |  91.62 $\pm$ 0.15 |  10.93 $\pm$ 2.30
DT |  91.29 $\pm$ 2.02 |  88.46 $\pm$ 3.22 |  88.80 $\pm$ 2.57 |  70.05 $\pm$ 12.41 |  47.53 $\pm$ 0.73 |  66.18 $\pm$ 0.47 |  89.35 $\pm$ 0.13 |  10.81 $\pm$ 0.80
GBDT |  95.28 $\pm$ 2.97 |  94.15 $\pm$ 3.32 |  94.22 $\pm$ 3.41 |  89.23 $\pm$ 4.81 |  58.93 $\pm$ 1.57 |  70.82 $\pm$ 0.39 |  91.60 $\pm$ 0.31 |  22.86 $\pm$ 1.04
XGBoost |  94.99 $\pm$ 3.16 |  93.66 $\pm$ 3.01 |  93.75 $\pm$ 3.21 |  90.33 $\pm$ 4.87 |  57.48 $\pm$ 0.00 |  70.83 $\pm$ 0.00 |  91.26 $\pm$ 0.00 |  27.75 $\pm$ 0.00
CatBoost |  94.82 $\pm$ 2.79 |  93.57 $\pm$ 3.15 |  93.66 $\pm$ 3.19 |  90.30 $\pm$ 5.66 |  60.80 $\pm$ 0.93 |  71.95 $\pm$ 0.52 |  91.93 $\pm$ 0.17 |  23.64 $\pm$ 0.77
MLP |  98.78 $\pm$ 1.19 |  98.85 $\pm$ 1.16 |  95.09 $\pm$ 3.87 |  89.76 $\pm$ 8.91 |  68.09 $\pm$ 4.99 |  91.61 $\pm$ 1.14 |  92.38 $\pm$ 0.71 |  36.22 $\pm$ 8.39
RNN |  99.00 $\pm$ 1.27 |  99.04 $\pm$ 1.18 |  95.54 $\pm$ 3.62 |  89.94 $\pm$ 5.47 |  70.18 $\pm$ 2.73 |  91.92 $\pm$ 0.67 |  92.40 $\pm$ 0.61 |  44.51 $\pm$ 2.57
LSTM |  98.66 $\pm$ 1.69 |  98.78 $\pm$ 1.43 |  94.63 $\pm$ 4.03 |  88.03 $\pm$ 6.47 |  66.22 $\pm$ 3.49 |  91.33 $\pm$ 0.82 |  91.72 $\pm$ 0.76 |  38.51 $\pm$ 4.11
GRU |  98.45 $\pm$ 1.47 |  98.54 $\pm$ 1.37 |  94.03 $\pm$ 3.45 |  85.66 $\pm$ 6.72 |  67.75 $\pm$ 3.18 |  91.35 $\pm$ 0.80 |  91.83 $\pm$ 0.50 |  39.95 $\pm$ 5.84
TCN |  99.21 $\pm$ 0.99 |  99.21 $\pm$ 1.04 |  95.62 $\pm$ 3.31 |  93.19 $\pm$ 5.08 |  69.96 $\pm$ 7.90 |  91.33 $\pm$ 2.19 |  92.10 $\pm$ 1.22 |  41.23 $\pm$ 11.74
Transformer |  98.32 $\pm$ 1.62 |  98.40 $\pm$ 1.50 |  93.32 $\pm$ 3.67 |  87.73 $\pm$ 6.62 |  64.66 $\pm$ 4.30 |  90.77 $\pm$ 1.17 |  91.75 $\pm$ 0.62 |  30.09 $\pm$ 4.49
RETAIN |  99.24 $\pm$ 0.72 |  99.21 $\pm$ 0.86 |  95.13 $\pm$ 2.88 |  88.43 $\pm$ 8.03 |  68.77 $\pm$ 3.59 |  91.02 $\pm$ 1.09 |  93.00 $\pm$ 0.43 |  36.75 $\pm$ 3.94
StageNet |  98.31 $\pm$ 1.67 |  98.38 $\pm$ 1.52 |  91.92 $\pm$ 4.59 |  73.08 $\pm$ 16.58 |  68.18 $\pm$ 1.91 |  91.73 $\pm$ 0.65 |  91.84 $\pm$ 0.40 |  42.56 $\pm$ 2.17
Dr. Agent |  99.10 $\pm$ 1.05 |  99.14 $\pm$ 0.96 |  95.07 $\pm$ 3.38 |  91.67 $\pm$ 5.76 |  67.87 $\pm$ 0.79 |  91.30 $\pm$ 0.28 |  91.85 $\pm$ 0.24 |  40.19 $\pm$ 2.48
AdaCare |  99.29 $\pm$ 0.89 |  99.29 $\pm$ 0.93 |  95.83 $\pm$ 3.02 |  91.16 $\pm$ 5.25 |  69.55 $\pm$ 0.53 |  91.70 $\pm$ 0.14 |  92.25 $\pm$ 0.06 |  39.83 $\pm$ 2.19
GRASP |  98.66 $\pm$ 1.50 |  98.75 $\pm$ 1.33 |  94.55 $\pm$ 2.79 |  86.26 $\pm$ 3.94 |  67.19 $\pm$ 1.46 |  91.39 $\pm$ 0.46 |  92.19 $\pm$ 0.47 |  31.95 $\pm$ 6.53
ConCare |  98.90 $\pm$ 0.97 |  98.86 $\pm$ 0.97 |  96.04 $\pm$ 2.42 |  91.14 $\pm$ 6.99 |  75.33 $\pm$ 2.99 |  93.35 $\pm$ 0.83 |  93.33 $\pm$ 0.34 |  38.09 $\pm$ 9.56

We plot the ROC and PRC curves for the 3 best models on both datasets, as
shown in Figure 5 and 6.

(a) TJH ROC Curves

(b) CDSL ROC Curves

Figure 5: ROC curves of the 3 best models on TJH and CDSL datasets.

(a) TJH PRC Curves

(b) CDSL PRC Curves

Figure 6: PRC curves of the 3 best models on TJH and CDSL datasets.

####  2.4.2 Benchmarking performance of outcome-specific length-of-stay
prediction

We report the benchmarking performances of the outcome-specific length-of-stay
prediction task under both multi-task and two-stage settings in Table 2. On
the TJH dataset, TCN with two-stage learning achieves the lowest OSMAE. RETAIN
with two-stage learning achieves the lowest MSE, RMSE and MAE. On the CDSL
dataset, TCN with two-stage learning achieves the lowest MAE, MSE and RMSE.
TCN with multi-task learning achieves the lowest OSMAE.

The deep learning models achieve significantly better performance on both
datasets. This may because predicting length-of-stay heavily relies on the
extraction of patients’ temporal disease progressions. Comparing the two-stage
and multi-task learning settings, we find that on the TJH dataset, models
trained with two-stage learning generally perform better on all four metrics.
However, on the CDSL dataset, models trained with two-stage learning have
lower MAE, MSE and RMSE, but high OSMAE. We leave a detailed discussion about
this issue in the discussion section.

Table 2: Benchmarking performance of outcome-specific length-of-stay prediction. (Top rows) ML models; (Middle rows) Basic DL models; (Bottom rows) EHR-specific DL models. The reported score is of the form $mean\pm std$. Subscript $m$ denotes multi-task learning strategy, $t$ denotes two-stage learning strategy. Dataset | TJH | CDSL
---|---|---
Metric |  MAE ($\downarrow$) |  MSE ($\downarrow$) |  RMSE ($\downarrow$) |  OSMAE ($\downarrow$) |  MAE ($\downarrow$) |  MSE ($\downarrow$) |  RMSE ($\downarrow$) |  OSMAE ($\downarrow$)
$\text{RF}_{t}$ |  5.31 $\pm$ 0.74 |  59.88 $\pm$ 16.89 |  7.66 $\pm$ 1.12 |  4.73 $\pm$ 1.11 |  4.38 $\pm$ 0.00 |  51.02 $\pm$ 0.04 |  7.14 $\pm$ 0.00 |  4.46 $\pm$ 0.05
$\text{DT}_{t}$ |  6.39 $\pm$ 0.91 |  79.01 $\pm$ 21.24 |  8.80 $\pm$ 1.24 |  7.86 $\pm$ 1.19 |  4.60 $\pm$ 0.01 |  54.18 $\pm$ 0.11 |  7.36 $\pm$ 0.01 |  5.18 $\pm$ 0.03
$\text{GBDT}_{t}$ |  5.40 $\pm$ 0.75 |  61.31 $\pm$ 17.44 |  7.75 $\pm$ 1.15 |  4.56 $\pm$ 0.91 |  4.34 $\pm$ 0.01 |  50.14 $\pm$ 0.14 |  7.08 $\pm$ 0.01 |  4.40 $\pm$ 0.08
$\text{XGBoost}_{t}$ |  5.41 $\pm$ 0.82 |  62.08 $\pm$ 17.71 |  7.79 $\pm$ 1.18 |  4.74 $\pm$ 0.76 |  4.35 $\pm$ 0.00 |  50.06 $\pm$ 0.00 |  7.08 $\pm$ 0.00 |  4.54 $\pm$ 0.00
$\text{CatBoost}_{t}$ |  5.68 $\pm$ 0.77 |  67.71 $\pm$ 18.53 |  8.14 $\pm$ 1.17 |  4.96 $\pm$ 0.95 |  4.73 $\pm$ 0.01 |  56.81 $\pm$ 0.15 |  7.54 $\pm$ 0.01 |  4.66 $\pm$ 0.04
$\text{MLP}_{m}$ |  4.68 $\pm$ 0.53 |  43.52 $\pm$ 10.96 |  6.54 $\pm$ 0.83 |  4.62 $\pm$ 0.81 |  3.84 $\pm$ 0.04 |  39.90 $\pm$ 0.62 |  6.32 $\pm$ 0.05 |  3.88 $\pm$ 0.07
$\text{MLP}_{t}$ |  4.75 $\pm$ 0.61 |  43.93 $\pm$ 12.39 |  6.56 $\pm$ 0.94 |  4.18 $\pm$ 1.04 |  3.59 $\pm$ 0.16 |  28.37 $\pm$ 6.44 |  5.29 $\pm$ 0.58 |  4.09 $\pm$ 0.31
$\text{RNN}_{m}$ |  4.63 $\pm$ 0.61 |  42.74 $\pm$ 12.64 |  6.46 $\pm$ 0.99 |  5.24 $\pm$ 1.10 |  3.69 $\pm$ 0.05 |  38.05 $\pm$ 0.69 |  6.17 $\pm$ 0.06 |  3.67 $\pm$ 0.07
$\text{RNN}_{t}$ |  4.64 $\pm$ 0.49 |  41.90 $\pm$ 9.96 |  6.43 $\pm$ 0.75 |  3.95 $\pm$ 0.92 |  3.45 $\pm$ 0.19 |  27.22 $\pm$ 6.37 |  5.18 $\pm$ 0.59 |  3.76 $\pm$ 0.21
$\text{LSTM}_{m}$ |  4.05 $\pm$ 0.67 |  34.74 $\pm$ 12.57 |  5.81 $\pm$ 1.00 |  4.24 $\pm$ 1.35 |  3.68 $\pm$ 0.08 |  37.98 $\pm$ 0.89 |  6.16 $\pm$ 0.07 |  3.65 $\pm$ 0.13
$\text{LSTM}_{t}$ |  4.56 $\pm$ 0.56 |  40.71 $\pm$ 10.80 |  6.32 $\pm$ 0.85 |  4.09 $\pm$ 1.09 |  3.21 $\pm$ 0.24 |  25.26 $\pm$ 7.08 |  4.98 $\pm$ 0.66 |  3.84 $\pm$ 0.22
$\text{GRU}_{m}$ |  4.48 $\pm$ 0.56 |  40.88 $\pm$ 12.51 |  6.32 $\pm$ 0.94 |  4.71 $\pm$ 0.96 |  3.64 $\pm$ 0.04 |  37.26 $\pm$ 0.65 |  6.10 $\pm$ 0.05 |  3.64 $\pm$ 0.05
$\text{GRU}_{t}$ |  4.48 $\pm$ 0.70 |  39.54 $\pm$ 12.27 |  6.22 $\pm$ 0.91 |  4.29 $\pm$ 1.12 |  3.37 $\pm$ 0.22 |  26.53 $\pm$ 6.72 |  5.11 $\pm$ 0.63 |  3.97 $\pm$ 0.22
$\text{TCN}_{m}$ |  3.94 $\pm$ 0.77 |  32.71 $\pm$ 13.52 |  5.59 $\pm$ 1.19 |  4.02 $\pm$ 1.05 |  3.43 $\pm$ 0.05 |  34.36 $\pm$ 0.77 |  5.86 $\pm$ 0.07 |  3.26 $\pm$ 0.15
$\text{TCN}_{t}$ |  4.19 $\pm$ 0.65 |  35.48 $\pm$ 10.55 |  5.89 $\pm$ 0.88 |  3.52 $\pm$ 0.94 |  3.12 $\pm$ 0.11 |  22.87 $\pm$ 4.79 |  4.76 $\pm$ 0.48 |  3.66 $\pm$ 0.32
$\text{Transformer}_{m}$ |  4.58 $\pm$ 0.58 |  40.60 $\pm$ 10.53 |  6.32 $\pm$ 0.85 |  4.31 $\pm$ 1.16 |  3.45 $\pm$ 0.10 |  33.66 $\pm$ 1.76 |  5.80 $\pm$ 0.15 |  3.47 $\pm$ 0.24
$\text{Transformer}_{t}$ |  4.62 $\pm$ 0.59 |  39.66 $\pm$ 12.14 |  6.23 $\pm$ 0.95 |  4.68 $\pm$ 1.01 |  3.30 $\pm$ 0.19 |  24.54 $\pm$ 6.07 |  4.92 $\pm$ 0.59 |  4.03 $\pm$ 0.19
$\text{RETAIN}_{m}$ |  4.12 $\pm$ 0.63 |  32.47 $\pm$ 11.91 |  5.61 $\pm$ 1.02 |  4.37 $\pm$ 1.09 |  3.58 $\pm$ 0.07 |  36.56 $\pm$ 1.00 |  6.05 $\pm$ 0.08 |  3.45 $\pm$ 0.10
$\text{RETAIN}_{t}$ |  3.83 $\pm$ 0.99 |  30.43 $\pm$ 13.98 |  5.37 $\pm$ 1.24 |  3.57 $\pm$ 1.12 |  3.27 $\pm$ 0.16 |  25.27 $\pm$ 5.99 |  4.99 $\pm$ 0.58 |  3.97 $\pm$ 0.31
$\text{StageNet}_{m}$ |  4.14 $\pm$ 0.61 |  35.57 $\pm$ 12.73 |  5.87 $\pm$ 1.06 |  3.62 $\pm$ 0.68 |  3.70 $\pm$ 0.09 |  37.41 $\pm$ 1.22 |  6.12 $\pm$ 0.10 |  3.74 $\pm$ 0.17
$\text{StageNet}_{t}$ |  4.91 $\pm$ 0.64 |  45.17 $\pm$ 10.85 |  6.67 $\pm$ 0.83 |  5.56 $\pm$ 1.80 |  3.60 $\pm$ 0.18 |  29.08 $\pm$ 6.40 |  5.36 $\pm$ 0.56 |  4.10 $\pm$ 0.15
$\text{Dr. Agent}_{m}$ |  4.31 $\pm$ 0.63 |  38.30 $\pm$ 11.79 |  6.11 $\pm$ 0.97 |  4.00 $\pm$ 0.77 |  3.61 $\pm$ 0.08 |  37.12 $\pm$ 1.07 |  6.09 $\pm$ 0.09 |  3.61 $\pm$ 0.07
$\text{Dr. Agent}_{t}$ |  4.28 $\pm$ 0.56 |  36.30 $\pm$ 10.41 |  5.96 $\pm$ 0.87 |  3.86 $\pm$ 0.85 |  3.34 $\pm$ 0.18 |  25.92 $\pm$ 6.25 |  5.06 $\pm$ 0.59 |  3.91 $\pm$ 0.27
$\text{AdaCare}_{m}$ |  4.12 $\pm$ 0.64 |  34.40 $\pm$ 12.29 |  5.77 $\pm$ 1.02 |  4.81 $\pm$ 1.69 |  3.59 $\pm$ 0.09 |  36.72 $\pm$ 1.04 |  6.06 $\pm$ 0.09 |  3.57 $\pm$ 0.12
$\text{AdaCare}_{t}$ |  4.46 $\pm$ 0.57 |  39.60 $\pm$ 11.72 |  6.22 $\pm$ 0.93 |  3.64 $\pm$ 0.78 |  3.13 $\pm$ 0.28 |  24.02 $\pm$ 7.37 |  4.85 $\pm$ 0.70 |  3.75 $\pm$ 0.19
$\text{GRASP}_{m}$ |  4.46 $\pm$ 0.64 |  40.29 $\pm$ 12.42 |  6.27 $\pm$ 0.96 |  4.89 $\pm$ 0.98 |  3.56 $\pm$ 0.03 |  36.72 $\pm$ 0.28 |  6.06 $\pm$ 0.02 |  3.64 $\pm$ 0.09
$\text{GRASP}_{t}$ |  4.61 $\pm$ 0.63 |  40.98 $\pm$ 11.33 |  6.34 $\pm$ 0.87 |  4.26 $\pm$ 0.81 |  3.57 $\pm$ 0.26 |  28.91 $\pm$ 7.14 |  5.34 $\pm$ 0.65 |  4.06 $\pm$ 0.30
$\text{ConCare}_{m}$ |  4.50 $\pm$ 0.56 |  42.88 $\pm$ 10.36 |  6.50 $\pm$ 0.81 |  5.01 $\pm$ 1.49 |  3.25 $\pm$ 0.15 |  28.72 $\pm$ 2.88 |  5.35 $\pm$ 0.26 |  3.32 $\pm$ 0.16
$\text{ConCare}_{t}$ |  4.52 $\pm$ 0.60 |  41.48 $\pm$ 10.48 |  6.38 $\pm$ 0.86 |  3.81 $\pm$ 0.71 |  3.38 $\pm$ 0.14 |  25.91 $\pm$ 4.48 |  5.07 $\pm$ 0.44 |  3.71 $\pm$ 0.34

###  2.5 Code structure and deployment

Figure 7: Code structure. The blue box shows the folder name, and the gray box
below contains files of the folder. Dashed cells denote Python files. main.py
is the program’s main entry point.

The code structure of this benchmark pipeline is shown in Figure 7. The figure
is basically self-explanatory. All running configs for model parameters and
data splitting are stored in config.yaml. All deep learning and machine
learning models are implemented with different modules, including backbones
and prediction heads. This clearly structured and modularized design allows
users to easily extend the existing models, design new models and tasks, or
even add new datasets by adding corresponding components without affecting the
downstream benchmark prediction and evaluation process.

(a) Screenshot of the online benchmark results visualization system.

(b) Screenshot of the online prediction system with benchmark models.

Figure 8: Screenshots of our deployed online benchmark system.

We also deploy the benchmark system online, as shown in Figure 8. First, all
model performances with all hyper-parameter combinations on both tasks are
available in the system. The tables are easy to query, compare and sort. Users
can select specific rows and download the csv files or LaTex codes. We also
deploy all trained models online, allowing clinicians and researchers to
easily use these models to get prediction results by uploading their own data.

##  3 Discussion

In this section, we aim to analyze the difference between model performances
in terms of ES, OSMAE and traditional metrics. We also provide case studies to
show the predicted risk curves.

###  3.1 Performance comparison between ES, OSMAE and traditional metrics

ES and OSMAE show models’ performances on the early and outcome-specifc
perspectives of two tasks. For the early mortality prediction task, the early
prediction score (ES) and AUPRC of deep learning models on two datasets are
shown in Figure 9. For most models, the performance gaps between AUPRC and ES
are not large on both datasets. However, some models achieve high AUPRC but
low ES, for example, the StageNet model on the TJH dataset and the LSTM on the
CDSL dataset. This indicates that these models may be good at predicting last-
time status but fail to raise the alarm at the early stage. Interestingly,
though StageNet achieves low ES score on the TJH dataset, it achieves the
second highest ES on the CDSL dataset. This proves that with sufficient
training data, StageNet can still achieve good predictive performances.

Figure 9: ES and AUPRC of deep learning models on two datasets for early
mortality prediction.

For the outcome-specific length-of-stay prediction task, the OSMAE and MAE in
both multi-task and two-stage settings on two datasets are shown in Figure 10.
For the TJH dataset, we find that the MAE under both multi-task MAE and two-
stage MAE (dark and light green lines) are similar. But the OSMAE under the
two-stage setting is generally lower than in the multi-task setting. However,
in the CDSL dataset, the OSMAE under the multi-task setting is much lower
(dark and light blue lines). Considering the definition of OSMAE, these
results indicate that on the CDSL dataset, different patients’ statuses may be
more complex. Using multi-task learning on the CDSL dataset can better
discriminate patients’ status with different outcomes by predicting them
simultaneously. While in the TJH dataset, the discrepancy between patient
status with positive and negative outcomes may be more significant. The
prediction process is simpler than on the CDSL dataset (this is also proved in
the mortality prediction performance). Therefore, introducing another
prediction target may not further help the models’ learning process.

Figure 10: Comparison between OSMAE and MAE score for deep learning models.

Furthermore, we find that the values of MAE and OSMAE for most models are more
similar in the multi-task setting on both datasets (dark green and dark blue
lines), while there is a gap between MAE and OSMAE under two-stage setting
(light green and light blue lines). This shows that training models in the
multi-task setting can naturally associate length-of-stays with outcomes by
sharing the same model backbone parameters. While in the two-stage setting,
their optimization direction may not necessarily be the same, which leads to
the discrepancy between OSMAE and MAE.

To further prove this, we plot the learned hidden states in the ConCare model
with t-SNE on the CDSL dataset, as shown in Figure 11. Each green point is an
embedding of one alive patient, and a red x is an embedding of one dead
patient. We find that the embeddings of patients with the same outcomes are
more clustered in the multi-task ConCare. Therefore, multi-task models have
more consistent OSMAE and MAE scores. Future research may focus on proposing a
better multi-task model structure to optimize OSMAE.

(a) Patient embeddings of the two-stage ConCare

(b) Patient embeddings of the multi-task ConCare

Figure 11: Patient embedding visualizations for two-stage and multi-task LOS
prediction settings using t-SNE.

We also explore how the threshold $\gamma$ affects OSMAE and ES values. We
plot the OSMAE and ES values with different $\gamma$ using the same TCN multi-
task model on CDSL dataset in Figure 12. Obviously, as $\gamma$ increases,
OSMAE will increase and ES will decrease. This is because increasing the
$\gamma$ requires the model to correctly predict the target earlier, which
increases the difficulty of the task. When applied in real-world clinical
settings, clinicians can decide the threshold based on their needs.

(a) OSMAE with $\gamma$ from 0-10

(b) ES with $\gamma$ from 0-10

Figure 12: OSMAE and ES values with different $\gamma$. Scores are calculated
using the same TCN multi-task model on CDSL dataset.

###  3.2 Case studies

We present two case patients in the CDSL outcome-specific length-of-stay task
to show the prediction results. As shown in Figure 13, green and red curves
denote an alive case and a dead case, and the blue curves are the predicted
mortality risk. Each point denotes a record and the perfect LOS prediction is
the diagonal line. All prediction curves are from the results of the multi-
task ConCare model. The first alive case has a low mortality risk over the
entire ICU stay. The predicted length-of-stay curve is also decreasing and is
close to the ground-truth. We notice that there are fluctuations for both risk
and length-of-stay curves at the first few timesteps, which may be because the
patient’s status is not stable or the input sequence is too short to allow the
model to make accurate predictions. For the dead case, the predicted risk
increases rapidly during the first few records and maintains at a high-level.
This rising trend also appears in the length-of-stay curve. This may be
because the patient’s status is not very severe at the first few timesteps, so
the initially predicted outcome may be leaving the ICU. But the risk curve
shows that the patient’s status is rapidly deteriorating toward another
direction, so the length-of-stay curve shows a rise-then-decline trend. This
shows that predicting outcome and length-of-stay simultaneously can help
clinicians better understand patients’ progression.

Figure 13: The length-of-stay curve and risk curve for patients with two
different mortality outcomes.

##  4 Method

###  4.1 Background

Here, we provide a brief overview of the research background for this study,
in terms of COVID-19 predictive modeling using EHR data and benchmark for EHR
predictive models.

####  4.1.1 COVID-19 Predictive Modeling Using EHR Data

In the past two years, many machine learning and deep learning models have
been proposed to conduct COVID-19 clinical prediction tasks, including
diagnosis prediction [13, 14], length-of-stay prediction [15, 16], severity
and mortality prediction [3, 4, 5, 6, 7, 8, 9, 10, 11, 12], etc. Yan et al.
[3] conducted mortality prediction for COVID-19 patients from the Tongji
Hospital in China. Bennett et al. [10] compared multiple machine learning
models including gradient boosted tree and linear regression to predict
hospitalization severity for COVID-19 patients in the US. Gao et al. [12] used
deep learning and tree-based models to predict COVID-19 severity and
hospitalization risks. Ma et al. [15] conducted length-of-stay prediction for
hospitalized COVID-19 patients from the HM Hospitals in Spain. Though these
works have achieved good prediction performance on their own data, these
different models are applied to different datasets, and most of them are not
publicly available or have strict access restrictions. Therefore, their
results are not directly comparable and difficult to reproduce. This issue
limits these models’ practice usage and further research. Besides, the
prediction tasks in most existing works still face the task practicality issue
discussed in the introduction. Therefore, it is non-trivial to compare these
models using the same data and more practical clinical prediction tasks.

####  4.1.2 Benchmark for EHR Predictive Models

Existing EHR benchmarks mainly rely on the publicly available Intensive Care
Units (ICU) datasets MIMIC-III [44] and MIMIC-II [45]. These datasets consist
of physiological measurements, laboratory tests, diagnoses and patient
demographics. Pirracchio et al. [46] compares multiple machine learning models
and traditional clinical ICU risk scores using the MIMIC-II data. Johnson et
al. [47] studied the reproducibility issue using the results from publications
that used the MIMIC-III dataset. Purushotham et al. [48], Harutyunyan et al.
[17] and Yeche et al. [49] proposed three standardized benchmarks using the
MIMIC-III datasets. They all compared various machine learning and deep
learning models using multiple standardized prediction tasks, including
mortality prediction, patient phenotyping, length-of-stay prediction, etc.
They also released the code to reproduce the benchmarking results. Same with
the COVID-19 predictive tasks, some of the benchmark tasks are not well-
designed for practical clinical usage and also face the task practicality
issue. Besides, these benchmarks only compare naive machine learning and deep
learning models’ performance, such as naive RNN or LSTM, while there have been
many advanced deep learning models designed specifically for EHR data and
tasks. In this work, we aim to fill this gap by proposing a standardized and
comprehensive benchmark for COVID-19 tasks by comparing machine learning,
basic deep learning and also state-of-the-art EHR-specified models.

###  4.2 Pipeline design

The benchmark data process pipeline consists of four steps: data cleaning,
merging, normalization and imputation.

####  4.2.1 Pre-Processing:

Though both datasets are well-formatted data tables, there are still some
artifacts in the data requiring pre-processing steps. Previous machine
learning and deep learning works on COVID-19 predictive modeling do not have a
unified setting for these pre-processing details, which may cause subtle
performance differences. In this work, we propose a reproducible and reliable
pre-processing pipeline to lay the foundation for a fair comparison between
different models.

Data cleaning and merging: To build a structured EHR data format that can be
fed into a machine learning (ML) or deep learning (DL) model, we first pre-
process the dataset into feature matrices. Each patient’s longitudinal EHR
records will be represented as a matrix, where each column represents a
certain feature or label, and each row represents a record. We also extract
patients’ demographic information as static data. At this step, we calculate
the ground truth labels (i.e., length of stays and mortality outcomes).

For error records cleaning, we first calculate the statistics of each feature,
including mean, standard deviations, min, max, median and missing rate. For
example, in the TJH dataset, we observe that `2019-nCoV nucleic acid
detection` feature are all the same and some other feature values are
negative. We exclude these apparent error records. However, we also find that
there are also some outliers that may be incorrect but are difficult to
identify with simple observations in the TJH dataset. To filter these
outliers, we follow the three-sigma rule. Feature values with a z-score larger
than 3 are excluded. We find that the CDSL dataset has fewer errors, we only
remove apparent errors such as oxygen saturation values above 100, and max
blood pressure higher than 220. We replace all filtered values with NaN (Not a
Number).

In the raw dataset, the record-level missing rates are 84.25% in the TJH
dataset and 98.38% in the CDSL dataset. Such high missing rates may affect
models’ prediction performances. Therefore, following the previous benchmark
preprocess settings [17], we merge data into day-level, thereby reducing the
missing rate of recorded physiological characteristics. Such data merging will
not lose significant information since most records (over 95%) do not record
different values for the same feature on the same day. If there are multiple
records of the same feature on the same day, we record their mean value.
Furthermore, for the CDSL dataset, the feature dimensions are much larger and
features are sparser than in the TJH dataset: nearly 85 percent of the
features have a missing rate beyond 90%, meaning over 90% patients never have
records for these features. Therefore, we remove features with a missing rate
higher than 90%.

Statistics of the cleaned dataset are shown in Table 3. The distributions of
the length of stays are shown in Figure 14. We plot the distributions of 16
features with the lowest missing rates in two datasets in Figure 15 and 16. We
also report the detailed feature lists and statistics in the Appendix.

Table 3: Statistics for TJH and CDSL dataset. The reported statistics are of the form $Median[Q1,Q3]$. Dataset | TJH Dataset | CDSL Dataset
---|---|---
Mortality Outcome | Total | Alive | Dead | Total | Alive | Dead
# Patients | 361 | 195 (54.02%) | 166 (45.98%) | 4255 | 3715 (87.31%) | 540 (12.69%)
# Records | 1338 | 718 (53.66%) | 620 (46.34%) | 42204 | 34211 (81.06%) | 5013 (18.94%)
# Avg. records | 3 [2, 5] | 3 [2, 5] | 3 [2, 5] | 8 [5, 12] | 8 [5, 12] | 7 [4, 12.25]
Age | 62.0 [46.0, 70.0] | 51.0 [37.0, 62.0] | 69.0 [62.25, 77.0] | 67.2 [56.0, 80.0] | 65.1 [54.0, 77.0] | 81.6 [75.0, 89.0]
Gender | 58.7% Male | 47.2% Male | 72.3% Male | 59.1% Male | 58.5% Male | 63.3% Male
# Features | 73 | 97
Length of stay | 10.0 [4.0, 15.0] | 14.0 [9.0, 17.0] | 5.0 [3.0, 10.0] | 7.0 [4.0, 11.0] | 7.0 [4.0, 11.0] | 7.0 [4.0, 13.0]
Figure 14: Length of stay distributions in two datasets. To keep the figure
informative, we only show the statistics in the 0%-95% range for the CDSL
dataset. Figure 15: Distributions of 16 features with the lowest missing rates
in the TJH dataset. We plot the data distributions of 16 features with the
lowest missing rates for the overall, alive, and dead patients. The blue bars
are distributions of total patients. The green and pink curves are
distributions for alive and dead patients, respectively. To keep the figure
informative, we only show the statistics in the 0%-95% range. Figure 16:
Feature distributions of the CDSL dataset. We plot the data distributions of
16 features with the lowest missing rates for the overall, alive, and dead
patients. The blue bars are distributions of total patients. The green and
pink curves are distributions for alive and dead patients, respectively. To
keep the figure informative, we only show the statistics in the 0%-95% range.

Data normalization and imputation:

Most deep learning models require numeric scale stability in feature
distributions. We apply the Z-score normalization for all demographic
features, vital signs, lab test features and also the length of stays. The
Z-score normalization is calculated as:

$\hat{x}=\frac{x-\mu}{\sigma}$

We calculate the mean $\mu$ and standard deviation $\sigma$ based on the data
in 5% to 95% quantile range.

We take a forward-filling imputation strategy to fill missing values to
prevent future information leakage. Concretely, we fill missing values with
the last recent values. In other cases where the patient has no former record
of a feature, we fill the missing value with the median value of all patients.

The data preprocessing details are shown in Figure 17.

Figure 17: Data pre-processing details of two datasets.

##  5 Conclusion

In this work, we propose a comprehensive benchmark for COVID-19 predictive
modeling with EHR data in critical care. We set up the data pre-processing
pipeline, including cleaning, filtering, missing value imputation and cohort
construction for two real-world COVID-19 EHR datasets. We propose two tasks
based on the clinical practice of COVID-19 patients: Outcome-specific length-
of-stay prediction and Early mortality prediction. We implement 17 state-of-
the-art machine learning and deep learning prediction models from existing
COVID-19 as well as more general EHR predictive works. We conduct fair
comparisons using these models on both datasets and report the detailed
experiment results.

Accessibility and reproducibility are the keys of our benchmark. We build an
online platform to deploy all benchmark experiment results, including
experiment results for all model performances with different hyper-parameter
combinations on both tasks. We also deploy all trained models online, which
allows clinicians to easily use these models to get prediction results using
their own data directly. We hope our efforts can further facilitate deep
learning and machine learning research for COVID-19 predictive modeling.

## Code availability

The code for this benchmark work is available on GitHub
(https://github.com/yhzhu99/covid-ehr-benchmarks). The online benchmark system
is available at http://106.15.42.91:8000/.

## Acknowledgements

Liantao Ma was supported by China Postdoctoral Science Foundation
(2021TQ0011). Junyi Gao was supported by Health Data Research UK, an
initiative funded by UK Research and Innovation, the Department of Health and
Social Care (England) and the devolved administrations, and leading medical
research charities.

## Author contributions statement

Junyi Gao and Liantao Ma designed this study, including tasks, pipelines,
experiments and result analysis. Yinghao Zhu and Wenqing Wang processed the
datasets and built the models. All authors wrote and reviewed the manuscript.

## Competing interests

All authors declare that they have no conflicts of interest.

## References

  * [1] Organization, W. H.  Who coronavirus (covid-19) dashboard (2022).
  * [2] Callaway, E., Ledford, H. _et al._ How bad is omicron? what scientists know so far.  _Nature_ 600, 197–199 (2021).
  * [3] Yan, L. _et al._ An interpretable mortality prediction model for covid-19 patients.  _Nature machine intelligence_ 2, 283–288 (2020).
  * [4] Wynants, L. _et al._ Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal.  _bmj_ 369 (2020).
  * [5] Jamshidi, E. _et al._ Using machine learning to predict mortality for covid-19 patients on day 0 in the icu.  _Frontiers in digital health_ 3 (2021).
  * [6] Martin, B. _et al._ Characteristics, outcomes, and severity risk factors associated with sars-cov-2 infection among children in the us national covid cohort collaborative.  _JAMA network open_ 5, e2143151–e2143151 (2022).
  * [7] Nachega, J. B. _et al._ Assessment of clinical outcomes among children and adolescents hospitalized with covid-19 in 6 sub-saharan african countries.  _JAMA pediatrics_ 176, e216436–e216436 (2022).
  * [8] Domínguez-Rodríguez, S. _et al._ A bayesian model to predict covid-19 severity in children.  _The Pediatric Infectious Disease Journal_ 40, e287–e293 (2021).
  * [9] Oliveira, E. A. _et al._ Comparison of the first and second waves of the coronavirus disease 2019 pandemic in children and adolescents in a middle-income country: Clinical impact associated with severe acute respiratory syndrome coronavirus 2 gamma lineage.  _The Journal of pediatrics_ 244, 178–185 (2022).
  * [10] Bennett, T. D. _et al._ Clinical characterization and prediction of clinical severity of sars-cov-2 infection among us adults using data from the us national covid cohort collaborative.  _JAMA network open_ 4, e2116901–e2116901 (2021).
  * [11] Elliott, J. _et al._ Covid-19 mortality in the uk biobank cohort: revisiting and evaluating risk factors.  _European journal of epidemiology_ 36, 299–309 (2021).
  * [12] Gao, J. _et al._ Medml: Fusing medical knowledge and machine learning models for early pediatric covid-19 hospitalization and severity prediction.  _Iscience_ 104970 (2022).
  * [13] Zoabi, Y., Deri-Rozov, S. & Shomron, N.  Machine learning-based prediction of covid-19 diagnosis based on symptoms.  _npj digital medicine_ 4, 1–5 (2021).
  * [14] Feng, Z. _et al._ Early prediction of disease progression in covid-19 pneumonia patients with chest ct and clinical characteristics.  _Nature communications_ 11, 1–9 (2020).
  * [15] Ma, L. _et al._ Distilling knowledge from publicly available online emr data to emerging epidemic for prognosis.  In _Proceedings of the Web Conference 2021_ , 3558–3568 (2021).
  * [16] Dan, T. _et al._ Machine learning to predict icu admission, icu mortality and survivors’ length of stay among covid-19 patients: toward optimal allocation of icu resources.  In _2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)_ , 555–561 (IEEE, 2020).
  * [17] Harutyunyan, H., Khachatrian, H., Kale, D. C., Ver Steeg, G. & Galstyan, A.  Multitask learning and benchmarking with clinical time series data.  _Scientific data_ 6, 1–18 (2019).
  * [18] Hospitales, H.  Covid data save lives (2022).
  * [19] Reyna, M. A. _et al._ Early prediction of sepsis from clinical data: the physionet/computing in cardiology challenge 2019.  In _2019 Computing in Cardiology (CinC)_ , Page–1 (IEEE, 2019).
  * [20] Noh, J. _et al._ Prediction of the mortality risk in peritoneal dialysis patients using machine learning models: a nation-wide prospective cohort in korea.  _Scientific reports_ 10, 1–11 (2020).
  * [21] Iwendi, C. _et al._ Covid-19 patient health prediction using boosted random forest algorithm.  _Frontiers in public health_ 8, 357 (2020).
  * [22] Li, S. _et al._ Development and external evaluation of predictions models for mortality of covid-19 patients using machine learning method.  _Neural Computing and Applications_ 1–10 (2021).
  * [23] Chen, T. & Guestrin, C.  Xgboost: A scalable tree boosting system.  In _Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining_ , 785–794 (2016).
  * [24] Dorogush, A. V., Ershov, V. & Gulin, A.  Catboost: gradient boosting with categorical features support.  _arXiv preprint arXiv:1810.11363_ (2018).
  * [25] Kim, J. S.  Covid-19 prediction and detection using machine learning algorithms: Catboost and linear regression.  _American Journal of Theoretical and Applied Statistics_ 10, 208–215 (2021).
  * [26] Tomašev, N. _et al._ A clinically applicable approach to continuous prediction of future acute kidney injury.  _Nature_ 572, 116–119 (2019).
  * [27] Rumelhart, D. E., Hinton, G. E. & Williams, R. J.  Learning representations by back-propagating errors.  _nature_ 323, 533–536 (1986).
  * [28] Raket, L. L. _et al._ Dynamic electronic health record detection (detect) of individuals at risk of a first episode of psychosis: a case-control development and validation study.  _The Lancet Digital Health_ 2, e229—e239 (2020).
  * [29] Choi, E., Bahadori, M. T., Schuetz, A., Stewart, W. F. & Sun, J.  Doctor ai: Predicting clinical events via recurrent neural networks.  In _Machine learning for healthcare conference_ , 301–318 (PMLR, 2016).
  * [30] Hochreiter, S. & Schmidhuber, J.  Long short-term memory.  _Neural computation_ 9, 1735–1780 (1997).
  * [31] Thorsen-Meyer, H.-C. _et al._ Dynamic and explainable machine learning prediction of mortality in patients in the intensive care unit: a retrospective study of high-frequency data in electronic patient records.  _The Lancet Digital Health_ 2, e179–e191 (2020).
  * [32] Chung, J., Gulcehre, C., Cho, K. & Bengio, Y.  Empirical evaluation of gated recurrent neural networks on sequence modeling.  _arXiv preprint arXiv:1412.3555_ (2014).
  * [33] Meyer, A. _et al._ Machine learning for real-time prediction of complications in critical care: a retrospective study.  _The Lancet Respiratory Medicine_ 6, 905–914 (2018).
  * [34] Bai, S., Kolter, J. Z. & Koltun, V.  An empirical evaluation of generic convolutional and recurrent networks for sequence modeling.  _arXiv preprint arXiv:1803.01271_ (2018).
  * [35] Zhang, J., Pathak, H. S., Snowdon, A. & Greiner, R.  Learning models for forecasting hospital resource utilization for covid-19 patients in canada.  _Scientific reports_ 12, 1–14 (2022).
  * [36] Vaswani, A. _et al._ Attention is all you need.  _Advances in neural information processing systems_ 30 (2017).
  * [37] Nitski, O. _et al._ Long-term mortality risk stratification of liver transplant recipients: real-time application of deep learning algorithms on longitudinal data.  _The Lancet Digital Health_ 3, e295–e305 (2021).
  * [38] Choi, E. _et al._ Retain: An interpretable predictive model for healthcare using reverse time attention mechanism.  _Advances in neural information processing systems_ 29 (2016).
  * [39] Gao, J. _et al._ Stagenet: Stage-aware neural networks for health risk prediction.  In _Proceedings of The Web Conference 2020_ , 530–540 (2020).
  * [40] Gao, J., Xiao, C., Glass, L. M. & Sun, J.  Dr. agent: Clinical predictive model via mimicked second opinions.  _Journal of the American Medical Informatics Association_ 27, 1084–1091 (2020).
  * [41] Ma, L. _et al._ Adacare: Explainable clinical health status representation learning via scale-adaptive feature extraction and recalibration.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, 825–832 (2020).
  * [42] Ma, L. _et al._ Concare: Personalized clinical feature embedding via capturing the healthcare context.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, 833–840 (2020).
  * [43] Zhang, C. _et al._ Grasp: Generic framework for health status representation learning based on incorporating knowledge from similar patients.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 35, 715–723 (2021).
  * [44] Johnson, A. E. _et al._ Mimic-iii, a freely accessible critical care database.  _Scientific data_ 3, 1–9 (2016).
  * [45] Lee, J. _et al._ Open-access mimic-ii database for intensive care research.  In _2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society_ , 8315–8318 (IEEE, 2011).
  * [46] Pirracchio, R.  Mortality prediction in the icu based on mimic-ii results from the super icu learner algorithm (sicula) project.  _Secondary Analysis of Electronic Health Records_ 295–313 (2016).
  * [47] Johnson, A. E., Pollard, T. J. & Mark, R. G.  Reproducibility in critical care: a mortality prediction case study.  In _Machine learning for healthcare conference_ , 361–376 (PMLR, 2017).
  * [48] Purushotham, S., Meng, C., Che, Z. & Liu, Y.  Benchmarking deep learning models on large healthcare datasets.  _Journal of biomedical informatics_ 83, 112–134 (2018).
  * [49] Yèche, H. _et al._ Hirid-icu-benchmark–a comprehensive machine learning benchmark on high-resolution icu data.  _arXiv preprint arXiv:2111.08536_ (2021).

##  Appendix A Feature statistics

The statistics of all lab test features in two datasets are shown in Table
LABEL:tab:tjh_lab_tests_charactistics and
LABEL:tab:cdsl_lab_tests_charactistics. The reported values are statistics
after merging into day-level, so the missing rates of some features are larger
than 90%.

Table 4: Statistics of lab test features in the TJH dataset. The reported statistics are of the form Median[Q1, Q3]. Characteristics | Overall | Alive | Dead | Missing Rate
---|---|---|---|---
Hypersensitive cardiac troponinI | 15.60 [3.30, 64.15] | 2.65 [1.90, 6.73] | 38.40 [15.20, 199.90] | 67.49%
hemoglobin | 126.00 [114.00, 137.00] | 127.00 [117.00, 138.00] | 124.00 [112.00, 135.50] | 31.84%
Serum chloride | 101.70 [98.90, 104.40] | 100.90 [98.89, 103.12] | 102.80 [98.95, 106.90] | 36.77%
Prothrombin time | 14.50 [13.50, 16.20] | 13.60 [13.10, 14.12] | 15.90 [14.80, 17.20] | 54.78%
procalcitonin | 0.09 [0.03, 0.24] | 0.04 [0.02, 0.06] | 0.26 [0.13, 0.65] | 68.46%
eosinophils(%) | 0.05 [0.00, 0.70] | 0.50 [0.00, 1.30] | 0.00 [0.00, 0.10] | 33.71%
Interleukin 2 receptor | 660.50 [456.25, 1060.75] | 526.00 [392.00, 730.50] | 1147.00 [735.00, 1481.00] | 81.32%
Alkaline phosphatase | 67.50 [53.00, 85.00] | 59.00 [50.00, 74.00] | 78.00 [60.00, 105.12] | 36.47%
albumin | 32.50 [27.80, 36.72] | 36.20 [33.42, 39.20] | 27.90 [24.50, 31.30] | 32.44%
basophil(%) | 0.10 [0.10, 0.20] | 0.20 [0.00, 0.30] | 0.10 [0.10, 0.20] | 34.23%
Interleukin 10 | 5.20 [5.00, 10.05] | 5.00 [5.00, 7.57] | 9.90 [5.68, 14.55] | 81.91%
Total bilirubin | 10.00 [7.10, 14.00] | 8.40 [6.10, 11.72] | 12.40 [8.90, 17.30] | 38.34%
Platelet count | 177.00 [112.00, 245.25] | 227.00 [175.00, 284.00] | 113.00 [59.00, 173.00] | 31.24%
monocytes(%) | 5.60 [2.85, 8.50] | 8.10 [6.20, 9.90] | 2.95 [2.00, 4.62] | 30.57%
antithrombin | 86.00 [76.00, 97.00] | 93.00 [86.00, 102.00] | 81.00 [73.00, 92.00] | 76.98%
Interleukin 8 | 14.40 [8.20, 28.25] | 10.30 [6.70, 16.88] | 27.10 [16.45, 49.50] | 81.84%
indirect bilirubin | 5.10 [3.60, 7.30] | 4.80 [3.35, 6.70] | 5.70 [4.00, 8.00] | 38.04%
Red blood cell distribution width | 12.50 [11.90, 13.40] | 12.20 [11.80, 12.80] | 13.10 [12.40, 13.90] | 35.80%
neutrophils(%) | 82.40 [65.20, 92.20] | 66.25 [56.78, 76.10] | 92.40 [88.10, 94.60] | 29.82%
total protein | 66.55 [62.20, 70.50] | 68.30 [65.00, 71.82] | 63.55 [59.42, 68.40] | 37.37%
Quantification of Treponema pallidum antibodies | 0.05 [0.04, 0.06] | 0.05 [0.04, 0.06] | 0.05 [0.04, 0.07] | 80.49%
Prothrombin activity | 82.00 [67.00, 95.00] | 94.00 [87.00, 102.00] | 69.00 [58.62, 79.00] | 53.74%
HBsAg | 0.01 [0.00, 0.01] | 0.00 [0.00, 0.01] | 0.01 [0.00, 0.01] | 80.49%
mean corpuscular volume | 90.00 [87.10, 93.50] | 89.80 [87.10, 91.90] | 91.10 [87.10, 95.57] | 33.86%
hematocrit | 36.70 [33.70, 39.80] | 37.05 [34.30, 39.85] | 36.10 [33.00, 39.75] | 32.36%
White blood cell count | 7.29 [4.92, 11.50] | 5.50 [4.27, 7.68] | 11.10 [7.60, 14.74] | 27.73%
Tumor necrosis factor$\alpha$ | 8.30 [6.70, 10.70] | 7.70 [6.10, 9.62] | 10.30 [7.43, 13.30] | 81.61%
mean corpuscular hemoglobin concentration | 343.00 [334.00, 350.00] | 343.00 [335.00, 350.00] | 343.00 [333.00, 350.00] | 33.78%
fibrinogen | 4.12 [3.09, 5.41] | 4.38 [3.56, 5.32] | 3.92 [2.52, 5.46] | 59.94%
Interleukin $1\beta$ | 5.00 [5.00, 5.00] | 5.00 [5.00, 5.00] | 5.00 [5.00, 5.00] | 81.91%
Urea | 5.40 [3.88, 9.30] | 4.20 [3.30, 5.20] | 9.40 [6.40, 13.20] | 37.22%
lymphocyte count | 0.77 [0.46, 1.27] | 1.19 [0.85, 1.54] | 0.46 [0.31, 0.69] | 31.76%
PH value | 6.50 [6.00, 7.00] | 6.50 [6.00, 7.00] | 6.50 [6.00, 7.41] | 73.24%
Red blood cell count | 4.09 [3.63, 4.55] | 4.16 [3.72, 4.62] | 4.00 [3.55, 4.47] | 28.62%
Eosinophil count | 0.00 [0.00, 0.04] | 0.02 [0.00, 0.07] | 0.00 [0.00, 0.01] | 35.35%
Corrected calcium | 2.36 [2.27, 2.44] | 2.37 [2.28, 2.44] | 2.35 [2.27, 2.44] | 34.16%
Serum potassium | 4.36 [3.94, 4.76] | 4.27 [3.91, 4.62] | 4.46 [4.01, 4.94] | 35.13%
glucose | 6.68 [5.36, 8.94] | 5.66 [4.96, 7.26] | 8.30 [6.70, 10.77] | 47.98%
neutrophils count | 5.52 [2.97, 9.88] | 3.46 [2.43, 5.20] | 9.96 [6.68, 13.22] | 33.18%
Direct bilirubin | 4.40 [3.00, 6.60] | 3.50 [2.50, 4.75] | 6.20 [4.40, 9.00] | 38.79%
Mean platelet volume | 10.70 [10.10, 11.50] | 10.40 [9.90, 11.00] | 11.20 [10.70, 11.90] | 37.97%
ferritin | 661.40 [342.60, 1270.18] | 504.30 [234.80, 832.45] | 1390.95 [865.25, 1986.82] | 80.27%
RBC distribution width SD | 40.50 [38.30, 43.70] | 39.40 [37.60, 41.30] | 42.80 [39.55, 46.20] | 36.17%
Thrombin time | 16.60 [15.58, 17.82] | 16.40 [15.60, 17.30] | 16.90 [15.50, 18.30] | 61.73%
(%)lymphocyte | 11.45 [4.00, 24.70] | 23.75 [15.38, 32.12] | 3.90 [2.40, 6.90] | 29.90%
HCV antibody quantification | 0.06 [0.04, 0.08] | 0.06 [0.04, 0.08] | 0.06 [0.04, 0.09] | 80.72%
D-D dimer | 2.10 [0.60, 21.00] | 0.57 [0.33, 1.24] | 18.38 [2.72, 21.00] | 53.36%
Total cholesterol | 3.61 [3.01, 4.18] | 3.88 [3.34, 4.35] | 3.32 [2.74, 3.87] | 34.75%
aspartate aminotransferase | 26.00 [19.00, 38.75] | 21.00 [17.00, 28.75] | 35.00 [24.00, 48.25] | 35.58%
Uric acid | 236.90 [181.00, 317.00] | 238.60 [193.00, 303.00] | 227.00 [157.80, 331.00] | 34.90%
HCO3- | 23.60 [21.30, 25.90] | 24.65 [22.80, 26.60] | 22.10 [19.82, 24.70] | 35.43%
calcium | 2.09 [1.99, 2.19] | 2.17 [2.10, 2.25] | 2.00 [1.92, 2.08] | 31.84%
NT-proBNP | 475.00 [110.50, 1752.25] | 61.00 [22.75, 159.25] | 1001.00 [472.00, 2786.75] | 68.31%
Lactate dehydrogenase | 314.50 [212.00, 536.50] | 220.00 [189.00, 278.25] | 554.00 [417.00, 725.50] | 35.58%
platelet large cell ratio | 30.50 [25.40, 36.38] | 27.60 [23.38, 32.62] | 34.60 [29.82, 39.58] | 37.97%
Interleukin 6 | 16.95 [4.12, 47.11] | 7.57 [2.36, 21.13] | 54.88 [28.05, 100.50] | 81.39%
Fibrin degradation products | 17.80 [4.00, 150.00] | 4.00 [4.00, 4.10] | 111.90 [17.58, 150.00] | 75.56%
monocytes count | 0.39 [0.26, 0.54] | 0.42 [0.32, 0.57] | 0.34 [0.20, 0.51] | 33.26%
PLT distribution width | 12.30 [11.00, 14.00] | 11.60 [10.70, 13.00] | 13.30 [11.95, 15.10] | 38.94%
globulin | 32.60 [29.80, 36.02] | 31.80 [29.50, 35.00] | 33.90 [30.30, 37.50] | 35.13%
$\gamma$-glutamyl transpeptidase | 32.00 [21.00, 49.00] | 28.00 [19.00, 44.00] | 35.00 [25.00, 58.00] | 37.29%
International standard ratio | 1.12 [1.03, 1.28] | 1.04 [0.99, 1.09] | 1.27 [1.15, 1.39] | 54.86%
basophil count(#) | 0.01 [0.01, 0.02] | 0.01 [0.01, 0.02] | 0.01 [0.01, 0.02] | 33.63%
mean corpuscular hemoglobin | 30.90 [29.80, 32.10] | 30.70 [29.71, 31.70] | 31.15 [30.00, 32.40] | 35.43%
Activation of partial thromboplastin time | 38.80 [35.10, 42.90] | 38.65 [35.25, 42.42] | 39.00 [35.10, 43.20] | 61.21%
Hypersensitive c-reactive protein | 44.45 [4.72, 102.62] | 6.60 [1.60, 35.17] | 100.65 [61.35, 156.78] | 48.73%
HIV antibody quantification | 0.09 [0.07, 0.10] | 0.09 [0.08, 0.11] | 0.08 [0.07, 0.10] | 80.27%
serum sodium | 140.10 [137.60, 142.40] | 139.90 [137.70, 141.48] | 140.50 [137.43, 144.10] | 38.12%
thrombocytocrit | 0.20 [0.15, 0.26] | 0.24 [0.19, 0.29] | 0.15 [0.10, 0.21] | 37.89%
ESR | 28.00 [14.00, 43.50] | 25.00 [13.00, 40.00] | 35.00 [15.00, 54.00] | 72.27%
glutamic-pyruvic transaminase | 22.00 [15.00, 34.25] | 21.00 [14.00, 32.00] | 24.00 [17.00, 37.00] | 36.92%
eGFR | 89.70 [69.90, 105.00] | 100.00 [86.00, 113.90] | 75.30 [52.25, 92.65] | 35.05%
creatinine | 72.00 [57.00, 90.00] | 64.00 [54.00, 83.00] | 82.00 [64.75, 101.00] | 37.44%
Table 4: (Continued) TJH dataset lab tests measures Table 5: Statistics of lab test features in the CDSL dataset. The reported statistics are of the form Median[Q1, Q3]. Characteristics | Overall | Alive | Dead | Missing Rate
---|---|---|---|---
Heart rate value | 76.6 [67.0, 85.0] | 75.9 [66.0, 84.0] | 82.2 [70.0, 92.0] | 44.88%
Oxygen saturation value | 94.9 [94.0, 97.0] | 95.2 [94.0, 97.0] | 92.2 [90.0, 96.0] | 32.36%
Maximum blood pressure value | 123.4 [110.0, 135.0] | 122.8 [110.0, 135.0] | 127.6 [111.0, 142.0] | 61.70%
Minimum blood pressure value | 71.0 [62.0, 79.0] | 71.0 [63.0, 79.0] | 70.4 [61.0, 80.0] | 61.71%
Temperature value | 36.3 [35.9, 36.6] | 36.2 [35.9, 36.6] | 36.4 [36.0, 36.7] | 29.39%
ADW – Coeficiente de anisocitosis | 13.46 [12.10, 14.60] | 13.22 [11.90, 14.30] | 14.56 [13.10, 15.80] | 32.39%
ADW – SISTEMATICO DE SANGRE | 13.94 [12.30, 15.45] | 13.72 [12.10, 15.20] | 14.86 [13.20, 16.30] | 77.28%
ALB – ALBUMINA | 3.02 [2.60, 3.40] | 3.10 [2.70, 3.50] | 2.69 [2.30, 3.00] | 88.83%
AMI – AMILASA | 108.41 [47.00, 120.50] | 115.82 [50.00, 131.00] | 86.60 [39.00, 100.70] | 88.67%
AP – ACTIVIDAD DE PROTROMBINA | 77.83 [70.00, 88.00] | 78.04 [70.00, 87.00] | 76.97 [67.00, 89.00] | 50.88%
APTT – TIEMPO DE CEFALINA (APTT) | 31.45 [28.00, 33.70] | 31.31 [28.20, 33.60] | 32.01 [27.27, 34.20] | 54.75%
AU – ACIDO URICO | 5.24 [3.50, 6.50] | 5.00 [3.40, 6.10] | 6.19 [3.70, 8.20] | 95.64%
BAS – Bas¢filos | 0.03 [0.01, 0.04] | 0.03 [0.01, 0.04] | 0.03 [0.01, 0.04] | 32.57%
BAS – SISTEMATICO DE SANGRE | 0.03 [0.01, 0.04] | 0.03 [0.01, 0.04] | 0.04 [0.01, 0.03] | 77.06%
BAS% – Bas¢filos % | 0.36 [0.10, 0.50] | 0.39 [0.20, 0.50] | 0.24 [0.10, 0.30] | 32.55%
BAS% – SISTEMATICO DE SANGRE | 0.40 [0.10, 0.50] | 0.44 [0.20, 0.60] | 0.25 [0.10, 0.30] | 77.06%
BD – BILIRRUBINA DIRECTA | 0.49 [0.18, 0.40] | 0.36 [0.17, 0.37] | 1.13 [0.22, 0.76] | 85.12%
BE(b) – BE(b) | 3.07 [-0.05, 6.03] | 3.30 [0.47, 5.90] | 2.53 [-1.60, 6.30] | 77.75%
BE(b)V – BE (b) | 1.86 [-0.60, 4.60] | 2.49 [0.20, 4.90] | -0.18 [-3.50, 3.00] | 91.97%
BEecf – BEecf | 3.47 [-0.30, 6.90] | 3.62 [0.20, 6.73] | 3.12 [-1.80, 7.35] | 77.75%
BEecfV – BEecf | 2.35 [-0.60, 5.50] | 3.05 [0.30, 5.80] | 0.10 [-3.83, 3.60] | 91.97%
BT – BILIRRUBINA TOTAL | 0.55 [0.28, 0.59] | 0.53 [0.27, 0.56] | 0.63 [0.30, 0.74] | 85.35%
BT – BILIRRUBINA TOTAL | 0.76 [0.36, 0.78] | 0.63 [0.35, 0.73] | 1.28 [0.38, 1.05] | 70.07%
CA – CALCIO | 8.28 [7.90, 8.70] | 8.32 [8.00, 8.70] | 8.14 [7.70, 8.60] | 93.52%
CA++ – Ca++ Gasometria | 4.36 [4.18, 4.63] | 4.39 [4.21, 4.65] | 4.31 [4.13, 4.60] | 78.58%
CHCM – Conc. Hemoglobina Corpuscular Media | 33.31 [32.50, 34.20] | 33.39 [32.60, 34.20] | 32.93 [31.90, 34.00] | 32.00%
CHCM – SISTEMATICO DE SANGRE | 33.01 [32.15, 34.00] | 33.07 [32.20, 34.00] | 32.77 [31.70, 33.90] | 77.06%
CK – CK (CREATINQUINASA) | 163.57 [36.20, 139.00] | 130.31 [34.00, 124.00] | 284.43 [51.15, 237.60] | 74.02%
CL – CLORO | 103.27 [98.80, 106.67] | 102.69 [98.70, 105.90] | 104.96 [99.40, 110.22] | 80.32%
CREA – CREATININA | 0.96 [0.64, 1.07] | 0.91 [0.63, 1.01] | 1.21 [0.70, 1.46] | 31.95%
DD – DIMERO D | 3498.14 [544.00, 2309.75] | 2722.87 [504.00, 1935.25] | 7720.24 [1163.25, 6076.33] | 49.58%
EOS – Eosin¢filos | 0.11 [0.00, 0.14] | 0.11 [0.00, 0.14] | 0.08 [0.00, 0.07] | 32.57%
EOS – SISTEMATICO DE SANGRE | 0.18 [0.01, 0.23] | 0.19 [0.03, 0.25] | 0.13 [0.00, 0.14] | 77.06%
EOS% – Eosin¢filos % | 1.41 [0.00, 1.90] | 1.55 [0.00, 2.20] | 0.78 [0.00, 0.80] | 32.57%
EOS% – SISTEMATICO DE SANGRE | 2.29 [0.20, 3.30] | 2.54 [0.40, 3.60] | 1.24 [0.00, 1.40] | 77.06%
FA – FOSFATASA ALCALINA | 99.18 [56.42, 109.00] | 95.84 [56.00, 104.00] | 113.05 [62.00, 127.20] | 72.18%
FER – FERRITINA | 1297.82 [442.90, 1635.80] | 1199.15 [416.52, 1560.38] | 1873.89 [672.10, 2257.45] | 84.90%
FIB – FIBRINàGENO | 573.25 [400.00, 724.00] | 578.93 [412.00, 723.50] | 554.74 [356.25, 725.75] | 75.02%
FOS – FOSFORO | 3.39 [2.70, 3.85] | 3.32 [2.70, 3.80] | 3.54 [2.70, 4.10] | 79.95%
G-CORONAV (RT-PCR) | 0.70 [0.00, 1.00] | 0.68 [0.00, 1.00] | 0.91 [1.00, 1.00] | 96.33%
GGT – GGT | 131.87 [31.60, 150.00] | 126.20 [31.00, 140.10] | 158.11 [34.30, 210.62] | 54.18%
GLU – GLUCOSA | 128.25 [94.60, 146.38] | 121.88 [92.90, 137.00] | 157.05 [113.62, 184.78] | 36.75%
GOT – GOT (AST) | 49.39 [21.00, 49.40] | 40.94 [20.00, 47.00] | 88.18 [25.85, 61.00] | 39.19%
GPT – GPT (ALT) | 59.98 [19.00, 61.73] | 53.24 [19.00, 61.00] | 92.23 [19.00, 66.00] | 41.02%
HCM – Hemoglobina Corpuscular Media | 29.66 [28.60, 30.80] | 29.63 [28.60, 30.80] | 29.79 [28.73, 31.00] | 32.01%
HCM – SISTEMATICO DE SANGRE | 29.78 [28.80, 30.90] | 29.70 [28.70, 30.80] | 30.09 [28.90, 31.10] | 77.06%
HCO3 – HCO3- | 27.89 [24.20, 31.20] | 27.79 [24.45, 30.75] | 28.10 [23.37, 32.10] | 77.75%
HCO3V – HCO3- | 27.25 [24.05, 30.10] | 27.82 [24.80, 30.30] | 25.39 [22.10, 28.50] | 91.97%
HCTO – Hematocrito | 37.35 [33.00, 41.90] | 37.61 [33.60, 41.90] | 36.17 [30.18, 41.70] | 32.00%
HCTO – SISTEMATICO DE SANGRE | 34.12 [29.00, 38.90] | 34.48 [29.60, 39.00] | 32.60 [26.40, 38.00] | 77.06%
HEM – Hemat¡es | 4.21 [3.67, 4.76] | 4.26 [3.77, 4.78] | 4.01 [3.30, 4.65] | 31.99%
HEM – SISTEMATICO DE SANGRE | 3.80 [3.18, 4.39] | 3.86 [3.25, 4.43] | 3.56 [2.90, 4.17] | 77.06%
HGB – Hemoglobina | 12.46 [10.80, 14.10] | 12.58 [11.10, 14.10] | 11.92 [9.80, 13.80] | 32.00%
HGB – SISTEMATICO DE SANGRE | 11.27 [9.50, 12.90] | 11.41 [9.70, 13.00] | 10.67 [8.80, 12.30] | 77.06%
INR – INR | 1.26 [1.09, 1.27] | 1.24 [1.09, 1.27] | 1.34 [1.08, 1.31] | 50.87%
K – POTASIO | 4.26 [3.83, 4.67] | 4.25 [3.84, 4.64] | 4.32 [3.82, 4.76] | 33.25%
LAC – LACTATO | 1.70 [1.10, 2.07] | 1.59 [1.05, 1.94] | 1.97 [1.23, 2.30] | 77.34%
LDH – LDH | 621.16 [414.70, 713.30] | 558.62 [398.40, 657.00] | 909.29 [574.75, 1015.00] | 40.32%
LEUC – Leucocitos | 8.68 [5.39, 10.33] | 7.88 [5.22, 9.43] | 12.36 [7.31, 15.07] | 31.99%
LEUC – SISTEMATICO DE SANGRE | 9.33 [5.74, 10.97] | 8.29 [5.48, 9.78] | 13.73 [8.24, 16.26] | 77.06%
LIN – Linfocitos | 1.28 [0.71, 1.52] | 1.35 [0.78, 1.59] | 0.93 [0.49, 1.11] | 32.03%
LIN – SISTEMATICO DE SANGRE | 1.51 [0.78, 1.74] | 1.63 [0.90, 1.84] | 0.98 [0.53, 1.16] | 77.06%
LIN% – Linfocitos % | 17.26 [8.50, 23.90] | 18.97 [10.30, 25.50] | 9.47 [4.30, 11.80] | 32.03%
LIN% – SISTEMATICO DE SANGRE | 18.36 [9.00, 25.50] | 20.67 [11.60, 27.80] | 8.61 [4.25, 10.80] | 77.06%
MG – MAGNESIO | 2.09 [1.79, 2.31] | 2.02 [1.76, 2.24] | 2.25 [1.91, 2.51] | 78.00%
MONO – Monocitos | 0.61 [0.36, 0.77] | 0.60 [0.38, 0.77] | 0.69 [0.30, 0.80] | 32.51%
MONO – SISTEMATICO DE SANGRE | 0.66 [0.42, 0.83] | 0.65 [0.43, 0.82] | 0.67 [0.34, 0.88] | 77.06%
MONO% – Monocitos % | 7.86 [4.80, 10.30] | 8.35 [5.40, 10.70] | 5.64 [3.00, 7.20] | 32.50%
MONO% – SISTEMATICO DE SANGRE | 8.09 [5.20, 10.40] | 8.75 [6.10, 10.90] | 5.31 [3.20, 6.70] | 77.06%
NA – SODIO | 138.50 [135.50, 140.80] | 138.05 [135.50, 140.15] | 140.52 [135.10, 144.80] | 33.22%
NEU – Neutr¢filos | 6.65 [3.50, 8.33] | 5.78 [3.29, 7.26] | 10.62 [5.87, 13.04] | 32.02%
NEU – SISTEMATICO DE SANGRE | 6.94 [3.55, 8.57] | 5.77 [3.27, 7.16] | 11.86 [6.69, 14.05] | 77.06%
NEU% – Neutr¢filos % | 73.04 [63.50, 84.90] | 70.70 [61.30, 82.10] | 83.69 [79.30, 91.25] | 32.02%
NEU% – SISTEMATICO DE SANGRE | 70.75 [60.20, 83.40] | 67.51 [57.02, 79.10] | 84.42 [80.12, 90.90] | 77.06%
PCO2 – pCO2 | 43.14 [36.00, 48.67] | 41.35 [35.33, 46.33] | 47.23 [37.88, 54.00] | 77.74%
PCO2V – pCO2 | 44.84 [38.00, 50.00] | 45.03 [39.00, 50.00] | 44.22 [36.38, 50.00] | 91.97%
PCR – PROTEINA C REACTIVA | 69.70 [9.20, 97.38] | 62.24 [8.28, 85.13] | 105.13 [18.57, 155.19] | 34.23%
PH – pH | 7.42 [7.38, 7.47] | 7.44 [7.41, 7.48] | 7.39 [7.33, 7.46] | 77.75%
PHV – pH | 7.39 [7.36, 7.44] | 7.40 [7.37, 7.44] | 7.37 [7.33, 7.43] | 91.97%
PLAQ – Recuento de plaquetas | 259.14 [174.00, 322.00] | 268.25 [182.00, 333.00] | 217.61 [142.00, 273.00] | 31.99%
PLAQ – SISTEMATICO DE SANGRE | 251.23 [169.00, 311.00] | 256.05 [174.00, 315.00] | 230.89 [141.25, 294.75] | 77.06%
PO2 – pO2 | 91.43 [65.00, 111.10] | 92.52 [64.00, 114.00] | 88.95 [68.46, 105.00] | 77.74%
PO2V – pO2 | 48.19 [27.50, 58.00] | 48.05 [28.00, 58.00] | 48.63 [27.00, 60.00] | 91.97%
PROCAL – PROCALCITONINA | 0.91 [0.08, 0.36] | 0.59 [0.07, 0.28] | 1.68 [0.12, 0.64] | 93.28%
PT – PROTEINAS TOTALES | 5.54 [5.00, 6.10] | 5.62 [5.10, 6.10] | 5.26 [4.75, 5.70] | 92.62%
SO2C – sO2c (Saturaci¢n de ox¡geno) | 93.23 [93.00, 98.00] | 93.58 [93.00, 98.00] | 92.42 [92.00, 97.33] | 77.75%
SO2CV – sO2c (Saturaci¢n de ox¡geno) | 67.87 [49.00, 90.00] | 68.43 [50.00, 90.00] | 66.09 [46.00, 89.00] | 91.98%
TCO2 – tCO2(B)c | 29.19 [25.30, 32.60] | 29.05 [25.60, 32.14] | 29.53 [24.58, 33.69] | 77.75%
TCO2V – tCO2 (B) | 28.59 [25.12, 31.59] | 29.16 [26.10, 31.80] | 26.73 [23.15, 30.00] | 92.01%
TP – TIEMPO DE PROTROMBINA | 14.02 [12.10, 14.10] | 13.79 [12.10, 14.10] | 14.92 [12.00, 14.50] | 50.88%
TROPO – TROPONINA | 44.51 [8.49, 36.40] | 27.55 [7.30, 27.45] | 101.95 [17.44, 66.75] | 91.90%
U – UREA | 54.43 [29.80, 64.10] | 47.90 [28.00, 55.90] | 84.31 [46.00, 102.00] | 33.66%
VCM – SISTEMATICO DE SANGRE | 90.26 [87.10, 93.40] | 89.86 [86.90, 93.00] | 91.93 [88.50, 94.50] | 77.06%
VCM – Volumen Corpuscular Medio | 89.09 [86.10, 92.30] | 88.77 [85.90, 91.90] | 90.55 [87.03, 94.10] | 32.00%
VPM – SISTEMATICO DE SANGRE | 10.58 [9.80, 11.30] | 10.48 [9.70, 11.20] | 11.03 [10.20, 11.80] | 77.38%
VPM – Volumen plaquetar medio | 10.46 [9.70, 11.10] | 10.36 [9.60, 11.00] | 10.93 [10.10, 11.65] | 32.72%
VSG – VSG | 50.27 [17.00, 75.00] | 50.30 [16.00, 76.00] | 50.09 [21.50, 72.00] | 96.27%
Table 5: (Continued) CDSL dataset lab tests measures

##  Appendix B Hyperparameter Search Settings

For machine learning models, we decide their hyper-parameters using the grid-
search strategy on the validation set for each task. The search range and the
final hyper-parameter settings are shown in Table 6.

Table 6: Hyper-parameter search range of machine learning models. Numbers inside the big parenthesis represent the candidates of the grid search technique, whereas the number outside denotes the chosen one with the best performance. Dataset | Task | Model |  Detail
---|---|---|---
TJH | outcome | RF | max depth 20{5, 10, 20}, n-estimators 10{10, 50, 100}, min samples split 2{2, 5, 10}
length of stay | RF | max depth 20{5, 10, 20}, n-estimators 100{10, 50, 100}, min samples split 5{2, 5, 10}
outcome | DT | max depth 10{10, 50, 100}
length of stay | DT | max depth 10{10, 50, 100}
outcome | GBDT | sub sample 0.6{0.6, 0.8, 1}, n-estimators 100{10, 50, 100}, learning rate 0.1{0.01, 0.05, 0.1}
length of stay | GBDT | sub sample 0.8{0.6, 0.8, 1}, n-estimators 100{10, 50, 100}, learning rate 0.1{0.01, 0.05, 0.1}
outcome | XGBoost | max depth 5{3, 5, 7}, min child weight 5{1, 3, 5}, learning rate 0.1{0.01, 0.05, 0.1}
length of stay | XGBoost | max depth 5{3, 5, 7}, min child weight 1{1, 3, 5}, learning rate 0.01{0.01, 0.05, 0.1}
outcome | CatBoost | max depth 3{3, 6, 9}, iterations 150{50, 100, 150}, learning rate 0.1{0.1, 0.5, 1}
length of stay | CatBoost | max depth 3{3, 6, 9}, iteration 150{50, 100, 150}, learning rate 0.1{0.1, 0.5, 1}
CDSL | outcome | RF | max depth 20{5, 10, 20}, n-estimators 100{10, 50, 100}, min samples split 10{2, 5, 10}
length of stay | RF | max depth 10{5, 10, 20}, n-estimators 100{10, 50, 100}, min samples split 2{2, 5, 10}
outcome | DT | max depth 10{10, 50, 100}
length of stay | DT | max depth 10{10, 50, 100}
outcome | GBDT | sub sample 0.6{0.6, 0.8, 1}, n-estimators 100{10, 50, 100}, learning rate 0.1{0.01, 0.05, 0.1}
length of stay | GBDT | sub sample 0.8{0.6, 0.8, 1}, n-estimators 100{10, 50, 100}, learning rate 0.1{0.01, 0.05, 0.1}
outcome | XGBoost | max depth 7{3, 5, 7}, min child weight 3{1, 3, 5}, learning rate 0.1{0.01, 0.05, 0.1}
length of stay | XGBoost | max depth 5{3, 5, 7}, min child weight 3{1, 3, 5}, learning rate 0.01{0.01, 0.05, 0.1}
outcome | CatBoost | max depth 3{3, 6, 9}, iterations 150{50, 100, 150}, learning rate 0.1{0.1, 0.5, 1}
length of stay | CatBoost | max depth 6{3, 6, 9}, iteration 150{50, 100, 150}, learning rate 0.1{0.1, 0.5, 1}