# Distillation to Enhance the Portability of Risk Models Across Institutions
with Large Patient Claims Databases

Steve Nyemba1, Chao Yan1, Ziqi Zhang2, Amol Rajmane3, Pablo Meyer4,
Prithwish Chakraborty4, Bradley Malin1,2 1 Vanderbilt University Medical
Center, Nashville, TN, USA
2 Vanderbilt University, Nashville, TN, USA
3 IBM Watson Health, MA, USA
4 Center for Computational Health, IBM Research, NY, USA

###### Abstract.

Artificial intelligence, and particularly machine learning (ML), is
increasingly developed and deployed to support healthcare in a variety of
settings. However, clinical decision support (CDS) technologies based on ML
need to be portable if they are to be adopted on a broad scale. In this
respect, models developed at one institution should be reusable at another.
Yet there are numerous examples of portability failure, particularly due to
naive application of ML models. Portability failure can lead to suboptimal
care and medical errors, which ultimately could prevent the adoption of ML-
based CDS in practice. One specific healthcare challenge that could benefit
from enhanced portability is the prediction of 30-day readmission risk.
Research to date has shown that deep learning models can be effective at
modeling such risk. In this work, we investigate the practicality of model
portability through a cross-site evaluation of readmission prediction models.
To do so, we apply a recurrent neural network, augmented with self-attention
and blended with expert features, to build readmission prediction models for
two independent large scale claims datasets. We further present a novel
transfer learning technique that adapts the well-known method of born-again
network (BAN) training. Our experiments show that direct application of ML
models trained at one institution and tested at another institution perform
worse than models trained and tested at the same institution. We further show
that the transfer learning approach based on the BAN produces models that are
better than those trained on just a single institution’s data. Notably, this
improvement is consistent across both sites and occurs after a single
retraining, which illustrates the potential for a cheap and general model
transfer mechanism of readmission risk prediction.

ai for health, model portability, transfer learning

††copyright: none††conference: DSHealth ’22: 2022 KDD Workshop on Applied Data
Science for Healthcare; August 2022; Washington DC, USA††ccs: Computing
methodologies Artificial intelligence††ccs: Applied computing Health
informatics††ccs: Computing methodologies Learning paradigms

##  1\. Introduction

The quantity of patient data continues to grow at an unprecedented pace. This
is supported, in part, by the continued adoption of electronic health record
(EHR) systems and use of greater detail in insurance claims. The large scale
nature of such resources provide an opportunity to develop artificial
intelligence, and machine learning (ML) in particular, to solve problems in a
variety of settings.

From a patient health perspective, ML applications have shown promise for the
analysis of radiology images (Sorantin et al., 2021), modeling co-morbidity of
chronic disorders (Kwon et al., 2022; John et al., 2021; Dey et al., 2021),
and forecasting of risk in developing particular diseases (Zhang et al.,
2022). From a healthcare administration perspective, there is an expectation
that ML can help detect and prevent costly healthcare events, such as the
prediction of all-cause 30-day readmission after discharge from a hospital
stay. This problem holds real world importance with both societal (preventing
readmissions reduce the quality of life for patients (Horwitz et al., 2014))
and economic implications (Fonarow et al., 2017). National payment agencies,
such as the U.S. Centers for Medicare and Medicaid Services (CMS), regularly
monitor the rate of such an event across healthcare facilities and penalize
for unplanned readmissions, There has been a non-trivial amount of research
into summary indicators of readmission (Horwitz et al., 2014). For instance,
Chakraborty et al. (Chakraborty et al., 2021) showed that effective ML models
for forecasting 30-day readmission risk can be learned by jointly representing
the complex, sparse, and potentially noisy sequential patient records along
with the summary indicators.

Demonstration of a model on a dataset from a single healthcare organization is
a critical first step in model development. However, it is important that
models developed using data from one site be tested at, and adapted to, other
sites.Yet there are various challenges that can lead to an ineffective
transfer of ML models. First, there are often inconsistencies in how data is
organized and recorded across healthcare organizations. Second, ML models may
be overfit to disease patterns observed at a specific site. While this can
arise from insufficient model evaluation, patient populations at disparate
healthcare organizations may be significantly different. These problems were
clearly illustrated by a failure in portability of Epic’s sepsis prediction
tool (Habib et al., 2021). Community efforts, such as the OHDSI network
(Hripcsak et al., 2015), have worked to solve the first problem by promoting
common data model formats across institutes; however, the second aspect is
often understudied and is thus the focus of this paper.

In this work, we investigate the practicality of model portability through a
cross-site evaluation of readmission prediction models. To do so, we apply a
recurrent neural network, augmented with self-attention and blended with
expert features, to build readmission prediction models for two independent
large scale claims datasets. Interestingly, it is often desirable from a data-
privacy and data-usage restrictions to allow transfer of built models between
datasets without actually sharing the underlying training data/gradients.
Thus, our focus is on such ‘black-box’ transfer of models (in contrast to
federated learning) where we transfer only the final trained model from one
dataset and apply it directly on a held-out test data from the second dataset.
Our experiments show that such ‘remote’ models perform worse than ‘local’
models that are trained on the training data from the second dataset. Thus, we
introduce a novel transfer learning technique by adapting the Born-again
network (BAN) training paradigm (Furlanello et al., 2018). BAN is a cheap
training routine where the network is trained sequentially over generations,
such that soft-decisions on training examples from the model of the previous
generation are utilized to achieve more effective models. This approach has
been found to be effective across a number of domains and applications
(Kodialam et al., 2021). However, to the best of our knowledge, this is the
first to adapt this strategy to transfer models across health institutes.
There specific contributions of this paper are:

  * •

We introduce a novel, computationally inexpensive, and effective model
transfer strategy based on the BAN training paradigm. It can consider a remote
ML model as a black box (in contrast to federated learning/data sharing
scenarios where regulated healthcare datasets may need to be
explicitly/implicitly combined) while still leading to an effective transfer
across institutions.

  * •

We report on a 30-day readmission prediction across two data marts with varied
data coverage patterns. We show that the new strategy can lead to more
effective models - achieving over 10% improvement in the area under the
precision-recall curve regardless of the transfer direction (i.e., site A to
site B and vice versa) over directly applying remote models and about 2%
improvement over local models.

  * •

We show the robustness of this process by analyzing the performance lift of
the BAN-transferred remote model compared to the corresponding local model for
various bootstrap fold and find consistent improvement over the local models.

The remainder of the paper is organized as follows. Next, we describe the
experimental setup, including a brief description of the data sets used in
this study and the prediction problem. Then, we provide a brief overview of
the BAN-driven transfer learning strategy. Finally, we report our results and
provide a discussion on future opportunities.

All available claimsLong-term stays? In-hospital death / transfer to acute /
left AMA? Received treatment for cancer? Visited Rehab / primary DX for rehab?
Starting Cohort Additional Claim within 2 days? Final Cohort 30 day
readmission flag Is Acute and unplanned proc? Unplanned readmission (Flag)
NNNNNY

Figure 1. The cohort construction process to study all-cause 30-day
readmission. Figure 2. BAN Transfer: Depiction of transfer learning using
Born-again-network training. Models share the same definition. Model trained
at Site A is used to generate soft-predictions on training data for site B. A
model trained on site B is then fine-tuned on these soft-predictions to
generate the final transferred model for site B. This process implicitly
transfers the knowledge from site A in the form of soft decision boundary.
Green lines denotes training whereas dotted red-line indicate application of
model

##  2\. Experimental Setup

We conducted the analysis on two separate datasets, specifically the IBM
Marketscan Commercial database (IBM Marketscan (Butler et al., 2021)) and
Vanderbilt University Medical Center (VUMC) data mart, over the same calendar
period (2017-2019). IBM Marketscan is intended to be nationally
representative, albeit with different geographical sampling, and covering more
than 220 million patient lives over the US. By contrast, the VUMC data mart
covers substantially fewer patient lives - but it provides more detail on a
specific geographic area. Both datasets cover various data sources such as
diagnoses, drugs, and laboratory reports from administrative claims. The
Marketscan dataset covers more than 220 million patient lives across the
entire United States of America. Diagnoses, procedures, and drugs are encoded
using the International Classification of Diseases (ICD), Current Procedural
Terminology (CPT), and National Drug Code (NDC) terminologies, respectively
(Butler et al., 2021). The second dataset is from the VUMC, where the data
mainly comes from Electronic Data Interchange (EDI) x12 387 reports. The VUMC
data is composed of patients from all fifty states, with a heavy skew towards
the middle Tennessee region. The EDI data is transactional by nature and is,
at times, redundant or contradictory as a result of interactions between
providers and various payers. Thus, we relied upon data from Vanderbilt’s EHR,
structured as Observational Medical Outcomes Partnership (OMOP), to
consolidate claim records into a single event per patient. The EDI reports
enable the VUMC data be semantically consistent with the IBM Marketscan data.

We constructed all-cause 30-day readmission prediction models following the
same cohort definition and considerations as described by Chakraborty et al.
(Chakraborty et al., 2021) . More specifically, Figure 1 depicts the
constructed cohort where we applied several exclusion criteria such as cancer
patients, patients who left the hospital against medical advice, and patients
who were in rehab. These criteria were aimed at analyzing a standard set of
patients for whom we may expect a similar readmission pattern. Using this
definition, starting from about $220$ million patient lives, we were able to
extract $1,112,958$ patients with more than $10$ million records from IBM
Marketscan dataset. Similarly, from a starting point of about $5$ million
patient records, we were able to extract $1,472,891$ records distributed
across $53,810$ patients from the VUMC dataset. For each of the selected
patients, we next defined their ‘index events’ (see (Chakraborty et al.,
2021)) as the discharge date of a single hospital care or the last date of a
set of contiguous hospital care. We also can label each such ‘index events’
with the ground truth label for 30-day readmission. The final dataset is
imbalanced with respect to target variables with less than $10\%$ positive
labels for each ‘index event’. We extracted a broad set of features for each
patients including demographic, medical, and hand-crafter features such as CCI
index (Chakraborty et al., 2021). Finally, we split the data using a
$70:10:5:15$ split for training, validation, calibration, and test
respectively using a stratified sampling approach. For each dataset, we
trained local models with the same architectures as in (Chakraborty et al.,
2021) using lightsaber, an open source model training framework
(Suryanarayanan et al., 2021) to ensure reproduciblity and auditability. The
best trained models were selected by running hyperparameter search over the
validation set and further calibrated using the calibration set. The test set
was used as the holdout and, in the next section, all performance scores are
reported based on this set for each dataset.

##  3\. Methods

Here, we provide an overview of the architecture for model transfer, the
process for which is illustrated in Figure 2. Let us represent the data for
two datasets as $\mathcal{D}_{1}:\\{X_{1},Y_{1}\\}$ and
$\mathcal{D}_{2}:\\{X_{2},Y_{2}\\}$, where $X_{i}=\\{x_{1i},x_{2i},\cdots
x_{N_{i}i}\\}$ denote the set of features for $N_{i}$ index-events for the
dataset $\mathcal{D_{i}}$ and $Y_{i}=\\{y_{1i},y_{2i},\cdots y_{N_{i}i}\\}$ is
a set of boolean values represent the corresponding ground truth for 30-day
readmission. Furthermore, let us denote the local models trained on their
respective dataset as $\mathcal{M}_{1}$ and $\mathcal{M}_{2}$. These are
referred to as the local models when used in context for the respective
datsets. In contrast, we refer to these as the remote models when using it for
the alternate dataset e.g. $\mathcal{M}_{1}$ used to predict readmission on
$\mathcal{D}_{2}$.

Without loss of generality, we now describe the transfer process by
considering models are trained on $\mathcal{D}_{1}$ and applied on
$\mathcal{D}_{2}$. With respect to $\mathcal{D}_{2}$, its local model
$\mathcal{M}_{2}$ is trained by minimizing the cross-entropy loss on the
corresponding training instances as:

(1) $L(\mathcal{D}_{2})=L(\hat{y}_{2},y_{2})$

where $L$ is the cross-entropy loss, $y_{2}$ is the target value in the
training instances from dataset $\mathcal{D}_{2}$, and $\hat{y}_{2}$ is the
soft-decision score ($\in[0,1]$) output by $\mathcal{M}_{2}$ for training
instances $\\{x_{j2};j\in\text{train-set}(\mathcal{D}_{2}\\}$.

In the proposed BAN-transfer procedure, this local model is further fine-tuned
using outputs $\hat{y}_{1\rightarrow 2}$ predicted on data mart
$\mathcal{D}_{2}$ using remote model $\mathcal{M}_{1}$ (trained on
$\mathcal{D}_{1}$). The final model, hereafter referred to as the BAN-
transferred model, is then trained by fine-tuning $\mathcal{M}_{2}$ and
minimizing the cross-entropy loss as below:

(2) $L(\mathcal{D}_{2},\mathcal{M}_{1})=L(\hat{y}_{2},\hat{y}_{1\rightarrow
2})$

##  4\. Results

Table 1. Performance evaluation of the locally learned and transfer learned models per site. | IBM | VUMC
---|---|---
Model | AUROC | AUPRC | AUROC | AUPRC
IBM | 0.585 (baseline) | 0.290 (baseline) | 0.600 (-1.15$\%$) | 0.259 (-13.67$\%$)
VUMC | 0.610 (4.3$\%$) | 0.260 (-10.34$\%$) | 0.607 (baseline) | 0.300 (baseline)
BAN transferred VUMC | 0.611 (4.4$\%$) | 0.295 (1.72$\%$) | - | -
BAN transferred IBM | - | - | 0.614 (1.15$\%$) | 0.308 (2.6 $\%$)

(a) Performance

(b) Lift over baseline

Figure 3. Performance (AUROC and AUPRC) evaluation on VUMC dataset.
Comparisons shown for (a) locally built model (baseline) against remote model
and transferred remote model and (b) achieved lift in performance over
baseline. Dots and diamonds represent the values for each fold and mean over
all folds, respectively.

The performance of the models on each site is reported in Table 1 with respect
to both area under the receiver operator characteristic curve (AUROC) and
precision-recall curve (AUPRC). We report the performance for the best models
as selected by hyperparameter selection over the validation set on the held
out test set. Performance is reported for the local models (when applied on
the same dataset as the one it was trained on), remote models (when applying
it on the alternate dataset), as well as the BAN-transferred models. With
respect to each dataset, we consider the corresponding local model as the
baseline and report the change in performance as a percentage lift over the
baseline. It can be seen that the directly transferred remote models generally
has worse performance than the local models. Notably, the degradation in model
performance is more pronounced for the AUPRC than the AUROC. Specifically, it
can be seen that model performance is over $10\%$ worse for the directly
transferred remote models with respect to AUPRC.

By contrast, the BAN-transferred models lead to slightly better AUROC
performance than their respective baselines. However, more importantly, it can
be seen that the AUPRC has now a slightly better performance than the
baselines. This suggests that the BAN-transferred model accounts for the
imbalanced data better than the local models.

To further investigate this improvement, we created $8$ random bootstraps of
the training, validation, calibration, and test folds on the VUMC dataset ;
trained local VUMC models; and applied both the local VUMC and the remote IBM
model on the VUMC dataset. Figure 3(a) depicts the performance across these
$8$ runs. It can be seen that, while the best runs for the remote model can
achieve performance on par with the local models, on average the local models
perform considerably better with respect to both AUROC and AUPRC.
Interestingly, the BAN-transferred remote model leads to an improvement on
average for both metrics over the local model.

Our experimental setup of $8$ distinct bootstrap folds also enables us to
compare the lift achieved by BAN-transferred model over the corresponding
local model (that was fine-tuned to realize the BAN-transferred model) for
each run. Figure 3(b) shows the performance lift and compares it against the
remote model as a reference. It can be seen that the BAN-transfer process
consistently increases the performance of local models (average greater than
$0\%$) and this lift was found to be statistically significant for AUROC (0.05
significance level for paired t-test with two side alternate hypothesis).
Furthermore, $87.5\%$ of the bootstrap runs led to a lift for AUPRC. These
show the consistency of the transfer procedure in improving a locally built
model.

##  5\. Discussion and Conclusions

This paper presents a new method of transferring models across sites using a
computationally cheap process by only considering the models as black-box. The
empirical findings suggest that a BAN-based transfer process shows promise in
transferring ML models across healthcare organizations. This approach is
notable in that allows the locally learned models to be treated as a black-
box. In future work, we plan to investigate various experimental settings to
ground the improvements in BAN-transfer and define the boundaries when such
transfer is successful.

##  6\. Acknowledgments

This work was funded in part by IBM Watson Health and NIH grant ULTR002243.

## References

  * (1)
  * Butler et al. (2021) Anne M Butler, Katelin B Nickel, Robert A Overman, and M Alan Brookhart. 2021.  IBM MarketScan research databases.  In _Databases for Pharmacoepidemiological Research_. Springer, 243–251.
  * Chakraborty et al. (2021) Prithwish Chakraborty, James Codella, Piyush Madan, Ying Li, Hu Huang, Yoonyoung Park, Chao Yan, Ziqi Zhang, Cheng Gao, Steve Nyemba, Xu Min, Sanjib Basak, Mohamed Ghalwash, Zach Shahn, Parthasararathy Suryanarayanan, Italo Buleje, Shannon Harrer, Sarah Miller, Amol Rajmane, Colin Walsh, Jonathan Wanderer, Gigi Yuen Reed, Kenney Ng, Daby Sow, and Bradley Malin. 2021\.  Blending Knowledge in Deep Recurrent Networks for Adverse Event Prediction at Hospital Discharge. In _Proceedings of the AMIA Informatics Summit_. 132–141.
  * Dey et al. (2021) Sanjoy Dey, Aritra Bose, Prithwish Chakraborty, Mohamed Ghalwash, Aldo Guzman Saenz, Filipp Ultro, NG Kenney, Jianying Hu, Laxmi Parida, and Daby Sow. 2021\.  Impact of Clinical and Genomic Factors on SARS-CoV2 Disease Severity. In _Proceedings of the AMIA Annual Symposium_. 378–387.
  * Fonarow et al. (2017) Gregg C Fonarow, Marvin A Konstam, and Clyde W Yancy. 2017\.  The hospital readmission reduction program is associated with fewer readmissions, more deaths: time to reconsider.  _Journal of the American College of Cardiology_ 70, 15 (2017), 1931–1934.
  * Furlanello et al. (2018) Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. 2018\.  Born again neural networks. In _Proceedings of the International Conference on Machine Learning_. 1607–1616.
  * Habib et al. (2021) Anand Habib, Anthony Lin, and Richard Grant. 2021.  The Epic Sepsis Model Falls Short—The Importance of External Validation.  _JAMA_ 181, 8 (2021), 1040–1041.
  * Horwitz et al. (2014) Leora Horwitz, Chohreh Partovian, Zhenqiu Lin, Jacqueline Grady, Jeph Herrin, Mitchell Conover, Julia Montague, Chloe Dillaway, Kathleen Bartczak, Lisa Suter, Joseph Ross, Susannah Bernheim, Harlan Krumholz, and Elizabeth Drye. 2014\.  Development and use of an administrative claims measure for profiling hospital-wide performance on 30-Day unplanned readmission.  _Annals of Internal Medicine_ 161, 10 Suppl (2014), S66–75.
  * Hripcsak et al. (2015) George Hripcsak, Jon D Duke, Nigam H Shah, Christian G Reich, Vojtech Huser, Martijn J Schuemie, Marc A Suchard, Rae Woong Park, Ian Chi Kei Wong, Peter R Rijnbeek, et al. 2015\.  Observational Health Data Sciences and Informatics (OHDSI): opportunities for observational researchers.  _Studies in Health Technology and Informatics_ 216 (2015), 574–578.
  * John et al. (2021) Christopher Clement John, Vijayakumar Ponnusamy, Sriharipriya Krishnan Chandrasekaran, and R Nandakumar. 2021.  A survey on mathematical, machine learning and deep learning models for COVID-19 transmission and diagnosis.  _IEEE Reviews in Biomedical Engineering_ (2021), 325–340.
  * Kodialam et al. (2021) Rohan Kodialam, Rebecca Boiarsky, Justin Lim, Aditya Sai, Neil Dixit, and David Sontag. 2021\.  Deep contextual clinical prediction with reverse distillation. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 35. 249–258.
  * Kwon et al. (2022) Bum Chul Kwon, Vibha Anand, Peter Achenbach, Jessica L Dunne, William Hagopian, Jianying Hu, Eileen Koski, Åke Lernmark, Markus Lundgren, Kenney Ng, et al. 2022\.  Progression of type 1 diabetes from latency to symptomatic disease is predicted by distinct autoimmune trajectories.  _Nature communications_ 13, 1 (2022), 1–9.
  * Sorantin et al. (2021) Erich Sorantin, Michael G Grasser, Ariane Hemmelmayr, Sebastian Tschauner, Franko Hrzic, Veronika Weiss, Jana Lacekova, and Andreas Holzinger. 2021\.  The augmented radiologist: Artificial intelligence in the practice of radiology.  _Pediatric Radiology_ (2021), 1–13.
  * Suryanarayanan et al. (2021) Parthasarathy Suryanarayanan, Prithwish Chakraborty, Piyush Madan, Kibichii Bore, William Ogallo, Rachita Chandra, Mohamed Ghalwash, Italo Buleje, Sekou Remy, Shilpa Mahatma, et al. 2021\.  Disease progression modeling workbench 360.  _arXiv preprint arXiv:2106.13265_ (2021).
  * Zhang et al. (2022) Ziqi Zhang, Chao Yan, Xinmeng Zhang, Steve Nyemba, and Bradley Malin. 2022.  Forecasting the future clinical event of a patient through contrastive learning.  _Journal of the American Medical Informatics Association_ (2022), in press.