# Data driven analysis of cosmic rays in the heliosphere: diffusion of cosmic
protons

N. Tomassetti 1, E. Fiandrini 1, B. Bertucci 1, F. Donnini 2, M. Graziani 1,
B. Khiali 3, A. Reina Conde 4 1 Dipartimento di Fisica e Geologia, University
of Perugia, Italy  2 INFN - Sezione di Perugia, Italy  3 INFN - Sezione di
Roma Tor Vergata, Italy  4 INFN - Sezione di Bologna, Italy

###### Abstract

Understanding the time-dependent relationship between the Sun’s variability
and cosmic rays (GCR) is essential for developing predictive models of
energetic radiation in space. When traveling inside the heliosphere, GCRs are
affected by magnetic turbulence and solar wind disturbances which result in
the so-called solar modulation effect. To investigate this phenomenon, we have
performed a data-driven analysis of the temporal dependence of the GCR flux
over the solar cycle. With a global statistical inference of GCR data
collected in space by AMS-02 and PAMELA on monthly basis, we have determined
the rigidity and time dependence of the GCR diffusion mean free path. Here we
present our results for GCR protons, we discuss their interpretation in terms
of basic processes of particle transport and their relations with the dynamics
of the heliospheric plasma.

##  I Introduction

When entering the heliosphere, Galactic Cosmic Rays (GCRs) are subjected to
the solar modulation effect which causes a significant modification in the
energy spectrum of their flux in comparison with the local interstellar
spectrum (LIS) outside the heliosphere. To understand solar modulation, it is
crucial to model the transport processes of GCRs in the solar wind and its
embedded magnetic field. The main processes are diffusion, drift, advection
and adiabatic deceleration. All these processes are time dependent and follow
the quasiperiodical 11-year solar cycle. Solar modulation is very important in
GCR physics and heliophysics Moraal2013 ; Potgieter2013 . Modeling the
temporal evolution of GCRs in interplanetary space is also important for
assessing radiation risks and hazards in long-duration crewed space missions.
In this respect, the recent high-precision and time-resolved data from AMS-02
Aguilar2018PHeVSTime ; Aguilar2018LeptonVSTime  and PAMELA Adriani2013 ;
Martucci2018  experiments offer a unique possibility to study the GCR
modulation over a long period of time.

##  II Methodology

###  II.1 The model

We implemented a 2D description of the heliosphere, modeled as a spherical
bubble centered on the Sun. The wind flows radially from the Sun, with a speed
$V_{sw}(r,\theta,t)$ that depends on helioradius $r$, heliolatitude $\theta$,
and time $t$ (Fiandrini2021, ). The solar wind drops to a subsonic speed
across the termination shock $r_{\rm{TS}}=85$ AU, and vanishes at the
heliopause $r_{\rm{HP}}=122$ AU. The Earth is placed in the equatorial plane,
at $r_{0}=$1 AU. The interplanetary magnetic field (IMF) $vec{B}$ is wound up
in a rotating spiral, where its angular aperture depends on the wind speed.
Similarly, the heliospheric current sheet (HCS) is modeled on that structure.
The HCS is a rotating layer which divides the IMF into two hemispheres of
opposite polarity. The angular size of the HCS amplitude depends, in
particular, on the tilt angle $\alpha$ between magnetic and solar rotational
axis. The tilt angle is time dependent. It ranges from $\sim{10}^{\circ}$
during solar minimum (flat HCS) to $\sim{80}^{\circ}$ during maximum and
reversal (wavy HCS). Measurements of the tilt angle are provided by the Wilcox
Solar Observatory (WSO), since the 1970’s to date, on a 10-day basis
Hoeksema1995 .

The transport of GCRs in the heliosphere is described by the Parker equation
Moraal2013 :

$\frac{\partial f}{\partial t}=\nabla\cdot[\mathbf{K}^{S}\cdot\nabla
f]-(\vec{V}_{sw}+\vec{V}_{D})\cdot\nabla
f+\frac{1}{3}(\nabla\cdot\vec{V}_{sw})\frac{\partial f}{\partial(lnR)}$ (1)

where $f$ is the phase space density of GCR particles, $R=p/Z$ is their
rigidity (momentum/charge ratio), $\mathbf{K}^{S}$ is the symmetric part of
the diffusion tensor, $\vec{V}_{sw}$ is the solar wind speed, and
$\vec{V}_{D}$ is the drift speed

$\vec{V}_{D}=\frac{\beta{R}}{3}\nabla\times\frac{\vec{B}}{B^{2}}\,.$ (2)

The GCR flux $J=J(t,R)$ is given by $J=\frac{\beta{c}}{4\pi}n$, where
$\beta{c}$ is their speed and $n=4{\pi}R^{2}f$ is their number density. In
this work, we solved Eq. 1 by means of the _stochastic differential equation_
method in steady-state conditions ($\partial/\partial{t}=0$) Strauss2017 ,

The diffusion of GCR particles arises from their scattering off the small-
scale irregularities of the turbulent IMF. Drift motion is caused by gradient
and curvature of the regular component of the IMF, and in particular across
HCS. Diffusion and drift can be formally incorporated in the diffusion tensor
$\mathbf{K}$ as symmetric and antisymmetric parts, respectively:
$\mathbf{K}=\mathbf{K}^{S}+\mathbf{K}^{A}$, with $K_{ij}^{S}=K_{ji}^{S}$ and
$K_{ij}^{A}=-K_{ji}^{A}$. However, in Eq. 1, drift is explicitly accounted by
the $\vec{V}_{D}$-term, and thus only the symmetric part of the diffusion
tensor appears in the $\mathbf{K}$-term Moraal2013 . The $\mathbf{K}^{S}$
tensor can be also split into parallel and perpendicular diffusion
$K_{\parallel}$ and $K_{\perp}$, where we assume $K_{\perp}=\xi
K_{\parallel}$, with $\xi\cong\,0.02$ Giacalone1999 . The corresponding _mean
free paths_ are $\lambda_{\parallel}$ and $\lambda_{\perp}$, respectively,
such that $K_{\parallel}=\beta c\lambda_{\parallel}/3$, where $\beta=v/c$ is
the particle speed. A large compilation of observational on the parallel mean
free path in the $\sim$ 0.5 MV - 5 GV rigidity range was reported in Palmer
(1982). The mean free path, however, is rigidity and time dependent. From the
condition of cyclotron resonance, the scattering of GCRs occurs when their
Larmor radius $r_{L}=r_{L}(R)$ is comparable with the typical size of the
irregularities $\hat{\lambda}$. From the condition $r_{L}\sim\hat{\lambda}$,
it turns out that GCRs with rigidity $R$ resonate at wave number
$k_{\rm{res}}\sim 1/R$. The IMF irregularities follows a distribution of the
type $w(k)\propto k^{-\eta}$, which is the spectrum of interplanetary
turbulence expressed in terms of wave numbers $k=2\pi/\lambda$. An important
parameter is the index $\eta$, on which different regimes can be distinguished
for the IMF power spectrum Kiyani2015 . The resulting rigidity dependence of
the diffusion mean free path (or coefficient) is $\lambda_{\parallel}\sim
R^{2-\eta}$. To account for different regimes in the IMF power spectrum
Kiyani2015 , the mean free paths are often parameterized as a _double power-
law_ function of the particle rigidity. For the parallel component, we have
adopted the following description:

$\lambda_{\parallel}=K_{0}\left(\frac{B_{0}}{B}\right)\left(\frac{R_{0}}{R}\right)^{a}\times\left[\frac{(R/R_{0})^{h}+(R_{k}/R_{0})^{h}}{1+(R_{k}/R_{0})^{h}}\right]^{\frac{b-a}{h}}\,,$
(3)

where $R_{0}\equiv$ 1 GV sets the rigidity scale, $B_{0}$ is the local value
of the IMF $B$ at $r_{0}=$ 1 AU, and the normalization factor $K_{0}$ is given
in units of $10^{23}$ $cm^{2}s^{-1}$. The spectral indices $a$ and $b$ set the
slopes of the rigidity dependence of $\lambda_{\parallel}$ below and above
$R_{k}$, respectively. The parameter $h$ sets the smoothness of the
transition. The perpendicular component follows from
$\lambda_{\perp}\equiv\xi\lambda_{\parallel}$, with the addition of small
corrections in the polar regions Heber1998 .

###  II.2 The key parameters

In general, all parameters entering Eq. 3 might be time-dependent Manuel2014 .
We identify a minimal set of _diffusion parameters_ as ${K_{0},a,b}$. These
parameters and their temporal dependence will be determined using time-
resolved GCR proton data from AMS-02 and PAMELA Aguilar2018PHeVSTime ;
Adriani2013 ; Martucci2018 . Along with diffusion parameters, we define a
minimal set of _heliospheric parameters_ $\\{\alpha,B_{0},A\\}$ that describe
the time-dependent conditions of the heliosphere in a given epoch: the HCS
tilt angle $\alpha$, the local IMF intensity $B_{0}$, and its polarity $A$.
Magnetic polarity is defined as the sign of the IMF in the outward (inward)
direction from the Sun’s North (South) pole. To obtain the solution of Eq. 1
for a given GCR species, the LIS has to be specified as boundary condition.
For GCR protons, our LIS model is obtained by Galactic propagation
calculations and GCR flux data Tomassetti2015TwoHalo ; Tomassetti2012Hardening
; Feng2016 ; Tomassetti2018PHeVSTime .

Figure 1: Compilation of proton LIS modesl from various works: long-dashed red
(Tomassetti2017TimeLag, ), dot-dashed black (Corti2019, ) dotted blue
(Tomassetti2017BCUnc, ), dotted green (Boschini2017, ), dashed pink
(Tomassetti2018PHeVSTime, ), solid orange line (Tomassetti2017Universality, ).
Data are from Voyager-1 Cummings2016  and AMS-02 (Aguilar2015Proton, ).

The data, used to constrain the GCR propagation model, are from the Voyager-1
spacecraft at $\sim$ 100 -500 MeV of kinetic energy Cummings2016 , and from
AMS-02 experiment at $E\sim$ 100 GeV – 2 TeV Aguilar2018PHeVSTime ;
Aguilar2015Proton ; Aguilar2015Helium . Our proton LIS agrees fairly well with
other recent models Boschini2017 ; Corti2019 ; Tomassetti2017TimeLag ;
Tomassetti2015PHeAnomaly ; Tomassetti2017Universality ; Tomassetti2017BCUnc .
A compilation of proton LIS models is shown in Fig. 1, along with the data
from Voyager-1 and AMS-02.

###  II.3 The analysis

We determine the time-dependent GCR diffusion parameters by means of a
statistical inference on the monthly measurements of the GCR proton fluxes
reported by AMS-02 and PAMELA Aguilar2018PHeVSTime ; Adriani2013 ;
Martucci2018 . For every month, however, the heliospheric parameters have to
be specified as well. They are evaluated from observations of the WSO
observatory ($\alpha$, $A$) and by _in situ_ measurements of the ACE space
probe ($B_{0}$). For a given epochs $t$, a backward moving average is
calculated within a time window $[t-\Delta{T},t]$, with $\Delta{T}=6-12$
months. This ensures that the average values $\hat{\alpha}$, $\hat{A}$, and
$\hat{B}_{0}$ reflect the average IMF conditions sampled by GCRs arriving
Earth at the epoch $t$ Fiandrini2021 ; Tomassetti2017TimeLag . Hence, for each
epoch, the diffusion parameters $K_{0}$, $a$, and $b$ can be determined with a
global fit on the GCR proton measurements from AMS-02 and PAMELA. In practice,
to make the fit, we have built a 6D parameter grid where each node corresponds
to a parameter configuration $\vec{q}=$ ($\alpha$, $B_{0}$, $A$, $K_{0}$, $a$,
$b$). The grid has 938,400 nodes. With the stochastic method, the GCR proton
spectrum $J_{m}(E,\vec{q})$ was calculated for each node of the grid at
several values of kinetic energies between 20 MeV and 200 GeV.

Figure 2: One-dimensional projections of the $\chi^{2}$ surfaces as function
of the transport parameters $K_{0}$, $a$, and $b$ evaluated with CR proton
flux data in two epochs: April 2014, corresponding to solar maximum (pink
dashed lines, from AMS-02), and March 2009 corresponding to solar minimum
(blue solid lines, from PAMELA).

The simulation was highly CPU consuming. It required the simulation of 14
billion pseudo-particle trajectories, backward-propagated from Earth to the
heliopause and then re-weighted according to their LIS. Once the proton grid
was fully sampled, the parameters were determined as follows. From measured
fluxes $J_{d}(E,t)$ made at epoch $t$, the model calculation $J(E,\vec{q})$
was evaluated as function of its parameters. The heliospheric parameters
${\hat{\alpha},\hat{B_{0}},\hat{A}}$ were kept fixed at their evaluation at
epoch $t$. For a GCR flux measurements $J_{m}(E,t)$, as function of energy $E$
and observed at epoch $t$, the diffusion parameters are determined by the
minimization of the function:

$\chi^{2}(K_{0},a,b)=\sum_{i}\frac{\left[J_{d}(E_{i},t)-J_{m}(E_{i},\vec{q})\right]^{2}}{\sigma^{2}(E_{i},t)}\,,$
(4)

where the errors are given by
$\sigma^{2}(E_{i},t)=\sigma_{d}^{2}(E_{i},t)+\sigma_{mod}^{2}(E_{i},t)$. The
errors account for various contributions: experimental uncertainties in the
data, theoretical uncertainties of the model, and uncertainties associated
with the minimization procedure. The projections of the $\chi^{2}$ surfaces
calculated for two flux measurements are illustrated in Fig. 2 as function of
the GCR diffusion parameters $K_{0}$, $a$, and $b$. The figure shows two
distinct epoch of solar minimum (March 2009) and solar maximum (April 2014).
The best-fit parameter is shown in each curve together with its uncertainty
band. The data come from PAMELA (March 2009) and AMS-02 experiment (April
2014). It can be seen that the parameters $K_{0}$ and $b$ are tightly
constrained by the AMS-02 data. The parameter $a$ is sensitive to low-rigidity
data and thus it is better constrained by PAMELA. In general AMS-02 gives
larger $\chi^{2}$-values in comparison with PAMELA, but the convergence of the
fit is overall good.

##  III Results and Discussion

Figure 3: Best-fit results for the diffusion parameters ${K_{0},a,b}$ obtained
with the monthly flux measurements of CR protons made by PAMELA (blue
triangles) and AMS-02 (pink circles). The greenish band indicates the magnetic
reversal epoch. During this period, the IMF polarity is not well defined.
Figure 4: Envelope of the diffusion mean free paths $\lambda_{\parallel}$ as
function of GCR rigidity inferred in the examined period. The pink and blue
bands correspond to the AMS-02 and PAMELA dataset, respectively. Note that the
two bands are largely overlapped. The shaded green box corresponds to the so-
called _Palmer consensus_ on the mean value of $\lambda_{\parallel}$ at
rigidity below 5 GV(Palmer, 1982).

Along with the two epochs of Fig. 2, the fits have been performed for the
whole time-series of CR proton flux measurements of AMS-02 and PAMELA. The
AMS-02 time series consists in 79 proton fluxes measured on 27-day basis
between May 2011 and May 2017. The PAMELA series are 83 proton fluxes measured
on 27-day basis between June 2006 and January 2014. Their data, provided on
monthly basis, cover large fractions of the solar cycles 23 and 24. With the
least-square minimization described in Sect. II.3, we obtained time-series of
best-fit diffusion parameters ${K_{0},a,b}$, along with their uncertainties.
These results are shown in Fig. 3. The fit covers epochs of solar activity
that include minimum, maximum, and IMF reversal. From the fit, we found that
the diffusion parameters show a distinct time dependence that associated with
solar activity (Fiandrini2021, ). The parameter $K_{0}$ is in anti-correlation
with the monthly sunspot number, which can be understood easily within the
force field model, as the modulation parameter is $\phi\propto 1/K_{0}$
(Tomassetti2017BCUnc, ). Smaller $K_{0}$ values imply slower diffusion and a
more significant modulation effect, i.e., a stronger suppression of the low-
energy GCR flux. In contrast, larger $K_{0}$ values imply faster GCR
diffusion, therefore causing a minor modification of the LIS. We found, for
instance, a positive correlation between $K_{0}(t)$ and the flux $J(E_{0},t)$
evaluated at a given reference energy (Fiandrini2021, ). Our finding are in
good agreement with other works (Manuel2014, ; Tomassetti2017TimeLag, ;
Corti2019, ). In all these works, the GCR transport is dominated by parallel
diffusion. We also find that the parameter $b$ shows a remarkable time
dependence, reflecting the connection between solar variability and the
spectrum of magnetic turbulence in the inertial range. In this range, the
associated spectral index evolves from $\eta=0.74\pm 0.08$ at solar minimum to
$\approx{1.3}\pm 0.15$ during the solar maximum. This shows that the IMF
turbulence is subjected to variations Usoskin2019 ; Horbury2005 . Regarding
the parameter $a$, we found a milder temporal dependence, i.e., a nearly
constant spectral index of $\eta=0.79\pm 0.13$. In both ranges, our findings
are in agreement with direct measurements of the IMF power spectrum Kiyani2015
.

Once the full time-series of the diffusion parameters ${K_{0},a,b}$ is
reconstructed, from their best-fit values of Fig. 3 it is possible to
calculate the time- and rigidity-dependent diffusion mean free path
$\lambda_{\parallel}(t,R)$ using Eq. 3. The result is shown in Fig. 4, where
we plot the envelope of all mean free paths as function of the GCR rigidity
inferred in the examined periods from AMS-02 (pink circles) and PAMELA (blue
triangles). The two bands are largely overlapped. The resulting mean free path
for parallel diffusion in good accordance with the so-called Palmer consensus
on the observations of $\lambda_{\parallel}$, shown in the figure as green
shaded box (Palmer, 1982).

##  IV Acknowledgements

The present work has been developed in the framework of the joint research
program between University of Perugia and Italian Space Agency (ASI) under
agreement ASI-UniPG 2019-2-HH.0. B. K., M. G., and F. D. acknowledges support
from agreement ASI-INFN 2019-19-HH.0. It is also acknowledged the support of
Fondo Ricerca di Base of the University of Perugia. The GCR data used in this
work have been retrieved through the cosmic-ray database of the ASI Space
Science Data Center.

##  V Declarations

All authors have approved this manuscript, agree to the order in which their
names are listed, and declare that no conflict of interest exists. The authors
are responsible for the content and writing of this article.

## References

  * (1) Moraal, H., _Cosmic-Ray Modulation Equations_ , Space Sci Rev. 176 299 (2013);
  * (2) Potgieter, M. S., _Solar modulation of cosmic rays_ , Living Rev. Solar Phys., 10, 3 (2013);
  * (3) Cummings, A. C., et alii, _Galactic cosmic rays in the local interstellar medium: Voyager 1 observations and model results_ , Astrophys. J. 831, 18 (2016);
  * (4) Aguilar, M., et alii, _Observation of Fine Time Structures in the Cosmic Proton and Helium Fluxes with the Alpha Magnetic Spectrometer on the International Space Station_ , Phys. Rev. Lett. 121, 051101 (2018);
  * (5) Aguilar, M., et alii, _Observation of Complex Time Structures in the Cosmic-Ray Electron and Positron Fluxes with the Alpha Magnetic Spectrometer on the International Space Station_ , Phys. Rev. Lett. 121, 0511012 (2018);
  * (6) Adriani, O., et alii, _Time dependence of the proton flux measured by PAMELA during the 2006 July - 2009 December solar minimum_ , Astrophys. J. 765, 91, (2013);
  * (7) Martucci, M., et alii, _Proton Fluxes Measured by the PAMELA Experiment from the Minimum to the Maximum Solar Activity for Solar Cycle 24_ Astrophys. J. 854, L2 (2018);
  * (8) Strauss, R. D., Effenberger, F., _A Hitch-hiker’s Guide to Stochastic Differential Equations. Solution Methods for Energetic Particle Transport in Space Physics and Astrophysics_ , Space Sci Rev (2017);
  * (9) Kappl, R., _SOLARPROP: Charge-sign dependent solar modulation for everyone_ , Comp. Phys. Comm. 207 (2016) 386-399;
  * (10) Tomassetti, N., _Solar and nuclear physics uncertainties in cosmic-ray propagation_ , Phys. Rev. D 96, 103005 (2017);
  * (11) Fiandrini, E., Tomassetti, N., Bertucci, et alii, _Numerical modeling of cosmic rays in the heliosphere: Analysis of proton data from AMS-02 and PAMELA_ , to appear in Phys. Rev. D arXiv:2010.08649; (2021)
  * (12) Hoeksema, J. T., _The large-scale structure of the heliospheric current sheet during the Ulysses epoch_ , Space Sci. Rev. 72, 137-148 (1995);
  * (13) Giacalone, J., & Jokipii, J. R., _The Transport of Cosmic Rays across a Turbulent Magnetic Field_ , Astrophys. J. 520, 204 (1999);
  * (14) Kiyani K. H., Osman K. T., Chapman S.C., _Dissipation and heating in solar wind turbulence: from the macro to the micro and back again_ , Phil. Trans. R. Soc. A 373: 20140155 (2015);
  * (15) Heber, B., et alii, _Latitudinal distribution of greater than 106 MeV protons and its relation to the ambient solar wind in the inner southern and northern heliosphere - ULYSSES Cosmic and Solar Particle Investigation Kiel Electron Telescope results_ , J. Geophys. Res., 103, 4809 (1998);
  * (16) Manuel, R., Ferreira, S. E. S., Potgieter, M. S., _Time-Dependent Modulation of Cosmic Rays in the Heliosphere_ , Sol. Phys. 289, 2207 (2014);
  * (17) Tomassetti, N., _Cosmic-ray protons, nuclei, electrons, and antiparticles under a two-halo scenario of diffusive propagation_ , Phys. Rev. D 92, 081301 (2015);
  * (18) Tomassetti, N. _Origin of the cosmic ray spectral hardening_ , Astrophys. J. Lett. 752, L13 (2012);
  * (19) Feng, J., Tomassetti, N., Oliva, A., _Bayesian analysis of spatial-dependent cosmic-ray propagation: Astrophysical background of antiprotons and positrons_ , Phys. Rev. D 94, 123007 (2016);
  * (20) Tomassetti, N., Bertucci, B., Barão, F., et alii, _Testing Diffusion of Cosmic Rays in the Heliosphere with Proton and Helium Data from AMS_ , Phys. Rev. Lett. 121, 251104 (2018);
  * (21) Aguilar, M., et alii, _Precision Measurement of the Proton Flux in Primary Cosmic Rays from Rigidity 1 GV to 1.8 TV with the Alpha Magnetic Spectrometer on the International Space Station_ , Phys. Rev. Lett. 114, 171103 (2015);
  * (22) Aguilar, M., et alii, _Precision Measurement of the Helium Flux in Primary Cosmic Rays of Rigidities 1.9 GV to 3 TV with the Alpha Magnetic Spectrometer on the International Space Station_ , Phys. Rev. Lett. 115, 211101 (2015);
  * (23) Boschini, M. J., et alii, _Solution of Heliospheric Propagation: Unveiling the Local Interstellar Spectra of Cosmic-ray Species_ , Astrophys. J. 840, 115 (2017);
  * (24) Corti, C., et alii, _Numerical modeling of galactic cosmic ray proton and helium observed by AMS-02 during the solar maximum of Solar Cycle 24_ , Astrophys. J. 871, 253 (2019);
  * (25) Tomassetti, N., Orcinha, M., Barão, F., Bertucci, B., _Evidence for a Time Lag in Solar Modulation of Galactic Cosmic Rays_ , Astrophys. J. Lett. 849, 32 (2017);
  * (26) Tomassetti, N., _Origin of the proton-to-helium ratio anomaly in cosmic rays_ , Astrophys. J. Lett. 815, L1 (2015);
  * (27) Tomassetti, N., _Testing universality of cosmic-ray acceleration with proton/helium data from AMS and Voyager-1_ Adv. Space Res. 60, 815-825 (2017);
  * Palmer (1982) Palmer, I. D., _Transport coefficients of low-energy cosmic rays in interplanetary space_ , Rev. Geophys. Space Phys. 20, 335 (1982);
  * (29) Vaisanen, P., Usoskin, I., Mursula, K., _Long-Term and Solar Cycle Variation of Galactic Cosmic Rays: Evidence for Variable Heliospheric Turbulence_ , J. Geophys. Res.: Space Phys. 124, 804-811 (2019);
  * (30) Horbury, T. S., Forman, M. A., & Oughton, S., _Spacecraft observations of solar wind turbulence: an overview_ , Plasma Phys. Contr. Fusion, 47, B703-B717 (2005);


# Machine Learning for Deferral of Care Prediction

Muhammad Aurangzeb Ahmad1, Raafia Ahmed1, Dr. Steve Overman1, Patrick
Campbell1,
Corinne Stroum1, Bipin Karunakaran2
1\. KenSci Inc.
2\. Novant Health

{muhammad, raafia.ahmed, drsteve, patrick.campbell, corinne} @kensci.com
<EMAIL_ADDRESS>
###### Abstract

Care deferral is the phenomenon where patients defer or are unable to receive
healthcare services, such as seeing doctors, medications or planned surgery.
Care deferral can be the result of patient decisions, service availability,
service limitations, or restrictions due to cost. Continual care deferral in
populations may lead to a decline in population health and compound health
issues leading to higher social and financial costs in the long term. [1].
Consequently, identification of patients who may be at risk of deferring care
is important towards improving population health and reducing care total
costs. Additionally, minority and vulnerable populations are at a greater risk
of care deferral due to socioeconomic factors. In this paper, we (a) address
the problem of predicting care deferral for well-care visits; (b) observe that
social determinants of health are relevant explanatory factors towards
predicting care deferral, and (c) compute how fair the models are with respect
to demographics, socioeconomic factors and selected comorbidities. Many health
systems currently use rules-based techniques to retroactively identify
patients who previously deferred care. The objective of this model is to
identify patients at risk of deferring care and allow the health system to
prevent care deferrals through direct outreach or social determinant
mediation.

##  I Introduction

Disruption in access to healthcare can have long term negative consequences to
a person’s health. For example, a study conducted by the BMJ indicated that
even a four week treatment delay in certain cancers increases the risk of
death by 6$\%$ to 13$\%$ [2]. Socio-economic factors like physical disability,
lack of transportation, limited financial resources etc. have been associated
with care deferral, as these factors become barriers to attaining appropriate
healthcare [3]. A study by Palm et al. found that COVID-19 propagated
healthcare deferrals by persons avoiding public transportation and persons
without access to their own vehicle [3]. At the onset of the pandemic,
government and healthcare leaders cancelled many elective procedures and
preventative care services, and certain healthcare services continue to be
restricted. Although these measures are deemed necessary, they created a
”crisis within the crisis.” Healthcare personnel had to urgently implement
strategies to mediate this crisis through telehealth access, screening,
diagnosis and online treatment consultations [4]. Even as health systems have
re-opened to elective procedures, many patients continued to defer care due to
the COVID-19 outbreak [1]. Researchers from the National Cancer Institute
estimate 10,000 excess deaths from breast or colorectal cancer over ten years
due to missed preventative care [5]. Care deferral for urgent conditions like
stroke and other medical emergencies can also lead to increased in-home deaths
[6]. Recent studies have also shown that another impact of the pandemic was a
dramatic reduction in admissions of persons with chronic conditions [7] [8].

Prior to COVID-19, minorities and other vulnerable groups had barriers to
accessing healthcare [9]; the pandemic has worsened this problem
significantly. COVID-19 studies show that care deferral accelerated its
spread, especially within vulnerable groups [10]. Care deferral may not be
explicitly identified unless patient interviews are conducted to determine the
factors driving deferral, such as cost of care, loss of insurance cover and
extent of cover, loss of employment, access to transportation, fear of
contracting COVID-19, or perceptions that care is not needed. Regardless of
the deferral reasons, long term effects from condition onset or exacerbation
may detrimentally impact a person’s health and/or lead to increased healthcare
costs.

In this paper we address the problem of identifying patients at risk for
deferring care. If such patients can be proactively identified then health
system user groups, such as Population Health or Health Equity teams, may
conduct outreach and mitigate or address social determinant factors to reduce
care deferral. We developed a predictive model that allows the patient
population to be risk stratified for deferring well-care encounters, resulting
in a health system being able to integrate this predictive model into their
workflow. Studies show that limited access to healthcare contributes to ill
health and Poverty cycles [11]. This model aims to generate special attention
to minority and vulnerable populations, in order to lessen care disparities
and enhance equity.

Novant Health, with the support of The Duke Endowment, partnered with KenSci
to build a Machine Learning system that identifies patients at risk for
deferring their well-care encounters. Novant Health is a healthcare system
comprising of 15 hospitals across four states on the East Coast. The Duke
Endowment is one of the nation’s largest private foundation, investing in
communities across North Carolina, while ensuring racial equity. The code for
the care deferral model is available in an open source GitHub
repository.111https://github.com/KenSciResearch/CareDeferral

The rest of the paper is organized as follows: In section II we describe
related work, sections III and IV we describe the data and the experiments. In
addition to determining the predictive performance, we focus on model
explanations and fairness of predictive models in section IV-D. In healthcare
applications, model explanations are an integral part of model usage, as they
are used by healthcare end users to make decisions on individual patient or
population action plans. [12]. In section IV-C we summarize observations
related to top factors for model explanations where many social determinants
of health show up. In section IV-D we explore a critical component of
responsible AI - if and how model fairness varies across cohorts.

##  II Related Work

There are several studies which assess the impacts of care deferral on health
outcomes, and populations most affected by care deferral. In a 10 year
longitudinal study of non-elderly patients with atherosclerotic cardiovascular
disease (ASCVD) Khera et al [13], it was reported that one in five patients
deferred care due to cost. Jatrana and Crampton [14] observed health
deterioration due to care deferral for a longitudinal study in New Zealand and
found that cost barriers to doctor visits negatively impacted the mental
health of males and young adults. He and Wu [15] observed care deferral due to
economic disparities between urban and rural parts of China. Care deferral was
prevalent BY migrants to urban areas because they could no longer afford it.
Morales et al [16] observed that short term medical missions can greatly
reduce care deferral in certain locales.

a growing body of research indicates COVID-19 influenced care deferral for a
large segment of the population. De Jong et al [6] highlighted the negative
long term hidden effect of COVID-19 on non-COVID conditions due to care
deferral. Blecker et al [17] observed substantial changes in hospitalization
patterns in New York City in 2020 due to COVID-19 for chronic conditions.
Atherly et al [18] observed a marked increase in care deferral associated with
COVID-19 and that telemedicine lead to partially mitigating negative impacts.
Reuter et al [19] observed widespread care deferral due to COVID-19 across 28
different European countries.

Powis et al [20] observed the negative impact the quality of care and patient
outcomes for care deferral among cancer patients, and they predicted that the
fallout from cancer care deferral may be observed for years to come. Palm et
al [3] studied the effects of lack transportation during the early months of
COVID-19 and showed that minorities, disabled people, and low-income people
were more likely to be impacted. Sun et al [21] reported worse cardiac arrest
outcomes during COVID-19 in Boston could be attributed to patient reluctance
to seek care because of fears related to COVID-19. Lastly, a large scale
national longitudinal study by Whaley et al [22] observed similar patterns of
deferred care.

Feature Category | Example Features | Total Features
---|---|---
Demographics | Age, Ethnicity, Gender | 9
Diagnosis | Diabetes, Heart Disease, COPD | 41
Utilization | Total ED Encounter, Total Encounters | 12
Medications | Anticoagulants Med, Cardiovascular Med | 14
Labs | Hemoglobin A1C, Sodium, BUN | 3 (22 aggregated labs)
SDOH |  $\%$ Unemployed, $\%$ No Vehicle, $\%$ Minority | 21
TABLE I: Feature sets used for prediction

##  III Data & Methodology

Novant Health’s Electronic Health Record (EHR) was used to develop, experiment
with and validate a predictive model for Care Deferral. An additional source
of data that augmented the EHR data was the Social Vulnerability Index (SVI)
developed by CDC and ATSDR [23]. Our inclusion criteria consisted of patients
who had well-care visit during a prior 2-year period. We define well-care
Visits according to a value set in the Healthcare Effectiveness Data and
Information Set (HEDIS)222The measure results have not been certified via
NCQA’s Measure Certification Program and constitute ”Adjusted, Uncertified,
Unaudited HEDIS Rates”. NCQA disclaims all use or interpretation of the
results. The Healthcare Effectiveness Data and Information Set (HEDIS) is a
registered trademark of NCQA. The HEDIS measure for Adult Access to
Preventative/Ambulatory Health Services (AAP) measure reports health plan
member rates of ambulatory or preventative care visits [9], and our use of
this value set ensures alignment with other healthcare analyses. This value
set includes a set of CPT, HCPCS and Diagnosis Codes for annual physical
exams, telemedicine visits, office visit, Medicare wellness visits and similar
encounters. Care visits are separate from other preventive services such as
standard vaccinations or screenings such as Mammograms or Colonoscopies. In
other applications of this model, the value sets predicted could be swapped or
added to learn about other types of deferred preventive care. Therefore, we
thus only selected patient cohorts who had at least one well-care encounter.
Consequently, patients who never received well-care (e.g., patients who had
some encounter in the EHR but no well-care encounter) were excluded from this
cohort. We determined that simple rules (e.g., based on utilization) could
find those patients who have never had a well-care visit in the system.

During the selection process for our inclusion criteria, we identified five
potential cohorts / sub-cohorts the model could learn from.

  * •

All community members in Novant Health’s Service Areas

  * •

Members from the Novant Health service areas with no encounter at Novant
Health (these persons do not show in the EHR)

  * •

Members from the Novant Health service areas with at least encounter at Novant
Health (these patients do show in the EHR)

  * •

Patients with at least one well-care encounter

  * •

Patients with no well-care encounter

##  IV Experiments & Results

###  IV-A Problem Setup

We posit the problem of predicting care deferral as a binary classification
problem. A positive instance is defined as a patient who has had a well-care
encounter in the training period but not in the test period. A negative
instance is when the patient has had a well-care encounter in both the
training and the test period. We note that an instance is considered to be a
negative instance if there is even a single well-care follow up encounter
within the span of the test period as long as it is within a year of last
well-care visit in the training period. The training period spans from January
2017 to December 2018 while the test period spans from January 2019 to
December 2019. The data from 2020 was not employed since the time frame was
non-representative of normal patterns of care deferral due to the pandemic.
The problem setup is visualized in Figure 1.

Figure 1: Definition of Positive and Negative examples for Care Deferral

After applying the criteria, the final cohort consisted of 779,506 patients of
which 322,667 (41.39$\%$) deferred care and 456,839 (58.61 $\%$) did not defer
care. Representation across gender is fairly balanced. The patient data that
we used comes from EHR encounter data, which includes demographics, diagnoses,
utilization, medications, and labs. Publicly available data for Social
Vulnerability Index, as described in the previous section, was also used. We
note that there is some loss of granularity since all EHR derived variables
are characteristics of the individual while the features that are derived from
SVI are linked to the locale (county) that the patient resides in. We
acknowledge that the Census Tract can also be utilized, as it is provided in
the SVI, however we did not have the patient census tracts available during
the timeframe of this project. Social Determinants of Health Features
utilizing census tracts is preferred, as counties can be quite large and may
not represent the patient well.

###  IV-B Model Performance

In building the model, we compared a standard set of standard models: Logistic
Regression, Ada Boost, Random Forest, Naive Bayes, SVM, XGBoost, and Decision
Trees. A summary of prediction results is given in Table II for the best
model, XGBoost, and the baseline models. From the results it is clear that the
best set of results are obtained from the XGBoost model with auto-tuning of
hyperparameters. We also use a statistical baseline where the prediction is
done at random with respect to the relative distribution of the two classes.

Metrics | XGBoost | Baseline (Statistical)
---|---|---
Accuracy | 0.52 | 0.73
Specificity | 0.59 | 0.74
Precision | 0.41 | 0.72
Recall | 0.41 | 0.65
F-Score | 0.41 | 0.68
AUC | 0.50 | 0.79
TABLE II: Model Performance

###  IV-C Model Explanation

The goal of predicting deferral of care is to employ the model as a tool to
reduce the number of such instances. Thus, it is important to understand what
factors may be driving patients to defer care. One way to accomplish this is
via explanation of model predictions. While model explanations are not causal
in nature, a domain expert can still look at the model explanations and
determine what factors may be driving a patient to defer care. These insights
could then be used to create policies that can help reduce instances of
deferral of care. We employed the SHAP framework [24] for model explanations.
A condensed list of top 15 risk factors and their average values of those risk
factors for the cohort are given in Figure 2.

####  IV-C1 Utilization Features

From the model explanations in 2, a number of features stand out: maximum
difference between previous well-care encounters appear to be an important
factor. We see high values for Max days between well-care encounters impacts
patients deferring care. Other features associated with deferring well-care
were the average days difference between well-care encounters and number of
prior well-care encounters. This illustrates that past diligence in keeping up
with well-care visits is a good indicator of future compliance for well-care
visits (e.g., High number of well-care visits, indicates patient is less
likely to defer care). Additional utilization factors that associate with
deferring well-care are overall number of encounters in the previous 30,
number of days from the last well-care encounter, and the total number of ED
encounters.

####  IV-C2 Lab Features

The feature set consisted of two labs features (chemistry labs and common
labs), both of which show up in the most associated feature lists. An earlier
iteration of the model had individuals labs encoded as separate features,
however, combining them led to slight improvement in model performance. Both
labs features are binary functions i.e., the value is one if the test was done
and zero otherwise. The common labs feature consists of the following labs:
Liver function tests, Renal Panel, Lipid labs, Thyroid function tests, urine
tests and blood tests. The chemistry labs consist of the following: Albumin,
Sodium, Potassium, Bilirubin, Blood urea nitrogen (Bun), and Alkaline
Phosphate.

####  IV-C3 Social Determinants of Health Features

A large number of social determinants of health (SDoH) show up in the list of
most important features. Most of these features are aggregate features at the
county level, which are linked to the individual patients residing in that
county. The SDoH features that are in the top list of features are: Percentage
of households with no vehicles, percentage of people unemployed, percentage of
people living in group quarters, percentage of people who are minorities,
percentage of disabled people, percentage of people living in poverty, per
capita income, and percentage of people who speak English less than well.

Figure 2: SHAP values for Care Deferral Model Metric | Measure | Female | Asthma | COPD | HF | CVD | PUD | PVD | MLD
---|---|---|---|---|---|---|---|---|---
Group Fairness | AUC Difference | 0.0295 | 0.0181 | 0.0188 | 0.0292 | 0.0361 | 0.0206 | 0.0315 | 0.0283
Balanced Accuracy Difference | 0.0206 | 0.0279 | 0.0296 | 0.0362 | 0.0518 | 0.0248 | 0.0483 | 0.0371
Balanced Accuracy Ratio | 1.0315 | 1.0438 | 1.0465 | 1.0576 | 1.0845 | 1.0388 | 1.0784 | 1.0592
Disparate Impact Ratio | 0.9457 | 1.3274 | 1.3792 | 1.1972 | 1.6324 | 1.4083 | 1.6933 | 1.3443
Equal Odds Difference | -0.0439 | 0.1265 | 0.1413 | 0.1165 | 0.2117 | 0.1351 | 0.2131 | 0.1348
Equal Odds Ratio | 0.8481 | 1.3351 | 1.4034 | 1.2407 | 1.6405 | 1.4582 | 1.7274 | 1.2885
Positive Predictive Parity Difference | 0.0436 | 0.0226 | 0.0197 | -0.0179 | 0.0267 | 0.0182 | 0.0308 | 0.0543
Positive Predictive Parity Ratio | 1.0702 | 1.0366 | 1.0317 | 0.9727 | 1.0436 | 1.0293 | 1.0506 | 1.0928
Statistical Parity Difference | -0.0231 | 0.1062 | 0.1197 | 0.0689 | 0.1643 | 0.1211 | 0.1734 | 0.1077
Individual Fairness | Between-Group Gen. Entropy Error | 0.0001 | 0.0003 | 0.0004 | 0.0001 | 0.0003 | 0 | 0.0003 | 0.0001
Data Metrics | Prevalence of Privileged Class ($\%$) | 60 | 13 | 16 | 2 | 2 | 1 | 4 | 4
TABLE III: Measurement of Model Fairness Across Cohorts

###  IV-D Model Fairness

There are multiple notions of fairness of machine learning models that map to
various measures and metrics of fairness [25]. We computed fairness of our
model across demographics, socioeconomic factors as well as comorbidities. The
focus of fairness analysis is to determine differences across cohorts for
protected attributes like race, gender, ethnicity, socioeconomic status etc.
In addition to measuring differences across protected attributes, we also
measured differences in model fairness across comorbidities e.g., Diabetes,
COPD etc. This analysis was done to highlight that model performance across
conditions can vary and predictive models may adversely affect patients who
may have certain comorbidities.

The results are summarized in Table III. Due to limitations in space we only
show a subset of variables for which differences across cohorts were observed.
The metrics given in the tables either compute ratios or statistical
differences between the predictive performance across cohorts of interest. In
general, a statistical ratio measure $\theta$ is considered relatively fair if
$0.8\leq\theta\leq 1.2$. And any fairness measure of statistical difference
$\phi$ is considered fair if $-0.1\leq\phi\leq 0.1$. No noticeable differences
were found in fairness metrics across different demographic groups or across
socio-economic status. We however note that there are differences in model
performance across co-morbid condition groups of Asthma, COPD (Chronic
Obstructive Pulmonary Disease), HF (Heart Failure), CVD (Cardiovascular
Disease), PUD (Peptic Ulcer Disease), PVD (Peripheral Vascular Disease), and
MLD (Mild Liver Disease). Differences in model fairness been extensively
documented in literature when evaluated across chronic conditions and
vulnerable populations. [25].

##  V Discussion

Many health systems currently use rules-based techniques to retroactively
identify patients who have deferred care. The objective of this project is to
proactively predict which patients are at risk for not receiving preventative
care services. Novant Health, along with many other healthcare systems, is
concerned about care deferral, especially among historically marginalized
populations. The risk scores from this model are to be used to prioritize
patient lists for individual outreach and cohort mitigation strategies. These
predictions are used with SDoH data to identify community factors that impact
specific populations, such as public transportation. The model explanations
indicate that social factors play an important role in deferral of care
predictions. This observation also validates the hypothesis that care deferral
is driven by socio-economic factors for a segment of the population.

To optimally reduce well-care deferral, care managers, healthcare provider
teams, and population health professionals will need to work together to not
only identify and reach out to high-risk patients, but also to implement
strategies to encourage patients to follow-up on their preventative care
visits, expand remote options, and to connect patients to appropriate
community resources where necessary. There may be additional constraints when
applying predictive models in practice e.g., limitations in available
resources may imply that precision may need to be maximized at the cost of
recall. On the other hand if the goal is maximizing the reduction of deferral
of care then the opposite trade-off i.e., maximize recall even if precision
suffers is desirable. Both scenarios can be satisfied by adjusting the
threshold for prediction for any given classifier.

##  VI Conclusion

In this paper we considered the problem of predicting care deferral for well-
care visits, especially among vulnerable populations. While care deferral is
not a new crisis, COVID-19 has exacerbated care deferrals as many people were
forced to postpone elective care or forgo health screenings. [6]. Our results
indicate that machine learning can be used to risk-stratify patients in danger
of deferring care and a subset of care deferral is associated with
socioeconomic factors. In the future we would like to extend the analysis of
care deferral during the COVID-19 pandemic and how it differs from baseline
rates of care deferral as it may take some time for healthcare usage patterns
to return to normal, if at all [20]. We would also like to extend this
analysis to other value sets to learn about care deferral for vaccinations and
preventative cancer screenings.

For future work we recommend focusing on feature refinement e.g., usage of
more granular SDoH features, ideally at the patient level. For cases where
such data may not be available to use Census Tract level data to understand
the neighborhoods that the patients reside in. Additional features that may be
helpful in care deferral prediction are (a) mental health related, (b)
medications, (c) Compliance, and insurance. Lastly, well-care is one value set
to consider for this project. Other preventative care procedures can be
implemented in this model, such as cancer screenings or vaccinations. A study
of the change during the following three time periods: before, during and
after COVID-19 may be valuable in understanding baseline utilization of
preventive vs ambulatory care services to patients to help in better resource
allocation and preventing care deferral.

##  VII Acknowledgment

The project was funded by a grant from The Duke Endowment. The project was
done as partnership between Novant Health and KenSci with the support of The
Duke Endowment.

## References

  * [1] M. G. Findling, R. J. Blendon, and J. M. Benson, “Delayed Care with Harmful Health Consequences - Reported Experiences from National Surveys During Coronavirus Disease 2019,” _JAMA Health Forum_ , vol. 1, no. 12, p. e201463, Dec. 2020. [Online]. Available: https://doi.org/10.1001/jamahealthforum.2020.1463
  * [2] T. Hanna, “Mortality due to cancer treatment delay: systematic review and meta-analysis | The BMJ.” [Online]. Available: https://www.bmj.com/content/371/bmj.m4087
  * [3] M. Palm, S. L. Sturrock, N. A. Howell, S. Farber, and M. J. Widener, “The uneven impacts of avoiding public transit on riders’ access to healthcare during COVID-19,” _Journal of Transport & Health_, vol. 22, p. 101112, Sep. 2021. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S2214140521001420
  * [4] N. Purcell, J. Sells, S. McGrath, H. Mehlman, D. Bertenthal, and K. H. Seal, “Then COVID happened Veterans Health, Wellbeing, and Engagement in Whole Health Care During the COVID-19 Pandemic,” _Global Advances in Health and Medicine_ , vol. 10, p. 21649561211053828, Jan. 2021, publisher: SAGE Publications Inc. [Online]. Available: https://doi.org/10.1177/21649561211053828
  * [5] N. E. Sharpless, “COVID-19 and cancer,” _Science_ , vol. 368, no. 6497, pp. 1290–1290, Jun. 2020, publisher: American Association for the Advancement of Science. [Online]. Available: https://www.science.org/doi/10.1126/science.abd3377
  * [6] C. DeJong, M. H. Katz, and K. Covinsky, “Deferral of Care for Serious Non-COVID-19 Conditions: A Hidden Harm of COVID-19,” _JAMA Internal Medicine_ , vol. 181, no. 2, p. 274, Feb. 2021. [Online]. Available: https://doi.org/10.1001/jamainternmed.2020.4016
  * [7] H. P. Bhambhvani, A. J. Rodrigues, J. S. Yu, J. B. Carr, II, and M. Hayden Gephart, “Hospital Volumes of 5 Medical Emergencies in the COVID-19 Pandemic in 2 US Medical Centers,” _JAMA Internal Medicine_ , vol. 181, no. 2, pp. 272–274, Feb. 2021. [Online]. Available: https://doi.org/10.1001/jamainternmed.2020.3982
  * [8] S. Blecker, S. A. Jones, C. M. Petrilli, A. J. Admon, H. Weerahandi, F. Francois, and L. I. Horwitz, “Hospitalizations for Chronic Disease and Acute Conditions in the Time of COVID-19,” _JAMA Internal Medicine_ , vol. 181, no. 2, pp. 269–271, Feb. 2021. [Online]. Available: https://doi.org/10.1001/jamainternmed.2020.3978
  * [9] “Adults Access to preventative Ambulatory Health Services.” [Online]. Available: https://www.ncqa.org/hedis/measures/adults-access-to-preventive-ambulatory-health-services/
  * [10] L. Ponnambalam, L. Samavedham, H. R. Lee, and C. S. Ho, “Understanding the socioeconomic heterogeneity in healthcare in US counties: the effect of population density, education and poverty on H1N1 pandemic mortality,” _Epidemiology and Infection_ , vol. 140, no. 5, pp. 803–813, May 2012. [Online]. Available: https://www.cambridge.org/core/product/identifier/S0950268811001464/type/journal_article
  * [11] M. Fisher and E. Bubola, “As Coronavirus Deepens Inequality, Inequality Worsens Its Spread,” _The New York Times_ , Mar. 2020. [Online]. Available: https://www.nytimes.com/2020/03/15/world/europe/coronavirus-inequality.html
  * [12] M. A. Ahmad, C. Eckert, A. Teredesai, and G. McKelvey, “Interpretable Machine Learning in Healthcare,” p. 7, 2018.
  * [13] R. Khera, J. Valero-Elizondo, A. Saxena, J. Butler, Z. Samad, S. S. Virani, S. Das, J. de Lemos, H. M. Krumholz, and K. Nasir, “Abstract 15418: Deferral of Care Due to Healthcare Costs in the Uninsured With Atherosclerotic Cardiovascular Disease in the United States, 2006-2015,” _Circulation_ , vol. 138, no. Suppl_1, pp. A15 418–A15 418, Nov. 2018, publisher: American Heart Association. [Online]. Available: https://www.ahajournals.org/doi/abs/10.1161/circ.138.suppl_1.15418
  * [14] S. Jatrana and P. Crampton, “Do financial barriers to access to primary health care increase the risk of poor health? Longitudinal evidence from New Zealand,” _Social Science & Medicine_, vol. 288, p. 113255, Nov. 2021. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0277953620304743
  * [15] A. J. He and S. Wu, “Towards Universal Health Coverage via Social Health Insurance in China: Systemic Fragmentation, Reform Imperatives, and Policy Alternatives,” _Applied Health Economics and Health Policy_ , vol. 15, no. 6, pp. 707–716, Dec. 2017. [Online]. Available: https://doi.org/10.1007/s40258-016-0254-1
  * [16] D. Morales, W. Clay, R. Khamishon, R. Zaragoza, R. Eldnaf, A. T. Nagy, M. Ong, J. Morales, and M. Ryan, “Bridging the Gap: Including Patient Voices in Short-Term Medical Mission Evaluations,” _Annals of Global Health_ , vol. 85, no. 1, p. 86. [Online]. Available: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6634370/
  * [17] S. Blecker, S. A. Jones, C. M. Petrilli, A. J. Admon, H. Weerahandi, F. Francois, and L. I. Horwitz, “Hospitalizations for Chronic Disease and Acute Conditions in the Time of COVID-19,” _JAMA Internal Medicine_ , vol. 181, no. 2, pp. 269–271, Feb. 2021. [Online]. Available: https://doi.org/10.1001/jamainternmed.2020.3978
  * [18] A. Atherly, E. V. D. Broek-Altenburg, V. Hart, K. Gleason, and J. Carney, “Consumer Reported Care Deferrals Due to the COVID-19 Pandemic, and the Role and Potential of Telemedicine: Cross-Sectional Analysis,” _JMIR Public Health and Surveillance_ , vol. 6, no. 3, p. e21607, Sep. 2020, company: JMIR Public Health and Surveillance Distributor: JMIR Public Health and Surveillance Institution: JMIR Public Health and Surveillance Label: JMIR Public Health and Surveillance Publisher: JMIR Publications Inc., Toronto, Canada. [Online]. Available: https://publichealth.jmir.org/2020/3/e21607
  * [19] A. Reuter, S. Smolic, T. Barnighausen, and N. Sudharsanan, “Predicting missed health care visits during the COVID-19 pandemic using machine learning methods: Evidence from 55,500 individuals from 28 European Countries,” Public and Global Health, preprint, Mar. 2022. [Online]. Available: http://medrxiv.org/lookup/doi/10.1101/2022.03.01.22271611
  * [20] M. Powis, C. Milley-Daigle, S. Hack, S. Alibhai, S. Singh, and M. K. Krzyzanowska, “Impact of the early phase of the COVID pandemic on cancer treatment delivery and the quality of cancer care: a scoping review and conceptual model,” _International Journal for Quality in Health Care_ , vol. 33, no. 2, Jun. 2021. [Online]. Available: https://doi.org/10.1093/intqhc/mzab088
  * [21] C. Sun, S. Dyer, J. Salvia, L. Segal, and R. Levi, “Worse Cardiac Arrest Outcomes During The COVID-19 Pandemic In Boston Can Be Attributed To Patient Reluctance To Seek Care,” _Health Affairs_ , vol. 40, no. 6, pp. 886–895, Jun. 2021, publisher: Health Affairs. [Online]. Available: https://www.healthaffairs.org/doi/abs/10.1377/hlthaff.2021.00250
  * [22] C. M. Whaley, M. F. Pera, J. Cantor, J. Chang, J. Velasco, H. K. Hagg, N. Sood, and D. M. Bravata, “Changes in Health Services Use Among Commercially Insured US Populations During the COVID-19 Pandemic,” _JAMA Network Open_ , vol. 3, no. 11, p. e2024984, Nov. 2020\. [Online]. Available: https://doi.org/10.1001/jamanetworkopen.2020.24984
  * [23] B. E. Flanagan, E. W. Gregory, E. J. Hallisey, J. L. Heitgerd, and B. Lewis, “A Social Vulnerability Index for Disaster Management,” _Journal of Homeland Security and Emergency Management_ , vol. 8, no. 1, Jan. 2011, publisher: De Gruyter. [Online]. Available: https://www.degruyter.com/document/doi/10.2202/1547-7355.1792/html
  * [24] S. M. Lundberg and S.-I. Lee, “A Unified Approach to Interpreting Model Predictions,” in _Advances in Neural Information Processing Systems_ , vol. 30. Curran Associates, Inc., 2017. [Online]. Available: https://proceedings.neurips.cc//paper/2017/hash/8a20a8621978632d76c43dfd28b67767-Abstract.html
  * [25] M. A. Ahmad, A. Patel, C. Eckert, V. Kumar, and A. Teredesai, “Fairness in Machine Learning for Healthcare,” in _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. New York, NY, USA: Association for Computing Machinery, Aug. 2020, pp. 3529–3530. [Online]. Available: https://doi.org/10.1145/3394486.3406461