# On the Origin of Superconductivity at Nickel-Bismuth Interfaces

Matthew Vaughan  School of Physics and Astronomy, University of Leeds, Leeds,
LS2 9JT, United Kingdom  Nathan Satchell  School of Physics and Astronomy,
University of Leeds, Leeds, LS2 9JT, United Kingdom  Mannan Ali  School of
Physics and Astronomy, University of Leeds, Leeds, LS2 9JT, United Kingdom
Christian J. Kinane  ISIS Neutron and Muon Source, Rutherford Appleton
Laboratory, OX11 0QX, United Kingdom  Gavin B. G. Stenning  ISIS Neutron and
Muon Source, Rutherford Appleton Laboratory, OX11 0QX, United Kingdom  Sean
Langridge  ISIS Neutron and Muon Source, Rutherford Appleton Laboratory, OX11
0QX, United Kingdom  Gavin Burnell<EMAIL_ADDRESS>School of Physics
and Astronomy, University of Leeds, Leeds, LS2 9JT, United Kingdom

###### Abstract

Unconventional superconductivity has been suggested to be present at the
interface between bismuth and nickel in thin-film bilayers. In this work, we
study the structural, magnetic and superconducting properties of sputter
deposited Bi/Ni bilayers. As-grown, our films do not display a superconducting
transition, however, when stored at room temperature, after about
14$\,\mathrm{days}$ our bilayers develop a superconducting transition up to
3.8$\,\mathrm{K}$. To systematically study the effect of low temperature
annealing on our bilayers, we perform structural characterization with X-ray
diffraction and polarized neutron reflectometry, along with magnetometry and
low temperature electrical transport measurements on samples annealed at
$70^{\circ}\,\mathrm{C}$. We show that the onset of superconductivity in our
samples is coincident with the formation of ordered $\mathrm{NiBi_{3}}$
intermetallic alloy, a known $s$-wave superconductor. We calculate that the
annealing process has an activation energy of $(0.86\pm 0.06)\,\mathrm{eV}$.
As a consequence, gentle heating of the bilayers will cause formation of the
superconducting $\mathrm{NiBi_{3}}$ at the Ni/Bi interface, which poses a
challenge to studying any distinct properties of Bi/Ni bilayers without
degrading that interface.

##  I Introduction

Superconductivity and ferromagnetism are normally considered incompatible
phases as the strong exchange field of a ferromagnet will act to break
superconducting Cooper pairs Ginzburg (1957). It is therefore unusual to find
a superconducting transition at about 4 K in Bi/Ni bilayers, when
ferromagnetic Ni has no known such transition, and crystalline Bi is only
superconducting below 0.5 $\,\mathrm{mK}$ Prakash  _et al._ (2017). Higher
critical temperatures in Bi have been reported under certain conditions, for
example, $T_{c}$ $\approx 6\,\mathrm{K}$ in amorphous Bi, $T_{c}$ $\approx
4\,\mathrm{K}$ induced under pressures of a few $\,\mathrm{GPa}$, a $T_{c}$
range of between 2-5.5$\,\mathrm{K}$ on the surface of grain boundaries and
$T_{c}$ of 1.3$\,\mathrm{K}$ in nanowires Petrosyan  _et al._ (1975); Buckel
and Wittig (1965); Weitzel and Micklitz (1991); Li  _et al._ (2017); Tian  _et
al._ (2009). None of these can, however, explain the superconductivity in
Bi/Ni bilayers.

Bi/Ni bilayer superconductivity was initially discovered in Bi layers grown on
a dusting of Ni in tunneling measurements, which showed that the
superconductivity extends across the entire thickness of the Bi Moodera and
Meservey (1990). Later, similar measurements showed that in such bilayers,
superconductivity and ferromagnetism coexist LeClair  _et al._ (2005).
Recently, there is a renewed interest in Bi/Ni bilayers as the combination of
superconductivity, ferromagnetism and strong spin-orbit coupling may lead to
exotic new physics. In particular, epitaxial bilayers of Bi/Ni grown by
molecular beam epitaxy have been heavily studied Gong  _et al._ (2015, 2017);
Wang  _et al._ (2017); Zhou and Jin (2018); Zhou  _et al._ (2017); Chauhan
_et al._ (2019); Tokuda  _et al._ (2019). There is speculation that results on
these epitaxial bilayers show $p$-wave superconductivity Gong  _et al._
(2015), time-reversal symmetry breaking Gong  _et al._ (2017), and chiral
superconductivity Wang  _et al._ (2017).

An alternative explanation for the origin of the superconductivity in the
Bi/Ni bilayer is by the presence of the alloy $\mathrm{NiBi_{3}}$ which is
established to superconduct with a similar $T_{c}$ of 4 K Alekseevskii  _et
al._ (1952); Siva  _et al._ (2015). Measurements on bulk crystals of
$\mathrm{NiBi_{3}}$ suggest that it shows coexistence of superconductivity and
ferromagnetism Silva  _et al._ (2013), however it is expected to be a singlet,
$s$-wave, superconductor Zhao  _et al._ (2018). Silva et al observed this
alloy in thin-film Bi-Ni interfaces, which they attribute to spontaneous
formation during sample growth at a temperature of $60^{\circ}\,\mathrm{C}$
Silva  _et al._ (2013). Liuet al also observe interdiffusion during sample
growth at $300\,\mathrm{K}$ but no interdiffusion when samples are grown
colder than $110\,\mathrm{K}$ Liu  _et al._ (2018). Whilst formation during
growth will be dominated by comparatively rapid surface diffusion, it is also
important to establish whether, and under what conditions, formation of
$\mathrm{NiBi_{3}}$ can occur post-growth when the as-grown samples show
initially clean and distinct interfaces of Bi/Ni.

In this work, we set out to determine at what timescales and temperatures
intermixing from an initially distinct Bi/Ni interface becomes significant.
Annealing at low temperatures, we use SQUID magnetometry, x-ray diffraction
(XRD), and polarized neutron reflectometry (PNR) to measure changes in Bi/Ni
samples. X-ray reflectometry is not used as the x-ray scattering density for
bulk Bi and NI are too similar for effective contrast at the interface. We
observe the onset of a superconducting $T_{c}$, evolution of the magnetic
moment, and changing structure of the interface. Our results suggest that
special handling of the Bi/Ni samples and refrigerated storage is necessary to
prevent the formation of $\mathrm{NiBi_{3}}$, which otherwise occurs after a
few days at room temperature, or a few minutes at temperatures typical of many
cleanroom processing steps.

##  II Methods

Samples are grown by DC sputtering from pure metal targets of Bi (99.99%) and
Ni (99.95%) at ambient temperature ($21^{\circ}\,\mathrm{C}$) on to thermally
oxidized Si substrates. To prevent oxidation, the samples are capped with
5$\,\mathrm{nm}$ thick Ta layers. It is found that optimal layer thicknesses
for maximum $T_{c}$ are 50$\,\mathrm{nm}$ of Bi and 6$\,\mathrm{nm}$ of Ni,
consistent with previous work Gong  _et al._ (2015). See the Supplemental
Material for full details of the deposition and $T_{c}$ dependence on Bi and
Ni thicknesses Not (a).

To prevent unintentional annealing, directly after removing from the
deposition system, the samples are stored in a freezer at
$\approx-20^{\circ}\,\mathrm{C}$ and transported in a portable refrigerator at
$\approx 4^{\circ}\,\mathrm{C}$. We either anneal our samples at room
temperature (21${}^{\circ}\,\mathrm{C}$), or perform controlled annealing of
our samples between 50${}^{\circ}\,\mathrm{C}$ and 150${}^{\circ}\,\mathrm{C}$
on a hotplate under a cover to maintain a uniform temperature.

Polarized neutron reflectometry (PNR) is performed on the PolRef beamline at
the ISIS neutron and muon source. Polarized neutron reflectometry data are
analyzed using the GenX software Björck and Andersson (2007). X-ray
diffraction (XRD) is performed on a Rigaku SmartLab diffractometer using Cu
K${}_{\alpha}\ \lambda=1.54$Å radiation and magnetization loops are measured
using a Quantum Design MPMS 3 SQUID magnetometer both courtesy of the ISIS R53
characterization lab. We also employ an Oxford Instruments MagLab 8T VSM for
additional magnetization measurements for the temperature dependent annealing.
Electrical transport measurements are performed using a standard 4-point probe
AC method utilizing a lock-in amplifier and a
$77\,\mathrm{Hz},100\,\mathrm{\mu A}$ current inside a 4He variable
temperature cryostat with a 3$\,\mathrm{T}$ superconducting magnet in a
horizontal Helmholtz Coil configuration.

##  III Results

###  III.1 Polarized Neutron Reflectometry Measurements

Figure 1: Polarized neutron reflectometry (PNR) measurements at 10 K and 0.2 T
of Si/SiO (100 nm)/Bi (50 nm)/Ni (6 nm)/Ta (5 nm) sample. (a),(b) PNR and spin
asymmetry in the as-grown state. (c),(d) PNR and spin asymmetry after
3600$\,\mathrm{s}$ annealing at 70${}^{\circ}\,\mathrm{C}$. The solid lines
are fits to the data and the returned fit parameters are given in Table 1. (e)
The nuclear and magnetic scattering length density (SLD) with depth returned
from fitting showing the evolution of SLD with successive annealing.

By measuring the neutron reflectivity as a function of the wavevector transfer
and neutron spin eigenstate, PNR allows the scattering length density (SLD) to
be obtained. Careful fitting to the two obtained reflectivity curves in PNR
allows the extraction of depth dependent magnetization and structure. PNR is
widely employed in the successful characterization of spintronic materials
Zabel  _et al._ (2008) and is particularly useful in this study due to the
large SLD contrast between the Ni and Bi layers. The sample is annealed
repeatedly at 70${}^{\circ}\,\mathrm{C}$. Films are loaded at room temperature
and cooled in an 0.2$\,\mathrm{T}$ field to 10$\,\mathrm{K}$ where we perform
PNR measurements.

Figure 1 (a,c) shows the obtained PNR curves for the sample with the
corresponding fit to each spin state. (a) shows the as-grown state, while (c)
is after 3600$\,\mathrm{s}$ annealing at 70${}^{\circ}\,\mathrm{C}$. The
higher frequency oscillations arise from the thick SiO2 layer on the
substrate. The thinner, rougher, layers of the film modulate these
oscillations. The spin asymmetry as calculated by $\frac{u-d}{u+d}$ is shown
in Figure 1 (b,d). The spin asymmetry directly scales with the sample’s
magnetization. The spin asymmetry shows significant reduction between the as-
grown and annealed states indicating a reduction of magnetization.

We employ a box model to fit the PNR data with the layers
Ta2O5/Ta/Ni/$\mathrm{NiBi_{3}}$ /Bi/SiO2/Si. Ta2O5 is the most common oxide of
Ta and the SiO2 density is taken as amorphous. Each layer is assigned a
thickness, roughness, magnetization, scattering length, and density. The
scattering lengths are fixed from the known bulk values. Magnetism in the
structure is limited to the Ni and $\mathrm{NiBi_{3}}$ layers,
$\mathrm{NiBi_{3}}$ is not ferromagnetic and we do not find a significant
moment inside this layer. All other parameters except the substrate density
and thickness (the substrate is infinitely thick) are free fitting parameters
with physically realistic bounds defined from either the known bulk or as-
grown values. The results of fitting are shown in Figure 1 (e), Table (Fig.1)
and the full fitted parameters are listed in the Supplemental Material Not
(a).

The results of fitting are shown in Figure 1.(e). As grown the samples do show
a thin (2$\,\mathrm{nm}$) $\mathrm{NiBi_{3}}$ layer that has either formed
during growth, or in the time taken to remove them from the deposition system.
Consequently, the Ni and Bi layers are slightly thinner than expected
(4.4$\,\mathrm{nm}$ and 44$\,\mathrm{nm}$ versus the nominal 6$\,\mathrm{nm}$
and 50$\,\mathrm{nm}$). The thickness of the $\mathrm{NiBi_{3}}$ layer in the
model is less than the roughness of any of the Bi, $\mathrm{NiBi_{3}}$ or Ni
interfaces suggesting that it is an intermixed region rather than a distinct
layer. The Magnetic SLD of bulk Ni is $\approx 1\times
10^{-6}\,\mathrm{\AA^{-2}}$ where our magnetic SLD for the Ni in the as-grown
state is $\approx 0.4\times 10^{-6}\,\mathrm{\AA^{-2}}$ as the roughness
distributes the Ni over a wider range. The area under the Magnetic SLD and the
area derived from the box model with no roughness is $3.98\times
10^{-6}\,\mathrm{\AA^{-1}}$ and $3.97\times 10^{-6}\,\mathrm{\AA^{-1}}$
respectively.

The main changes to the structure with annealing the sample are the diffusion
of Ni atoms into the Bi layer. The stronger nuclear scattering of the Ni
increases the SLD of the Bi layer where atoms have diffused, Figure 1.(e). The
structural profile of the final annealed state suggests that the Ni can
diffuse across the entire thickness of the Bi as the SLD of the entire layer
is raised. At the Bi/Ni interface, we report that a layer with the correct SLD
for ordered $\mathrm{NiBi_{3}}$ intermetallic layer is found. The thickness of
this layer increases with annealing from (2$\,\mathrm{nm}$) in the unannealed
state, to (38.8$\,\mathrm{nm}$) in the final annealed state.

As the Ni diffuses; less ferromagnetically ordered Ni is left, in turn
decreasing the magnetic SLD profile and moment measured independently from
SQUID. For the final two annealing steps, the fitting returns a very thin
layer of Ni (0.7$\,\mathrm{nm}$). The large roughness (4$\,\mathrm{nm}$) of
this layer suggests it is no longer continuous and has been replaced by an
intermixing layer. In an attempt to improve the modeling, we try replacing
this Ni layer with a layer approximating a NiBi intermixing layer. Doing so,
we find the fitting returns a lower figure of merit, indicating that model has
a closer resemblance to the physical sample. This suggests that in the final
annealed state, Bi contaminates the Ni layer in addition to the Ni diffusion
into the Bi layer. There is very little change to the scattering length
density shown Figure 1 (e) between the two models as the roughness tends to
smear such fine details. The full fitting parameters from each model are
provided in the Supplemental Material Not (a).

Layer (nm) | Initial | Final
---|---|---
Ta2O5 | $1.6\pm 0.3$ | $4\pm 0.5$
Ta | $4.9\pm 0.4$ | $0.5\pm 0.5$
Ni | $4.5\pm 0.7$ | $0.7\pm 0.1$
$\mathrm{NiBi_{3}}$ | $2.3\pm 0.4$ | $38.8\pm 0.7$
Bi | $44.1\pm 0.4$ | $9.3\pm 0.7$
SiO2 | $95.0\pm 0.3$ | $93.9\pm 0.5$
Table 1: The initial (0$\,\mathrm{s}$) and final (3600$\,\mathrm{s}$) annealed
fitted thicknesses at 70${}^{\circ}\,\mathrm{C}$. The full fitting parameters
are given in the Supplemental Material Not (a).

###  III.2 Magnetometry and X-Ray Diffraction

To study the magnetic and structural properties of our Bi/Ni bilayer samples
we employ SQUID magnetometry and Cu K${}_{\alpha}\ \lambda=1.54$Å x-ray
diffraction (XRD). We use a sample grown in the same vacuum cycle as the
sample we study by PNR, which we dice into smaller (4x4$\,\mathrm{mm}$)
cuttings. The individual cuttings are treated with the same annealing process
at 70${}^{\circ}\,\mathrm{C}$ as the PNR sample. After annealing, we first
measure the magnetic hysteresis (moment vs field) of the cuttings at 10 K (the
same temperature as the PNR), then check for superconducting transition by
measuring moment vs temperature from the base temperature of the magnetometer
at small applied field (2-10 K sweep at 5 mT), finally we transfer the
cuttings to the x-ray diffractometer where XRD is measured at room
temperature. To minimize annealing during the x-ray measurements, the total
time to align and record a XRD scan is optimized to take about 30 minutes.
Nevertheless, we find by re-measuring a cutting’s magnetic response after the
room temperature XRD scan has finished, that some annealing occurs during the
x-ray measurement. We do not believe this to have an influence on the results
presented in this section, however as a precaution do not measure the same
cutting more than once and each annealing time for SQUID and XRD measurements
are from a different cutting of the pristine sample.

Figure 2: Magnetic characterization at 10 K of Si/SiO (100 nm)/Bi (50 nm)/Ni
(6 nm)/Ta (5 nm) samples. The moment per area calculated from the hysteresis
loops, inset, are shown with annealing time at 70${}^{\circ}\,\mathrm{C}$. The
diamagnetic contribution due to the substrate has been subtracted. The
uncertainty in moment/area is dominated by the area measurements (different
cuttings of the sample are used for each annealing step), and is less than 5%
. Also shown is the moment/area extracted by fitting the PNR measurements. The
line on the main figure is a guide for the eye.

The magnetic characterization of our Bi/Ni bilayer samples are shown in Figure
2. We extract the saturation (applied field of 1 T) moment/area by measuring
hysteresis loops for cuttings with different annealing times. Exemplar
hysteresis loops are shown in the inset of Figure 2. For the as-grown sample
cutting, the saturation magnetization of the 4.4 nm Ni layer is
318$\,\mathrm{emu/cm^{3}}$, reduced from the bulk value of
600$\,\mathrm{emu/cm^{3}}$. The reduced magnetization is consistent with the
formation of magnetic dead layers, which are often observed in thin film Ni
Robinson  _et al._ (2007). For all sample cuttings, we find that a large
magnetic field (nearly 1 T) is required to achieve full saturation, hence the
hysteresis loops shown in Figure 2 inset do not fully close within the field
range shown. As the film is annealed, two changes to the hysteresis loops are
observed; firstly, the saturation moment/area of the sample reduces and
secondly, the coercive field increases. Both observations are consistent with
the PNR modeling which shows that as the sample is annealed non-magnetic
$\mathrm{NiBi_{3}}$ intermetallic forms reducing the ferromagnetic Ni
thickness.

It is possible to compare the measured magnetic moment returned from the SQUID
measurements and PNR fitting (Figure 1) by normalizing datasets to the areas
of the samples. Collated moment/area with annealing time at
70${}^{\circ}\,\mathrm{C}$ for the samples are shown Figure 2. As the sample
is annealed we observe an exponential type decay of the sample moment/area
from about 188$\,\mathrm{emu/cm^{2}}$ in the as-grown state to
32$\,\mathrm{emu/cm^{2}}$ after 3600 s annealing at
70${}^{\circ}\,\mathrm{C}$. The time constant of the decay is
530$\,\mathrm{s}$. The extracted moment/area by the two techniques show close
agreement in both trend and absolute value for annealed samples, we note that
there is some disagreement in the magnetic moment of the as-grown state.

Figure 3: X-ray diffraction characterization of 70${}^{\circ}\,\mathrm{C}$
annealed Si/SiO (100 nm)/Bi (50 nm)/Ni (6 nm)/Ta (5 nm) samples. (a) Surfaced
aligned XRD scans with annealing time. Peaks due to the Si substrate, Bi layer
and $\mathrm{NiBi_{3}}$ intermetallic are indexed. (b) The normalized area
under the Bi (012) and $\mathrm{NiBi_{3}}$ (302) peaks to show evolution of
the sample with annealing step. Lines are a guide to the eye.

Fig. 3 (a) shows the results of XRD characterization. In the as-grown state,
structural peaks due to the Bi layer and Si substrate are present in the
sample. We do not expect Ni or Ta peaks to appear as these layers are too
thin. After annealing, peaks appear in the XRD scan which correspond to the
$\mathrm{NiBi_{3}}$ intermetallic and equally the Bi peak intensity drops,
suggesting that the textured Bi layer is being replaced by a textured
$\mathrm{NiBi_{3}}$ intermetallic. The timescale for these structural changes
to occur to the sample is similar to the timescale where changes are observed
in PNR and SQUID measurements annealed at the same temperature. The
$\mathrm{NiBi_{3}}$ has a weakly preferred orientations toward to (203).

###  III.3 Superconducting Properties

Low temperature 4-point probe transport measurements of a
70${}^{\circ}\,\mathrm{C}$ fully annealed sample (3600 s) show a clear
superconducting transition at 3.8$\,\mathrm{k}$, Figure 4 (a), the critical
magnetic field in-plane and out-of-plane was also measured, Figure4 (b). The
out-of-plane and in-plane $H_{c2}$ temperature dependence can be fitted
effectively by the following Ginzburg-Landau (GL) model for a thin
superconductor;

$\mu_{0}H_{c2}^{\bot}=\frac{\Phi_{\text{0}}}{2\pi\xi_{\text{GL}}(0)^{2}}(1-\frac{T}{T_{c}})$
(1)

$\mu_{0}H_{c2}^{\|}=\frac{\Phi_{\text{0}}\sqrt{12}}{2\pi\xi_{\text{GL}}(0)d_{\text{SC}}}(1-\frac{T}{T_{c}})^{\frac{1}{2}}$
(2)

where $\Phi_{\text{0}}$, $\xi_{\text{GL}}(0)$ and $d_{\text{SC}}$ stands for
flux quantum, in-plane coherence length and effective thickness of the
superconductivity respectively. Taking the values from both in-plane and out-
of-plane Hc2(0) fits the coherence length $\xi_{\text{GL}}$ as
13.8$\,\mathrm{nm}$ and the effective superconducting thickness
$d_{\text{SC}}$ as 36.2$\,\mathrm{nm}$, similar to the thickness of the
$\mathrm{NiBi_{3}}$ layer as obtained from the PNR data. The GL-theory for
thin superconductors assumes that the $d_{\text{SC}}<\xi_{\text{GL}}$ which is
not the case here and is visible in Figure 4 (b) from the inadequate fitting
for the in-plane $H_{c2}$ data.Alternatively, making the power of the in-plane
equation a free parameter (instead of a fixed $\frac{1}{2}$) returns a value
of 0.659 and a more satisfactory fit to the experimental data. A returned
power of 1 is expected for bulk behavior, suggesting this sample is in some
intermediate state between bulk and thin superconductivity.

Figure 4: Electrical transport characteristic of Si/SiO (100 nm)/Bi (50 nm)/Ni
(6 nm)/Ta (5 nm) sample after 70${}^{\circ}\,\mathrm{C}$ annealing for 3600 s.
(a) Resistance verses temperature with the superconducting transition at
3.8$\,\mathrm{K}$ shown in the inset. (b) The out-of-plane (red) and in-plane
(black) $H_{c2}$ data, solid lines a model fit for a thin superconductor (eq.
1,2). The dashed line has the power of $\frac{1}{2}$ eq. 2 as a free fitting
parameter and the $T_{c}$ is fixed to the $T_{c}$ measured by the RvT.
$\mu_{0}H_{c2}^{\|}=2.28\,\mathrm{T}$ and
$\mu_{0}H_{c2}^{\bot}=1.72\,\mathrm{T}$, the power of the dash line is 0.659.

###  III.4 Influence of Annealing Temperature

Samples as-grown were not immediately observed to be superconducting, but
after annealing or leaving at room temperature for several days the
magnetization of the Ni layer was reduced although it is non vanishing and an
increased coercivity (Fig.5.a), after that a superconducting transition
appeared and increased until stabilizing at 3.8$\,\mathrm{k}$ (Fig.5.b). The
thickness of the $\mathrm{NiBi_{3}}$ layer grows quickly as the sample is
annealed once the layer is thick enough to support a superconducting
transition there is a meissner response that can be measured to deduce the
$T_{c}$. By measuring the $T_{c}$ at several points along the annealing
process a time constant for the onset can be fitted from a decaying
exponential. As long as the sample is far from saturating its $T_{c}$ the time
constant will be not be affect by small amounts of annealing that may have
already taken place. The inverse time constant against annealing temperature
can be described well by the Arrhenius equation for a thermally activated
reaction with an association activation energy of $(0.86\pm
0.06)\,\mathrm{eV}$ (Fig.6.b). The activation energy is lower than for similar
systems of interface mixing and diffusion with typical activation energies of
$\approx$ 1$\,\mathrm{eV}$ Nicolet (1978); Höfler  _et al._ (1993).

Figure 5: (a) Sample of Ta(5)/Ni(6)/Bi(50)$\,\mathrm{nm}$ stored at room
temperature and hysteresis measured at 3$\,\mathrm{K}$. (b) MvT of
Ta(5)/Ni(4)/Bi(50)$\,\mathrm{nm}$ samples at two different temperatures, black
lines are logistic function fitting for which the $T_{c}$ is taken at 20%
height.

##  IV Discussion

Figure 6: (a) Fitted thickness extracted from PNR data for each annealing
step, omitting the roughness at the interface for simplicity. The annealing at
70${}^{\circ}\,\mathrm{C}$takes place between the dashed lines. (b) Arrhenius
plot of Bi/Ni bilayers annealed in the temperature range of
50${}^{\circ}\,\mathrm{C}$– 150${}^{\circ}\,\mathrm{C}$with an activation
energy of $(0.86\pm 0.06)\,\mathrm{eV}$, time constants taken from the
superconducting $T_{c}$ onset rate.

Figure.6 (a) shows the fitted thickness for each annealing step as extracted
from fitting to PNR. The Ta2O5 increases as expected for heating in air and
has protected the lower layers from oxidation. The thickness of the Bi and Ni
layers reduce as the $\mathrm{NiBi_{3}}$ layer becomes thicker. We also
observe an overall reduction in film thickness as the $\mathrm{NiBi_{3}}$
alloy is denser than the individual Bi or Ni layers. To optimize the $T_{c}$
50$\,\mathrm{nm}$ of Bi and 6$\,\mathrm{nm}$ of Ni are used, the ratio maybe
suggest that to obtain the highest $T_{c}$ annealed samples one should
minimize the remaining Ni and maximize the thickness of the
$\mathrm{NiBi_{3}}$Not (a).

Extrapolating from the Arrhenius plot at 21${}^{\circ}\,\mathrm{C}$ the time
constant for the annealing rate will be 4$\,\mathrm{days}$ which is comparable
to the timescale measured at room temperature, the exponential has a doubling
rate of $\approx$ 6${}^{\circ}\,\mathrm{C}$ such that if kept at
3${}^{\circ}\,\mathrm{C}$ the timescale extends to 12$\,\mathrm{days}$
(Fig.6). The low annealing temperatures can be understood from the low melting
point of Bi of 544$\,\mathrm{K}$ being about twice room temperature ($\approx
294\,\mathrm{K}$). It is not uncommon to anneal thin films to form alloys at
50% of the constituent layers melting points . The Bi atoms become mobile at
higher temperatures diffusing across the interface and the same for Ni as it
diffuses into the Bi layer forming a $\mathrm{NiBi_{3}}$ alloy.

When first measured in PNR, the as-grown sample has a thin layer at the
interface with a density similar to $\mathrm{NiBi_{3}}$ , although in the
corresponding XRD the $\mathrm{NiBi_{3}}$ peaks are at the limit of signal to
noise and so the volume fraction of ordered $\mathrm{NiBi_{3}}$ is very small.
In comparison, in samples that have been intentionally annealed, the ratio for
$\mathrm{NiBi_{3}}$ XRD peak areas and the fitted layer thickness suggest that
2$\,\mathrm{nm}$ of $\mathrm{NiBi_{3}}$ should have a greater intensity of XRD
peaks than is measured in the as-grown sample. This thin intermixed layer is
not unexpected. The roughness of the Bi layer, implantation of Ni adatoms
during growth and, probably most significantly, the time taken to get the
samples from a room temperature vacuum chamber to the freezer all contribute
to some intermixing of the interface.

Liu et al perform pulsed laser deposition (PLD) of Bi/Ni bilayers where they
report the $\mathrm{NiBi_{3}}$ alloy forms during growth. Their interpretation
is that PLD is a nonequilibrium process such that Ni atoms arrive at the Bi
layer with enough energy to implant deep into the film. Ni implantation stops
at the SiO2, where it accumulates forming a NiBi alloy and a
$\mathrm{NiBi_{3}}$ layer further from the SiO2. We find that ordered
$\mathrm{NiBi_{3}}$ does not exist in our samples at growth, but forms during
annealing and is confined to the Bi/Ni interface. We do not find evidence for
ordered NiBi alloy in our films, however in the final annealed state Bi may
have contaminated any remaining Ni layer in the structure. We conclude that
during growth the Ni implantation depth is confined to near the top surface of
the Bi, as fitted by the 2$\,\mathrm{nm}$ of $\mathrm{NiBi_{3}}$ in the as-
grown state.

By measuring PNR below $T_{c}$ in an applied field, it is possible to observe
Meissner screening under the right conditions (that the film thickness,
roughness, and the superconducting penetration depth are balanced to allow for
measurable screening to occur) Zhang  _et al._ (1995); Drew  _et al._ (2009).
It is also possible that exotic superconducting states can influence the
magnetic response of a superconducting sample Luke  _et al._ (1998); Flokstra
_et al._ (2016, 2019). Here, we measure both the as-grown and fully annealed
states of the sample at 3 K (below the superconducting transition for the
annealed sample) to look for changes to magnetic response from the sample. No
such changes are observed below $T_{c}$ in the PNR, most likely as the
$\mathrm{NiBi_{3}}$ superconductor is too thin to observe Meissner screening.

Experimental results by other techniques suggest that superconductivity in
bilayers of Bi/Ni may be spin-triplet ($p$-wave) in nature Gong  _et al._
(2015, 2017); Wang  _et al._ (2017); Zhou and Jin (2018); Zhou  _et al._
(2017); Chauhan  _et al._ (2019); Tokuda  _et al._ (2019). The symmetric spin-
triplet states are found in only a handful of superconductors, where Sr2RuO4
is currently the best candidate Luke  _et al._ (1998). In these materials, the
antisymmetry requirements are satisfied by the condensates of these
superconducting materials being spatially antisymmetric, that is, odd in
angular momentum. It is also possible to generate spin-triplet states in
proximity coupled thin films where $s$-wave pairing is retained by introducing
a Berizinskii state with spontaneous breaking of time-reversal symmetry
Bergeret  _et al._ (2001). Our Bi/Ni samples contain all the necessary
ingredients for such a state to occur; a source of $s$-wave superconductivity
($\mathrm{NiBi_{3}}$), ferromagnetism (Ni) and strong spin-orbit coupling (Bi)
Bergeret and Tokatly (2014); Jeon  _et al._ (2018); Satchell and Birge (2018);
Satchell  _et al._ (2019).

##  V Conclusions

In this work samples of Bi/Ni bilayers are observed initially to be non-
superconducting until either left at room temperature for several days or a
short period of heating as low as $+50^{\circ}\,\mathrm{C}$. The
superconductivity is attributed to intermixing of the Bi/Ni interface forming
an alloy of $\mathrm{NiBi_{3}}$ identified by XRD Bragg peaks. PNR data is
consistent with initially distinct Bi/Ni layers with minimal
$\mathrm{NiBi_{3}}$ in the as-grown states that when annealed at
$70^{\circ}\,\mathrm{C}$ for 1$\,\mathrm{hour}$ the diffusion across the
interface increases the $\mathrm{NiBi_{3}}$ thickness and reduces the pure
Bi/Ni layers.

From this it seems that to properly study clean and distinct interfaces that
maintaining a low temperature for preparation and storage is important. Normal
device fabrication recipes which use heating to bake resist or growth methods
that allow the sample to heat up will fully anneal Bi/Ni bilayers. Although
the superconductivity origins from a known bulk superconductor the existence
is still interesting as it is in close proximity to a ferromagnetic layer, a
strong spin-orbit coupling Bi layer and also the likely proximity effect in
the Bi layer.

The data associated with this paper are openly available from the University
of Leeds and ISIS Neutron and Muon source data repositories Not (b).

###### Acknowledgements.

We thank the ISIS neutron and muon source for allocating beamtime
(RB:1920455). The work was supported financially through the EPSRC Doctoral
Training Partnership and grant EP/M000923/1. This project has received funding
from the European Unions Horizon 2020 research and innovation programme under
the Marie Skłodowska-Curie Grant Agreement No. 743791 (SUPERSPIN).

## References

  * Ginzburg (1957) V. L. Ginzburg, Sov. Phys. JETP 4, 153 (1957).
  * Prakash  _et al._ (2017) O. Prakash, A. Kumar, A. Thamizhavel, and S. Ramakrishnan, Science (80-. ). 355, 52 (2017).
  * Petrosyan  _et al._ (1975) V. I. Petrosyan, V. N. Molin, O. I. Vasin, P. A. Skripkina, S. I. Stenin, and E. G. Batyev, Vacuum 25, 181 (1975).
  * Buckel and Wittig (1965) W. Buckel and J. Wittig, Phys. Lett. 17, 187 (1965).
  * Weitzel and Micklitz (1991) B. Weitzel and H. Micklitz, Phys. Rev. Lett. 66, 385 (1991).
  * Li  _et al._ (2017) Y. Li, E. Wang, X. Zhu, and H.-H. Wen, Phys. Rev. B 95, 024510 (2017).
  * Tian  _et al._ (2009) M. Tian, J. Wang, Q. Zhang, N. Kumar, T. E. Mallouk, and M. H. W. Chan, Nano Lett. 9, 3196 (2009).
  * Moodera and Meservey (1990) J. S. Moodera and R. Meservey, Phys. Rev. B 42, 179 (1990).
  * LeClair  _et al._ (2005) P. LeClair, J. S. Moodera, J. Philip, and D. Heiman, Phys. Rev. Lett. 94, 037006 (2005).
  * Gong  _et al._ (2015) X.-X. Gong, H.-X. Zhou, P.-C. Xu, D. Yue, K. Zhu, X.-F. Jin, H. Tian, G.-J. Zhao, and T.-Y. Chen, Chinese Phys. Lett. 32, 067402 (2015).
  * Gong  _et al._ (2017) X. Gong, M. Kargarian, A. Stern, D. Yue, H. Zhou, X. Jin, V. M. Galitski, V. M. Yakovenko, and J. Xia, Sci. Adv. 3, e1602579 (2017).
  * Wang  _et al._ (2017) J. Wang, X. Gong, G. Yang, Z. Lyu, Y. Pang, G. Liu, Z. Ji, J. Fan, X. Jing, C. Yang, F. Qu, X. Jin, and L. Lu, Phys. Rev. B 96, 054519 (2017).
  * Zhou and Jin (2018) H. Zhou and X. Jin, J. Magn. Magn. Mater. 458, 171 (2018).
  * Zhou  _et al._ (2017) H. Zhou, X. Gong, and X. Jin, J. Magn. Magn. Mater. 422, 73 (2017).
  * Chauhan  _et al._ (2019) P. Chauhan, F. Mahmood, D. Yue, P.-C. Xu, X. Jin, and N. P. Armitage, Phys. Rev. Lett. 122, 017002 (2019).
  * Tokuda  _et al._ (2019) M. Tokuda, N. Kabeya, K. Iwashita, H. Taniguchi, T. Arakawa, D. Yue, X. Gong, X. Jin, K. Kobayashi, and Y. Niimi, Appl. Phys. Express 12, 53005 (2019).
  * Alekseevskii  _et al._ (1952) N. E. Alekseevskii, N. B. Brandt, and T. I. Kostina, Izv. Akad. Nauk SSSR 16, 233 (1952).
  * Siva  _et al._ (2015) V. Siva, K. Senapati, B. Satpati, S. Prusty, D. K. Avasthi, D. Kanjilal, and P. K. Sahoo, J. Appl. Phys. 117, 83902 (2015).
  * Silva  _et al._ (2013) B. Silva, R. F. Luccas, N. M. Nemes, J. Hanko, M. R. Osorio, P. Kulkarni, F. Mompean, M. García-Hernández, M. A. Ramos, S. Vieira, and H. Suderow, Phys. Rev. B 88, 184508 (2013).
  * Zhao  _et al._ (2018) G. J. Zhao, X. X. Gong, P. C. Xu, B. C. Li, Z. Y. Huang, X. F. Jin, X. D. Zhu, and T. Y. Chen, Supercond. Sci. Technol. 31, 125005 (2018).
  * Liu  _et al._ (2018) L. Y. Liu, Y. T. Xing, I. L. C. Merino, H. Micklitz, D. F. Franceschini, E. Baggio-Saitovitch, D. C. Bell, and I. G. Solórzano, Phys. Rev. Mater. 2, 014601 (2018).
  * Not (a) (a), See Supplemental Material at [URL will be inserted by publisher] for additional details on the experimental methodology, the full set of fitting parameters for the PNR model, and additional data on the formation of superconducting phases in these samples.
  * Björck and Andersson (2007) M. Björck and G. Andersson, J. Appl. Crystallogr. 40, 1174 (2007).
  * Zabel  _et al._ (2008) H. Zabel, K. Theis Brohl, M. Wolff, and B. Toperverg, IEEE Trans. Magn. 44, 1928 (2008).
  * Robinson  _et al._ (2007) J. W. A. Robinson, S. Piano, G. Burnell, C. Bell, and M. G. Blamire, Phys. Rev. B 76, 094522 (2007).
  * Nicolet (1978) M.-A. Nicolet, Thin Solid Films 52, 415 (1978).
  * Höfler  _et al._ (1993) H. J. Höfler, R. S. Averback, H. Hahn, and H. Gleiter, J. Appl. Phys. 74, 3832 (1993).
  * Zhang  _et al._ (1995) H. Zhang, J. W. Lynn, C. F. Majkrzak, S. K. Satija, J. H. Kang, and X. D. Wu, Phys. Rev. B 52, 10395 (1995).
  * Drew  _et al._ (2009) A. J. Drew, M. W. Wisemayer, D. O. G. Heron, S. Lister, S. L. Lee, A. Potenza, C. H. Marrows, R. M. Dalgliesh, T. R. Charlton, and S. Langridge, Phys. Rev. B 80, 134510 (2009).
  * Luke  _et al._ (1998) G. M. Luke, Y. Fudamoto, K. M. Kojima, M. I. Larkin, J. Merrin, B. Nachumi, Y. J. Uemura, Y. Maeno, Z. Q. Mao, Y. Mori, H. Nakamura, and M. Sigrist, Nature 394, 558 (1998).
  * Flokstra  _et al._ (2016) M. G. Flokstra, N. Satchell, J. Kim, G. Burnell, P. J. Curran, S. J. Bending, J. F. K. Cooper, C. J. Kinane, S. Langridge, A. Isidori, N. Pugach, M. Eschrig, H. Luetkens, A. Suter, T. Prokscha, and S. L. Lee, Nat. Phys. 12, 57 (2016).
  * Flokstra  _et al._ (2019) M. G. Flokstra, R. Stewart, N. Satchell, G. Burnell, H. Luetkens, T. Prokscha, A. Suter, E. Morenzoni, S. Langridge, and S. L. Lee, Appl. Phys. Lett. 115, 072602 (2019).
  * Bergeret  _et al._ (2001) F. S. Bergeret, A. F. Volkov, and K. B. Efetov, Phys. Rev. Lett. 86, 4096 (2001).
  * Bergeret and Tokatly (2014) F. S. Bergeret and I. V. Tokatly, Phys. Rev. B 89, 134517 (2014).
  * Jeon  _et al._ (2018) K.-R. Jeon, C. Ciccarelli, A. J. Ferguson, H. Kurebayashi, L. F. Cohen, X. Montiel, M. Eschrig, J. W. A. Robinson, and M. G. Blamire, Nat. Mater. 17, 499 (2018).
  * Satchell and Birge (2018) N. Satchell and N. O. Birge, Phys. Rev. B 97, 214509 (2018).
  * Satchell  _et al._ (2019) N. Satchell, R. Loloee, and N. O. Birge, Phys. Rev. B 99, 174519 (2019).
  * Not (b) (b), M. Vaughan et al (2019), University of Leeds data repository. URL to be added. ISIS Nerutropn and Muon Sopurce data Repository doi:10.5286/ISIS.E.RB1920455.


# CheXpedition: Investigating Generalization Challenges for Translation of
Chest X-Ray Algorithms to the Clinical Setting

Pranav Rajpurkar<EMAIL_ADDRESS>1234-5678-9012 Stanford University
,  Anirudh Joshi<EMAIL_ADDRESS>Stanford University ,  Anuj Pareek
<EMAIL_ADDRESS>0000-0002-1526-3685 Stanford University ,  Phil Chen  ,
Amirhossein Kiani  ,  Jeremy Irvin  Stanford University ,  Andrew Y. Ng
Stanford University and  Matthew P. Lungren  Stanford University

###### Abstract.

Although there have been several recent advances in the application of deep
learning algorithms to chest x-ray interpretation, we identify three major
challenges for the translation of chest x-ray algorithms to the clinical
setting. We examine the performance of the top 10 performing models on the
CheXpert challenge leaderboard on three tasks: (1) TB detection, (2) pathology
detection on photos of chest x-rays, and (3) pathology detection on data from
an external institution. First, we find that the top 10 chest x-ray models on
the CheXpert competition achieve an average AUC of 0.851 on the task of
detecting TB on two public TB datasets without fine-tuning or including the TB
labels in training data. Second, we find that the average performance of the
models on photos of x-rays (AUC = 0.916) is similar to their performance on
the original chest x-ray images (AUC = 0.924). Third, we find that the models
tested on an external dataset either perform comparably to or exceed the
average performance of radiologists. We believe that our investigation will
inform rapid translation of deep learning algorithms to safe and effective
clinical decision support tools that can be validated prospectively with large
impact studies and clinical trials.

generalization, chest x-ray classification

††conference: Preprint; 2020; USA††booktitle: 2020††ccs: Computing
methodologies Neural networks††ccs: Applied computing Health informatics
Figure 1. We examine the performance of the top 10 available models on the
CheXpert competition leaderboard on three tasks: (1) TB detection, (2)
pathology detection on photos of chest x-rays, and (3) pathology detection on
data from an external institution

##  1\. Introduction

There have been several recent advances in the application of deep learning
algorithms to chest x-ray interpretation at a high level of performance
(Rajpurkar et al., 2018b; Singh et al., 2018; Nam et al., 2018). Although
these advancements have led many to suggest a near-term potential of these
algorithms to provide accurate chest x-ray interpretation and increase access
to radiology expertise, a few major challenges remain to their translation to
the clinical setting.

There remain major challenges for the translation of chest x-ray algorithms to
the clinical setting. First, the performance of deep learning chest x-ray
algorithms, trained with mainly US-based chest x-ray datasets, on endemic and
globally relevant diseases not commonly found in the US, such as tuberculosis
(TB) is unknown (Qin et al., 2018; Qin et al., 2019). Second, most chest x-ray
algorithms have been developed and validated on digital x-rays, while the vast
majority of the world relies on film for X-ray interpretation, a barrier that
denies these populations from the advancements of automated interpretation
(Schwartz et al., 2014). In order to apply an interim digital solution,
digital photographs of films for storage, interpretation, and consultation can
be performed as a ”workaround” (Handelman et al., 2018). Third, chest x-ray
algorithms which are developed using the data from one institution have not
shown sustained performance when externally validated in application data from
a different unrelated institution, and instead, these models have been
criticized as vulnerable to bias and non-medically relevant cues (Zech et al.,
2018). We believe that tackling each of these challenges will serve to inform
improved translation of deep learning algorithms into safe and effective
clinical decision support tools that can be validated prospectively with large
impact studies and clinical trials.

The purpose of this work is to systematically address the aforementioned
translation challenges for chest x-ray models. We validate the performance of
chest x-ray models on the tasks of (1) TB detection (2) pathology detection on
digital photographs of chest x-rays, and (3) pathology detection on chest
x-rays from a separate institution. Rather than choosing one model
architecture or approach, we evaluate performance under each of the conditions
using the top 10 performing models on the CheXpert challenge, a large public
competition for chest x-ray analysis (Irvin et al., 2019).

In this work we report performance metrics for the generalizability of
existing chest x-ray models on the three aforementioned tasks. First, we find
that the top 10 chest x-ray models on the CheXpert competition without fine-
tuning or including the TB labels in training data, achieve an average AUC of
0.851 on the task of detecting TB on two public TB datasets, competitive with
previously published approaches that trained and tested their models
specifically on these same TB datasets (Qin et al., 2018; Qin et al., 2019).
Second, we find that the average performance of the models on photos of x-rays
(AUC = 0.916) is similar to their performance on the original chest x-ray
images (AUC = 0.924). Third, we find that the models tested on an external
dataset either perform comparably to or exceed the average performance of
radiologists.

Figure 2. The average AUC of the models on all five CheXpert tasks was a
stronger predictor (R2=0.78) of TB performance than the performance of on the
consolidation task in CheXpert (R2=0.011).

##  2\. Experimental Setup

### Top Models on CheXpert Leaderboard

We investigated the generalization performance of the top 10 models on the
CheXpert (Irvin et al., 2019) competition leaderboard. CheXpert is a
competition for automated chest x-ray interpretation that has been running
from January 2019 featuring a strong radiologist-labeled reference standard.
As of November, 2019, there were 94 models that had been submitted to the
CheXpert leaderboard from both academic and industry teams. The top 10
available models on the CheXpert competition leaderboard as of November 2019
were selected. All of the selected models were ensembles with the number of
models in the ensemble ranging from 8 to 32; the majority of these models
featured Densely Connected Convolutional Networks (Huang et al., 2017) as part
of their ensemble.

### Running Models on New Test Sets

CheXpert is a unique competition in that it uses a hidden test set for
official evaluation of models. Teams submit their executable code, which is
then run on a test set that is not publicly readable. Such a setup preserves
the integrity of the test results. Models can be rerun on new test sets to
evaluate the ability of the model to generalize to new domains.

We make use of the CodaLab platform to re-run these chest x-ray models on the
new test sets. CodaLab is an online platform for collaborative and
reproducible computational research. The system exposes a simple command-line
interface using which one can upload code and data and subsequently submit
jobs to run them. Once a team has submitted their model on CodaLab and
successfully inferred on the hidden CheXpert test set, they get added to the
leaderboard. We reproduced the runs of the top 10 teams using their model
checkpoints and inference scripts by substituting the hidden CheXpert test set
for the other datasets used in this study.

### Evaluation Metrics

Our primary evaluation metric is the area under the receiver operating
characteristic curve (AUC). We report the average AUC of the top 10 models on
the new test sets, averaged over the available tasks. Additionally, in
experiments comparing the models to board-certified radiologists, we compute
the average sensitivities of all models per task thresholded at the
specificities of the radiologists at each task. The sensitivities of the
average of the radiologists are compared to the sensitivities of the models
per task.

### Saliency Maps

Gradient-weighted class activation maps (CAMs) (Selvaraju et al., 2016; Zhou
et al., 2015) were used to highlight regions with the greatest influence on a
model’s decision. For a given x-ray, the CAM was produced for every class by
taking the weighted average across the final convolutional feature map, with
weights determined by the linear layer. The CAM was then scaled according to
the output probability, so that more confident predictions appeared brighter.
Finally, the map was upsampled to the input image resolution, and overlaid
onto the input image. The Stanford baseline model on the CheXpert leaderboard
was used as the model of choice to generate the CAMs.

Figure 3. The average performance of the models on photos of x-rays (AUC =
0.916) is similar to their performance on the original x-ray images (AUC =
0.924). Figure 4. The CAM for the TB detection task on the consolidation class
correctly highlights the region of the image consisting with TB.

##  3\. TB Detection

### Task

We evaluated the models on the task of detecting tuberculosis (TB). TB is the
leading cause of death from a single infectious disease agent and the leading
cause of death for people living with human immunodeficiency virus (HIV)
infection (MacNeil et al., 2019). Currently, chest x-ray models trained using
large datasets from American institutions (Irvin et al., 2019; Johnson et al.,
2019; Wang et al., 2017) do not include TB as one of the labeled pathologies
because the pathology is not prevalent in their settings. However, the
application of chest x-ray models to the global setting requires their high
performance on this globally relevant task. We hypothesized that we could use
existing models trained on the CheXpert dataset to detect TB without any fine-
tuning on the TB task or the TB datasets. Because consolidation is one of the
most common chest x-ray manifestations of pulmonary TB, we considered the use
of the consolidation label as a proxy for the task of detecting TB.

### Datasets

We tested the performance of the models on two datasets: the Shenzhen and
Montgomery datasets released by the NIH (Jaeger et al., 2014). The Shenzhen
dataset was collected in the Shenzhen No.3 Hospital, China. Of the 662 x-rays
in the dataset, 326 are normal and 336 are abnormal with manifestations of TB;
34 cases are pediatric cases (defined as age $<$ 18 years). The Montgomery set
was collected by the Department of Health and Human Services in Montgomery
County, USA. Of the 138 x-rays in the dataset, 80 are normal and 58 are
abnormal; 17 cases are pediatric cases.

### Results

We evaluated the performance of the models using their probability on the
consolidation label as the predicted score for TB on an x-ray (see Figure 2).
The average AUC of the models on the TB test sets ranged from 0.815 to 0.893
with an average of 0.851.

### Analysis

We analyzed the strength of the relationship between the performance of the
models on the source tasks and the target TB dataset. We ran a linear
regression to predict the average AUC of the models on the TB datasets using
(1) the average AUC of the models on the consolidation task in CheXpert, and
(2) the average AUC across all 5 competition tasks in CheXpert.

We found that the strength of relationship was smaller for the AUC on
consolidation in CheXpert ($R^{2}=0.011$) compared to the average AUC on all
five CheXpert tasks ($R^{2}=0.78$).

### Discussion

There have been a number of studies developing models for TB detection. Hwang
et al. (2016) tested on the Shenzhen TB dataset without training on the data,
but their models were explicitly trained on the TB task, and achieved an AUC
of 0.884 on the Shenzhen dataset. Pasa et al. (2019) reported AUCs of 0.811 on
Montgomery and 0.900 on the Shenzhen dataset when their model was trained on a
combination of the two datasets and additional data. Similarly, Vajda et al.
(2018) reported AUCs of 0.870 on Montgomery and 0.990 on the Shenzhen dataset
after training on the same two datasets. Finally, Lakhani and Sundaram (2017)
trained on a combination of four different TB datasets, and achieved an AUC of
0.990 on their test set with their ensemble model.

In our study, we found that the average AUC of the models on the TB test sets
(average AUC of 0.851) without exposure to TB datasets was competitive to that
of models that had been directly trained on these datasets for the task of
tuberculosis detection. We also found that the average performance of a model
across tasks was a stronger predictor of performance on the tuberculosis
dataset as compared to the performance of the model on any of the individual
tasks. This suggests that training models to perform well across tasks may
allow them to perform better on unseen images than models that optimize for a
single task. A possible reason for this finding may be that the shared
representations learnt by optimizing for multitask performance are exploited
for better performance on different data distributions (Caruana, 1997).

Figure 5. The average performance of models tested on the external dataset is
either comparable to or exceeds the average performance of radiologists on all
5 tasks. Figure 6. The CAM for the photo of the portable frontal radiograph of
the chest demonstrates cardiomegaly and bilateral mid and lower lung
interstitial predominant opacities consistent with pulmonary edema.

##  4\. Smartphone Photos

### Task

We evaluated the models on the task of detecting pathologies on smartphone
photos of chest x-rays. While most deep learning models are trained on digital
x-rays, scaled deployment demands a solution that can navigate an endless
array of medical imaging / IT infrastructures. An appealing solution to scaled
deployment is to leverage the ubiquity of smartphones: clinicians and
radiologists in parts of the world take smartphone photos of medical imaging
studies to share with other experts or clinicians using messaging services
like WhatsApp (Handelman et al., 2018). While using photos of chest x-rays to
input into chest-xray algorithms could enable any physician with a smartphone
to get instant AI algorithm assistance, the performance of chest x-ray
algorithms on photos of chest x-rays has not been thoroughly investigated.
Outside chest x-ray classification, deep learning algorithms for image
classification have been shown to attain lower performance on photos of images
than on the images themselves (Kurakin et al., 2016). We conducted an
experiment to determine whether existing chest x-ray models could generalize
well to photos of chest x-rays.

### Datasets

We generated a dataset of photos of the CheXpert test set, consisting of
studies from 500 patients. Chest X-rays from each test study were displayed on
a non-diagnostic computer monitor. Photos of the monitor were taken with an
Apple iPhone 7 by a physician. The physician was instructed to keep the mobile
camera stable and center the lung fields in the camera view. A time-
restriction of 5 seconds per image was imposed to simulate a busy healthcare
environment. Subsequent inspection of photos showed that they were taken with
slightly varying angles; some photos included artefacts such as Moiré patterns
and subtle screen-glares. Photos were labeled using the ground truth for the
corresponding digital x-ray image.

### Results

The models achieved a mean AUC of 0.916 on photos of the chexpert test set,
compared with an AUC of 0.924 on the original chexpert test set. All of the
models had mean AUCs higher than 0.9, and were within 0.01 AUC of their
performance on the original images. The average AUCs of each of the top 10
models across the 5 CheXpert competition tasks are detailed in Figure 3.

### Discussion

Several studies have highlighted the importance of generalizability of
computer vision models with noise in images (Hendrycks and Dietterich, 2019).
Dodge and Karam (2017) demonstrated that deep neural networks perform poorly
compared to humans on image classification on distorted images. Schmidt et al.
(2018); Geirhos et al. (2019) have found that convolutional neural networks
trained on specific image corruptions did not generalize, and the error
patterns of network and human predictions were not similar on noisy and
elastically deformed images.

In our study, the dataset we generated for this experiment allows for the
direct comparison of the effect of photos against the source images on model
performance, addressing a key deployment and generalization challenge. We
found that the performance across top teams on photos of chest x-rays was
comparable to their performance on the original x-rays. Figure 6 demonstrates
that the model is able to detect the location of the pathology on a
characteristic example where the distortion generated by taking photos of the
x-rays did not affect the ability of the model to identify clinically relevant
information in the x-rays.

Example |  Category |  Count
---|---|---
|  Failed to correctly localize (False Negative). CAMs failed to localize the actual consolidation. Typically, the consolidation was smaller or less opaque than average; in some cases, the CAMs highlighted a feature that was visually similar but unrelated to consolidation. |  36 (44.44%)
|  Failed to confidently detect (False Negative). CAMs accurately localized the consolidation, but wasn’t confident enough to make a positive diagnosis. This was found to occur when the consolidation was overlapping with other diseases (such as severe pulmonary edema) or anatomical structures. |  29 (35.80%)
|  Mistaken for mimicking feature (False Positive). CAMs detected a visual feature which mimics consolidation and made a false positive diagnosis. This was often the case in the presence of severe pulmonary edema, and cases with other pulmonary opacities such as fibrosis, scarring and lung lesion. |  13 (16.05%)
|  Mistaken for non-mimicking feature (False Positive). The x-ray contains enlarged cardiac contours and bilateral mid and lower lung interstitial predominant opacities consistent with cardiomegaly and pulmonary edema. CAMs highlighted an area of the cardiac border and chest wall which bear no apparent visual resemblance to consolidations. |  3 (3.70%)
Table 1. Qualitative CAM Analysis of CheXpert model mistakes (73 images out of
420 images) on the consolidation task on external institution test data.

##  5\. External Institution

### Task

We evaluated the performance of the top 10 CheXpert models on a dataset from
an external institution. Chest x-ray algorithms which are developed using the
data from one institution have not shown sustained performance when externally
validated in application data from a different unrelated institution and have
been criticized as vulnerable to bias and non-medically relevant cues (Zech et
al., 2018). Furthermore, certain institutions may not allow access to patient
data for privacy reasons. This makes it important for models trained on one
institution’s data to be generalizable to others without finetuning or
retraining for wider deployment in the healthcare system.

### Dataset

We used a set of 420 frontal chest x-rays curated in the test set of Rajpurkar
et al. (2018b). These x-rays contained images from the ChestXray-14 dataset
collected at the National Institutes of Health Clinical Center (Wang et al.,
2017), sampled to contain at least 50 cases of each pathology according to the
original labels provided in the dataset.

### Results

The models achieved an average performance of 0.897 AUC across the 5 CheXpert
competition tasks on the test set from the external institution. On
Atelectasis, Cardiomegaly, Edema, and Pleural Effusion, the mean sensitivities
of the models of 0.750, 0.617, 0.712, and 0.806 respectively, are higher than
the mean radiologist sensitivities of 0.646, 0.485, 0.710, and 0.761 (at the
mean radiologist specificities of 0.806, 0.924, 0.925, and 0.883
respectively). On Consolidation, the mean sensitivity of the models of 0.443
is lower than the mean radiologist sensitivity of 0.456 (at the mean
radiologist specificity of 0.935).

### Analysis

Because our primary performance measures do not reveal any information on
patterns of mistakes or systematic biases, we qualitatively analyzed chest
x-rays where the model output was wrong compared to ground truth diagnosis of
consolidation. We used CAMs to reason about model mistakes. The analysis
revealed that the type of model mistakes could be pooled into four distinct
categories as shown in Table 1. Each chest x-ray was categorized into one or
more of four categories: Failure to correctly localize the consolidation,
Failure to confidently detect consolidation, Mistaking a mimicking feature for
consolidation, Mistaking a non-mimicking feature for consolidation. The most
common mistake was failure to detect to consolidation, and as can be expected
this was often the case for faint or small consolidations.

### Discussion

Given the variety of healthcare systems and patient populations, it is
critical for deep learning models in healthcare to be able to generalize to
new patient populations from different institutions (Kelly et al., 2019; Chen
et al., 2019). There have been several studies investigating the
generalization of models to different institutions. Particularly for chest
x-ray interpretation models, Zech et al. (2018) trained image classifiers on
chest x-ray from three different institutions and found that models trained on
data from one institution failed to generalize to other institutions. Chen et
al. (2019) raised concerns about whether deep learning based approaches could
generalize to smaller healthcare institutions with limited data. Kelly et al.
(2019) detailed limitations of deep learning towards generalization to new
populations given that the models may learn confounders present in one
population. However, McKinney et al. (2020) recently showed that the
performance of deep learning models on the task of breast cancer detection
entirely trained on data from the UK generalized to healthcare data from the
US. Kim et al. (2019) reported that only 6% of studies evaluating the
performance of AI algorithms for diagnostic analysis of medical images
performed external validation.

In our study, we found that CheXpert-trained models demonstrated
generalizability to another institution’s data without any additional site
specific training. Furthermore, the models exceeded radiologists on
sensitivity for majority of the tasks when thresholded on radiologists’
specificity despite not having been trained on the dataset. The CAMs
demonstrate that the model is learning clinically relevant information in the
chest x-rays and not confounders.

##  6\. Limitations

Our primary assumption in testing the generalization of these models for these
different tasks and circumstances is that these models had not been exposed to
data used for the external test sets. All models used in the study were
trained exclusively to classify CheXpert pathologies (and did not include TB
or NIH-specific pathologies): we verified that the output of all models had
complete intersection with the CheXpert pathologies.

Furthermore, the results of our study do not suggest guaranteed generalization
of chest x-ray models to new clinical settings; future work should evaluate
evaluate the performance in clinical trials for further verification, a
necessary step for the successful translation of diagnostic or predictive
artificial intelligence tools into practice (Park and Han, 2018).

##  7\. Conclusion

Despite advances in the performance of chest x-ray algorithms (Lakhani and
Sundaram, 2017; Kallianos et al., 2019; Shih et al., 2019; Kashyap et al.,
2019; Qin et al., 2019; Qin et al., 2018), the ability of these models to
generalize has not been systematically explored. The purpose of this study was
to systematically evaluate the generalization capabilities of existing models
to (1) detect diseases not explicitly included in model development, (2)
smartphone photos of x-rays, and (3) x-rays from institutions not included in
model development. Our results suggest the possibility for existing chest
x-ray models to generalize to new clinical settings without fine-tuning.

Deep learning models, including for chest x-ray interpretation, have been
criticized for their inability to generalize to new clinical settings (Kelly
et al., 2019). For instance, Zech et al. (2018) reported that chest x-ray
models failed to generalize to new populations or institutions separate from
the training data, relying on institution specific and/or confounding cues to
infer the label of interest. In contrast, our results suggest that existing
models may generalize across institutions, modalities, and diseases without
further engineering. Importantly, in evaluation of the models there was no
indication of bias toward institution specific features in model decision
making or a reliance on unrelated features for classification as evident from
the class activation maps.

Our systematic examination of the generalization capabilities of existing
models can be extended to other tasks in medical AI (Rajpurkar et al., 2018a;
Hannun et al., 2019; Park et al., 2019; Uyumazturk et al., 2019; Varma et al.,
2019; Duan et al., 2019; Topol, 2019), and provide a framework for tracking
technical readiness towards clinical translation.

###### Acknowledgements.

We would like to acknowledge the Stanford Machine Learning Group
(stanfordmlgroup.github.io) and the Stanford Program for Artificial
Intelligence in Medicine and Imaging for infrastructure support
(AIMI.stanford.edu). We would also like to acknowledge those among the top
submitters in the competition who helped us understand the data and techniques
used in their models: Wenwu Ye from JF healthcare, Hieu Pham from the Medical
Imaging Team at Vingroup Big Data Institute (VinBDI), Desmond from Beihang
University, Vu Hoang and Hoang Ngoc Nguyen from the VinBrain Applied Scientist
Team.

## References

  * (1)
  * Caruana (1997) Rich Caruana. 1997\.  Multitask Learning.  _Mach. Learn._ 28, 1 (July 1997), 41–75.  https://doi.org/10.1023/A:1007379606734
  * Chen et al. (2019) David Chen, Sijia Liu, Paul Kingsbury, Sunghwan Sohn, Curtis B. Storlie, Elizabeth B. Habermann, James M. Naessens, David W. Larson, and Hongfang Liu. 2019\.  Deep learning and alternative learning strategies for retrospective real-world clinical data.  _Nature News_ (May 2019).  https://www.nature.com/articles/s41746-019-0122-0
  * Dodge and Karam (2017) Samuel Dodge and Lina Karam. 2017.  A Study and Comparison of Human and Deep Learning Recognition Performance under Visual Distortions. In _2017 26th International Conference on Computer Communication and Networks (ICCCN)_. 1–7.  https://doi.org/10.1109/ICCCN.2017.8038465
  * Duan et al. (2019) Tony Duan, Pranav Rajpurkar, Dillon Laird, Andrew Y Ng, and Sanjay Basu. 2019.  Clinical Value of Predicting Individual Treatment Effects for Intensive Blood Pressure Therapy: A Machine Learning Experiment to Estimate Treatment Effects from Randomized Trial Data.  _Circulation: Cardiovascular Quality and Outcomes_ 12, 3 (2019), e005010.
  * Geirhos et al. (2019) Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. 2019.  ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness.  _arXiv:1811.12231 [cs, q-bio, stat]_ (Jan. 2019).  arXiv: 1811.12231.
  * Handelman et al. (2018) Guy S. Handelman, Ailin C. Rogers, Zafir Babiker, Michael J. Lee, and Morgan P. McMonagle. 2018\.  Media messaging in diagnosis of acute CXR pathology: an interobserver study among residents.  _SpringerLink_ (Apr 2018).
  * Hannun et al. (2019) Awni Y Hannun, Pranav Rajpurkar, Masoumeh Haghpanahi, Geoffrey H Tison, Codie Bourn, Mintu P Turakhia, and Andrew Y Ng. 2019\.  Cardiologist-level arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network.  _Nature Medicine_ 25, 1 (2019), 65.
  * Hendrycks and Dietterich (2019) Dan Hendrycks and Thomas Dietterich. 2019.  Benchmarking Neural Network Robustness to Common Corruptions and Perturbations.  _arXiv:1903.12261 [cs, stat]_ (March 2019).  arXiv: 1903.12261.
  * Huang et al. (2017) Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. 2017.  Densely Connected Convolutional Networks. 4700–4708.
  * Hwang et al. (2016) Sangheum Hwang, Hyo-Eun Kim, Jihoon Jeong M.d, and Hee-Jin Kim. 2016.  A novel approach for tuberculosis screening based on deep convolutional neural networks. In _Medical Imaging 2016: Computer-Aided Diagnosis_ , Vol. 9785. International Society for Optics and Photonics, 97852W.  https://doi.org/10.1117/12.2216198
  * Irvin et al. (2019) Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, Jayne Seekins, David A. Mong, Safwan S. Halabi, Jesse K. Sandberg, Ricky Jones, David B. Larson, Curtis P. Langlotz, Bhavik N. Patel, Matthew P. Lungren, and Andrew Y. Ng. 2019\.  CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison.  _arXiv:1901.07031 [cs, eess]_ (Jan. 2019).  arXiv: 1901.07031.
  * Jaeger et al. (2014) Stefan Jaeger, Sema Candemir, Sameer Antani, Yì-Xiáng J Wáng, Pu-Xuan Lu, and George Thoma. 2014\.  Two public chest X-ray datasets for computer-aided screening of pulmonary diseases.  _Quantitative imaging in medicine and surgery_ (Dec 2014).
  * Johnson et al. (2019) Alistair E. W. Johnson, Tom J. Pollard, Nathaniel R. Greenbaum, Matthew P. Lungren, Chih-ying Deng, Yifan Peng, Zhiyong Lu, Roger G. Mark, Seth J. Berkowitz, and Steven Horng. 2019.  MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs.  _arXiv:1901.07042 [cs, eess]_ (Nov. 2019).  arXiv: 1901.07042.
  * Kallianos et al. (2019) K. Kallianos, J. Mongan, S. Antani, T. Henry, A. Taylor, J. Abuya, and M. Kohli. 2019\.  How far have we come? Artificial intelligence for chest radiograph interpretation.  _Clinical Radiology_ 74, 5 (May 2019), 338–345.  https://doi.org/10.1016/j.crad.2018.12.015
  * Kashyap et al. (2019) Satyananda Kashyap, Mehdi Moradi, Alexandros Karargyris, Joy T. Wu, Michael Morris, Babak Saboury, Eliot Siegel, and Tanveer Syeda-Mahmood. 2019.  Artificial intelligence for point of care radiograph quality assessment. In _Medical Imaging 2019: Computer-Aided Diagnosis_ , Vol. 10950. International Society for Optics and Photonics, 109503K.  https://doi.org/10.1117/12.2513092
  * Kelly et al. (2019) Christopher Kelly, Alan Karthikesalingam, Mustafa Suleyman, Greg Corrado, and Dominic King. 2019\.  Key challenges for delivering clinical impact with artificial intelligence.  (Oct 2019).
  * Kim et al. (2019) Dong Wook Kim, Hye Young Jang, Kyung Won Kim, Youngbin Shin, and Seong Ho Park. 2019.  Design Characteristics of Studies Reporting the Performance of Artificial Intelligence Algorithms for Diagnostic Analysis of Medical Images: Results from Recently Published Papers.  _Korean journal of radiology_ (Mar 2019).
  * Kurakin et al. (2016) Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. 2016\.  Adversarial examples in the physical world.  _CoRR_ abs/1607.02533 (2016).  arXiv:1607.02533
  * Lakhani and Sundaram (2017) Paras Lakhani and Baskaran Sundaram. 2017.  Deep Learning at Chest Radiography: Automated Classification of Pulmonary Tuberculosis by Using Convolutional Neural Networks.  _Radiology_ 284, 2 (April 2017), 574–582.  https://doi.org/10.1148/radiol.2017162326
  * MacNeil et al. (2019) Adam MacNeil, Philippe Glaziou, Charalambos Sismanidis, Susan Maloney, and Katherine Floyd. 2019\.  Global Epidemiology of Tuberculosis and Progress Toward Achieving Global Targets - 2017.
  * McKinney et al. (2020) Scott Mayer McKinney, Marcin Sieniek, Varun Godbole, Jonathan Godwin, Natasha Antropova, Hutan Ashrafian, Trevor Back, Mary Chesus, Greg C. Corrado, Ara Darzi, and et al. 2020\.  International evaluation of an AI system for breast cancer screening.  _Nature News_ (Jan 2020).
  * Nam et al. (2018) Ju Gang Nam, Sunggyun Park, Eui Jin Hwang, Jong Hyuk Lee, Kwang-Nam Jin, Kun Young Lim, Thienkai Huy Vu, Jae Ho Sohn, Sangheum Hwang, Jin Mo Goo, et al. 2018\.  Development and validation of deep learning–based automatic detection algorithm for malignant pulmonary nodules on chest radiographs.  _Radiology_ 290, 1 (2018), 218–228.
  * Park et al. (2019) Allison Park, Chris Chute, Pranav Rajpurkar, Joe Lou, Robyn L Ball, Katie Shpanskaya, Rashad Jabarkheel, Lily H Kim, Emily McKenna, Joe Tseng, et al. 2019\.  Deep Learning–Assisted Diagnosis of Cerebral Aneurysms Using the HeadXNet Model.  _JAMA Network Open_ 2, 6 (2019), e195600–e195600.
  * Park and Han (2018) Seong Ho Park and Kyunghwa Han. 2018.  Methodologic Guide for Evaluating Clinical Performance and Effect of Artificial Intelligence Technology for Medical Diagnosis and Prediction.  _Radiology_ 286, 3 (Jan. 2018), 800–809.  https://doi.org/10.1148/radiol.2017171920
  * Pasa et al. (2019) F. Pasa, V. Golkov, F. Pfeiffer, D. Cremers, and D. Pfeiffer. 2019.  Efficient Deep Network Architectures for Fast Chest X-Ray Tuberculosis Screening and Visualization.  _Scientific Reports_ 9, 1 (April 2019), 1–9.  https://doi.org/10.1038/s41598-019-42557-4
  * Qin et al. (2018) Chunli Qin, Demin Yao, Yonghong Shi, and Zhijian Song. 2018\.  Computer-aided detection in chest radiography based on artificial intelligence: a survey.  _BioMedical Engineering OnLine_ 17, 1 (Aug. 2018), 113.  https://doi.org/10.1186/s12938-018-0544-y
  * Qin et al. (2019) Zhi Zhen Qin, Melissa S. Sander, Bishwa Rai, Collins N. Titahong, Santat Sudrungrot, Sylvain N. Laah, Lal Mani Adhikari, E. Jane Carter, Lekha Puri, Andrew J. Codlin, and Jacob Creswell. 2019.  Using artificial intelligence to read chest radiographs for tuberculosis detection: A multi-site evaluation of the diagnostic accuracy of three deep learning systems.  _Scientific Reports_ 9, 1 (Oct. 2019), 1–10.  https://doi.org/10.1038/s41598-019-51503-3
  * Rajpurkar et al. (2018a) Pranav Rajpurkar, Jeremy Irvin, Aarti Bagul, Daisy Ding, Tony Duan, Hershel Mehta, Brandon Yang, Kaylie Zhu, Dillon Laird, Robyn L Ball, et al. 2018a.  MURA: Large Dataset for Abnormality Detection in Musculoskeletal Radiographs. In _1st Conference on Medical Imaging with Deep Learning_.
  * Rajpurkar et al. (2018b) Pranav Rajpurkar, Jeremy Irvin, Robyn L. Ball, Kaylie Zhu, Brandon Yang, Hershel Mehta, Tony Duan, Daisy Ding, Aarti Bagul, Curtis P. Langlotz, Bhavik N. Patel, Kristen W. Yeom, Katie Shpanskaya, Francis G. Blankenberg, Jayne Seekins, Timothy J. Amrhein, David A. Mong, Safwan S. Halabi, Evan J. Zucker, Andrew Y. Ng, and Matthew P. Lungren. 2018b.  Deep learning for chest radiograph diagnosis: A retrospective comparison of the CheXNeXt algorithm to practicing radiologists.  _PLOS Medicine_ 15, 11 (Nov. 2018), e1002686.  https://doi.org/10.1371/journal.pmed.1002686
  * Schmidt et al. (2018) Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. 2018\.  Adversarially Robust Generalization Requires More Data.  In _Advances in Neural Information Processing Systems 31_ , S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.). Curran Associates, Inc., 5014–5026.
  * Schwartz et al. (2014) Adam B Schwartz, Gina S Siddiqui, John L Barbieri, Amana F Akhtar, Woojin K Kim, Ryan A Littman-Quinn, Emily S Conant, Narainder D Gupta, Bryan A Pukenas, Parvati H Ramchandani, and et al. 2014\.  The accuracy of mobile teleradiology in the evaluation of chest X-rays.  _Journal of Telemedicine and Telecare_ (Oct 2014).
  * Selvaraju et al. (2016) Ramprasaath R. Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. 2016.  Grad-CAM: Why did you say that? Visual Explanations from Deep Networks via Gradient-based Localization.  _CoRR_ abs/1610.02391 (2016).  arXiv:1610.02391 http://arxiv.org/abs/1610.02391
  * Shih et al. (2019) George Shih, Carol C. Wu, Safwan S. Halabi, Marc D. Kohli, Luciano M. Prevedello, Tessa S. Cook, Arjun Sharma, Judith K. Amorosa, Veronica Arteaga, Maya Galperin-Aizenberg, Ritu R. Gill, Myrna C.B. Godoy, Stephen Hobbs, Jean Jeudy, Archana Laroia, Palmi N. Shah, Dharshan Vummidi, Kavitha Yaddanapudi, and Anouk Stein. 2019\.  Augmenting the National Institutes of Health Chest Radiograph Dataset with Expert Annotations of Possible Pneumonia.  _Radiology: Artificial Intelligence_ 1, 1 (Jan. 2019), e180041.  https://doi.org/10.1148/ryai.2019180041
  * Singh et al. (2018) Ramandeep Singh, Mannudeep K. Kalra, Chayanin Nitiwarangkul, John A. Patti, Fatemeh Homayounieh, Atul Padole, Pooja Rao, Preetham Putha, Victorine V. Muse, Amita Sharma, and Subba R. Digumarthy. 2018\.  Deep learning in chest radiography: Detection of findings and presence of change.  _PLoS ONE_ 13, 10 (Oct. 2018).  https://doi.org/10.1371/journal.pone.0204155
  * Topol (2019) Eric J Topol. 2019\.  High-performance medicine: the convergence of human and artificial intelligence.  _Nature medicine_ 25, 1 (2019), 44.
  * Uyumazturk et al. (2019) Bora Uyumazturk, Amirhossein Kiani, Pranav Rajpurkar, Alex Wang, Robyn L Ball, Rebecca Gao, Yifan Yu, Erik Jones, Curtis P Langlotz, Brock Martin, et al. 2019\.  Deep Learning for the Digital Pathologic Diagnosis of Cholangiocarcinoma and Hepatocellular Carcinoma: Evaluating the Impact of a Web-based Diagnostic Assistant.  _arXiv preprint arXiv:1911.07372_ (2019).
  * Vajda et al. (2018) Szilárd Vajda, Alexandros Karargyris, Stefan Jaeger, KC Santosh, Sema Candemir, Zhiyun Xue, Sameer Antani, and George Thoma. 2018.  Feature selection for automatic tuberculosis screening in frontal chest radiographs.  _Journal of medical systems_ 42, 8 (2018), 146\.
  * Varma et al. (2019) Maya Varma, Mandy Lu, Rachel Gardner, Jared Dunnmon, Nishith Khandwala, Pranav Rajpurkar, Jin Long, Christopher Beaulieu, Katie Shpanskaya, Li Fei-Fei, et al. 2019\.  Automated abnormality detection in lower extremity radiographs using deep learning.  _Nature Machine Intelligence_ (2019), 1–6.
  * Wang et al. (2017) Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers. 2017.  Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 2097–2106.
  * Zech et al. (2018) John R. Zech, Marcus A. Badgeley, Manway Liu, Anthony B. Costa, Joseph J. Titano, and Eric Karl Oermann. 2018.  Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study.  _PLOS Medicine_ 15, 11 (Nov. 2018), e1002683.  https://doi.org/10.1371/journal.pmed.1002683
  * Zhou et al. (2015) Bolei Zhou, Aditya Khosla, Àgata Lapedriza, Aude Oliva, and Antonio Torralba. 2015\.  Learning Deep Features for Discriminative Localization.  _CoRR_ abs/1512.04150 (2015).  arXiv:1512.04150 http://arxiv.org/abs/1512.04150