Lennart Alexander Van der Gotenlavdg@kth.se1,2 Tobias
Hepptobias.hepp@tuebingen.mpg.de3,4 Zeynep Akatazeynep.akata@uni-
tuebingen.de3,4 Kevin Smithksmith@kth.se1,2 KTH Royal Institute of Technology
Stockholm, SWEDEN Science for Life Laboratory
Solna, SWEDEN Max Planck Institute for Intelligent Systems
Tübingen, GERMANY University of Tübingen
Tübingen, GERMANY CONDITIONAL DE-IDENTIFICATION OF MRI SCANS

# Conditional De-Identification of 3D Magnetic Resonance Images

###### Abstract

Privacy protection of medical image data is challenging. Even if metadata is
removed, brain scans are vulnerable to attacks that match renderings of the
face to facial image databases. Solutions have been developed to de-identify
diagnostic scans by obfuscating or removing parts of the face. However, these
solutions either fail to reliably hide the patient’s identity or are so
aggressive that they impair further analyses. We propose a new class of de-
identification techniques that, instead of removing facial features, remodels
them. Our solution relies on a conditional multi-scale GAN architecture. It
takes a patient’s MRI scan as input and generates a 3D volume conditioned on
the patient’s brain, which is preserved exactly, but where the face has been
de-identified through remodeling. We demonstrate that our approach preserves
privacy far better than existing techniques, without compromising downstream
medical analyses. Analyses were run on the OASIS-3 and ADNI corpora.

##  1 Introduction

The digitalization of heath records has increased the risk of –and impact of–
large scale data leaks. Although data compliance standards have been enacted
to protect health records (HIPAA and GDPR), privacy of medical data is a
growing concern. Three-dimensional scans such as magnetic resonance images
(MRI) and computed tomography (CT), for example, contain an intrinsic privacy
risk [Lotan et al.(2020)Lotan, Tschider, Sodickson, Caplan, Bruno, Zhang, and
Lui]. Detailed renderings of the head can be crafted from 3D scans using
techniques such as volumetric raycasting, as in Figure 1. This vulnerability
can expose the patient’s identity if the renderings are matched to a face
database [Mazura et al.(2012)Mazura, Juluru, Chen, Morgan, John, and Siegel,
Lotan et al.(2020)Lotan, Tschider, Sodickson, Caplan, Bruno, Zhang, and Lui].

To prevent these types of attack, medical scans are currently de-identified
using crude removal-based techniques [Bischoff-Grethe et al.(2007)Bischoff-
Grethe, Ozyurt, Busa, Quinn, Fennema-Notestine, Clark, Morris, Bondi,
Jernigan, Dale, Brown, and Fischl, Schimke et al.(2011)Schimke, Kuehler, and
Hale, Milchenko and Marcus(2013)] which seek to remove privacy-sensitive parts
of the head (examples in Figure 3). However, as we demonstrate, these existing
techniques fail to reliably hide the patient’s identity – or they are so
aggressive that they impair further medical analyses. A better solution is
needed.

One might ask _why de-identify the face when one can just remove everything
except the brain?_ This approach, known as skull-stripping [Ségonne et
al.(2004)Ségonne, Dale, Busa, Glessner, Salat, Hahn, and Fischl], does provide
excellent privacy guarantees, but unfortunately renders the scan useless for
many types of clinical analysis. Automated tools for analyzing MRI scans rely
on landmarks within the head, and fail when they are removed [De Sitter et
al.(2020)De Sitter, Visser, Brouwer, Cover, van Schijndel, Eijgelaar, Müller,
Ropele, Kappos, Rovira, Filippi, Enzinger, Frederiksen, Ciccarelli, Guttmann,
Wattjes, Witte, de Witt Hamer, Barkhof, and Vrenken]. Furthermore, skull-
stripping corrupts measurements of important tissues and fluids, such as
extra-cranial CSF [Bischoff-Grethe et al.(2007)Bischoff-Grethe, Ozyurt, Busa,
Quinn, Fennema-Notestine, Clark, Morris, Bondi, Jernigan, Dale, Brown, and
Fischl]. For these reasons, _remodeling the head_ rather than deleting
privacy-sensitive regions is desirable, because it protects privacy and at the
same time ensures robustness of downstream medical analyses.

---
Figure 1: _De-identification through facial remodeling._ Face renderings from
medical scans such as these MRIs represent a privacy vulnerability. To address
this, we define a new class of de-identification techniques that aims to
realistically _remodel_ privacy-sensitive regions, such as the face, while
_preserving_ essential regions, such as the brain (illustrated OASIS-3).

Therefore, in this work, we define a new class of de-identification techniques
that _remodels_ the privacy-sensitive regions without altering the content of
medically relevant data (see Figure 1). Under such a remodeling approach, the
face, eyes, oral and nasal cavities, _etc._ should exhibit realistic
appearance and structure of appropriate size, but should otherwise be
independent of the original data. To solve this task, we propose a novel model
called _Convex Privacy_ GAN, or CP-GAN, that conditions on a convex hull of
the skull extracted from the scan to be de-identified. The generator learns to
synthesize volumes that preserve medically-sensitive regions such as the
brain, while non-invertibly _remodeling_ privacy-sensitive characteristics
from the original scan.

The main contributions of this work are as follows: (1) We define a novel
methodology to ensure privacy in medical imagery in which medically relevant
regions are preserved and privacy-sensitive regions are de-identified. (2) We
propose CP-GAN, a conditional multi-scale volumetric GAN that realizes a
solution to the aforementioned methodology. (3) Through human- and model-based
experiments, we show that CP-GAN preserves privacy in MRI scans more reliably
than removal-based techniques without adversely affecting downstream analyses.
In addition, we make technical contributions towards the generation of the
convex hull and surface representations necessary for the privacy conditioning
of the GAN. Source code as well as a video demonstration can be found in the
_supplementary material_.

##  2 Related Work

A handful of de-identification techniques exist for MRI scans, which are
conventionally used for sharing and distribution of MRI data. These existing
methods rely on a _removal_ approach to privacy. DEFACE [Bischoff-Grethe et
al.(2007)Bischoff-Grethe, Ozyurt, Busa, Quinn, Fennema-Notestine, Clark,
Morris, Bondi, Jernigan, Dale, Brown, and Fischl] estimates the probabilities
of voxels belonging to the face based on an atlas of healthy control subjects.
The scan is de-identified by setting intensities of voxels whose probabilities
are small enough to zero. QUICKSHEAR [Schimke et al.(2011)Schimke, Kuehler,
and Hale] is a fast but simple approach that computes a hyperplane to divide
the MRI into two regions: one containing facial structures, and the other
containing the brain of the scan. Voxels in the first part are set to zero.
FACE MASK [Milchenko and Marcus(2013)] uses a filtering method to blur the
facial features. These existing de-identification approaches are based on
traditional computer vision techniques; we believe that the proposed algorithm
is the first to adopt a learning-based approach.

While not a de-identification method, _Shin_ _et al_ [Shin et al.(2018)Shin,
Tenenholtz, Rogers, Schwarz, Senjem, Gunter, Andriole, and Michalski] recently
proposed a _pix2pix_ -inspired model [Isola et al.(2016)Isola, Zhu, Zhou, and
Efros] to generate synthetic abnormal MRI images with brain tumors. In this
work, the authors argue that, in principle, their approach can be used to
generate a completely artificial corpus where none of the scans can be
attributed to actual patients. However, as the brain data is hallucinated,
this method is not useful for our task.

The literature covering removal of privacy-sensitive information from image
data largely focuses on de-identification of photographs of faces [Jourabloo
et al.(2015)Jourabloo, Yin, and Liu, Newton et al.(2005)Newton, Sweeney, and
Malin]. Among these, _Deep Privacy_ [Hukkelås et al.(2019)Hukkelås, Mester,
and Lindseth] is the closest to our approach as it was the first to suggest
GANs to de-identify faces. It conditions on an _a priori_ binary segmentation,
guiding the generator to inpaint privacy-sensitive regions while preserving
insensitive regions. Similar to our approach, _Deep Privacy_ seeks to
anonymize faces – but in 2D images. To identify face regions for conditional
inpainting, it relies on an SSD detector [Liu et al.(2015)Liu, Anguelov,
Erhan, Szegedy, Reed, Fu, and Berg]. We develop an alternative approach
because a 3D analogue does not exist, and this allows us to comprehensively
remodel interior regions of the head such as the neck and oral/nasal cavities.
In particular, we define a convex hull enclosing the head and mask of the
brain for conditioning. Finally, whereas _Deep Privacy_ de-identifies
conventional images of size $128{\times}128$, our goal is to generate much
higher dimensional 3D volumes at $128^{3}$ voxels – the equivalent of a
$1448{\times}1448$ image.

##  3 Conditional De-Identification of 3D Images

Given a set of 3D images $(X^{(i)})_{i=1,\ldots
N}\overset{\text{i.i.d.}}{\sim}\mathcal{P}_{X}$ with values in
$\mathcal{I}^{S\times S\times S}$ over some intensity space
$\mathcal{I}\subset\mathbb{R}$, we are interested in finding a function of the
form $Y{=}f_{\Phi}(\gamma(X)){\sim}\mathcal{P}_{Y}$ that maps a 3D image $X$
to its de-identified counterpart $Y$. The task of the function $\gamma(X)$ is
to filter out any sensitive information in order to make it impossible to
infer the subject’s identity given only $Y$; _i.e_ to create a _privacy
preserving representation_. In this work we consider MRI data111MRI scans can
be acquired under different conditions, data availability limits us to common
T1-weighted MRI., and choose $\gamma(X)$ to be a function of the convex hull
of the head $c(X){\in}\\{0,1\\}^{S\times S\times S}$ and the brain mask
$b(X){\in}\\{0,1\\}^{S\times S\times S}$.

Within this _remodeling_ -based privacy framework, we impose three
requirements: (i) _Anonymity_ , _i.e_ $\gamma(X)$ is non-invertible; (ii)
_Distribution preservation_ , _i.e_ $\mathcal{P}_{X}$ and $\mathcal{P}_{Y}$
are stochastically indistinguishable and finally (iii) _Brain preservation_ ,
_i.e_ $\forall(i,j,k):b(X)_{i,j,k}=1\implies
X_{i,j,k}=f_{\Phi}(\gamma(X))_{i,j,k}$. In other words, we are interested in
deriving a function $f_{\Phi}$ that maps some original scan $X$ to some de-
identified scan $Y$, while retaining medically relevant information (_e.g_ the
brain) but preventing other information specific to $X$ to leak into $Y$
(_e.g_ the face). This makes it impossible to infer a person’s identity from
facial renderings. Figure 2 depicts the de-identification process, described
below, including the privacy transform $\gamma(X)$ and the mapping function
$f_{\Phi}$ implemented with a conditional multi-scale volumetric GAN.

###  3.1 The Privacy Transform $\gamma(X)$

The goal of the privacy transform is to non-invertibly change an individual
MRI representation $x$ into a form $\gamma(x)$ that removes detailed privacy-
sensitive information and replaces it with a convex hull filled with 1’s,
smoothing away detailed face information (_e.g_ eyes, nose, and mouth). The
transform guides the GAN, showing which regions should be hallucinated via a
convex hull $c(x)$ and which regions should be retained through a brain mask
$b(x)$. It also includes the brain data $b(x)\circ x$. The convex hull can
afterwards be used to suppress noise patterns surrounding the head. Following
the preprocessing (see _Appendix_), we define a function $c(x)$ that maps a
scan $x\in\mathcal{I}^{S\times S\times S}$ to a binary convex hull volume of
the same shape. As no efficient off-the-shelf algorithm exists, we developed a
probabilistic solution that first constructs a surface representation from the
MRI scan, and from this we compute the convex hull of the head. These steps
are described below.

Figure 2: _Overview of our approach_ : We apply standard preprocessing to
register and correct the scans (illustrated OASIS-3). Then, using a novel
technique, we construct a _head surface_ representation from which we extract
a convex hull $c(x)$, a brain mask $b(x)$, and the brain intensities
$b(x)\circ x$. Combined, these form the privacy transform $\gamma(x)$ which
serves as a conditioning variable of our model. CP-GAN learns to convert the
distribution $\mathcal{P}_{X}$ of original MR scans to a de-identified
counterpart $\mathcal{P}_{Y}$. Note that $\gamma(x)$ does not contain any
privacy-sensitive information.

Surface Representation. To extract a surface representation $Z$ from an MRI
scan $x$, we compute maps where rays cast from each direction intersect the
head at random rotations. We then rotate these measurements back to the
reference coordinates and treat each as the probability of it belonging to the
surface. The rotations are randomized to sample the subject from all sides
uniformly. We begin by converting a given scan into a sequence of $K$
_binarized_ and _rotated_ scans, _i.e_
$m^{(i)}=\text{Rot}(\mathds{1}[x\geq\delta];R_{i})\in\mathcal{I}^{S\times
S\times S}$ for sampled rotations
$R_{1},\ldots,R_{K}\overset{\text{i.i.d}}{\sim}\mathcal{U}(\text{SO}(3))$,
where $\mathcal{U}(\text{SO}(3))$ denotes the uniform distribution over all
rotations in three-dimensional space and $\delta\in\mathcal{I}$ represents a
suitably chosen binarization threshold222The threshold $\delta$ is chosen to
be larger than the noise values surrounding the skull. for the binarization
operator $\mathds{1}[\cdot]$. Let us further introduce the concept of the
$\zeta_{a,d}$-distance of some voxel at position $(k_{0},k_{1},k_{2})$ for
some axis $a\in\\{0,1,2\\}$ and some direction $d\in\\{-1,+1\\}$:

$\displaystyle\zeta_{a,d}(k_{0},k_{1},k_{2})\hat{=}\begin{cases}(S-1)-k_{a}&\text{if
}d=+1\\\ k_{a}&\,\text{otherwise.}\end{cases}$ (1)

For fixed $a$ and $d$, we can use this to create an _intersection map_
$\Lambda_{a,d}^{i}$ for each binary image $m_{S}^{i}$:

$\displaystyle\begin{aligned}
\Lambda_{a,d}^{(i)}[k_{0},k_{1},k_{2}]=\mathds{1}\bigg{[}\bigg{(}m_{k_{0},k_{1},k_{2}}^{(i)}=1\bigg{)}\wedge\bigg{(}\zeta_{a,d}(k_{0},k_{1},k_{2})=\mspace{-30.0mu}\min_{\begin{subarray}{c}s\in\\{0,\ldots,S-1\\},\\\
m_{k_{a-1}\mid s\mid
k_{a+1}}^{(i)}=1\end{subarray}}\mspace{-30.0mu}\zeta_{a,d}(k_{a-1}\mid s\mid
k_{a+1})\bigg{)}\bigg{]}&&\end{aligned}$ (2)

where $(k_{a-1}\mid s\mid k_{a+1})$ indicates that the $a$-th index is set to
$s$ and the two others to their associated value in $k_{0},k_{1},k_{2}$. We
average the intersection map over all axis-direction combinations, _i.e_
$\Lambda^{(i)}=\nicefrac{{1}}{{6}}\sum_{a}\sum_{d}\Lambda_{a,d}^{i}$. This
process can be thought of as casting rays from each principle direction and
recording the location of the intersection with the rotated, binarized head in
$m^{(i)}$. Voxels on the surface of the head will exhibit high values of
$\Lambda^{(i)}$. The final step is to back-rotate
$\Lambda^{(1)},\ldots,\Lambda^{(K)}$ to the reference coordinate system and
average among the $K$ randomly sampled rotations to create the _surface_
representation:

$Z=\nicefrac{{1}}{{K}}\sum_{i=1}^{K}Rot(\Lambda^{(i)};R_{i}^{-1})\in[0,1]^{S\times
S\times S}$ (3)

Note that $Z$ is a random variable induced by the sampled rotations
$R_{1},\ldots,R_{K}$. We interpret individual voxel values of $Z$ as Bernoulli
parameters characterizing the probability of some voxel belonging to the
surface. This justifies binarizing $Z$ by considering it as a three-
dimensional Bernoulli tensor and sampling from it on a voxel-wise basis in the
next step.

Convex Hull. From $Z$, we sample a set of non-zero indices and use Chan’s
Algorithm [Chan(1996)] to compute the triangles $\mathcal{T}$ making up the
convex hull. We initialize a uniform volume filled with 1’s, then randomly
select a sufficient number of triangles ($100$ suffice) from $\mathcal{T}$.
For each triangle, we find its corresponding hyperplane and the half-spaces
within $c(x)$ defined by it. Voxels in the outward half-space of $c(x)$ are
set to 0 while the rest are unchanged, yielding a binary convex hull volume.

Privacy Transform. The binary convex hull volume $c(x)$ instructs the GAN as
to which regions should be hallucinated. A binary brain mask $b(x)$ obtained
by applying [Iglesias et al.(2011)Iglesias, Liu, Thompson, and Tu] indicates
which regions should be preserved. Together, these volumes along with the
masked _continuous_ values of the brain $b(x)\circ x$, are concatenated to
make the privacy transform $\gamma(x)$. The GAN is conditioned on $\gamma(x)$
in the following subsection, as depicted in Figure 2.

###  3.2 Conditional De-identification GAN

The CP-GAN architecture depicted in Figure 2 is capable of generating volumes
at multiple scales and passing gradients between each scale during training.
We start from a 2D generation framework akin to MSG-GAN [Karnewar and
Wang(2019)] and adapt it to our task by means of the following: _(1)_ we
incorporate _conditional_ information via the privacy transform, _(2)_ we make
architectural improvements described below, _(3)_ we use a new resampling
strategy, _(4)_ we adopt relativistic (non-averaging) R-LSGAN loss, and _(5)_
we operate on 3D volumes. We use _bottlenecks_ between scales as recently
suggested by [Karras et al.(2019)Karras, Laine, Aittala, Hellsten, Lehtinen,
and Aila], in which the generator outputs single-channel maps instead of
multi-channel maps. To reduce the memory footprint, we use modified
MobileNetV2 convolutions as suggested in [Howard et al.(2017)Howard, Zhu,
Chen, Kalenichenko, Wang, Weyand, Andreetto, and Adam].

Both the generator $G_{\Phi}(\gamma(x))$ and the discriminator
$D_{\Theta}(\gamma(x),v)$ are conditioned on $\gamma(x)$, where $v$ either
denotes a multi-resolutional original or fake sample. Regarding scales –
suppose that $S$ and $s$ are powers of two that denote the maximum/minimum
resolution synthesized by $G_{\Phi}$. Then both $G_{\Phi}$ and $D_{\Theta}$
are defined to have $N_{B}=\log_{2}(\nicefrac{{S}}{{s}})+1$ blocks (indexed by
$k$) that either double ($G_{\Phi}$) or halve ($D_{\Theta}$) their input
resolution. Here, we generate scales from $4{\times}4{\times}4$ to
$128{\times}128{\times}128$.

Generator. The generator $G_{\Phi}=G_{\Phi}^{(N_{B})}\circ\ldots\circ
G_{\Phi}^{(1)}$ for
$G_{\Phi}^{(k)}:\mathbb{R}^{r_{G}(k-1)}\times\mathbb{R}^{r_{G}(k)}\rightarrow\mathbb{R}^{r_{G}(k)}$
and $r_{G}(k)=1\times 2^{k-1}s\times 2^{k-1}s\times 2^{k-1}s$ synthesizes a
sequence of fake images $g_{1},\ldots,g_{N_{B}}$ of increasing resolutions as
$g_{k}=G_{\Phi}^{(k)}(g_{k-1},\gamma_{k})\ \text{for}\ k=1,\ldots,N_{B}$ where
$g_{0}\sim\mathcal{N}(0,1)$ and
$\gamma_{k}=\downdownarrows_{r_{G}(k)}\gamma(x)$ is $\gamma(x)$ downsampled to
a resolution of $r_{G}(k)$.

Discriminator. The discriminator $D_{\Theta}=F\circ
D_{\Theta}^{(N_{B})}\circ\ldots\circ D_{\Theta}^{(1)}$ for
$D_{\Theta}^{(1)}:\mathbb{R}^{r_{D}(1)}\times\mathbb{R}^{r_{D}(1)}\rightarrow\mathbb{R}^{r_{D}(1)}$
resp.
$D_{\Theta}^{(k)}:\mathbb{R}^{r_{D}(k-1)}\times\mathbb{R}^{r_{D}(k)}\times\mathbb{R}^{r_{D}(k)}\rightarrow\mathbb{R}^{r_{D}(k)}(k>1)$
and
$r_{D}(k)=1\times\nicefrac{{S}}{{2^{k-1}}}\times\nicefrac{{S}}{{2^{k-1}}}\times\nicefrac{{S}}{{2^{k-1}}}$
assigns a _scalar_ to a sequence of images333$x_{1},\ldots,x_{N_{B}}$ in case
of an original image and $g_{1},\ldots,g_{N_{B}}$ in case of a fake image of
decreasing resolutions $v_{1},\ldots,v_{N_{B}}$ as
$d_{1}=D_{\Theta}(v_{1},\gamma_{1})$ resp.
$d_{k}=D_{\Theta}(d_{k-1},v_{k},\gamma_{k})$ for $k=2,\ldots,N_{B}$ where
$\gamma_{k}=\downdownarrows_{r_{D}(k)}\gamma(x)$ is $\gamma(x)$ downsampled to
a resolution of $r_{D}(k)$ and $F$ is a fully-connected layer that computes a
scalar summary of the output of $D_{\Theta}^{(N_{B})}$.

Resampling blocks. [Karras et al.(2018)Karras, Laine, and Aila, Karras et
al.(2019)Karras, Laine, Aittala, Hellsten, Lehtinen, and Aila] recently
proposed to use bilinear interpolation for downsampling, but adapting this
approach is problematic as it will create undesirable interpolation effects in
the binary volumes. Therefore, we suggest a _probabilistic_ interpretation of
_average pooling_ which guarantees that the proportion of non-zero voxels is
preserved (in expectation) while maintaining voxel-wise correspondence to
conventional average pooling performed on non-binary images. Specifics can be
found in the _Appendix_.

Loss Function. We use the relativistic (_non-averaging_) _R-LSGAN_ loss
[Jolicoeur-Martineau(2018)]: We opt for relativistic losses as they induce a
lower memory footprint than, for instance, the widely-established WGAN-GP
[Gulrajani et al.(2017)Gulrajani, Ahmed, Arjovsky, Dumoulin, and Courville]
requiring an additional forward/backward pass.

Brain Preservation. One of the requirements defined above in the Problem
Definition is to perfectly preserve medically relevant information. Therefore,
in a similar process to image inpainting in which original image content is
masked and retained, we use the brain mask $b(x)$ to embed the original brain
data into the volume synthesized by the generator, _i.e_
$f_{\Phi}(\gamma(x))=b(x)\circ x+(1-b(x))\circ G_{\Phi}(\gamma(x))$ where
$\circ$ denotes the _Hadamard_ product.

Original | Renderings after de-identification | Original | MRI slices after de-identification
---|---|---|---
|  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |
| CP-GAN | FACE MASK | DEFACE | QUICKSHEAR |  | CP-GAN | FACE MASK | DEFACE | QUICKSHEAR

Figure 3: _De-identification examples_ : De-identified face renderings
_(left)_ and the corresponding MRI scans _(right)_ for several examples from
OASIS-3. All methods perfectly preserve the brain, but only CP-GAN
successfully de-identifies the patient while maintaining realistic appearance
and structure.

##  4 Experiments

Above, we proposed a new and modern approach to de-identify medical image
data. To judge its utility, we must address the following questions: _(1) Does
remodeling preserve privacy better than existing removal-based de-
identification methods? (2) Does our approach adversely affect the performance
of common medical applications?_ Below, we compare our approach to other de-
identification methods to answer these questions experimentally.

###  4.1 Setup

Datasets. In this work, we use two standard, publicly available large-scale
Alzheimer’s disease imaging studies which feature T1-weighted volumetric MR
scans of the head for each subject: A selection of 2,172 MRIs from ADNI
[Weiner et al.(2017)Weiner, Veitch, Aisen, Beckett, Cairns, Green, Harvey,
Jack, Jagust, Morris, Petersen, Salazar, Saykin, Shaw, Toga, and Trojanowski,
Wyman et al.(2013)Wyman, Harvey, Crawford, Bernstein, Carmichael, Cole, Crane,
Decarli, Fox, Gunter, Hill, Killiany, Pachai, Schwarz, Schuff, Senjem, Suhy,
Thompson, Weiner, and Jack] and 2,168 MRIs from OASIS-3 [LaMontagne et
al.(2019)LaMontagne, Benzinger, Morris, Keefe, Hornbeck, Xiong, Grant,
Hassenstab, Moulder, Vlassenko, Raichle, Cruchaga, and Marcus]. Both datasets
are split ($80\%$-$20\%$ train-test) on a patient level to avoid data leakage
by memorizing the patient. The ADNI data used in the preparation of this
article were obtained from the Alzheimer’s Disease Neuroimaging Initiative
database (adni.loni.usc.edu). ADNI was launched in 2003 as a public-private
partnership, led by Principal Investigator Michael W. Weiner, MD. The primary
goal of ADNI has been to test whether serial magnetic resonance imaging (MRI),
positron emission tomography (PET), other biological markers, and clinical and
neuropsychological assessment can be combined to measure the progression of
mild cognitive impairment (MCI) and early Alzheimer’s disease (AD). For up-to-
date information, see www.adni-info.org. Scanner types and acquisition
protocols differ between and within the datasets, details can be found in the
_Appendix_.

Benchmark De-Identification Methods. We compare our result with three publicly
available and widely-established methods for de-identification of MRI head
scans, depicted in Figure 3. All methods have in common that they _(1)_ are
not deep-learning-driven, _(2)_ require no additional training and _(3)_ , are
used on a day-to-day basis in neuroscience and clinical research. All
procedures were applied with default settings on images of resolution
$128{\times}128{\times}128$. The methods include QUICKSHEAR [Schimke et
al.(2011)Schimke, Kuehler, and Hale], FACE MASK [Milchenko and Marcus(2013)],
and DEFACE [Bischoff-Grethe et al.(2007)Bischoff-Grethe, Ozyurt, Busa, Quinn,
Fennema-Notestine, Clark, Morris, Bondi, Jernigan, Dale, Brown, and Fischl].
Descriptions of the methods are provided in the _Appendix_. We also include
MRI WATERSHED [Ségonne et al.(2004)Ségonne, Dale, Busa, Glessner, Salat, Hahn,
and Fischl], a skull-stripping method that removes everything except the
brain.

Training. We use the AdamP [Heo et al.(2020)Heo, Chun, Oh, Han, Yun, Uh, and
Ha] optimizer with a learning rate of $2\cdot 10^{-3}$ and $\beta=(0,0.99)$
and a batch size of 2. See the _Appendix_ for a complete list of
hyperparameters.

| USER-BASED | MODEL-BASED
---|---|---
| OASIS-3 | ADNI | OASIS-3 | ADNI
ORIGINAL | $64.3$ | $\pm 2.1$ | $63.2$ | $\pm 2.1$ | $100.0$ | $\pm 0.0$ | $100.0$ | $\pm 0.0$
BLURRED | $57.2$ | $\pm 2.1$ | $53.1$ | $\pm 2.2$ | $46.7$ | $\pm 8.2$ | $51.2$ | $\pm 7.8$
FACE MASK | $48.7$ | $\pm 2.3$ | $47.5$ | $\pm 2.2$ | $68.5$ | $\pm 7.6$ | $63.3$ | $\pm 8.2$
DEFACE | $54.0$ | $\pm 2.3$ | $46.7$ | $\pm 2.3$ | $70.7$ | $\pm 6.4$ | $73.1$ | $\pm 7.0$
QUICKSHEAR | $45.6$ | $\pm 2.3$ | $39.9$ | $\pm 2.2$ | $44.0$ | $\pm 7.4$ | $47.0$ | $\pm 6.7$
CP-GAN | $\mathbf{28.3}$ | $\mathbf{\pm 2.0}$ | $\mathbf{22.4}$ | $\mathbf{\pm 1.8}$ | $\mathbf{29.5}$ | $\mathbf{\pm 5.9}$ | $\mathbf{27.4}$ | $\mathbf{\pm 5.7}$
BLACK | $18.9$ | $\pm 1.6$ | $20.5$ | $\pm 1.8$ | $19.7$ | $\pm 4.7$ | $19.8$ | $\pm 4.4$
MRI WATERSHED | $20.7$ | $\pm 2.0$ | $18.7$ | $\pm 1.6$ | $21.0$ | $\pm 4.8$ | $21.6$ | $\pm 4.4$

OASIS-3

Relative rank $\alpha$

ADNI

Relative rank $\alpha$

$\mathrm{Pr}[\mathcal{C}\in\mathrm{Top}(\alpha)]\ \
{\scriptstyle(\hat{=}\mathrm{Pr}[\alpha_{\mathcal{C}}\leq\alpha])}$

Figure 4: (_left_) A user-based study on AMT and a model-based study with
_Siamese_ networks to determine how well de-identified faces can be
identified. Correct identification rates ($\pm$s.d.) are reported for two
datasets (lower is better); (_right_) A Siamese network takes a face rendering
as an input, and aims to identify the correct de-identified rendering in a
retrieval scenario. We measure the probability of the correct face appearing
in its top $\alpha$-percentile ranked predictions. BLACK = _optimal_ ,
ORIGINAL = _pessimal_.

###  4.2 Results

In this section, we present results on (1) studies comparing the
identification rate of our model with existing de-identification methods, and
(2) the effects of de-identification on common medical image analysis tasks.
In the _Appendix_ , we provide a comparison of execution times. Video results
are provided in _supplementary material._

De-identification quality user study. The privacy attack described in [Mazura
et al.(2012)Mazura, Juluru, Chen, Morgan, John, and Siegel] relied on
prospectively collected data, meaning the authors had access to CT scans as
well as photographs of patient faces. Replicating that study for MRI scans is
impossible, because photographs of ADNI and OASIS-3 patients do not exist.
Therefore, we conduct a similarly-spirited study using _Amazon Mechanical
Turk_ in which workers are asked to defeat the various de-identification
methods given renderings of MRI scans. Workers were presented with an
unaltered rendering of a query patient along with five renderings de-
identified using a single method444An exemplary question can be found in the
_Appendix_ – one of which is a de-identified rendering of the query patient.
The task was then to pick out the de-identified rendering which corresponds to
the unaltered query rendering. We considered the following de-identification
methods: QUICKSHEAR [Schimke et al.(2011)Schimke, Kuehler, and Hale], FACE
MASK [Milchenko and Marcus(2013)], and DEFACE [Bischoff-Grethe et
al.(2007)Bischoff-Grethe, Ozyurt, Busa, Quinn, Fennema-Notestine, Clark,
Morris, Bondi, Jernigan, Dale, Brown, and Fischl], and CP-GAN (ours). In
addition to the four de-identification methods, we added four control tasks,
ORIGINAL, which signifies the absence of any de-identification scheme,
BLURRED, in which the 2D renderings are blurred to mildly obscure the patient
identity, BLACK, which features the same all-black image for each option, and
MRI WATERSHED [Ségonne et al.(2004)Ségonne, Dale, Busa, Glessner, Salat, Hahn,
and Fischl] which completely removes all tissue except the brain. We asked 800
distinct questions per dataset. Each question was given to five workers, for a
total of 4,000 assignments. The mean and the standard deviation are estimated
by bootstrapping over 1,000 resamples.

In Figure 4 (_left_), we report the identification rate, or how often the
workers were able to defeat each method, see _Appendix_ for details. The upper
performance bound from random guessing corresponds to 20%. The results
substantiate the claim that CP-GAN performs extraordinarily well at de-
identification. Our model outperforms the other de-identification methods by
gaps of 17%–25% on both datasets. We note that for both datasets, CP-GAN
performs close to the theoretical optimum of 20%.

De-identification quality model-based study. In a similar fashion to the last
experiment, we assess the de-identification performance of the various models
by attempting to defeat them. This time, however, we leverage a neural network
to assess similarity.

To this end, we use a metric learning approach to train a Siamese network
$\mathcal{S}(\cdot,\cdot)$ to quantify whether its two input renderings belong
to the same patient or not.

| _Sørensen-Dice coefficient $\uparrow$_ |  _Intersection-over-Union_ (IoU) $\uparrow$
---|---|---
| OASIS-3 | ADNI | OASIS-3 | ADNI
| BRAIN | VCSF | WHITE | GREY | BRAIN | VCSF | WHITE | GREY | BRAIN | VCSF | WHITE | GREY | BRAIN | VCSF | WHITE | GREY
ORIGINAL | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000
[2pt/1pt] FACE MASK | 0.991 | 0.984 | 0.989 | 0.996 | 0.986 | 0.977 | 0.976 | 0.987 | 0.982 | 0.968 | 0.978 | 0.992 | 0.973 | 0.955 | 0.953 | 0.975
DEFACE | 0.993 | 0.986 | 0.986 | 0.995 | 0.982 | 0.966 | 0.965 | 0.981 | 0.985 | 0.972 | 0.973 | 0.990 | 0.965 | 0.934 | 0.932 | 0.963
QUICKSHEAR | 0.994 | 0.989 | 0.990 | 0.997 | 0.986 | 0.975 | 0.972 | 0.985 | 0.987 | 0.978 | 0.980 | 0.994 | 0.972 | 0.952 | 0.946 | 0.971
CP-GAN | 0.995 | 0.991 | 0.992 | 0.998 | 0.989 | 0.979 | 0.978 | 0.989 | 0.989 | 0.981 | 0.983 | 0.996 | 0.977 | 0.960 | 0.957 | 0.978
[2pt/1pt] MRI WATERSHED | 0.675 | 0.415 | 0.564 | 0.718 | 0.717 | 0.570 | 0.589 | 0.732 | 0.509 | 0.262 | 0.393 | 0.560 | 0.559 | 0.399 | 0.417 | 0.578

Table 1: _Brain segmentation:_ We measure the Sørensen-Dice coefficient and
the IoU computed between segmentations on the original scan and de-identified
scans using standard software, SIENAX. We test on the whole brain, VCSF, white
matter, and grey matter. Ideally, segmentation should not be affected by de-
identification, indicated by a Dice score and IoU of 1. CP-GAN outperforms all
other de-identification methods. Note that MRI WATERSHED, which removes
everything but the brain, has a catastrophic effect.

Given two inputs $x$ and $y$, the network $\mathcal{S}$ is constructed by
applying a sub-network $\tilde{\mathcal{S}}$ (conv. block/flatten/fully-
connected layer) on $x$ and $y$ independently, followed by summarizing both
embeddings with the _Euclidean_ distance, _i.e_
$\mathcal{S}(x,y)=\lVert\tilde{\mathcal{S}}(x)-\tilde{\mathcal{S}}(y)\rVert_{2}$.
We use the _Triplet Margin_ loss function as described in [Vassileios Balntas
and Mikolajczyk(2016)], choosing the margin to be equal to $5$. We split the
previously defined (hold-out) data set, and _randomly_ select 80% of its
patients as a training set $\mathcal{D}_{i}$ and the remaining 20% as its
complement $\overline{\mathcal{D}_{i}}$, where $i=1,\ldots,100$ denotes the
$i$-th (resampled) fold. Specifics on the nature of the training can be found
in the _Appendix_.

In Figure 4 (_left_) we report the ability of the network to defeat the de-
identification methods in similar fashion to the _user-based_ study. We first
sample a patient $p$ from $\overline{\mathcal{D}}_{i}$ from whom, in turn, we
sample a scan $s\in S(p)$. Afterwards, we sample a method $m$ and and consider
the $m$-rendering of $s$ to be the _correct_ option $\mathcal{C}$. The
remaining $5-1=4$ options are obtained by randomly selecting $m$-renderings
from other patients’ scans. Denoting the five options by $x_{1},\ldots,x_{5}$,
we obtain the predicted option by
$k=\operatorname*{arg\,min}_{j=1,\ldots,5}\mathcal{S}(x_{O},x_{j})$ where
$x_{O}$ denotes the original rendering of $s$. As in the _user-based_ study,
CP-GAN outperforms the other de-identification methods. For FACE MASK and
DEFACE, the network was able to de-identify between 16 to 20% more renderings
than its human counterparts. The effect on QUICKSHEAR is more moderate.

In Figure 4 (_right_) we evaluate the Siamese network’s ability to defeat de-
identification methods in a retrieval-inspired setting. We observe that CP-
GAN’s de-identification capabilities are strikingly close to the _optimal_
case while other methods perform substantially worse. For some given original
rendering $x^{(\textrm{orig.})}$ and some method $m$, we analyze how often the
correct choice $\mathcal{C}$ falls within the top $\alpha$ (relative) ranks,
_e.g_ $\mathrm{Top}(\alpha{=}0.1)$ is a subset of the $10\%$ top-ranked scans.
An _optimal_ de-identification method induces a _uniformly random_ rank
$\alpha_{\mathcal{C}}\sim\mathcal{U}(0,1)$ of $\mathcal{C}$ (c.f. BLACK),
whereas a _pessimal_ method induces a _Dirac_ placement
$\alpha_{\mathcal{C}}=0$ (c.f. ORIGINAL). Confidence bands
($\textrm{CI}{=}0.95$) are calculated over the $100$ previously defined data
splits $\overline{\mathcal{D}}_{i}$.

Overall, we conclude that CP-GAN outperforms established methods by a
substantial double-digit margin, withstanding both human and model-based
attacks. We note a gap between CP-GAN and optimal performance, which can be
explained by a property of the model: it preserves head size. The attacker may
exploit this to eliminate some candidates. We explore this concept further in
the _Appendix_.

Effect of De-Identification on Medical Analyses. Beyond ensuring patient
privacy, de-identification methods should not adversely affect software tools
commonly used on medical scans. However, it has been shown that facial de-
identification methods _do_ adversely impact automated image analysis on MRI
scans used in research and in the clinic [De Sitter et al.(2020)De Sitter,
Visser, Brouwer, Cover, van Schijndel, Eijgelaar, Müller, Ropele, Kappos,
Rovira, Filippi, Enzinger, Frederiksen, Ciccarelli, Guttmann, Wattjes, Witte,
de Witt Hamer, Barkhof, and Vrenken]. In line with this study, we conduct two
experiments. In the first, we assess how the de facto standard brain tissue
segmentation tool, SIENAX [Smith et al.(2004)Smith, Jenkinson, Woolrich,
Beckmann, Behrens, Johansen-Berg, Bannister, De Luca, Drobnjak, Flitney,
Niazy, Saunders, Vickers, Zhang, De Stefano, Brady, and Matthews], performs on
de-identified MRI scans in comparison to the originals. In Table 1, we report
the (Sørensen-) Dice scores [Sørensen(1948), Dice(1945)] and IoU between the
original and de-identified scans for various brain segmentation tasks. We
observe that CP-GAN outperforms all of its contenders, proving that brain
volume estimations are reliable after the subject is de-identified using CP-
GAN. _Note that removing everything except the brain using_ MRI WATERSHED _has
a catastrophic effect_ , replicating the effect observed in [Fennema-Notestine
et al.(2006)Fennema-Notestine, Ozyurt, Clark, Morris, Bischoff-Grethe, Bondi,
Jernigan, Fischl, Segonne, Shattuck, Leahy, Rex, Toga, Zou, and Brown].

In the second experiment, we investigate whether de-identification adversely
affects brain age estimation – an important task as the difference between
predicted and chronological age has links to brain disease [Jónsson et
al.(2019)Jónsson, Bjornsdottir, Thorgeirsson, Ellingsen, Walters,
Gudbjartsson, Stefansson, Stefansson, and Ulfarsson]. This is a challenging
task for CP-GAN since age information captured in the MRI is filtered out in
$\gamma(x)$ in contrast to the other methods that preserve head information in
addition to the brain. Nonetheless, we find that our de-identification
introduces less bias than DEFACE and QUICKSHEAR on both datasets, though FACE
MASK slightly outperforms our model. Due to space limitations, these results
appear in the _Appendix_.

##  5 Conclusion

In this work, we defined a new paradigm for de-identification of medical
imagery and realized it for MRI scans. Our approach remodels privacy-relevant
information while keeping medically-relevant information untouched. _It can be
applied to other modalities, producing remodeled images that appear genuine
and preserve relevant medical information, but without revealing privacy-
sensitive information._ Our method protects privacy substantially better than
existing methods, without compromising analyses typically found in research
and clinical settings – a crucial deficiency of strong removal methods such as
skull-stripping. A future research direction is to extend our approach to
other MRI and CT modalities, adding new downstream tasks such as lesion and
brain tumor segmentation [Schmidt et al.(2012)Schmidt, Gaser, Arsic, Buck,
Förschler, Berthele, Hoshi, Ilg, Schmid, Zimmer, Hemmer, and Mühlau, Meier et
al.(2016)Meier, Knecht, Loosli, Bauer, Slotboom, Wiest, and Reyes].
Incorporating other pulse sequences, such as _T2-weighting_ or _FLAIR_ ,
requires a re-training of the network as well as a sufficient number of
training samples. Unfortunately, alternative pulse sequences are less readily
available in comparison to the _T1-weighted_ imagery used in this paper. Apart
from this apparent scarcity, we however do not expect any change in
complexity. We hope that the methods outlined here can help to better protect
patient privacy.

#### Acknowledgements

This work was partially supported by the Swedish Research Council (VR)
2017-04609, the ERC (853489 \- DEXIM) and by the DFG (2064/1 – Project number
390727645). Data collection and sharing for this project was funded by the
Alzheimer’s Disease Neuroimaging Initiative (ADNI) (National Institutes of
Health Grant U01 AG024904) and DOD ADNI (Department of Defense award number
W81XWH-12-2-0012). ADNI is funded by the National Institute on Aging, the
National Institute of Biomedical Imaging and Bioengineering, and through
generous contributions from the following: AbbVie, Alzheimer’s Association;
Alzheimer’s Drug Discovery Foundation; Araclon Biotech; BioClinica, Inc.;
Biogen; Bristol-Myers Squibb Company; CereSpir, Inc.; Cogstate; Eisai Inc.;
Elan Pharmaceuticals, Inc.; Eli Lilly and Company; EuroImmun; F. Hoffmann-La
Roche Ltd and its affiliated company Genentech, Inc.; Fujirebio; GE
Healthcare; IXICO Ltd.; Janssen Alzheimer Immunotherapy Research &
Development, LLC.; Johnson & Johnson Pharmaceutical Research & Development
LLC.; Lumosity; Lundbeck; Merck & Co., Inc.; Meso Scale Diagnostics, LLC.;
NeuroRx Research; Neurotrack Technologies; Novartis Pharmaceuticals
Corporation; Pfizer Inc.; Piramal Imaging; Servier; Takeda Pharmaceutical
Company; and Transition Therapeutics. The Canadian Institutes of Health
Research is providing funds to support ADNI clinical sites in Canada. Private
sector contributions are facilitated by the Foundation for the National
Institutes of Health (www.fnih.org). The grantee organization is the Northern
California Institute for Research and Education, and the study is coordinated
by the Alzheimer’s Therapeutic Research Institute at the University of
Southern California. ADNI data are disseminated by the Laboratory for Neuro
Imaging at the University of Southern California.

## References

  * [Bischoff-Grethe et al.(2007)Bischoff-Grethe, Ozyurt, Busa, Quinn, Fennema-Notestine, Clark, Morris, Bondi, Jernigan, Dale, Brown, and Fischl] Amanda Bischoff-Grethe, I. Burak Ozyurt, Evelina Busa, Brian T. Quinn, Christine Fennema-Notestine, Camellia P. Clark, Shaunna Morris, Mark W. Bondi, Terry L. Jernigan, Anders M. Dale, Gregory G. Brown, and Bruce Fischl.  A technique for the deidentification of structural brain MR images.  _Human Brain Mapping_ , 28(9):892–903, sep 2007\.  ISSN 10659471.  10.1002/hbm.20312.
  * [Chan(1996)] T. M. Chan.  Optimal output-sensitive convex hull algorithms in two and three dimensions.  _Discrete and Computational Geometry_ , 16(4):361–368, 1996.  ISSN 01795376.  10.1007/BF02712873.
  * [De Sitter et al.(2020)De Sitter, Visser, Brouwer, Cover, van Schijndel, Eijgelaar, Müller, Ropele, Kappos, Rovira, Filippi, Enzinger, Frederiksen, Ciccarelli, Guttmann, Wattjes, Witte, de Witt Hamer, Barkhof, and Vrenken] A. De Sitter, M. Visser, I. Brouwer, K. S. Cover, R. A. van Schijndel, R. S. Eijgelaar, D. M.J. Müller, S. Ropele, L. Kappos, Rovira, M. Filippi, C. Enzinger, J. Frederiksen, O. Ciccarelli, C. R.G. Guttmann, M. P. Wattjes, M. G. Witte, P. C. de Witt Hamer, F. Barkhof, and H. Vrenken.  Facing privacy in neuroimaging: removing facial features degrades performance of image analysis methods.  _European Radiology_ , 30(2):1062–1074, feb 2020\.  ISSN 14321084.  10.1007/s00330-019-06459-3.
  * [Dice(1945)] Lee R. Dice.  Measures of the Amount of Ecologic Association Between Species.  _Ecology_ , 26(3):297–302, jul 1945.  ISSN 00129658.  10.2307/1932409.  URL http://doi.wiley.com/10.2307/1932409.
  * [Fennema-Notestine et al.(2006)Fennema-Notestine, Ozyurt, Clark, Morris, Bischoff-Grethe, Bondi, Jernigan, Fischl, Segonne, Shattuck, Leahy, Rex, Toga, Zou, and Brown] Christine Fennema-Notestine, I. Burak Ozyurt, Camellia P. Clark, Shaunna Morris, Amanda Bischoff-Grethe, Mark W. Bondi, Terry L. Jernigan, Bruce Fischl, Florent Segonne, David W. Shattuck, Richard M. Leahy, David E. Rex, Arthur W. Toga, Kelly H. Zou, and Gregory G. Brown.  Quantitative evaluation of automated skull-stripping methods applied to contemporary and legacy images: Effects of diagnosis, bias correction, and slice location.  _Human Brain Mapping_ , 27(2):99–113, 2006.  https://doi.org/10.1002/hbm.20161.  URL https://onlinelibrary.wiley.com/doi/abs/10.1002/hbm.20161.
  * [Gulrajani et al.(2017)Gulrajani, Ahmed, Arjovsky, Dumoulin, and Courville] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville.  Improved training of wasserstein gans.  In _Advances in neural information processing systems_ , pages 5767–5777, 2017.
  * [Heo et al.(2020)Heo, Chun, Oh, Han, Yun, Uh, and Ha] Byeongho Heo, Sanghyuk Chun, Seong Joon Oh, Dongyoon Han, Sangdoo Yun, Youngjung Uh, and Jung-Woo Ha.  Slowing down the weight norm increase in momentum-based optimizers.  _arXiv preprint arXiv:2006.08217_ , 2020.
  * [Howard et al.(2017)Howard, Zhu, Chen, Kalenichenko, Wang, Weyand, Andreetto, and Adam] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  _CoRR_ , abs/1704.04861, 2017.  URL http://arxiv.org/abs/1704.04861.
  * [Hukkelås et al.(2019)Hukkelås, Mester, and Lindseth] Håkon Hukkelås, Rudolf Mester, and Frank Lindseth.  DeepPrivacy: A Generative Adversarial Network for Face Anonymization.  sep 2019.  URL http://arxiv.org/abs/1909.04538.
  * [Iglesias et al.(2011)Iglesias, Liu, Thompson, and Tu] Juan Eugenio Iglesias, Cheng Yi Liu, Paul M. Thompson, and Zhuowen Tu.  Robust brain extraction across datasets and comparison with publicly available methods.  _IEEE Transactions on Medical Imaging_ , 30(9):1617–1634, sep 2011.  ISSN 02780062.  10.1109/TMI.2011.2138152.
  * [Isola et al.(2016)Isola, Zhu, Zhou, and Efros] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros.  Image-to-Image Translation with Conditional Adversarial Networks.  _Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017_ , 2017-January:5967–5976, nov 2016.  URL http://arxiv.org/abs/1611.07004.
  * [Jolicoeur-Martineau(2018)] Alexia Jolicoeur-Martineau.  The relativistic discriminator: a key element missing from standard GAN.  _7th International Conference on Learning Representations, ICLR 2019_ , jul 2018.  URL http://arxiv.org/abs/1807.00734.
  * [Jónsson et al.(2019)Jónsson, Bjornsdottir, Thorgeirsson, Ellingsen, Walters, Gudbjartsson, Stefansson, Stefansson, and Ulfarsson] Benedikt Atli Jónsson, Gyda Bjornsdottir, TE Thorgeirsson, Lotta María Ellingsen, G Bragi Walters, DF Gudbjartsson, Hreinn Stefansson, Kari Stefansson, and MO Ulfarsson.  Brain age prediction using deep learning uncovers associated sequence variants.  _Nature communications_ , 10(1):1–10, 2019.
  * [Jourabloo et al.(2015)Jourabloo, Yin, and Liu] Amin Jourabloo, Xi Yin, and Xiaoming Liu.  Attribute preserved face de-identification.  In _2015 International conference on biometrics (ICB)_ , pages 278–285. IEEE, 2015.
  * [Karnewar and Wang(2019)] Animesh Karnewar and Oliver Wang.  MSG-GAN: Multi-Scale Gradient GAN for Stable Image Synthesis.  mar 2019.  URL http://arxiv.org/abs/1903.06048.
  * [Karras et al.(2018)Karras, Laine, and Aila] Tero Karras, Samuli Laine, and Timo Aila.  A Style-Based Generator Architecture for Generative Adversarial Networks.  _Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ , 2019-June:4396–4405, dec 2018.  URL http://arxiv.org/abs/1812.04948.
  * [Karras et al.(2019)Karras, Laine, Aittala, Hellsten, Lehtinen, and Aila] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and Improving the Image Quality of StyleGAN.  dec 2019.  URL http://arxiv.org/abs/1912.04958.
  * [LaMontagne et al.(2019)LaMontagne, Benzinger, Morris, Keefe, Hornbeck, Xiong, Grant, Hassenstab, Moulder, Vlassenko, Raichle, Cruchaga, and Marcus] Pamela J LaMontagne, Tammie L.S. Benzinger, John C. Morris, Sarah Keefe, Russ Hornbeck, Chengjie Xiong, Elizabeth Grant, Jason Hassenstab, Krista Moulder, Andrei Vlassenko, Marcus E. Raichle, Carlos Cruchaga, and Daniel Marcus.  OASIS-3: Longitudinal Neuroimaging, Clinical, and Cognitive Dataset for Normal Aging and Alzheimer Disease.  _medRxiv_ , page 2019.12.13.19014902, dec 2019.  10.1101/2019.12.13.19014902.
  * [Liu et al.(2015)Liu, Anguelov, Erhan, Szegedy, Reed, Fu, and Berg] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C. Berg.  SSD: Single Shot MultiBox Detector.  _Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)_ , 9905 LNCS:21–37, dec 2015.  10.1007/978-3-319-46448-0_2.  URL http://arxiv.org/abs/1512.02325http://dx.doi.org/10.1007/978-3-319-46448-0{_}2.
  * [Lotan et al.(2020)Lotan, Tschider, Sodickson, Caplan, Bruno, Zhang, and Lui] Eyal Lotan, Charlotte Tschider, Daniel K Sodickson, Arthur L Caplan, Mary Bruno, Ben Zhang, and Yvonne W Lui.  Medical imaging and privacy in the era of artificial intelligence: Myth, fallacy, and the future.  _Journal of the American College of Radiology_ , 17(9):1159–1162, 2020.
  * [Mazura et al.(2012)Mazura, Juluru, Chen, Morgan, John, and Siegel] Jan C Mazura, Krishna Juluru, Joseph J Chen, Tara A Morgan, Majnu John, and Eliot L Siegel.  Facial recognition software success rates for the identification of 3d surface reconstructed facial images: implications for patient privacy and security.  _Journal of digital imaging_ , 25(3):347–351, 2012.
  * [Meier et al.(2016)Meier, Knecht, Loosli, Bauer, Slotboom, Wiest, and Reyes] Raphael Meier, Urspeter Knecht, Tina Loosli, Stefan Bauer, Johannes Slotboom, Roland Wiest, and Mauricio Reyes.  Clinical evaluation of a fully-automatic segmentation method for longitudinal brain tumor volumetry.  _Scientific Reports_ , 6:23376, 03 2016.  10.1038/srep23376.
  * [Milchenko and Marcus(2013)] Mikhail Milchenko and Daniel Marcus.  Obscuring surface anatomy in volumetric imaging data.  _Neuroinformatics_ , 11(1):65–75, jan 2013.  ISSN 15392791.  10.1007/s12021-012-9160-3.  URL http://www.ncbi.nlm.nih.gov/pubmed/22968671http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=PMC3538950.
  * [Newton et al.(2005)Newton, Sweeney, and Malin] Elaine M Newton, Latanya Sweeney, and Bradley Malin.  Preserving privacy by de-identifying face images.  _IEEE transactions on Knowledge and Data Engineering_ , 17(2):232–243, 2005.
  * [Schimke et al.(2011)Schimke, Kuehler, and Hale] Nakeisha Schimke, Mary Kuehler, and John Hale.  Preserving privacy in structural neuroimages.  In _Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)_ , volume 6818 LNCS, pages 301–308. Springer, Berlin, Heidelberg, 2011.  ISBN 9783642223471.  10.1007/978-3-642-22348-8_26.
  * [Schmidt et al.(2012)Schmidt, Gaser, Arsic, Buck, Förschler, Berthele, Hoshi, Ilg, Schmid, Zimmer, Hemmer, and Mühlau] Paul Schmidt, Christian Gaser, Milan Arsic, Dorothea Buck, Annette Förschler, Achim Berthele, Muna Hoshi, Rüdiger Ilg, Volker J. Schmid, Claus Zimmer, Bernhard Hemmer, and Mark Mühlau.  An automated tool for detection of flair-hyperintense white-matter lesions in multiple sclerosis.  _NeuroImage_ , 59(4):3774 – 3783, 2012.  ISSN 1053-8119.  https://doi.org/10.1016/j.neuroimage.2011.11.032.  URL http://www.sciencedirect.com/science/article/pii/S1053811911013139.
  * [Ségonne et al.(2004)Ségonne, Dale, Busa, Glessner, Salat, Hahn, and Fischl] F. Ségonne, A. M. Dale, E. Busa, M. Glessner, D. Salat, H. K. Hahn, and B. Fischl.  A hybrid approach to the skull stripping problem in MRI.  _NeuroImage_ , 22(3):1060–1075, jul 2004.  ISSN 10538119.  10.1016/j.neuroimage.2004.03.032.
  * [Shin et al.(2018)Shin, Tenenholtz, Rogers, Schwarz, Senjem, Gunter, Andriole, and Michalski] Hoo-Chang Shin, Neil A Tenenholtz, Jameson K Rogers, Christopher G Schwarz, Matthew L Senjem, Jeffrey L Gunter, Katherine Andriole, and Mark Michalski.  Medical Image Synthesis for Data Augmentation and Anonymization using Generative Adversarial Networks.  _Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)_ , 11037 LNCS:1–11, jul 2018.  URL http://arxiv.org/abs/1807.10225.
  * [Smith et al.(2004)Smith, Jenkinson, Woolrich, Beckmann, Behrens, Johansen-Berg, Bannister, De Luca, Drobnjak, Flitney, Niazy, Saunders, Vickers, Zhang, De Stefano, Brady, and Matthews] Stephen M. Smith, Mark Jenkinson, Mark W. Woolrich, Christian F. Beckmann, Timothy E.J. Behrens, Heidi Johansen-Berg, Peter R. Bannister, Marilena De Luca, Ivana Drobnjak, David E. Flitney, Rami K. Niazy, James Saunders, John Vickers, Yongyue Zhang, Nicola De Stefano, J. Michael Brady, and Paul M. Matthews.  Advances in functional and structural MR image analysis and implementation as FSL.  In _NeuroImage_ , volume 23, pages S208–S219. Academic Press, jan 2004.  10.1016/j.neuroimage.2004.07.051.
  * [Sørensen(1948)] T.J. Sørensen.  _A Method of Establishing Groups of Equal Amplitude in Plant Sociology Based on Similarity of Species Content and Its Application to Analyses of the Vegetation on Danish Commons_.  Biologiske skrifter. I kommission hos E. Munksgaard, 1948.  URL https://books.google.de/books?id=rpS8GAAACAAJ.
  * [Vassileios Balntas and Mikolajczyk(2016)] Daniel Ponsa Vassileios Balntas, Edgar Riba and Krystian Mikolajczyk.  Learning local feature descriptors with triplets and shallow convolutional neural networks.  In Edwin R. Hancock Richard C. Wilson and William A. P. Smith, editors, _Proceedings of the British Machine Vision Conference (BMVC)_ , pages 119.1–119.11. BMVA Press, September 2016.  ISBN 1-901725-59-6.  10.5244/C.30.119.  URL https://dx.doi.org/10.5244/C.30.119.
  * [Weiner et al.(2017)Weiner, Veitch, Aisen, Beckett, Cairns, Green, Harvey, Jack, Jagust, Morris, Petersen, Salazar, Saykin, Shaw, Toga, and Trojanowski] Michael W. Weiner, Dallas P. Veitch, Paul S. Aisen, Laurel A. Beckett, Nigel J. Cairns, Robert C. Green, Danielle Harvey, Clifford R. Jack, William Jagust, John C. Morris, Ronald C. Petersen, Jennifer Salazar, Andrew J. Saykin, Leslie M. Shaw, Arthur W. Toga, and John Q. Trojanowski.  The Alzheimer’s Disease Neuroimaging Initiative 3: Continued innovation for clinical trial improvement, may 2017.  ISSN 15525279.  URL http://www.ncbi.nlm.nih.gov/pubmed/27931796http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=PMC5536850.
  * [Wyman et al.(2013)Wyman, Harvey, Crawford, Bernstein, Carmichael, Cole, Crane, Decarli, Fox, Gunter, Hill, Killiany, Pachai, Schwarz, Schuff, Senjem, Suhy, Thompson, Weiner, and Jack] Bradley T. Wyman, Danielle J. Harvey, Karen Crawford, Matt A. Bernstein, Owen Carmichael, Patricia E. Cole, Paul K. Crane, Charles Decarli, Nick C. Fox, Jeffrey L. Gunter, Derek Hill, Ronald J. Killiany, Chahin Pachai, Adam J. Schwarz, Norbert Schuff, Matthew L. Senjem, Joyce Suhy, Paul M. Thompson, Michael Weiner, and Clifford R. Jack.  Standardization of analysis sets for reporting results from ADNI MRI data, may 2013.  ISSN 15525279.


# When Performance is not Enough - A Multidisciplinary View on Clinical
Decision Support

Roland Roller1,2 Klemens Budde2 Aljoscha Burchardt1 Peter Dabrock3 Sebastian
Möller1 Bilgin Osmanodja2 Simon Ronicke2 David Samhammer3 Sven Schmeier1
1German Research Center for Artificial Intelligence (DFKI), Germany
2Charité - Universitätsmedizin Berlin, Germany
3University Erlangen-Nürnberg (FAU), Germany

###### Abstract

Scientific publications about machine learning in healthcare are often about
implementing novel methods and boosting the performance - at least from a
computer science perspective. However, beyond such often short-lived
improvements, much more needs to be taken into consideration if we want to
arrive at a sustainable progress in healthcare. What does it take to actually
implement such a system, make it usable for the domain expert, and possibly
bring it into practical usage? Targeted at Computer Scientists, this work
presents a multidisciplinary view on machine learning in medical decision
support systems and covers information technology, medical, as well as ethical
aspects. Along with an implemented risk prediction system in nephrology,
challenges and lessons learned in a pilot project are presented.

##  1 Introduction

The United Nations’ Sustainable Development Goal 3 “Good Health and Well-
being” aims to ensure healthy lives and promote well-being at all ages, e.g.,
by increasing life expectancy and improving access to physicians. AI-based
tools, such as intelligent decision support systems, are expected to
contribute to this goal, as they promise to increase diagnostic precision and
improve monitoring medical treatments. Even though the healthcare sector comes
with many well justified constraints regarding the use of data or the design
of experiments, it can be considered one of the drivers of applied research in
AI and machine learning (ML) in particular. In the past, many scientific
publications in ML evolved around the introduction of novel methods and
boosting performance, e.g., of classification or prediction tasks based on
readily available data Tan et al. (2021); Qiao et al. (2019); Karthikeyan et
al. (2021).

While implementing methods for the best possible system performance is
important and necessary, this is only the first step and sometimes it means
harvesting the low-hanging fruit. Many fundamental research questions follow
directly when the goal is to make practical use of such a system in
healthcare: Can it be further developed without medical expertise and if not,
how and where does it come into play? How can the evolving socio-technical
system be evaluated? How can we, e.g., assess criteria such as trust or
transparency? Does the relationship between medical experts and patients
change when the system enters the scene? It is clear that these questions
cannot be answered by computer scientists alone and likewise they cannot be
answered by medical or ethical experts on their own without taking technical
expertise into consideration.

In order to move a step forward, this article is written for Computer
Scientists working (and particularly starting to work) in the intersection of
AI and medicine. Based on a risk prediction model implemented into an
intelligent decision support system that we have developed in the context of
nephrology, we discuss the main challenges and lessons learned from a
technical perspective. Doing so, we take medical and ethical perspectives into
account, and try to shed light on aspects that are worth taking into
consideration to get beyond a simple performance gain. The main contribution
of this work are the learnings we made during the development of our risk
prediction system.

##  2 A Use Case in Nephrology

The risk prediction model we present here was developed in a project by some
of the authors of this paper, namely the computer scientists and the clinical
experts. The main goal of our risk prediction model was the detection of
kidney transplant patients at risk of an infection within the next 90 days.
Kidney transplantation requires immunosuppressive medication, and while too
little immunosuppressive medication can lead to rejection and transplant
failure, the majority of patients suffer from the opposite - infections.
Severe bacterial infections are strongly associated with a very high CRP
(C-reactive protein) level in the blood, e.g., a CRP level increases from a
normal value below 5 to above 100 mg/L.

Baseline for our work was a real and noisy patient database of a kidney
transplant center, consisting of demographics, lab values, vital parameters,
medications, clinical notes etc. As transplanted patients need to come to the
hospital for checkups 3-4 times a year, the database contains a lot of
valuable information on a fine-grained level over many years.

Figure 1: Simplified version of the dashboard including risk score, risk graph
over time and relevant features.

For our experiments data from 2009-2019 was considered, involving 1500
different patients with altogether approximately 100k data points. We define a
data point as a point in the life of a patient documented in the database.
Each time a new entry is made in the patient database for that patient, a new
data point is created. Given a data point, including previous information
until that point in time, the risk prediction model makes an estimation of how
likely an infection occurs within a given time frame. About 12% of the data
points include an infection within the next 90 days.

Infection was defined based on CRP levels in the blood. Since CRP elevations
can be caused by other non-infectious causes (e.g. rheumatic diseases or
cancer) and in order to focus on relatively severe infections a threshold of
100 mg/L has been set for endpoint definition.

For our implementation we use a Gradient Boosted Regression Tree (GBRT), which
can be trained quickly, also on less performant infrastructure. Overall, our
model was intended to be a baseline for a first prototype. It was developed
within the hospital, given the available infrastructure, in close
collaboration with medical experts. Before training, the data was
preprocessed, transformed and filtered. The final model uses about 300
different features, consisting of lab values, vital parameters, diagnoses,
medications, demographics etc. The resulting model, as well as further
descriptions about the data and usage of the model, can be made available on
request.

Days | ROC | PRC
---|---|---
90 | 0.80 | 0.41
180 | 0.79 | 0.46
360 | 0.77 | 0.50
Table 1: Result on 10 fold cross-validation to predict the endpoint
“infection”, according to Area under Receiver Operator Curve (ROC) and Area
under Precision Recall Curve (PRC).

###  2.1 Internal Validation

Within an internal validation we evaluated our system using a 10-fold cross
validation on the historic (retrospective) patient data. Each time, the data
was randomly split into training, development and test using a split of
70/15/15. The split was conducted on a patient level, to ensure that no data
point of the same patient will occur across different splits. In addition to
the endpoint prediction for the next 90 days, we also explore the prediction
within the next 180 and 360 days. Even though Area under Receiver Operator
Curve (ROC) is frequently used for the evaluation of this kind of tasks Topol
(2019), it is known that ROC cannot deal very well with unbalanced data Branco
et al. (2016), which is often the case in a clinical setup. Therefore for our
evaluation we use ROC, as well as Area under Precision Recall Curve (PRC).

The results of the internal validation are presented in Table 1. For the
intended prediction within 90 days, the ROC score is 0.80 and the PRC score
0.41. In comparison with the longer time frames, the ROC score is a bit higher
for the event in the near future while the PRC score increases for the long
term predictions.

###  2.2 Experiment: Machine Learning vs Physicians

Based on the internal validation, it was difficult to assess the quality and
efficiency of the system as to the best of our knowledge no related work
addressing this problem existed. Therefore it was also not clear if this model
would be useful for medical experts in some way or other. Thus, instead of
focusing on the further optimization of our machine learning model, we decided
to find out how well physicians can solve the given task, and if our approach
might have a benefit for them already.

In the course of this, we conducted an experiment, consisting of two parts: In
the first part, physicians received a data point of a patient, together with
all information up to this time, and had to make a risk estimation (0-100%) if
an infection as defined above will occur within the next 90 days. In the
second part the physician additionally received information of the risk
prediction model, in the form of a dashboard (see Figure 1). The dashboard
depicts the risk score over time within a graph, mapped to a traffic light
system, indicating if the patient has got a low, medium or high risk of an
infection, along with local and global features of the risk prediction model.
Whereas global features are generally important features for the overall
model, local features are those which had the strongest impact on the current
prediction of the system. The cutoff values for the traffic light system were
calculated by using the predictions on a separate development set. While the
dashed line in the middle represents the threshold for the optimal F1 score,
the yellow area is defined by F2, which highlights recall over precision, and
the red area is defined by F0.5, which highlights precision over recall.

While automatic risk estimations can be generated within seconds, humans
require time to carry out this task. This fact indeed limited our
possibilities for human evaluation. Overall, our study included 120 data
points of 120 different patients (38 infections, 82 without infections). 8
physicians participated in our study, four experienced (senior) and four
assistant (junior) physicians. Within each part of the study, each physician
received 15 data points to analyze. The study was accepted by the ethical
advisory board of the hospital, and according to the staff council each
physician had up to 30 minutes time to examine the data of each patient. Each
physician received some basic introduction about the dashboard, information
about how the machine learning model works, as well as its performance on the
internal validation study. Finally, the result of both parts of the study are
then compared to the performance of the risk prediction system itself. In the
following we refer to the first part of the study as MD (medical doctor) and
the second part as MD+AI (medical doctor including machine learning support).
The machine learning component itself will be referred to as AI.

For the experiment we re-trained the model, and made sure that no patient of
the test set occurs within the training and development set. The evaluation of
the experiment is carried out according to ROC and PRC, as well as sensitivity
(recall), specificity (true negative rate), and positive predictive value
(precision) using different cutoff-thresholds.

| ROC | PRC | SEN | SPEC | PPV | thrs
---|---|---|---|---|---|---
AI | 0.719 | 0.567 | 0.754 | 0.718 | 0.264 | F2
0.580 | 0.844 | 0.333 | F1
0.329 | 0.940 | 0.424 | F0.5
MD | 0.630 | 0.474 | 0.448 | 0.689 | 0.406 |  $>$50%
MD+AI | 0.622 | 0.469 | 0.368 | 0.829 | 0.500 |  $>$50%
Table 2: Result of the physicians predicting infections within the next 90
days - without (MD) and with (MD+AI) decision support, in comparison to the
automatic risk prediction (AI). Evaluation is carried out using ROC (Area
under Receiver Operator Curve) and PRC (Area under Precision Recall Curve), as
well as SEN (sensitivity; recall), SPEC (specificity; true negative rate), and
PPV (positive predictive value; precision) defined by different thresholds
(thrs).

The results presented in Table 2 show that overall the task is not too easy
for physicians, considering the ROC score of 0.63. Moreover, the results
indicate that physicians receiving the additional automatic decision support
do not increase performance. On the other hand, the risk prediction system
outperforms the physicians in both parts of the study regarding ROC and PRC.

While the ROC and PRC results of AI show a promising improvement in comparison
to the physicians, this does not tell us if patients would be labeled as
critical (infection will occur) or uncritical. Thus we use different cutoff
values to map from the probability estimation of the physicians and the
regression score of the risk prediction system, to either 1 (infection) and 0
(no infection). In this way, we can evaluate the risk prediction system, as
well as the physicians according to sensitivity, specificity, and positive
predictive value. The results in Table 2 show that AI achieves the highest
sensitivity, therefore finds the largest number of critical patients,
depending on the selection of the cutoff value. Conversely, all results
present only a moderate performance according to the positive predictive value
(PPV). In the case of MD+AI for instance every second precision would be a
true positive infection prediction, while the sensitivity is only about 37%.
Similarly it behaves for the others, which achieve a higher sensitivity, but
it comes at the price of a reduced PPV.

Table 3 compares the ROC performance of the two subject groups in both parts
of the study. The table shows that senior MDs are better than junior MDs in
solving the task. Moreover, the table indicates that junior MDs increase in
performance, together with the automatic decision support, while senior MDs
decrease.

| Junior MD | Senior MD
---|---|---
MD | 0.5781 | 0.6772
MD+AI | 0.6398 | 0.6149
Table 3: ROC performance of the two different subject groups, without (MD) and
with (MD+AI) automatic decision support.

##  3 Learnings made during the Development

#### Development constraints

In general, many circumstances affected the overall development of the risk
prediction model. The development had to be conducted within the hospital
infrastructure and network, due to the sensitivity of the data. This meant
that limited computing power was available, partially admin rights of the
working computer were missing and access to the internet was restricted.

#### Data constraints

Real clinical data tends to be noisy and to have grown over time. In our case,
some older data did not have the same quality as more recent data points.
Certain database fields were not used anymore or were used only occasionally.
This means, much domain and expert knowledge was required to interpret and
subsequently filter and normalize the data, including lab values which
occurred in different units and eventually even were assessed with different
lab methods over time.

#### Never trust a single score

A somewhat trivial advice has proven helpful: Never trust a single score.
While a ROC score of 0.80 in the internal validation appears to be promising,
it is surprising that the 180 and 360 day prediction achieve similar results,
as a prediction in those long term time frames seems to be quite difficult.
One reason could be that the model might be able to identify factors, which
generally increase the risk for infections, rather than those predicting a
particular problem for the near future. In addition, the prediction of an
infection within the next 360 days, also includes events which occur already
in the near future. Moreover, the PRC score for 90 days is only mediocre,
showing an even stronger counterintuitive increase for longer time frames.

#### The issue with false positives

In our scenario, depending on how the threshold is chosen, the results of
sensitivity, specificity and PPV change. From a patient perspective the
presented results are far away from being optimal. None of the approaches will
predict all possible patients at risk, and the more critical patients can be
detected the lower the PPV. Can such a setup be possibly useful for clinical
practice? Particularly if the number of false positives is too large the
automatic prediction might not be seen as trustworthy anymore, or might lead
to “alert fatigue” Ash et al. (2007). Reporting non-perfect machine learning
scores on medical prediction tasks appears to be common, as no system provides
a perfect score Tomašev et al. (2019); Guo et al. (2019). However, reporting
results without taking the medical expertise into account, and without knowing
the benefit might be not optimal.

#### Data analysis

The usage of multiple scores is certainly beneficial, to get a good impression
of the quality of a system. On the other hand looking into the data, and
analyzing the estimations of single patients is essential. Is there something
the automatic risk prediction can do that the physician can’t do? Or can only
the obvious cases be predicted? In our experiment for instance, a detailed
analysis revealed, that the risk prediction system can partially detect
different patients at risk, which is a strong argument for using the system as
a double check for physicians.

#### Communication of results

Even though our model outperformed physicians on the risk prediction task, the
experiment showed that physicians cannot make better estimations together with
automatic predictions. This shows that performance alone is not sufficient to
introduce novel technologies into clinical practice. Moreover, as we have seen
in Table 3, junior MDs increased in ROC performance, while senior MDs did not.
This indicates that the human-machine-interaction and in particular the way
how information is communicated might be essential and need to be further
studied for the success of the system.

##  4 Discussion and more general Learnings

#### Communicate with relevant stakeholders

In order to develop machine learning models on clinical data, communication is
key. Frequent interactions with relevant stakeholders certainly help to
understand the task, and to identify relevant factors to solve it, or to
understand which information is reliable (or has to be ignored) in the
database. As simple as this sounds, it is not, as for instance physicians have
got a different background and so do you. It is necessary to find a “common
language”. In an ongoing project not reported here, we additionally take the
patient’s perspective into account. While this is undoubtedly important, it
comes with another bunch of challenges.

#### Start with a simple system

From a machine learning perspective, the selection of the underlying model has
a huge influence on the outcomes. Therefore neural models seem to be the first
choice. However, in most cases we recommend starting with a simple system in
order to develop a solid and reliable baseline first. A model which can be
quickly developed, and quickly trained, also on less powerful computing
architecture - as we did. Moreover, starting generally simple, with fewer
features and slowly increasing the complexity can be beneficial, as
(sequential) medical data can be complex.

#### Make use of expert knowledge

Machine learning engineers often think that a model solely needs to be trained
on large data in order to solve the problem by itself. In fact, data driven
machine learning models might be able to learn new or at least different
correlations and patterns as compared to physicians who can reason in terms of
causation. In our setup for instance, the number of lab values appears to be a
good indication that something is going wrong - conversely physicians would
probably rather argue that some values are “borderline” or abnormal, therefore
additional lab examinations might be needed, which has been also observed in
other related work Rethmeier et al. (2020). Although different, both
observations indirectly refer to the same - more lab examinations are needed,
as something does not seem to be okay. In any way, we recommend taking the
opinion of physicians into account (ideally from the beginning) and examine if
their suggestions about relevant information/parameters contribute to the
system performance. For instance, is the suggested information covered by the
data going into the model, or can the information actually properly be
represented by the model (e.g. fluctuations of particular lab values)?

#### Use Case/Target Definition

For every research project in medicine, endpoint definition is crucial.
Endpoints need to be precise and clinically relevant. When using clinical
routine data to train machine learning algorithms, endpoints need to be
defined from preexisting data, which can be tricky. While endpoints in
prospective clinical trials are explicitly assessed by performing diagnostic
tests at a certain time point defined in a study protocol (e.g. kidney biopsy
one year after study inclusion), this is not the case for retrospective trials
using clinical routine data. Hence, it is not advisable to use endpoints, for
which diagnostic tests are not regularly performed. In our example, to predict
infections, we chose a laboratory-based definition, because for the majority
of data points in our dataset, this laboratory value is available and always
will be determined in case of severe infection. On the contrary, it would be
much more problematic to predict a rare event, which requires a specific
diagnostic examination that is not routinely performed. Every prediction based
on such data will be imprecise, because the diagnosis is often missed or at
least delayed by the physicians themselves. It is important to keep in mind
that no AI tool can detect cases it has not learned to detect. Even with that
in mind, defining a broad endpoint such as infection using a single laboratory
parameter can be too simplistic and again not optimal for the clinicians.
Specifically, our definition ignores less severe, and non-bacterial
infections, but includes certain rare, non-infectious reasons for increased
CRP-value. Therefore, to prevent the necessity of redefining endpoints later
in the model development, it is again advisable to consult several clinicians
about their opinion about specific endpoint definition, as the endpoint
definition not only has to meet clinical usability but also has to be accepted
by the physician.

#### Lack of comparability

The evaluation of a method is essential. On the other hand, comparability and
reproducibility is a major problem for AI in healthcare, as datasets are often
not available. Each data set has its own characteristics. Achieving good
results on one dataset is not a guarantee that you also achieve similar
results on a different one, addressing the same problem. One reason for this,
if the underlying dataset (cohort) is selected slightly differently.

#### Setting of the study and choice of study subjects

In addition to performance evaluation, most medical problems involve at least
a medical expert - and potentially an AI-based system - for resolving the
task. Thus, the interplay between medical experts and AI-based systems needs
to be considered in the evaluation. Human-machine interaction is just one
aspect of this interplay. Importantly, the set-up of the experiments in which
humans and the system collaborate, the information that is given to the human
participants about the AI-based system, the setting of the study, etc. are all
factors which are likely to impact the results. Thus, a simple comparison
between human and machine baseline is not really helpful for estimating how
such a sociotechnical system would perform in practice. If the patient’s view
is to be taken into account, depending on the state of the patients, it can be
advisable to also include nursing staff, relatives or even third parties such
as patient representative organizations.

#### Model explanations

The “explanations” of our system are based on local and global features – a
simple and pragmatic solution. In many cases this is a particular lab value,
the presence of a diagnosis, intake of a medication or the time since the last
transplantation. In contrast, in previous work we explored how physicians
would actually justify their risk estimations. Those justifications were short
paragraphs consisting of a few sentences and partially contain similar
information as the relevant features of the ML system. However, justifications
contain, aside from pure facts, also descriptions about fluctuation and
tendencies of certain parameter (“lab value is raising since the last few
months”), negations/speculations (e.g. explicit absence of symptoms) ,
additional world knowledge (“lab value above norm”, “patient does not seem to
be adherent”) and interpretations (“patient is fit”). Overall, medical
decisions do not only depend on medical knowledge/experience and parameters
provided in a database, but also from conversions to the patient or nursing
staff. This information is only partially contained in clinical databases, if
then only in clinical text. Moreover, we experienced that human justifications
strongly vary, not only depending on task and risk score, but also depending
on the individual physician. This indicates that a “one-size-fits-all” perfect
system explanation will probably not exist, as each physician might have their
own preferences. As understanding the system recommendation is essential to
build up trust, the topic of communicating and justifying system decisions is
still an open research domain - although much work is currently conducted in
the context of XAI (explainable AI) Markus et al. (2021).

#### A customizable user interface

Another relevant aspect we experience in our experiment is the wish of a
personalization of such an automatic decision support system, which might be
grounded in the need for professional communication mentioned above. The
current approach is static, it is optimized according to ROC, and presents
results within a Dashboard. However, at the end of our experiments, physicians
expressed the interest to interact and modify the system according to their
preferences. For instance by excluding parameters, or increasing the relevance
of some aspects. We assume that an automatic and interactive decision support
system might help to easier understand the technology behind, and understand
the potential, as well as limitations of such a system. In this way, a more
efficient supportive system could be created, according to the wishes and
needs of the treating physician, e.g. targeting only on high precision
predictions, or achieving a larger recall. This might help to build up more
trust in this technology. Techniques that are discussed under the heading of
XAI like counterfactual explanations or the display of similar cases from the
training data may further improve the usefulness of the system here.

#### From Model to Usage

From a clinical perspective, many applications of AI represent improvements on
a technological level, but approach problems that are either mastered by human
physicians or can and have been solved with classical statistical methods as
well. For AI researchers working with medical data, the long-term goal should
be to arrive at a system that is able to improve patient outcomes or reduce
physicians’ workload. Only then will it be considered beneficial by evidence-
based medicine or by physicians in daily practice. Great performance alone
will not be enough for that. We think that systems aiming above all for
successful translation into clinical practice will have a higher impact than
very performant prediction tools for a suboptimal use case. Unfortunately,
this is usually more difficult to achieve with respect to data collection,
preparation and outcome definition. Additionally, structural barriers of
research funding limit the possibility to follow long-term projects.

Nevertheless, for a tool to be eventually implemented in a clinical setting,
design questions should be addressed carefully in the beginning. Specifically,
use cases should be defined before the development of a tool by performing
user research involving physicians, nurses, or patients at the beginning. They
should be selected based on existing weaknesses in clinical settings and
consider the problems and limitations of clinical work and reasoning. By
choosing such tasks, the potential benefit can be maximized and is easier to
demonstrate in controlled trials studying the system’s impact. Finally, a
close interaction with the end user during implementation seems crucial as the
end users have to be trained not only on practical aspects but also on the
strengths and weaknesses of the model in order to make an informed treatment
decision.

###  4.1 Connecting different Stakeholders

It has become standard practice for better AI decision support systems to
claim that they are aligned with the concept of trustworthy AI. Numerous
criteria have been developed for this purpose, for which various reviews are
now available Morley et al. (2021); Jobin et al. (2019). If one also takes the
frame of the HLEG as a reference, one can say: It is important that any AI-
System should be compatible with applicable laws, meets ethical standards, and
does not entail unforeseen side effects HLE (2022). However, trust is not a
concept that is valid once achieved but must be constantly maintained Braun et
al. (2021a). Therefore, it is crucial how the path from principles to practice
is designed. Formally, approaches of co-creation AI Zicari et al. (2021) are
well suited for this challenge. But also terms of content need to be unfolded.
For this purpose, we decided in our experiment to analyze and evaluate
attitudes of the physicians involved.

Of particular interest for AI-driven support systems in the clinic seems to be
the question of how interaction processes change through the introduction of
such and how normative concepts like trust, responsibility, and transparency
have to be rethought in a new and interrelated way Braun et al. (2021b). To
approach these questions in the context of our experiments, qualitative
interviews were conducted with the physicians at the end of the study. We
decided to use semi-structured expert interviews as the data collection
method. The interviews were intended both to obtain an evaluation of the
conducted case study and to find out what impact the introduction of AI-driven
support systems have on the interviewed physicians. The evaluation is mainly
relevant for the further development of the tested system while other
estimations of the physicians can help to draw conclusions for further ethical
discussions.

The results are reassuring. The physicians tell us how they used the system,
to what extent they trusted the assessment and what suggestions they have for
improvement. Especially for the design of such experiments, these statements
are of major importance. For example, it became clear that some of the
physicians followed a certain procedure they gave themselves, e.g., by first
noting their own prediction and then checking the system result (“second
opinion”) or by first looking at the system result and then challenging it
with the available evidence. When designing further experiments, we can now
better decide if we want to impose a certain procedure or not.

At the same time, the physicians report on the complexity of clinical
decision-making in general. They confirm trust to be a prerequisite for being
able to make decisions. For an AI-driven support system to be transferred to
clinical practice at all, it must therefore be trusted. According to the
physicians, the evaluations of the systems must be explainable for this. Not
completely, but in a way that the physicians are able to present the
application of the system to the patient, to whom responsibility is always
borne.

Without being able to go into more detail about the content of the interviews
at this point, we want to emphasize the positive experience that took place
with the qualitative survey accompanying the case study. What becomes clear is
that through the interviews a connection is made between the profession of the
physicians and the researchers. Following the thesis of Noordegraaf that
professions have to be more and more in touch with their environment,
especially due to technical challenges, we can say from our perspective that
the qualitative data collection is definitely able to be a link between the
physicians and their environment. Our recommendation to future research
projects is therefore that they should be accompanied by such or similar
studies. These should not only serve as an evaluation of the individual
projects, but also as part of designing AI decision support systems and as a
contribution of the communication process which must be part of the
transformations that already take place.

##  5 Related Work

Due to limited space, here, we just briefly draw comparison to the most
relevant work. The work of Sutton et al. provides a good introduction into the
world of clinical decision support and discusses some of our learnings of our
work, such as the problem of too many false positives. Zicari et al. discuss
the development of an ethically aligned co-design to ensure trustworthiness.
However, the work remains more theoretical, with less and concrete practical
advice as the work reports on an early developmental stage and thus misses the
aspect of a real system and experiments to discuss. In Amann et al., the
authors came to a similar conclusion as we do, that performance alone is not
enough, and therefore stress the importance of explanabilty. The authors
discuss this topic from multiple perspectives, but provide only an overview
about the topic without getting too much into detail. Bruckert et al., again a
high level and interdisciplinary work, present a roadmap to develop
comprehensive, transparent and trustful systems, considering the “black-box-
nature” of many machine learning approaches. However, the work targets mainly
on the aspect of explainability and interaction, and does not validate the
ideas within a particular experiment.

Although not too recent, the work of Topol provides an overview about
different medical fields and the most relevant work regarding developments of
AI algorithms. The authors also present publications from different medical
fields in which the performance of AI methods have been compared to the
performance of doctors - nephrology is not listed here. Moreover, the author
highlights that AI can enhance the patient-doctor relationship - a perspective
which has not been analyzed in detail in our work reported here. Finally, the
work of Ho and Caals arrives at a similar conclusion to ours, that the
inclusion of multiple stakeholders is necessary to develop AI technology in
healthcare.

Concluding, there is a large range of publications addressing the aspect of
automatic AI-driven clinical decision support (or particular aspects of it),
its challenges and coming to partially similar conclusions as we do. However,
this work discusses multiple aspects developing such a system from a rather
technical aspect, meant for Computer Scientists, but taking also other
perspectives into account which are necessary. Moreover, the discussion does
not remain theoretical, as we substantiate it with challenges and learnings we
have experienced along a use case in nephrology.

##  6 Conclusion

This work presented an automatic risk prediction system to predict severe
infections after kidney transplantation within the next 90 days. Within a
small study we compared the performance of our system to the performance of
physicians with and without the support of our system. Main findings are that
our risk prediction system outperforms physicians on our small dataset,
physicians do not improve together with the automatic decision support, only
junior physicians do. Although from a technical point of view, our system
achieves promising results, our experiments leave us with more open questions
than before, such as what does a successful decision support system need to
look like to achieve the goal to actually help physicians treating their
patients in a better way? Along with our experiments we discussed the
challenges and lessons learned from a multidisciplinary view. Targeted to
Computer Scientists new in the field of AI in healthcare, the main
contribution of this work is to highlight that pure performance is not enough
to set up a practical tool to support clinicians. Moreover, the presented
challenges indicate particularly the need in multidisciplinary work with
different stakeholders, as well as shaping the AI-human interaction on
different levels.

## Acknowledgements

This research was supported by the German Federal Ministry of Education and
Research (BMBF) through the project vALID (01GP1903A).

## References

  * Amann et al. [2020] Julia Amann, Alessandro Blasimme, Effy Vayena, Dietmar Frey, and Vince I Madai.  Explainability for artificial intelligence in healthcare: a multidisciplinary perspective.  BMC Medical Informatics and Decision Making, 20(1):1–9, 2020.
  * Ash et al. [2007] Joan S Ash, Dean F Sittig, Emily M Campbell, Kenneth P Guappone, and Richard H Dykstra.  Some unintended consequences of clinical decision support systems.  In Amia annual Symposium proceedings, volume 2007, page 26. American Medical Informatics Association, 2007.
  * Branco et al. [2016] Paula Branco, Luís Torgo, and Rita P Ribeiro.  A survey of predictive modeling on imbalanced domains.  ACM Computing Surveys (CSUR), 49(2):1–50, 2016.
  * Braun et al. [2021a] Matthias Braun, Hannah Bleher, and Patrik Hummel.  A leap of faith: is there a formula for “trustworthy” ai?  Hastings Center Report, 51(3):17–22, 2021.
  * Braun et al. [2021b] Matthias Braun, Patrik Hummel, Susanne Beck, and Peter Dabrock.  Primer on an ethics of ai-based decision support systems in the clinic.  Journal of medical ethics, 47(12):e3–e3, 2021.
  * Bruckert et al. [2020] Sebastian Bruckert, Bettina Finzel, and Ute Schmid.  The next generation of medical decision support: A roadmap toward transparent expert companions.  Frontiers in artificial intelligence, page 75, 2020.
  * Guo et al. [2019] Xu Guo, Han Yu, Chunyan Miao, and Yiqiang Chen.  Agent-based decision support for pain management in primary care settings.  In IJCAI, pages 6521–6523, 2019.
  * HLE [2022] High-Level Expert Group on Artificial Intelligence: Ethics guidelines for trustworthy AI. European Commission.  https://ec.europa.eu/digital-single-market/en/news/ethics-guidelines-trustworthy-ai, 2022\.  Accessed: 2022-02-11.
  * Ho and Caals [2021] Calvin Wai-Loon Ho and Karel Caals.  A call for an ethics and governance action plan to harness the power of artificial intelligence and digitalization in nephrology.  In Seminars in nephrology, volume 41, pages 282–293. Elsevier, 2021\.
  * Jobin et al. [2019] Anna Jobin, Marcello Ienca, and Effy Vayena.  The global landscape of ai ethics guidelines.  Nature Machine Intelligence, 1(9):389–399, 2019.
  * Karthikeyan et al. [2021] Akshaya Karthikeyan, Akshit Garg, PK Vinod, and U Deva Priyakumar.  Machine learning based clinical decision support system for early covid-19 mortality prediction.  Frontiers in public health, 9, 2021.
  * Markus et al. [2021] Aniek F Markus, Jan A Kors, and Peter R Rijnbeek.  The role of explainability in creating trustworthy artificial intelligence for health care: a comprehensive survey of the terminology, design choices, and evaluation strategies.  Journal of Biomedical Informatics, 113:103655, 2021.
  * Morley et al. [2021] Jessica Morley, Luciano Floridi, Libby Kinsey, and Anat Elhalal.  From what to how: an initial review of publicly available ai ethics tools, methods and research to translate principles into practices.  In Ethics, Governance, and Policies in Artificial Intelligence, pages 153–183. Springer, 2021.
  * Noordegraaf [2020] Mirko Noordegraaf.  Protective or connective professionalism? how connected professionals can (still) act as autonomous and authoritative experts.  Journal of Professions and Organization, 7(2):205–223, 2020.
  * Qiao et al. [2019] Zhi Qiao, Xian Wu, Shen Ge, and Wei Fan.  Mnn: multimodal attentional neural networks for diagnosis prediction.  In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI-19), 2019.
  * Rethmeier et al. [2020] Nils Rethmeier, Necip Oguz Serbetci, Sebastian Möller, and Roland Roller.  Efficare: Better prognostic models via resource-efficient health embeddings.  In AMIA Annual Symposium Proceedings, volume 2020, page 1060. American Medical Informatics Association, 2020.
  * Sutton et al. [2020] Reed T Sutton, David Pincock, Daniel C Baumgart, Daniel C Sadowski, Richard N Fedorak, and Karen I Kroeker.  An overview of clinical decision support systems: benefits, risks, and strategies for success.  NPJ digital medicine, 3(1):1–10, 2020.
  * Tan et al. [2021] Qingxiong Tan, Mang Ye, Grace Lai-Hung Wong, and PongChi Yuen.  Cooperative joint attentive network for patient outcome prediction on irregular multi-rate multivariate health data.  In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence (IJCAI-21), 2021.
  * Tomašev et al. [2019] Nenad Tomašev, Xavier Glorot, Jack W Rae, Michal Zielinski, Harry Askham, Andre Saraiva, Anne Mottram, Clemens Meyer, Suman Ravuri, Ivan Protsyuk, et al.  A clinically applicable approach to continuous prediction of future acute kidney injury.  Nature, 572(7767):116–119, 2019.
  * Topol [2019] Eric J Topol.  High-performance medicine: the convergence of human and artificial intelligence.  Nature medicine, 25(1):44–56, 2019.
  * Zicari et al. [2021] Roberto V Zicari, Sheraz Ahmed, Julia Amann, Stephan Alexander Braun, John Brodersen, Frédérick Bruneault, James Brusseau, Erik Campano, Megan Coffee, Andreas Dengel, et al.  Co-design of a trustworthy ai system in healthcare: deep learning based skin lesion classifier.  Frontiers in Human Dynamics, page 40, 2021.