# Advancing diagnostic performance and clinical usability of neural networks
via adversarial training and dual batch normalization

Tianyu Han  Physics of Molecular Imaging Systems, Experimental Molecular
Imaging, RWTH Aachen University, Germany Sven Nebelung  Department of
Diagnostic and Interventional Radiology, University Hospital Düsseldorf,
Germany  Federico Pedersoli  Department of Diagnostic and Interventional
Radiology, University Hospital Aachen, Germany  Markus Zimmermann  Department
of Diagnostic and Interventional Radiology, University Hospital Aachen,
Germany  Maximilian Schulze-Hagen  Department of Diagnostic and Interventional
Radiology, University Hospital Aachen, Germany  Michael Ho  ARISTRA GmbH,
Berlin, Germany  Christoph Haarburger  ARISTRA GmbH, Berlin, Germany  Fabian
Kiessling  The Institute for Experimental Molecular Imaging, RWTH Aachen
University, Germany  Fraunhofer Institute for Digital Medicine MEVIS, Bremen,
Germany  Comprehensive Diagnostic Center Aachen (CDCA), University Hospital
RWTH Aachen, Aachen, Germany  Christiane Kuhl  Department of Diagnostic and
Interventional Radiology, University Hospital Aachen, Germany  Comprehensive
Diagnostic Center Aachen (CDCA), University Hospital RWTH Aachen, Aachen,
Germany  Volkmar Schulz  Physics of Molecular Imaging Systems, Experimental
Molecular Imaging, RWTH Aachen University, Germany Fraunhofer Institute for
Digital Medicine MEVIS, Bremen, Germany  Comprehensive Diagnostic Center
Aachen (CDCA), University Hospital RWTH Aachen, Aachen, Germany
Correspondence should be addressed to V.S<EMAIL_ADDRESS>and D.T.
<EMAIL_ADDRESS> Daniel Truhn  Department of Diagnostic and
Interventional Radiology, University Hospital Aachen, Germany  ARISTRA GmbH,
Berlin, Germany  Correspondence should be addressed to V.S<EMAIL_ADDRESS>aachen.de) and D.T<EMAIL_ADDRESS>
###### Abstract

Unmasking the decision making process of machine learning models is essential
for implementing diagnostic support systems in clinical practice. Here, we
demonstrate that adversarially trained models can significantly enhance the
usability of pathology detection as compared to their standard counterparts.
We let six experienced radiologists rate the interpretability of saliency maps
in datasets of X-rays, computed tomography, and magnetic resonance imaging
scans. Significant improvements were found for our adversarial models, which
could be further improved by the application of dual-batch normalization.
Contrary to previous research on adversarially trained models, we found that
accuracy of such models was equal to standard models, when sufficiently large
datasets and dual batch norm training were used. To ensure transferability, we
additionally validated our results on an external test set of 22,433 X-rays.
These findings elucidate that different paths for adversarial and real images
are needed during training to achieve state of the art results with superior
clinical interpretability.

## INTRODUCTION

Computer vision (CV) in medical imaging has been a focus of radiological
research in recent years. It is likely that CV methods will soon be used as
adjunct tools by radiologists: Computer-aided diagnosis can help to speed up
the diagnostic process by guiding radiologists to findings worth looking at
and maximize diagnostic accuracy by reducing subjectivity [1, 2, 3, 4, 5].
Prominent examples are deep convolutional neural networks (CNN), which had
their breakthrough when more conventional computer vision algorithms were far
surpassed by residual neural networks in 2015 [6]. Similar developments have
taken place in medicine, where CNNs performed comparable to experts in lung
cancer diagnosis [7, 8, 9], retinal disease detection [10, 11, 12], and skin
lesion classification [13, 14, 15].

However, state-of-the-art deep learning models behave like a black box in that
their decision process remains obscure and inexplainable to human readers
[16]. This gives rise to problems: First, deep learning models trained in a
standard fashion are vulnerable when facing attacks from adversaries. An
attacker might introduce a subtle change into the image - such as changing a
single pixel [17] \- and manipulate the output of the model towards a desired
direction, e.g., non-presence of disease when in reality the disease is
present. Recent studies suggest that adversarial attacks are capable of
manipulating predictions of deep learning models across diverse clinical
domains [18, 19]. The second problem is associated with the so-called ‘Clever
Hans’-type decision strategy [20] where a model focusses its attention towards
irrelevant details such as a hospital department’s tag present in the X-ray
when assessing the severity of diseases. Thus, patients with X-rays from the
intensive care unit might seemingly correctly get classified as pathological
even though the model does nothing else than to scan for that specific tag.
For the clinician, it is thus important to understand, why a model arrives at
a certain conclusion and if that reasoning aligns with human reasoning: A
reliable and robust mechanism to explain the model’s reasoning could support
the acceptance of deep learning models in clinical routine [21]. Numerous
efforts have been devoted to understanding such decision making processes and
to find an agreement between data-driven features and human expertise [22, 23,
24, 25, 26].

Our hypothesis is that robustly trained models, i.e. models that are to a
certain extent immune to adversarial attacks, are not always at odds with
accuracy as previously assumed. We examine, if this conflict can be resolved
when employing specific training paradigms and having sufficient amounts of
radiological data available. Moreover, we hypothesize that training the model
in this fashion allows for the model’s reasoning to be more closely aligned
with clinical expectations than when the model is trained in a conventional
fashion. We test these hypotheses in dedicated experiments on X-rays, computer
tomography images and magnetic resonance images and involve six radiologists
who rate the clinical validity of our results.

## RESULTS

### Robustness and Model Performance Trade-off

Supervised models are vulnerable to adversarial attacks in which an adversary
subtly changes the input to the model and thereby manipulates the prediction
of that model. Instead of optimizing the parameters $\theta$, e.g., the
weights of the neurons, of a model $h_{\theta}$ towards the minimum of the
loss function $\mathcal{L}$,

$\underset{\theta}{\text{minimize}}\>\underset{(x,y)\sim
D}{\mathbb{E}}\mathcal{L}(h_{\theta}(x),y),$ (1)

one is able to generate adversarial examples $(x+\delta)$ by solving the
optimization problem

$\underset{\delta\in\Delta}{\text{maximize}}\>\mathcal{L}(h_{\theta}(x+\delta),y),$
(2)

where $(x,y)$ is an input-label pair in the dataset D, $\delta$ is the applied
adversarial perturbation, and $\Delta$ is an allowable set of perturbations.
In practice, adversarial examples will always be designed to be as
unconspicious as possible to the human eye. One commonly define the allowed
perturbations set $\Delta$ to be a hypersphere ball around any data $x$ with a
constrained norm ($l_{\infty}\leq\epsilon$) [27]. To select the best model in
the task of pathology detection, we quantified the model’s performance via the
area under the receiver operating characteristic curve (ROC-AUC). In
accordance with previous research, we found that adversarial perturbations can
easily influence conventionally trained models when applied to disease
detection in thoracic X-rays, see Fig 1. D: the standard classifier was
significantly biased even by a small amount of adversarial perturbation
($\epsilon\leq 0.001$).

In a second experiment, we made the models robust to adversarial attacks by
employing an approach proposed by Madry et al. [28]. In this approach, we
minimized the expected adversarial loss via performing gradient descend on
adversarial samples - effectively presenting the model with adversarial
examples during training:

$\underset{\theta}{\text{minimize}}\underset{(x,y)\sim
D}{\mathbb{E}}[\underset{\delta\in\Delta}{\text{maximize}}\>\mathcal{L}(h_{\theta}(x+\delta),y)].$
(3)

Figure 1: Adversarially trained models are robust against adversarial attacks.
Adversarial perturbations with increasing strength ($\epsilon$) were generated
via an iterative fast gradient sign method (FGSM). To demonstrate the impact
of adversarial attacks on state-of-the-art classifiers, we trained a ResNet-50
models with a large chest X-ray dataset (CheXpert) containing nearly 200,000
X-rays. (A) Original unmanipulated chest radiograph. (B) Adversarial noise
with $\epsilon$ = 0.002. (C) Manipulated chest radiograph (original radiograph
+ noise), i.e. adversarial example. (D) The standard model was easily misled
by small adversarial perturbations (B) that are not perceptible to the human
eye (C) and accuracy in classifying the disease dropped drastically when
allowing more pronounced perturbations. (E) Only a limited amount of
performance degradation was observed when applying adversarial attacks on the
model trained adversarially ($\epsilon$ during training was set to 0.005).

Fig. 1 visualizes our result, that an adversarially trained (robust)
classifier (Fig. 1E) was less sensitive to adversarial perturbations than its
counterpart that was trained in a standard fashion (Fig. 1D). Nevertheless, as
other groups have previously shown, that robust models appear to be less
accurate than standard models [29, 30, 31]. We tested these results in the
context of medical datasets by performing adversarial training on the Luna16
[32], kneeMRI [33], and CheXpert [34] datasets (shown in Fig. 2). We found
that robust models were indeed less accurate when trained on limited datasets,
see blue and green curves in Fig. 2 A and B. However, we found that the
performance gap between standard and robust models was less pronounced when
sufficient amounts of data were available, see Fig. 2 C.

Figure 2: The use of dual batch norm boosts the classification performance of
neural networks. Three models were compared: blue: neural network without
adversarial training, green: neural network with adversarial training and red:
neural network with adversarial training employing dual batch norms. The
models’ performances were tested on three distinct datasets: (A) Rijeka knee
magnetic resonance imaging (MRI) dataset. (B) Luna16 dataset containing
computed tomography (CT) slices of malignant tumors and (C) CheXPert thoracic
X-ray dataset. In line with previous reports [29] we found that robustness and
good performance appear to be incompatible when data is limited. In both
experiments A and B, the AUC of naively adversarially trained models (green)
dropped significantly as compared to models trained in a standard fashion
(blue). However that performance gap was reduced when the models were trained
on a large dataset of 191,027 radiographs C. Adversarially trained models
performed best when employing dual batch norm (red), no significant difference
in performance to the naively trained models were found. As reflected by the
red curves, the performance of robust models was boosted across different
datasets when dual batch norm training was employed (A-C).

### Achieving both Robustness and Accuracy

Adversarial examples can also be viewed as a form of data augmentation guided
by gradient backpropagation [29, 35]. However, adversarial training faces
problems due to differing statistics between the original data and the
adversarially perturbed data [36]. To optimize the gradient flow of
adversarial training, we used separate batch normalization layers for the aim
of reparametrizing original ($x$) and adversarially perturbed ($x^{\ast}$)
batches [36, 37]. Giving the network the freedom to choose the
reparameterization parameter $\gamma$ independently for adversarial and
original batches resulted in differing distributions, see Fig. S1. More
importantly, the employment of separate normalization layers removed the
performance gap towards naively trained models: as demonstrated by the red AUC
in Fig. 2, no performance difference towards naively trained models was found,
even when the size of the training set was comparatively small. A more
detailed summary of performance metrics and confidence intervals can be found
in SUPPLEMENTARY MATERIALS, SUPPLEMENTARY MATERIALS and SUPPLEMENTARY
MATERIALS. No significant differences in ROC-AUC, sensitivity, and specificity
were found when comparing the naively trained non-robust model to the
adversarially trained model with dual batch norm training.

### Generalization of Robust Models

In general, overfitting of machine learning models can be a problem. To
explore whether this was the case with our robust training, we used the
external ChestX-ray8 dataset as a test dataset. A comparison of the
distribution of ground-truth labels of both datasets is listed in Table
SUPPLEMENTARY MATERIALS. Fig. 3 shows a comparison of the standard and robust
models solely trained on the CheXpert dataset. Models were validated on the
external ChestX-ray8 dataset that contained 22,433 radiographs and had never
before been presented to the models [38]. As before, the robust model
employing dual batch norms outperformed the conventional adversarially trained
model and performed comparably to the standard model.

Figure 3: Validation of models on an external dataset (ChestX-ray8). (A)
Schematic of the data selection process. (B) AUC of the standard model (blue)
and the adversarially trained models with (red) and without (green) dual batch
norms on an independent test set of 22,433 radiographs from the ChestX-ray8
dataset. Dual batch norm training resulted in better AUC, closely matching the
performance of the standard model.

### Adversarial Training Selects Clinically Meaningful Features

Machine learning is often seen as a black box algorithm [21] in the sense that
the way in which a machine learning model comes to its result is not
transparent. This hinders the acceptance of such algorithms, as humans put
more trust in conclusions which are comprehensible. Furthermore, such
information might be helpful in pointing the radiologist to the suspicious
finding, e.g., a tumor, in the radiological examination. To visualize the
parts of the image which are decisive for the algorithm’s conclusions, we
highlighted the input pixels that contribute most to the final model
prediction via gradient backpropagation. These pixel maps are then displayed
in a color map and are generally referred to as saliency maps. As previously
reported by Tsipras et al. [29] saliency maps of robust models were found to
be both sparse and well aligned with human expertise, see Fig. 5. In this
study, it was found that the saliency maps of the adversarially trained neural
network with dual batch norm (SDBN) agreed significantly better with human
expertise than both the adversarially trained models with a single batch norm
(SSBN) and those of the standard model (SSM): Six radiologists were given the
task to rate the meaningfulness of the saliency maps in guiding the
radiologist to the correct pathology on a scale from 0 (no correlation between
pathology/ies and hot spot(s) on saliency map) to 5 (clear and unambiguous
correlation between pathology/ies and hot spot(s) on saliency map). Across all
modalities - i.e. X-ray, MRI, and CT - all readers consistently found that
SDBN were more meaningful than SSBN which in turn were more meaningful than
the SSM, see table Adversarial Training Selects Clinically Meaningful
Features. These results were most significant for X-rays, followed by MRIs and
CTs, most likely due to the number of training samples: for the training of
the classifiers, about 200,000 X-rays were used, while only 917 MRIs and 888
CT scans were available respectively. The results of individual readers as
well as the pooled ratings are given in table SUPPLEMENTARY MATERIALS. Fig. 4
visualizes the distribution of all ratings for X-rays, MRIs, and CTs
respectively. In all modalities, the highest ratings were achieved for SDBN,
with significant proportions of the maps reaching ratings as high as four and
five, while only negligible fractions reached higher ratings for the standard
SSM.

Figure 4: Adversarially trained neural network with dual batch norm yields clinically interpretable saliency maps. Figure shows the assessment of diagnostic relevance in percentage for SSM, SSBN, and SDBN models as evaluated independently by six radiologists. Each color bar reveals percentage of gradient saliencies with same rating score.  Table 1: Mean ratings of radiologists in guiding the radiologist to the correct pathology on a scale from 0 (useless) to 5 (saliency map points clearly and unambiguously to the correct pathology). In total 100 images were rated by 6 radiologists for each dataset Dataset | Score for SSM | Score for SSBN | Score for SDBN | Friedman test
---|---|---|---|---
X-ray | $0.57\pm 0.94$ | $2.20\pm 1.33$ | $2.69\pm 1.56$ | $p<0.001$
MRI | $0.49\pm 0.74$ | $0.74\pm 1.09$ | $2.17\pm 1.57$ | $p<0.001$
CT | $0.47\pm 0.73$ | $2.32\pm 1.72$ | $2.50\pm 1.74$ | $p<0.001$

More precisely, a robust model, as shown in red in Fig. 5 A, was able to
detect common thoracic diseases such as cardiomegaly, atelectasis, and
pneumonia based on the organ shape and the lung opacity. While the SDBN
pointed more clearly to the areas of interest that were decisive for
diagnosis, the SSN had less focus on these areas and the SSM was almost
completely uncorrelated to these areas and useless in guiding the radiologist
to the correct conclusions. Similarly, for knee magnetic resonance images
(Fig. 5 B), the SDBN showed a more direct correlation with the pathology than
the SSBN, which more often pointed to accompanying, but more unspecific
phenomena such as joint effusion. Again, the SSM was almost completely
uncorrelated to the imaging pattern of the disease. Finally, for the
intrapulmonary malignancies in CT slices shown in Fig. 5 C, the finding that
SDBN, SSBN, and SSM were useful in descending order was again confirmed with
the borders of the malignancy being emphasized more pronounced in the SDBN.
These examples also illustrate that large data is needed to train a network
that generates meaningful maps: while the examples of Fig. 5 A showed a good
correlation between the disease pattern and the saliency maps - at least for
SDBN and SSBN - the correlation was less pronounced for the dataset of knee
MRI and CT slices (Fig. 5 B and C) due to the much smaller size of those
datasets.

Figure 5: Saliency maps can help in guiding specialists to the correct
diagnosis. Loss gradients were plotted with respect to their input pixels for
the X-ray (A), MRI (B), and CT (C) datasets. No extra preprocessing steps were
applied to the loss gradients. Here, blue and red colored pixels denote
negative and positive gradients individually. For comparison, saliency maps of
standard and robust models are presented next to the radiological input. In
all three datasets, the SDBN pointed more accurately and more distinctively to
the pathology than the SSBN, while the SSM was almost completely uncorrelated
and almost noise-like. It can also be observed that the saliency maps created
based on neural networks that have been trained on a very large dataset of
hundreds of thousands of images (A) were more precise in pointing to the
pathology, than those trained on datasets containing fewer examples (B and C).

The above finding suggests that the reparameterization (learnable parameters
$\gamma$ and $\beta$, supplement section Reparameterization for adversarial
mixture training) offered by batch norms affect the representation learned by
deep neural networks.

### Batch Normalization Influences Learned Representations

In the setting of adversarial training with dual batch norms, original and
perturbed batches are decoupled via passing them through separate batch norm
layers. To understand how the use of these two batch norms influenced the
features learned by the robust models, we quantified the similarity between
the layers within the respective deep neural network by using the linear
centered kernel alignment (Linear CKA) method [39]. In Fig. 6 A, we visualized
the typical representation learned by a model in a standard training setting,
with only one batch norm. We found that a certain degree of correlation
between succeeding layers was present. However, long-range correlations - i.e.
correlations between layers that were far apart - tended to be relatively
weak, indicating, that the information that was passed on in the network gets
continuously processed. The situation was different however for the same
network architecture when adversarial training with only one batch norm was
used, see Fig. 6 B: Long ranging correlations resulted in a block like
structure and the first 35 layers (about 65 % of the network) seemed to carry
approximately the same information. Such a high similarity of learned
representations may be part of the reason for the performance degradation of
robust models trained via vanilla adversarial training [29, 39]: it seems that
the networks may not be able to encompass the full complexity of the dataset
after adversarial training. The network seems to effectively reduce to a
simpler - less deep network since neighboring layers contained similar
activations. Using a dual set of batch norms for the original image samples
and the adversarial image samples seemed to preserve the complexity of the
network when fed with the original samples (Fig. 6 C), while at the same time
providing the same transition as in Fig. 6 B for the adversarial samples as
indicated by the similarity between the linear CKA maps of Fig. 6 B and D.

Figure 6: Linear centered kernel alignment (CKA) reveals representations are
influenced by batch norms. To explore the learned hidden representations, the
linear CKA between convolutional layers of the models was computed on the
CheXpert test set: a model trained with a single batch norm in a conventional
setting with real examples (A), a model trained with a single batch norm with
real and adversarial examples (B), and a model trained with a dual batch norm
with real and adversarial examples when the respective CKA was evaluated
separately with the batch norm used for real (C) and adversarial (D) examples.
It should be noted that the observed grid pattern in A was due to the residual
connections in the ResNet architecture [39]. When employing adversarial
training with a single batch norm, layers of the network seem to get more
similar to each other, as visualized by the block-like structure arising from
the high degree of similar neural activations in Fig. B. This indicates, that
the neural network loses complexity due to adversarial training which might
contribute to a loss in performance. When employing a dual batch norm for
original and adversarial examples respectively, the complexity of the network
seems to be preserved (note the similarity between A and C), when presented
real examples using the first batch norm, while simultaneously robustness to
adversarial examples arises due to the same changes when employing the second
batch norm (D) that the network from B underwent (note the similarity between
B and D).

## DISCUSSION

The purpose of this study was to investigate the applicability and potential
advantages of adversarially robust models in the field of medical imaging. A
limitation of deploying such models in clinics is a potential performance
degradation as compared to conventionally trained models that has been found
by other research groups [29]. In our experiments shown in Fig. 2, however, we
found that this effect appears almost negligible when training the models on
large image data sets (SUPPLEMENTARY MATERIALS, SUPPLEMENTARY MATERIALS and
SUPPLEMENTARY MATERIALS) and when applying dual-batch norms, i.e., no
significant difference in the AUC was found between the standard model and the
adversarially trained model with dual batch norms. Furthermore, we have
validated that robust models can generalize well on external datasets by
employing 22,433 X-rays from the ChestX-ray8 dataset, that had not been part
of the training process and originated from a different institution. Most
likely, the reason that other groups had found significant differences between
the performance of standard models and adversarially trained models is the use
of a single batch norm in adversarial training: we consistently found in all
our experiments, that to achieve the best results in adversarial training, it
was necessary to employ separate batch norms for real and adversarial
examples. By using dual batch norms, our adversarially trained model achieved
state-of-the-art results on thoracic pathology detection [34].

Not only are adversarially trained models less vulnerable to adversarial
attacks (see Fig. 1), but saliency maps generated by adversarially trained
models provide significantly more information to the clinicians than those
generated by standard models and may help to guide them to the right
diagnosis. This can also boost the acceptance of deep learning models in
clinical routine. Deep learning models are often regarded as a black box and
not much trust is put into their opaque decision-making process by clinicians.
By providing the clinician with a meaningful saliency map as generated by
adversarially trained deep neural networks with dual batch norms, the decision
of the neural network can be made more transparent resulting in a better
acceptance by experts.

We further investigated a potential reason for the better performance of
adversarially trained models with dual batch norms: while conventional
adversarial training seems to reduce the complexity of the neural network as
indicated by the increased long-range correlations of the linear CKA between
layers of the network (see Fig. 6), the use of dual batch norms preserves
complexity levels of the networks when feeding in real examples, while
simultaneously accommodating for the increased robustness to adversarial
examples.

This study is limited by the use of neural networks using two-dimensional
inputs. Medical data in CT, MRI, and positron emission tomography (PET) is
inherently volumetric (3D) or even volumetric plus time (4D) and it might be
expected that models encompassing such higher dimensional inputs (instead of a
series of two-dimensional slices) can improve upon their performance. More
research is needed to train a model with high dimensional inputs as
adversarial training commonly becomes more difficult in a high-dimensional
feature space. If higher-dimensional models become more widespread and
applicable, future studies should try to reproduce our results in such models.

In conclusion, we demonstrated, that adversarially trained models with a dual
batch norm are not only equivalent to standard models in terms of diagnostic
performance but offer additional advantages in conveying their reasoning
through the use of clinically useful saliency maps and being more robust to
adversarial attacks. We encourage fellow research groups to employ
adversarially trained neural networks in their applications and hope that this
will not only lead to more robust and better results in terms of diagnostic
performance, but also increased acceptance of such algorithms in clinical
practice.

## MATERIALS AND METHODS

### Study Datasets

A total number of four medical imaging datasets are used in this study: the
CheXpert dataset, which has been released by Irvin et al. in January 2019 and
contains 224,316 chest radiographs of 65,240 patients [34]. Only 191,027
frontal radiographs are downloadable for model training. To clean up CheXpert
labels, we assigned pathology labels not mentioned to 0.0. According to the
labeling performance comparison [34], the uncertainty labels (U) were assigned
to 1.0, except for the consolidation class (to 0.0). For testing, we compare
the performance of the trained models on the official validation set of 202
scans on which the concurrence of diagnosis from three radiologists serves as
ground truth [34]. Another X-ray dataset used in this study is the ChestX-ray8
dataset released by the National Institutes of Health (NIH) in 2017,
containing 112,120 frontal radiographs of 30,805 unique patients [38]. We
randomly select 20% out of 112,120 radiographs, i.e., 6,187 patients and
22.433 radiographs, to form an external test set. The seven overlapping labels
between the CheXpert and ChestX-ray8 datasets are listed in table
SUPPLEMENTARY MATERIALS.

The MRI dataset used in this study is the kneeMRI dataset, which has been
released by Štajduhar et al. [33] in 2017 and contains 917 sagittal proton-
density weighted knee scans from Clinical Hospital Centre Rijeka, Croatia.
Three degrees of anterior cruciate ligament (ACL) injuries were recorded by
radiologists in the Rijeka dataset: namely non-injured (692 scans), partially
injured (172 scans), and completely ruptured (55 scans). Bounding box
annotations of the region of interest (ROI) slices responsible for ACL tear
diagnosis were also provided along with labels. According to ROI labels, we
extracted a total of 3,081 diagnostic relevant MRI slices and randomly split
them into 80% training, 10% development, and 10% testing. Lastly, we
investigated the applicability of adversarial training on CT data with the
Luna16 dataset consisting of 888 CT scans with lung cancer ROI annotations
[32]. In total, a number of 6,691 lung cancer patches were extracted and
randomly split into 80% training, 10% development, and 10% testing.

### PGD Attack and Adversarial Training

Following equation 1, we let the model parameter be denoted as $\theta$, model
loss as $\mathcal{L}$ and training input&label as (x, y). The projected
gradient descent (PGD) method repeatedly adjusts the model’s inputs x in the
direction of maximizing the loss function, i.e.,
$sign(\nabla_{x}\mathcal{L}(\theta,x,y))$. To safeguard models against
adversarial threats, we trained our models against a PGD adversary via both
vanilla and dual batch norm adversarial training. The details of the
adversarial training procedure with separate batch norms are depicted in
algorithm 1.

Algorithm 1 Adversarial Training with Separate $\textbf{BN}_{\textbf{std}}$
and $\textbf{BN}_{\textbf{adv}}$. We use default values of $\epsilon$ = 0.005,
$\alpha$ = 0.0025, $k$ = 10, and $b$ = 64

1:Require: Dataset D; A model h with its current parameter $\theta$ and loss
function $\mathcal{L}$; batch norm layers for standard inputs $BN_{std}$;
batch norm layers for adversarial inputs $BN_{adv}$; the batch size $b$;
learning rate $\eta$.

2:Require: $l_{\infty}$ boundary $\epsilon$; stepsize $\alpha$; number of
attack iterations $k$.

3:Sample a batch of inputs $\\{x^{(j)}\\}_{j=1}^{b}\sim D$ and class labels
$\\{y^{(j)}\\}_{j=1}^{b}\sim D$.

4:for $j=1,...,b$ do

5: Sample input $x\sim\\{x\\}^{b}$ and label $y\sim\\{x\\}^{b}$

6: $x_{0}^{\ast}=x$

7: for $t=1,...,k$ do

8: Input $x_{t-1}^{\ast}$ to model $h_{\theta}(x_{t-1}^{\ast})$

9: $x_{t}^{\ast}\leftarrow
x_{t-1}^{\ast}+\alpha\text{sign}(\nabla_{x}(\theta,x_{t-1}^{\ast},y))$

10: $x_{t}^{\ast}\leftarrow\text{clip}_{x,\epsilon}(x_{t}^{\ast})$

11: end for

12:end for

13:$\\{x^{\ast}\\}^{b}$ = $\\{x_{k}^{\ast}\\}^{b}$

14:$\mathcal{L}_{std}=\mathcal{L}(\theta,BN_{std},\\{x\\}^{b},\\{y\\}^{b})$;
$\mathcal{L}_{adv}=\mathcal{L}(\theta,BN_{adv},\\{x^{\ast}\\}^{b},\\{y\\}^{b})$

15:$\theta\leftarrow\theta+\eta\nabla_{\theta}(\mathcal{L}_{std}^{j}+\mathcal{L}_{adv}^{j})$

### Model Architecture and Training

We used ResNet-50 architecture for our experiments in this study. For all
classification tasks, an Adam optimizer with default $\beta_{1}$ = 0.9,
$\beta_{2}$ = 0.99, and $\epsilon$ = 1e-8 [40] was used to optimize the loss.
In a total number of 300 training epochs, We decayed the initial learning rate
0.01 by a factor of 10 once the number of epochs reached 100 epochs. All
classifier models utilized development-based early stopping with sigmoid
binary cross-entropy loss as the criterion.

Medical images from CheXpert, ChestX-ray8, and kneeMRI were scaled to a fixed
resolution of 256$\times$256 pixels whereas tumor patches extracted from
Luna16 ROI slices were scaled to 64$\times$64 pixels. During training, random
color transformations such as adjusting contrast, brightness, saturation, and
hue factor were applied to each training image. Additionally, we also
performed spatial affine and random cropping augmentations before normalizing
each input to the range of 0 to 1.

All computations were performed on a GPU cluster equipped with two Intel
Xeon(R) Silver 4208 processor (Intel, Santa Clara, Calif) and three Nvidia
Titan RTX 24 GB GPUs (Nvidia, Santa Clara, Calif). When not otherwise
specified, code implementations were in-house developments based on python
3.6.5 (https://www.python.org) and on the software modules Numpy, Scipy, and
Pytorch 1.1.0.

### Model Interpretation

To reveal the connection between input features (pixels) and the model’s final
predictions, we back-propagate the loss gradients with respect to their input
pixels. For all generated gradients, we simply clipped their values to the
range of $\pm$ 3 $\times$ standard deviation around their mean value and
normalized them to [-1, 1] [29].

To investigate the interaction between learned representations in the deep
neural networks and their batch norm layers, we quantified representation
similarity via the linear centered kernel alignment (linear CKA). For a given
input and a model, a linear CKA is defined as:

$\text{CKA(X,
Y)}=\frac{\left\|Y^{T}X\right\|_{F}^{2}}{(\left\|X^{T}X\right\|_{F}\left\|Y^{T}Y\right\|_{F})}$
(4)

where $X$ and $Y$ correspond to a centered Gram matrix of layer activations.
In particular, in Fig. 6, we computed the linear CKA matrix across all 202
radiographs from the internal CheXpert test set.

### Feature Evaluation by Radiologists

To evaluate the clinical utility of the generated saliency maps for the three
models (standard model, adversarially trained model with a single batch norm,
and adversarially trained model with the dual batch norm), we randomly chose
100 images from each of the three datasets used in this study (in total 300
images) and let six radiologists assess how useful the map was in guiding a
radiologist to the correct diagnosis. We used a scale from zero, signifying no
correlation between the pathology and the saliency map, to five, signifying a
map that points clearly and unambiguously to the correct pathology - or
pathologies if multiple pathologies were present in the image, see table
Feature Evaluation by Radiologists. All readers performed the task
independently of each other.

Table 2: Rating standard used for evaluating the diagnostic value of generated saliency maps Score | Diagnostic Rating
---|---
0 |  | no correlation between pathology/ies and hot spot(s) on saliency map.
---
1 | highly doubtful low-degree correlation between pathology/ies and hot spot(s) on saliency map.
2 |  | doubtful moderate-degree correlation between pathology/ies and hot spot(s) on saliency map.
---
3 | definite partial correlation between pathology/ies and hot spot(s) on saliency map.
4 |  | definite substantial correlation between pathology/ies and hot spot(s) on saliency map.
---
5 |  | clear and unambiguous correlation between pathology/ies and hot spot(s) on saliency map.
---

### Statistical Analysis

For each of the experiments, we calculated the following parameters on the
test set: area under the curve (AUC) for the receiver operator characteristic
(ROC), sensitivity, and specificity. The cutoff value for deciding between the
presence or non-presence of a pathology was determined by minimizing
$(1-\text{sensitivity})^{2}+(1-\text{specificity})^{2}$ [41]. To assess errors
due to sampling of the specific test set and estimate the confidence intervals
we employed bootstrap analysis with 10,000 redraws. The difference in metrics,
such as AUC, sensitivity, and specificity, was defined as a $\Delta$metric.
For the total number of N = 1,000 bootstrapping, models were built after
randomly permuting predictions of two classifiers, and metric’s differences
$\Delta\text{metric}_{i}$ were computed from their respective scores. We
obtained the p-value of individual metrics by counting all
$\Delta\text{metric}_{i}$ above the threshold $\Delta$metric. Statistical
significance was defined as P $<$ 0.001. To determine if differences were
significant in the reader studies we employed the Friedman test to test for
the presence of differences within the three groups. If the Friedman test was
significant we tested for pairwise differences (i.e. SSM vs. SSBN, SSM vs.
SDBN, and SSBN vs. SDBN) employing the Wilcoxon signed-rank test.

#### Author contributions:

TH and DT devised the concept of the study, DT, SN, MS, MZ, FP, and MH
performed the reader tests. TH wrote the code and performed the accuracy
studies. TH and DT did the statistical analysis. TH, DT, and VS wrote the
first draft of the manuscript. All authors contributed to correcting the
manuscript.

#### Disclosures of Conflicts of Interest:

There are no conflicts of interest to declare.

#### Data and materials availability:

This study used four publicly available datasets: NIH ChestX-ray14 dataset:
https://nihcc.app.box.com/v/ChestXray-NIHCC; Stanford CheXpert dataset:
https://stanfordmlgroup.github.io/competitions/chexpert; kneeMRI dataset:
http://www.riteh.uniri.hr/~istajduh/projects/kneeMRI/; LUNA16 dataset:
https://luna16.grand-challenge.org/Data/ Details of the implementation, as
well as the weights of the neural networks after training and the full code
producing the results of this paper, are made publicly available under
https://github.com/peterhan91/Medical-Robust-Training. All data needed to
evaluate the findings in the paper are present in the paper and/or the
supplementary material. Additional data related to this paper may be requested
from the authors.

## References

  * [1] S. M. Lundberg, B. Nair, M. S. Vavilala, M. Horibe, M. J. Eisses, T. Adams, D. E. Liston, D. K.-W. Low, S.-F. Newman, J. Kim, et al., “Explainable machine-learning predictions for the prevention of hypoxaemia during surgery,” Nature biomedical engineering, vol. 2, no. 10, pp. 749–760, 2018\.
  * [2] B. He, L. Bergenstråhle, L. Stenbeck, A. Abid, A. Andersson, Å. Borg, J. Maaskola, J. Lundeberg, and J. Zou, “Integrating spatial gene expression and breast tumour morphology via deep learning,” Nature biomedical engineering, pp. 1–8, 2020.
  * [3] D. Truhn, S. Schrading, C. Haarburger, H. Schneider, D. Merhof, and C. Kuhl, “Radiomic versus convolutional neural networks analysis for classification of contrast-enhancing lesions at multiparametric breast mri,” Radiology, vol. 290, no. 2, pp. 290–297, 2019.
  * [4] P. Rajpurkar, J. Irvin, K. Zhu, B. Yang, H. Mehta, T. Duan, D. Ding, A. Bagul, C. Langlotz, K. Shpanskaya, et al., “Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning,” arXiv preprint arXiv:1711.05225, 2017.
  * [5] P. Rajpurkar, J. Irvin, R. L. Ball, K. Zhu, B. Yang, H. Mehta, T. Duan, D. Ding, A. Bagul, C. P. Langlotz, et al., “Deep learning for chest radiograph diagnosis: A retrospective comparison of the chexnext algorithm to practicing radiologists,” PLoS medicine, vol. 15, no. 11, p. e1002686, 2018\.
  * [6] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.
  * [7] D. Ardila, A. P. Kiraly, S. Bharadwaj, B. Choi, J. J. Reicher, L. Peng, D. Tse, M. Etemadi, W. Ye, G. Corrado, et al., “End-to-end lung cancer screening with three-dimensional deep learning on low-dose chest computed tomography,” Nature medicine, vol. 25, no. 6, pp. 954–961, 2019.
  * [8] N. Coudray, P. S. Ocampo, T. Sakellaropoulos, N. Narula, M. Snuderl, D. Fenyö, A. L. Moreira, N. Razavian, and A. Tsirigos, “Classification and mutation prediction from non–small cell lung cancer histopathology images using deep learning,” Nature medicine, vol. 24, no. 10, pp. 1559–1567, 2018.
  * [9] A. B. Levine, C. Schlosser, J. Grewal, R. Coope, S. J. Jones, and S. Yip, “Rise of the machines: Advances in deep learning for cancer diagnosis,” Trends in cancer, vol. 5, no. 3, pp. 157–169, 2019.
  * [10] J. De Fauw, J. R. Ledsam, B. Romera-Paredes, S. Nikolov, N. Tomasev, S. Blackwell, H. Askham, X. Glorot, B. O’Donoghue, D. Visentin, et al., “Clinically applicable deep learning for diagnosis and referral in retinal disease,” Nature medicine, vol. 24, no. 9, pp. 1342–1350, 2018\.
  * [11] H.-Y. Tsao, P.-Y. Chan, and E. C.-Y. Su, “Predicting diabetic retinopathy and identifying interpretable biomedical features using machine learning algorithms,” BMC bioinformatics, vol. 19, no. 9, p. 283, 2018.
  * [12] F. Arcadu, F. Benmansour, A. Maunz, J. Willis, Z. Haskova, and M. Prunotto, “Deep learning algorithm predicts diabetic retinopathy progression in individual patients,” NPJ digital medicine, vol. 2, no. 1, pp. 1–9, 2019\.
  * [13] A. Esteva, B. Kuprel, R. A. Novoa, J. Ko, S. M. Swetter, H. M. Blau, and S. Thrun, “Dermatologist-level classification of skin cancer with deep neural networks,” Nature, vol. 542, no. 7639, pp. 115–118, 2017.
  * [14] P. Tschandl, C. Rinner, Z. Apalla, G. Argenziano, N. Codella, A. Halpern, M. Janda, A. Lallas, C. Longo, J. Malvehy, et al., “Human–computer collaboration for skin cancer recognition,” Nature Medicine, vol. 26, no. 8, pp. 1229–1234, 2020.
  * [15] Y. Liu, A. Jain, C. Eng, D. H. Way, K. Lee, P. Bui, K. Kanada, G. de Oliveira Marinho, J. Gallegos, S. Gabriele, et al., “A deep learning system for differential diagnosis of skin diseases,” Nature Medicine, pp. 1–9, 2020.
  * [16] C. Rudin, “Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead,” Nature Machine Intelligence, vol. 1, no. 5, pp. 206–215, 2019.
  * [17] J. Su, D. V. Vargas, and K. Sakurai, “One pixel attack for fooling deep neural networks,” IEEE Transactions on Evolutionary Computation, vol. 23, no. 5, pp. 828–841, 2019.
  * [18] S. G. Finlayson, J. D. Bowers, J. Ito, J. L. Zittrain, A. L. Beam, and I. S. Kohane, “Adversarial attacks on medical machine learning,” Science, vol. 363, no. 6433, pp. 1287–1289, 2019.
  * [19] X. Han, Y. Hu, L. Foschini, L. Chinitz, L. Jankelson, and R. Ranganath, “Deep learning models for electrocardiograms are susceptible to adversarial attack,” Nature Medicine, pp. 1–4, 2020.
  * [20] S. Lapuschkin, S. Wäldchen, A. Binder, G. Montavon, W. Samek, and K.-R. Müller, “Unmasking clever hans predictors and assessing what machines really learn,” Nature communications, vol. 10, no. 1, pp. 1–8, 2019.
  * [21] W. N. Price, “Big data and black-box medical algorithms,” Science translational medicine, vol. 10, no. 471, 2018.
  * [22] Z. Zhang, P. Chen, M. McGough, F. Xing, C. Wang, M. Bui, Y. Xie, M. Sapkota, L. Cui, J. Dhillon, et al., “Pathologist-level interpretable whole-slide cancer diagnosis with deep learning,” Nature Machine Intelligence, vol. 1, no. 5, pp. 236–245, 2019.
  * [23] Z. Tang, K. V. Chuang, C. DeCarli, L.-W. Jin, L. Beckett, M. J. Keiser, and B. N. Dugger, “Interpretable classification of alzheimer’s disease pathologies with a convolutional neural network pipeline,” Nature communications, vol. 10, no. 1, pp. 1–14, 2019.
  * [24] H. Lee, S. Yune, M. Mansouri, M. Kim, S. H. Tajmir, C. E. Guerrier, S. A. Ebert, S. R. Pomerantz, J. M. Romero, S. Kamalian, et al., “An explainable deep-learning algorithm for the detection of acute intracranial haemorrhage from small datasets,” Nature Biomedical Engineering, vol. 3, no. 3, p. 173, 2019.
  * [25] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-cam: Visual explanations from deep networks via gradient-based localization,” in Proceedings of the IEEE international conference on computer vision, pp. 618–626, 2017.
  * [26] A. Hosny, C. Parmar, T. P. Coroller, P. Grossmann, R. Zeleznik, A. Kumar, J. Bussink, R. J. Gillies, R. H. Mak, and H. J. Aerts, “Deep learning for lung cancer prognostication: A retrospective multi-cohort radiomics study,” PLoS medicine, vol. 15, no. 11, p. e1002711, 2018.
  * [27] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” arXiv preprint arXiv:1412.6572, 2014.
  * [28] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” arXiv preprint arXiv:1706.06083, 2017.
  * [29] D. Tsipras, S. Santurkar, L. Engstrom, A. Turner, and A. Madry, “Robustness may be at odds with accuracy,” arXiv preprint arXiv:1805.12152, 2018.
  * [30] H. Zhang, Y. Yu, J. Jiao, E. P. Xing, L. E. Ghaoui, and M. I. Jordan, “Theoretically principled trade-off between robustness and accuracy,” arXiv preprint arXiv:1901.08573, 2019.
  * [31] D. Su, H. Zhang, H. Chen, J. Yi, P.-Y. Chen, and Y. Gao, “Is robustness the cost of accuracy?–a comprehensive study on the robustness of 18 deep image classification models,” in Proceedings of the European Conference on Computer Vision (ECCV), pp. 631–648, 2018.
  * [32] A. A. A. Setio, A. Traverso, T. De Bel, M. S. Berens, C. van den Bogaard, P. Cerello, H. Chen, Q. Dou, M. E. Fantacci, B. Geurts, et al., “Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the luna16 challenge,” Medical image analysis, vol. 42, pp. 1–13, 2017.
  * [33] I. Štajduhar, M. Mamula, D. Miletić, and G. Ünal, “Semi-automated detection of anterior cruciate ligament injury from mri,” Computer methods and programs in biomedicine, vol. 140, pp. 151–164, 2017\.
  * [34] J. Irvin, P. Rajpurkar, M. Ko, Y. Yu, S. Ciurea-Ilcus, C. Chute, H. Marklund, B. Haghgoo, R. Ball, K. Shpanskaya, et al., “Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 590–597, 2019.
  * [35] T. Miyato, S.-i. Maeda, M. Koyama, and S. Ishii, “Virtual adversarial training: a regularization method for supervised and semi-supervised learning,” IEEE transactions on pattern analysis and machine intelligence, vol. 41, no. 8, pp. 1979–1993, 2018.
  * [36] C. Xie and A. Yuille, “Intriguing properties of adversarial training at scale,” arXiv preprint arXiv:1906.03787, 2019.
  * [37] S. Santurkar, D. Tsipras, A. Ilyas, and A. Madry, “How does batch normalization help optimization?,” in Advances in Neural Information Processing Systems, pp. 2483–2493, 2018.
  * [38] X. Wang, Y. Peng, L. Lu, Z. Lu, M. Bagheri, and R. M. Summers, “Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2097–2106, 2017.
  * [39] S. Kornblith, M. Norouzi, H. Lee, and G. Hinton, “Similarity of neural network representations revisited,” arXiv preprint arXiv:1905.00414, 2019.
  * [40] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
  * [41] H. C. Kniep, F. Madesta, T. Schneider, U. Hanning, M. H. Schönfeld, G. Schön, J. Fiehler, T. Gauer, R. Werner, and S. Gellissen, “Radiomics of brain mri: utility in prediction of metastatic tumor type,” Radiology, vol. 290, no. 2, pp. 479–487, 2019.
  * [42] Y. LeCun, “The mnist database of handwritten digits,” http://yann. lecun. com/exdb/mnist/, 1998.
  * [43] J. C. Pérez, M. Alfarra, G. Jeanneret, A. Bibi, A. Thabet, B. Ghanem, and P. Arbeláez, “Robust gabor networks,” arXiv preprint arXiv:1912.05661, 2019.

## SUPPLEMENTARY MATERIALS

Table S1: Data characteristics of ChestX-ray8 and CheXpert dataset.

| ChestX-ray8 dataset | CheXpert dataset
---|---|---
Number of patient radiographs | 112,120 | 224,316
Number of patients | 30,805 | 65,240
Age, mean (SD), years | 46.9 (16.6) | 60.7 (18.4)
Percentage of females (%) | 43.5% | 40.6%
Number of pathology labels | 8 | 14
Cardiomegaly | 2,776 | 30,092
Edema | 2,303 | 61,493
Effusion | 13,317 | 86,477
Pneumothorax | 5,302 | 20,401
Atelectasis | 11,559 | 59,583
Consolidation | 4,667 | 12,983
Pneumonia | 1,431 | 20,656

The seven overlapping labels between the CheXpert and ChestX-ray8 datasets are
listed in the table above. Note that pathological radiographs seem to be more
common in CheXpert dataset due to the differences in the labeling process
[34].
Abbreviations: SD, standard deviation. Table S2: Individual and pooled ratings
of radiologists in guiding them to the correct pathology on a scale from 0
(useless) to 5 (saliency map points clearly and unambiguously to the correct
pathology).

| Dataset | Score for SSM | Score for SSBN | Score for SDBN | Friedman test
---|---|---|---|---|---
| X-ray | $0.32\pm 0.51$ | $1.56\pm 0.84$ | $2.12\pm 1.10$ | $p=2.1\cdot 10^{-34}$
Radiologist 1 (8 years) | MRI | $0.19\pm 0.39$ | $0.39\pm 0.57$ | $1.15\pm 0.92$ | $p=2.1\cdot 10^{-21}$
| CT | $0.65\pm 0.50$ | $1.59\pm 0.73$ | $1.73\pm 0.83$ | $p=1.5\cdot 10^{-33}$
| X-ray | $1.53\pm 1.16$ | $2.86\pm 0.51$ | $3.30\pm 0.83$ | $p=1.1\cdot 10^{-25}$
Radiologist 2 (8 years) | MRI | $1.65\pm 0.61$ | $1.05\pm 0.78$ | $2.70\pm 1.22$ | $p=3.2\cdot 10^{-21}$
| CT | $1.59\pm 0.83$ | $2.70\pm 1.08$ | $2.93\pm 1.18$ | $p=5.0\cdot 10^{-23}$
| X-ray | $0.29\pm 0.57$ | $2.47\pm 1.64$ | $3.10\pm 2.00$ | $p=7.9\cdot 10^{-30}$
Radiologist 3 (8 years) | MRI | $0.32\pm 0.62$ | $0.54\pm 1.21$ | $2.34\pm 1.97$ | $p=8.7\cdot 10^{-20}$
| CT | $0.11\pm 0.31$ | $2.50\pm 2.13$ | $2.67\pm 2.14$ | $p=1.6\cdot 10^{-22}$
| X-ray | $1.03\pm 1.21$ | $2.49\pm 1.32$ | $2.52\pm 1.34$ | $p=4.8\cdot 10^{-19}$
Radiologist 4 (6 years) | MRI | $0.24\pm 0.51$ | $1.24\pm 1.30$ | $2.83\pm 1.46$ | $p=2.0\cdot 10^{-28}$
| CT | $0.37\pm 0.58$ | $2.92\pm 1.83$ | $3.10\pm 1.72$ | $p=2.7\cdot 10^{-30}$
| X-ray | $0.14\pm 0.40$ | $1.90\pm 1.42$ | $2.50\pm 1.74$ | $p=4.2\cdot 10^{-29}$
Radiologist 5 (5 years) | MRI | $0.20\pm 0.49$ | $0.40\pm 1.09$ | $1.90\pm 1.71$ | $p=1.3\cdot 10^{-20}$
| CT | $0.06\pm 0.24$ | $2.11\pm 1.91$ | $2.29\pm 1.91$ | $p=2.5\cdot 10^{-22}$
| X-ray | $0.13\pm 0.34$ | $1.92\pm 1.45$ | $2.57\pm 1.76$ | $p=3.3\cdot 10^{-29}$
Radiologist 6 (5 years) | MRI | $0.09\pm 0.29$ | $0.81\pm 1.11$ | $2.11\pm 1.31$ | $p=2.6\cdot 10^{-26}$
| CT | $0.04\pm 0.20$ | $2.08\pm 1.85$ | $2.25\pm 1.92$ | $p=9.9\cdot 10^{-23}$
| X-ray | $0.57\pm 0.94$ | $2.20\pm 1.33$ | $2.69\pm 1.56$ | $p=1.0\cdot 10^{-160}$
All radiologists | MRI | $0.49\pm 0.74$ | $0.74\pm 1.09$ | $2.17\pm 1.57$ | $p=1.7\cdot 10^{-118}$
| CT | $0.47\pm 0.73$ | $2.32\pm 1.72$ | $2.50\pm 1.74$ | $p=7.0\cdot 10^{-150}$

Table S3: Comparison of standard and robust models on Luna16 test set.

Prediction |  | ROC-AUC
---
(95% CI)
p-value |  | Sensitivity
---
(95% CI)
p-value |  | Specificity
---
(95% CI)
p-value
Tumor malignancy |  |  |  |  |  |
Standard model |  | 0.952
---
(0.946, 0.959)
- |  | 0.846
---
(0.827, 0.865)
- |  | 0.933
---
(0.923, 0.943)
-
| Robust model,
---
with dual batch norms
| 0.955
---
(0.949, 0.961)
0.472 |  | 0.893
---
(0.878, 0.909)
0.308 |  | 0.878
---
(0.864, 0.891)
0.275

A p-value $<$ 0.001 indicates statistical significance. Abbreviations: ROC-
AUC, the area under the receiver operating characteristic curve; CI,
confidence interval.

Table S4: Comparison of standard and robust models on kneeMRI test set.

Prediction |  | ROC-AUC
---
(95% CI)
p-value |  | Sensitivity
---
(95% CI)
p-value |  | Specificity
---
(95% CI)
p-value
Healthy ACL |  |  |  |  |  |
Standard model |  | 0.824
---
(0.807, 0.841)
- |  | 0.788
---
(0.773, 0.803)
- |  | 0.730
---
(0.701, 0.758)
-
| Robust model,
---
with dual batch norms
| 0.825
---
(0.808, 0.841)
0.487 |  | 0.788
---
(0.773, 0.803)
0.493 |  | 0.717
---
(0.688, 0.745)
0.467
Partially injured ACL |  |  |  |  |  |
Standard model |  | 0.742
---
(0.721, 0.763)
- |  | 0.660
---
(0.625, 0.696)
- |  | 0.735
---
(0.719, 0.750)
-
| Robust model,
---
with dual batch norms
| 0.741
---
(0.720, 0.761)
0.482 |  | 0.626
---
(0.590, 0.661)
0.393 |  | 0.731
---
(0.716, 0.747)
0.485
Completely ruptured ACL |  |  |  |  |  |
Standard model |  | 0.921
---
(0.909, 0.933)
- |  | 0.945
---
(0.915, 0.975)
- |  | 0.802
---
(0.789, 0.815)
-
| Robust model,
---
with dual batch norms
| 0.918
---
(0.908, 0.929)
0.499 |  | 1.000
---
(1.000)
0.434 |  | 0.798
---
(0.785, 0.811)
0.494

A p-value $<$ 0.001 indicates statistical significance. Abbreviations: ACL,
anterior cruciate ligament; ROC-AUC, the area under the receiver operating
characteristic curve; CI, confidence interval.

Table S5: Comparison of standard and robust models on CheXpert test set.

Prediction |  | ROC-AUC
---
(95% CI)
p-value |  | Sensitivity
---
(95% CI)
p-value |  | Specificity
---
(95% CI)
p-value
Cardiomegaly |  |  |  |  |  |
Standard model |  | 0.798
---
(0.782, 0.814)
- |  | 0.727
---
(0.703, 0.752)
- |  | 0.757
---
(0.741, 0.774)
-
| Robust model,
---
with dual batch norms
| 0.853
---
(0.841, 0.866)
0.177 |  | 0.743
---
(0.718, 0.767)
0.426 |  | 0.809
---
(0.794, 0.824)
0.334
Edema |  |  |  |  |  |
Standard model |  | 0.937
---
(0.929, 0.946)
- |  | 0.881
---
(0.859, 0.904)
- |  | 0.881
---
(0.870, 0.893)
-
| Robust model,
---
with dual batch norms
| 0.927
---
(0.918, 0.936)
0.457 |  | 0.928
---
(0.911, 0.946)
0.401 |  | 0.831
---
(0.818, 0.845)
0.361
Consolidation |  |  |  |  |  |
Standard model |  | 0.918
---
(0.909, 0.927)
- |  | 0.937
---
(0.918, 0.957)
- |  | 0.806
---
(0.792, 0.820)
-
| Robust model,
---
with dual batch norms
| 0.936
---
(0.928, 0.943)
0.396 |  | 0.906
---
(0.883, 0.929)
0.465 |  | 0.841
---
(0.828, 0.854)
0.422
Pneumonia |  |  |  |  |  |
Standard model |  | 0.857
---
(0.837, 0.876)
- |  | 1.000
---
(1.000)
- |  | 0.727
---
(0.713, 0.741)
-
| Robust model,
---
with dual batch norms
| 0.794
---
(0.768, 0.820)
0.350 |  | 1.000
---
(1.000)
- |  | 0.588
---
(0.572, 0.604)
0.304
Atelectasis |  |  |  |  |  |
Standard model |  | 0.808
---
(0.794, 0.822)
- |  | 0.787
---
(0.765, 0.808)
- |  | 0.756
---
(0.739, 0.773)
-
| Robust model,
---
with dual batch norms
| 0.818
---
(0.804, 0.832)
0.426 |  | 0.800
---
(0.779, 0.821)
0.455 |  | 0.772
---
(0.755, 0.789)
0.454
Pneumothorax |  |  |  |  |  |
Standard model |  | 0.865
---
(0.836, 0.895)
- |  | 0.714
---
(0.636, 0.792)
- |  | 0.795
---
(0.782, 0.808)
-
| Robust model,
---
with dual batch norms
| 0.791
---
(0.748, 0.834)
0.330 |  | 0.569
---
(0.484, 0.655)
0.178 |  | 0.995
---
(0.993, 0.997)
0.179
Effusion |  |  |  |  |  |
Standard model |  | 0.912
---
(0.903, 0.921)
- |  | 0.781
---
(0.758, 0.805)
- |  | 0.870
---
(0.857, 0.882)
-
| Robust model,
---
with dual batch norms
| 0.931
---
(0.923, 0.939)
0.383 |  | 0.797
---
(0.774, 0.820)
0.429 |  | 0.891
---
(0.879, 0.903)
0.423

A p-value $<$ 0.001 indicates statistical significance. Abbreviations: ROC-
AUC, the area under the receiver operating characteristic curve; CI,
confidence interval.

### Reparameterization for adversarial mixture training

Despite the domain is distinguishable among standard ($x$) and adversarial
batches ($x^{\ast}$) [36], in our study, we also confirm different
reparameterization is applied to those batches. By definition, Adversarial
batches are difficult to optimize when compared to their standard
counterparts. To investigate, we constructed a simple fully-connected network
with one hidden layer followed by two parallel batch norm layers (one for $x$
and one for $x^{\ast}$). We perform our analysis on Luna16 and MNIST [42]
datasets in combination with different attack strength ($\epsilon$). Recall,
the Lipschitz constant of the loss, i.e., gradient magnitude
$\left\|\hat{g}\right\|$ of a network with batch norms, is governed by
reparameterization $\gamma$ [37]. Therefore, in adversarial batches, a smaller
$\gamma$ is favorable as it corresponds to a smaller Lipschitzness of the loss
function and thus facilitates optimization. As shown in Fig. S1 A-C and D-F,
the learned $\gamma$ of adversarial batch norms shift to smaller magnitudes
when applying stronger attacks, whereas the distribution of standard batch
norm remains unchanged.

Figure S1: A smaller magnitude of $\gamma$ was learned when increasing
adversarial attack strength during training. Analysis of the
reparameterization of batch norms versus adversarial strength $\epsilon=$
0.005, 0.04, and 0.3. A shallow network with just one hidden layer (n=256
neurons and ReLU activation) was trained on both MNIST (A-C) and Luna16 (D-F)
datasets. We observe the distribution of adversarial batch $\gamma$ shifts
towards zero under stronger adversarial attacks ($\epsilon=0.3$).

### The influence of perturbation strength on adversarial training

We also investigate how the performance of robust models changes when
increasing adversarial attack strength. Under the setting of training via
separate batch norm layers, the model performance is stable as no significant
performance decrease was observed when increasing $\epsilon$ from 0.005 to
0.04 (Fig. S2).

However, as visualized in Fig. S3, we observe the general learned Gabor
filters (Fig. S3 B) are replaced by checkerboard-like filters (Fig. S3 C and
D) when applying stronger adversarial attacks during training.

Figure S2: Model performance versus training adversarial strength. We observe
that when training with different attack strength, i.e., $\epsilon=$ 0.001,
0.005, and 0.04, the performance of adversarial training with separate batch
norms shows little difference.

Figure S3: The influence of training adversarial strength on learned
convolutional kernels. Visualization of Conv1 filters (7$\times$7) shows Gabor
edge detectors are typically learned through standard training (A). In
addition, by adversarial training, robust models learned Dirac-delta functions
(kernels) to gain higher robustness [43] (B). From B to D, we find that
checkerboard patterns appear within trained kernels when increasing
adversarial strength.


Robin Staabeth
Mark Veroeth
Mislav Balunovićeth
Martin Vecheveth

ethDepartment of Computer Science, ETH Zurich
Robin<EMAIL_ADDRESS>
Machine Learning, Large Language Models, Anonymization, Privacy, Adversarial Learning

Recent work in privacy research on large language models has shown that they achieve near human-level performance at inferring personal data from real-world online texts. With consistently increasing model capabilities, existing text anonymization methods are currently lacking behind regulatory requirements and adversarial threats. This raises the question of how individuals can effectively protect their personal data in sharing online texts. In this work, we take two steps to answer this question: We first present a new setting for evaluating anonymizations in the face of adversarial LLMs inferences, allowing for a natural measurement of anonymization performance while remedying some of the shortcomings of previous metrics. We then present our LLM-based adversarial anonymization framework leveraging the strong inferential capabilities of LLMs to inform our anonymization procedure. In our experimental evaluation, we show on real-world and synthetic online texts how adversarial anonymization outperforms current industry-grade anonymizers both in terms of the resulting utility and privacy.

§ INTRODUCTION

§ BACKGROUND AND RELATED WORK

§ ANONYMIZATION UNDER LLM INFERENCE

§ FEEDBACK-GUIDED ADVERSARIAL ANONYMIZATION

§ EVALUATION

§ DISCUSSION

§ CONCLUSION

^^JLASTBODYPAGE ^^J

§ BROADER IMPACT

The advancement of anonymization methods to cover a wider range of personal attributes and offer better privacy protection has an important societal impact. Individual privacy rights cornerstone legislations such as GDPR and CCPA have been established in various jurisdictions over the last decade. We, therefore, see progress in the field of text anonymization as beneficial to all individuals in order to express the freedom to exercise these rights more accessibly. While more powerful anonymization methods could theoretically be abused to hide malicious intent or individuals, this is far outside the range of our methods.

[Cal, 2018]
California Consumer Privacy Act (CCPA).
https://oag.ca.gov/privacy/ccpa, October 2018.

[Mic, 2021]
Microsoft/presidio: Context aware, pluggable and customizable data
protection and de-identification SDK for text and images.
https://github.com/microsoft/presidio, 2021.

[Aahill, 2023]
What is Azure AI Language - Azure AI services, July 2023.

[Anandan et al., 2012]
Anandan, B., Clifton, C., Jiang, W., Murugesan, M., Pastrana-Camacho, P., and
Si, L.
t-plausibility: Generalizing words to desensitize text.
Trans. Data Privacy, 50 (3):0 505–534, dec
ISSN 1888-5063.

[Brewster, 2023]
Brewster, T.
Chatgpt has been turned into a social media surveillance assistant,
Nov 2023.

[Bubeck et al., 2023]
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E.,
Lee, P., Lee, Y. T., Li, Y., Lundberg, S., Nori, H., Palangi, H., Ribeiro,
M. T., and Zhang, Y.
Sparks of Artificial General Intelligence: Early experiments
with GPT-4, April 2023.
URL <http://arxiv.org/abs/2303.12712>.
arXiv:2303.12712 [cs].

[Carlini et al., 2021]
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee,
K., Roberts, A., Brown, T., Song, D., Erlingsson, U., Oprea, A., and Raffel,
Extracting Training Data from Large Language Models, June

[Carlini et al., 2023]
Carlini, N., Ippolito, D., Jagielski, M., Lee, K., Tramer, F., and Zhang, C.
Quantifying Memorization Across Neural Language Models, March

[Chiang & Lee, 2023]
Chiang, C.-H. and Lee, H.-y.
Can Large Language Models Be an Alternative to Human
Evaluations?, May 2023.

[Dernoncourt et al., 2016]
Dernoncourt, F., Lee, J. Y., Uzuner, O., and Szolovits, P.
De-identification of patient notes with recurrent neural networks.
Journal of the American Medical Informatics Association,
240 (3):0 596–606, 12 2016.
ISSN 1067-5027.
URL <https://doi.org/10.1093/jamia/ocw156>.

[DOL, 2023]
DOL, 2023.
URL <https://www.dol.gov/general/ppii>.

[Dou et al., 2023]
Dou, Y., Krsek, I., Naous, T., Kabra, A., Das, S., Ritter, A., and Xu, W.
Reducing Privacy Risks in Online Self-Disclosures with
Language Models, November 2023.

[EU, 2016]
EU, E. U.
General data protection regulation, 2016.
URL <https://gdpr-info.eu/>.

[Hartford, 2023]
Hartford, E.
https://erichartford.com/dolphin, July 2023.

[Hintze, 2017]
Hintze, M.
Viewing the GDPR through a de-identification lens: a tool for
compliance, clarification, and consistency.
International Data Privacy Law, 80 (1):0
86–101, 12 2017.
ISSN 2044-3994.
URL <https://doi.org/10.1093/idpl/ipx020>.

[Jiang et al., 2023]
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las
Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud,
L. R., Lachaux, M.-A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix,
T., and Sayed, W. E.
Mistral 7B, October 2023.

[Jiang et al., 2024]
Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C.,
Chaplot, D. S., de las Casas, D., Hanna, E. B., Bressand, F., Lengyel, G.,
Bour, G., Lample, G., Lavaud, L. R., Saulnier, L., Lachaux, M.-A., Stock, P.,
Subramanian, S., Yang, S., Antoniak, S., Scao, T. L., Gervet, T., Lavril, T.,
Wang, T., Lacroix, T., and Sayed, W. E.
Mixtral of Experts, January 2024.

[Li et al., 2007]
Li, N., Li, T., Venkatasubramanian, S., and Labs, T.
T-Closeness: Privacy Beyond k-Anonymity and

[Lin, 2004]
Lin, C.-Y.
ROUGE: A package for automatic evaluation of summaries.
In Text Summarization Branches Out, pp.  74–81, Barcelona,
Spain, July 2004. Association for Computational Linguistics.
URL <https://aclanthology.org/W04-1013>.

[Lison et al., 2021]
Lison, P., Pilán, I., Sanchez, D., Batet, M., and Øvrelid, L.
Anonymisation models for text data: State of the art, challenges and
future directions.
In Zong, C., Xia, F., Li, W., and Navigli, R. (eds.),
Proceedings of the 59th Annual Meeting of the Association for
Computational Linguistics and the 11th International Joint Conference on
Natural Language Processing (Volume 1: Long Papers), pp.  4188–4203,
Online, August 2021a. Association for Computational Linguistics.
URL <https://aclanthology.org/2021.acl-long.323>.

[Lison et al., 2021]
Lison, P., Pilán, I., Sanchez, D., Batet, M., and Øvrelid, L.
Anonymisation Models for Text Data: State of the art,
Challenges and Future Directions.
In Proceedings of the 59th Annual Meeting of the
Association for Computational Linguistics and the 11th
International Joint Conference on Natural Language Processing
(Volume 1: Long Papers), pp.  4188–4203, Online,
2021b. Association for Computational Linguistics.

[Machanavajjhala et al., 2006]
Machanavajjhala, A., Gehrke, J., Kifer, D., and Venkitasubramaniam, M.
L-diversity: Privacy beyond k-anonymity.
In 22nd International Conference on Data Engineering
(ICDE'06), pp.  24–24, Atlanta, GA, USA, 2006. IEEE.
ISBN 978-0-7695-2570-9.

[OpenAI, 2023]
Gpt-4 technical report.
ArXiv, abs/2303.08774, 2023.

[Papineni et al., 2002]
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J.
Bleu: a method for automatic evaluation of machine translation.
In Isabelle, P., Charniak, E., and Lin, D. (eds.), Proceedings
of the 40th Annual Meeting of the Association for Computational Linguistics,
pp.  311–318, Philadelphia, Pennsylvania, USA, July 2002. Association for
Computational Linguistics.
URL <https://aclanthology.org/P02-1040>.

[Pilán et al., 2022]
Pilán, I., Lison, P., Øvrelid, L., Papadopoulou, A., Sánchez, D.,
and Batet, M.
The Text Anonymization Benchmark (TAB): A Dedicated
Corpus and Evaluation Framework for Text Anonymization.
Computational Linguistics, 480 (4):0
1053–1101, December 2022.
ISSN 0891-2017, 1530-9312.

[Rosso et al., 2016]
Rosso, P., Rangel Pardo, F., Potthast, M., Stamatatos, E., Tschuggnall, M., and
Stein, B.
Overview of pan’16.
volume 9822, pp.  332–350, 09 2016.
ISBN 978-3-319-44563-2.

[Schuster et al., 2020]
Schuster, T., Schuster, R., Shah, D. J., and Barzilay, R.
The Limitations of Stylometry for Detecting Machine-Generated Fake
Computational Linguistics, 460 (2):0 499–510,
06 2020.
ISSN 0891-2017.
URL <https://doi.org/10.1162/coli_a_00380>.

[28]
Shetty, R., Schiele, B., and Fritz, M.
A4NT: Author Attribute Anonymity by Adversarial Training
of Neural Machine Translation.

[Staab et al., 2023]
Staab, R., Vero, M., Balunović, M., and Vechev, M.
Beyond memorization: Violating privacy via inference with large
language models, 2023.

[Sweeney, 2002]
Sweeney, L.
k-anonymity: A model for protecting privacy.
International journal of uncertainty, fuzziness and
knowledge-based systems, 100 (05):0 557–570, 2002.

[Sánchez & Batet, 2016]
Sánchez, D. and Batet, M.
C-sanitized: A privacy model for document redaction and sanitization.
Journal of the Association for Information Science and
Technology, 670 (1):0 148–163, 2016.

[Touvron et al., 2023]
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix,
T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin,
A., Grave, E., and Lample, G.
LLaMA: Open and Efficient Foundation Language Models,
February 2023a.

[Touvron et al., 2023]
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y.,
Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L.,
Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu,
W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S.,
Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev,
A., Koura, P. S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y.,
Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y.,
Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva,
R., Smith, E. M., Subramanian, R., Tan, X. E., Tang, B., Taylor, R.,
Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A.,
Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and
Scialom, T.
Llama 2: Open Foundation and Fine-Tuned Chat Models, July

[Zheng et al., 2023]
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z.,
Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I.
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,
December 2023.

^^JLASTREFERENCESPAGE ^^J