11institutetext: CitAI Research Centre, City, University of London, London, UK
22institutetext: BioMedIA, Imperial College, London, UK 22email:
<EMAIL_ADDRESS>
# Harder synthetic anomalies to improve Out-of-Distribution detection in
Medical Images

Sergio Naval Marimont 11 0000-0002-7075-5586 Giacomo Tarroni 11 2 2
0000-0002-0341-6138

###### Abstract

Our method builds upon previous Medical Out-of-Distribution (MOOD) challenge
winners that empirically show that synthetic local anomalies generated copying
/ interpolating foreign patches are useful to train segmentation networks able
to generalize to unseen types of anomalies. In terms of the synthetic anomaly
generation process, our contributions makes synthetic anomalies more
heterogeneous and challenging by 1) using random shapes instead of squares and
2) smoothing the interpolation edge of anomalies so networks cannot rely on
the high gradient between image - foreign patch to identify anomalies. Our
experiments using the validation set of 2020 MOOD winners show that both
contributions improved substantially the method performance. We used a
standard 3D U-Net architecture as segmentation network, trained patch-wise in
both brain and abdominal datasets. Our final challenge submission consisted of
10 U-Nets trained across 5 data folds with different configurations of the
anomaly generation process. Our method achieved first position in both sample-
wise and pixel-wise tasks in the 2022 edition of the Medical Out-of-
Distribution held at MICCAI.

###### Keywords:

anomaly detection out-of-distribution detection unsupervised learning
synthetic anomaly detection MOOD Challenge.

##  1 Introduction

Supervised deep learning approaches achieve state of the art performance in
many medical image analysis tasks and are being progressively incorporated in
the clinical practice [7]. However, supervised deep learning methods have
important limitations. First, they rely on large and diverse annotated
datasets which are difficult and expensive to obtain. Ownership and patient
privacy create important constraints to the availability of medical datasets.
Furthermore, deep learning requires images to be manually annotated by
clinical experts in a process that is both onerous and costly.

In addition to the challenges associated with the acquisition of large
annotated datasets, supervised approaches are limited to the anomalies
annotated and cannot be expected to generalize to unseen/unannotated
anomalies. Similarly, supervised methods are known to generate unreliable
predictions for Out-of-Distribution test samples. For example, in the context
of medical images a sufficiently different acquisition protocol from the one
used in the training set could potentially make model predictions unreliable.

Out-of-Distribution (OoD) or unsupervised medical anomaly detection methods
addresses some of these challenges by seeking to identify anomalous images
leveraging only a training set of images of normal subjects [19]. One of the
most promising OoD strategies consists in corrupting images obtained from
normal / healthy subjects with localized synthetic anomalies and in then
training deep networks to identify these synthetic anomalies [15]. Approaches
based on this strategy [15, 4] won the 2020 and 2021 editions of the Medical
Out-of-Distribution Challenge (MOOD) [19] held at the Medical Image Computing
and Computer Assisted Interventions (MICCAI) conference.

The choice for the strategy used to generate and blend anomalies with normal
images is key to enhance the ability of these approches to generalize to
previously unseen anomalies. In this work, we propose a set of simple yet
highly effective modifications to the previously proposed anomaly generation
strategies. We validated our approach both in a synthetic anomaly dataset and
by participating in the MOOD 2022 challenge, where we obtained the first
position overall in both sample-wise and pixel-wise rankings. Specifically our
method ranked first in Brain and Abdominal datasets in sample-wise tasks and
first in Abdominal dataset and fourth in Brain dataset in pixel-wise tasks.

####  1.0.1 Related Works:

Out-of-Distribution detection task aims at identifying anomalies leveraging
only samples from a normative distribution. Self-supervised learning
approaches fit naturally the task. Within self-supervised learning approaches,
generative models have been broadly studied in the literature. Generically,
generative models are used to model the distribution of anomaly-free
anatomies. Once the model has been trained, the expectation is that test
images with anomalies will not conform to the learnt distribution, and that
some mechanism will be able to classify them as anomalous. Anomaly
localization can then also be achieved by comparing pixel-by-pixel the
original image and the one reconstructed by the model. [20, 2] proposed using
Variational Auto-Encoders (VAE), while in [13] it is proposed to leverage
Generative Adversarial Networks (GAN). More recently, [17, 8, 10] propose two
stage approaches using VQ-VAE to encode image and Auto-Regressive models to
model the latent feature distribution.

One drawback of generative models for localization is that they define anomaly
scores comparing image intensities of test samples and reconstructions and
consequently the localization performance decreases in those anomalies with
intensities are similar to healthy anatomies. To address this issue, Foreign
Patch Interpolation (FPI) [15] proposes a new self-supervised pretext task
consisting on identifying local corruptions of images which is well suited for
OoD task. FPI proposes local corruptions created copying and pasting square
foreign image patches, similarly to what was proposed in CutMix [18] as an
augmentation method. FPI achieved the first position in the MOOD Challenge
2020 [19].

Networks can learn to identify FPI patches relying on high gradients present
on the edges of foreign patches. While this supports the identification of
many synthetic anomalies, the trained methods might not be able to identify
medical anomalies. Poisson Image Interpolation (PII) [16] was proposed to
address this limitation of FPI. In PPI, foreign patches are blended with the
training images using Poisson image editing [11] technique to achieve more
natural looking synthetic anomalies. MOOD 2021 challenge winners, [4] use as
well synthetic anomalies pretext task. Their approach introduces additional
augmentations to the foreign patches as proposed in CutPaste [6] and features
a 3D U-Net architecture [12].

In NSA [14] additional constraints are added to the location of the synthetic
anomalies and foreign patches are augmented. nn-OOD [1] adapts the successful
image segmentation methodology described in nn-UNet [5] and comparatively
evaluates previously described methods to generate synthetic anomalies.

####  1.0.2 Contributions:

Building on previous synthetic anomaly approaches, we propose the following
contributions:

  * •

We propose a series of modifications in the synthetic anomaly generation
process aiming to avoid spurious patterns in the anomaly identification
learning. This includes the introduction of random shapes and smoother
blending of synthetic anomalies into the training images.

  * •

We systematically report the impact of these modifications on a dataset of
synthetic anomalies (different from the ones used for training), using a
similar validation strategy to that of FPI [15].

  * •

We present the details of the architecture and pipeline we adopted for our
submission to the MOOD 2022 competition. We release to the public our
implementation.

##  2 Methods

###  2.1 Synthetic anomaly generation process

We describe the anomaly generation process from three aspects: anomaly
texture, shape and interpolation mechanism.

Anomaly texture: In order to obtain natural looking anomalies we also leverage
foreign patches (FP) $x_{fp}$ extracted from the training dataset itself.
Specifically, we build a set of $K$ 3D image patches with dimensions
$(W_{FP},H_{FP},D_{FP})$. FP are sampled randomly adding one patch to the
set/memory from each image processed. Once the memory reaches the $K$ patch
capacity, new FP replace previously added patches.

Similarly to [4] approach, we augment FP with random Gaussian noise, random
intensity shifts and random rotations.

Anomaly shape: While previous approaches described in the previous sections
use square patches, we explore using different shapes, which include
rectangular cuboids, spheres and random 3D shapes. In order to generate random
shapes we use a random walk of a simulated brush with random width that moves
along the 3D canvas. Random rectangular cuboids are augmented with random
rotations and random shapes are augmented with random affine transformations.

Note that in all cases the shapes fit within the specified foreign patch size
$(W_{FP},H_{FP},D_{FP})$. The shapes are referred as mask in Fig.1 and in the
implementation.

Figure 1: Texture (foreign patch) and shape components (mask) of synthetic
anomaly generation process.

Interpolation mechanism: pixel intensities of the anomalous areas are defined
as the linear interpolation between original image intensities $x$ and foreign
patch intensities $x_{fp}$ described in the equation below.

$x^{\prime}=x\times(1-\alpha)+x_{fp}\times\alpha$

The $\alpha$ factor that controls the weighting of the foreign patch
intensities $x_{fp}$ is sampled uniformly from the range $[0.3,1]$. The
location of the foreign patch with size $(W_{FP},H_{FP},D_{FP})$ is also
uniformly sampled to fit within the image with size $(W,H,D)$.

Additionally, as proposed in the original FPI method [15], anomalies are
generated only in foreground areas.

Anomaly edge smoothing: The anomaly generation process as defined in previous
MOOD challenge winners will create high gradients in the edges of the
interpolations. We hypothesised that networks can learn to leverage these
edges to identify the synthetic anomalies and this spurious learning can be
detrimental to identifying natural or synthetic anomalies. We aim to alleviate
this issue by smoothing the edges of the shape prior to the interpolation.
Consequently we apply Gaussian filters to the mask anomaly component. The size
of the Gaussian kernel is selected randomly during the anomaly generation
process among size values $1,3,5,7$. Figure 2 illustrates the anomaly edge
smoothing results compared to the original foreign patch interpolation
process.

Figure 2: Qualitative results and anomaly components using smoothed edges vs
original foreign patch interpolation method

###  2.2 Network architecture

We use a 3D U-Net architecture [12] in our experiments. The U-Net architecture
is symmetric, with 5 down-sampling and up-sampling blocks each consisting of 2
groups of convolution, batch normalization and non-linearity. We use Leaky
ReLU as non-linearity and sigmoid as final activation. Downsamplings and
upsamplings are implemented with 2-strided convolutions and transposed
convolutions respectively. Additionally we use deep-supervision in the outputs
of the last 4 blocks. The network architecture is detailed in the diagram 4
below.

Figure 3: U-Net segmentation network architecture.

In all our experiments we used AdamW optimizer with OneCycle learning rate
schedule with maximum learning rate of $10^{-3}$ and weight decay $1\times
10^{-5}$. We used Pytorch [9] and MONAI [3]. Implementation and trained models
are made publicly available in 111https://github.com/snavalm/mood22.

###  2.3 Training objective

As proposed in [15], we find in our experiments that training the network to
estimate the interpolation factor $\alpha$ improve evaluation metrics.
Specifically, we use binary cross-entropy loss pixel-wise:

$\mathcal{L}_{bce}=-\alpha\times\log(f(x^{\prime}))-(1-\alpha)\times\log(1-f(x^{\prime}))$

where $\alpha$ is the interpolation factor previously introduced and
$f(x^{\prime})$ is the pixel-wise output of the network for the locally
corrupted image $x^{\prime}$.

###  2.4 Anomaly Scores

In unsupervised anomaly localization tasks we use directly the output of our
network trained to identify synthetic anomalies, consequently the pixel-wise
anomaly score (AS) is $AS_{pixel}=f(x)$.

For image classification task we define an image-wise AS relying on the pixel-
wise prediction. Specifically
$AS_{sample}=\sum^{\mathbb{M}}_{i}(f(x_{i}))/100$ where $\mathbb{M}$ is the
subset of 100 pixels with higher pixel-wise anomaly scores in the image.

##  3 Experiments and Results

###  3.1 Datasets:

We evaluated our approach participating in the Medical Out-of Distribution
Challenge 2022 [19]. The challenge consists of sample and pixel-wise tasks and
two datasets:

  * •

Brain: 800 MRI scans, each with spatial dimensions $(256\times 256\times
256)$.

  * •

Abdominal: 550 CT scans, each with spatial dimensions $(512\times 512\times
512)$.

Both datasets are anomaly-free and provided pre-processed with intensities
normalized in the range $[0,1]$. We note that images in the abdominal dataset
have diverse spacing so we resampled the abdominal dataset to a common
isotropic spacing of 1 mm. The MOOD challenge test set contains anomaly-free
images, images with natural anomalies and images with synthetic anomalies. It
is important to note that the MOOD challenge test set is kept confidential at
all times to prevent participants overfitting the anomalies. Additionally the
challenge organizers only provide test set results for the final submission
with the same objective of preventing overfitting to their test set.
Furthermore, the MOOD image pre-processing pipeline is also kept confidential
to prevent participants to leverage other datasets whentraining or evaluating
their models.

Considering MOOD constraints we developed an evaluation dataset leveraging
images provided by the MOOD organizers. Specifically, we set aside $1/5$ of
each of the two provided datasets for evaluation. Using these evaluation
images, and for each of the two modalities, we built a validation set with
synthetically-generated anomalies. Specifically, each validation set
comprised:

  * •

50 healthy / uncorrupted images

  * •

30 images - local additive noise

  * •

30 images - local additive noise (smoothed edges)

  * •

30 images - local synthetic deformations

  * •

30 images - local synthetic reflections

  * •

30 images - local synthetic shifts

  * •

30 images - local uniform noise

  * •

30 images - local uniform noise (smoothed edges)

Note that the evaluation dataset comprises the same type of evaluation
anomalies introduced in the FPI paper and the implementation can be found
here. In order to validate the issues with overfitting to anomaly edges, we
introduced noise anomalies with smoothed edges. The edge smoothing is done
using the methodology previously introduced.

####  3.1.1 Experimental set-up:

In all our experiments we train our network using randomly sampled image
patches of size $(160\times 160\times 160)$ for 35.000 training steps. At
inference time, we use sliding window-based inference with 0.5 patch overlap
and Gaussian weighting of patches.

In our final submission we ensembled 10 models trained with 5 data folds, with
and without mask edge smoothing.

####  3.1.2 Performance evaluation:

To validate the proposed contributions we run experiments focusing on the
Abdominal dataset. We used average precision (AP) as evaluation metric as in
the MOOD challenge evaluation process.

First, we attempted to validate the hypothesis that models trained on
anomalies interpolated using masks with hard edges rely on them to identify
anomalies. To verify this, we used a baseline method with only cuboid mask
shapes of maximum size $W_{FP}=H_{FP}=D_{FP}=64$. Results in Table 1 show that
when introducing 60 samples with smoothed edges, AP decreased from $0.845$ to
$0.740$, validating the reliance on the high gradients created in edges to
identify anomalies. Furthermore, we trained a new model with both smoothed and
hard interpolations and it achieved AP of $0.681$. Using qualitative analysis
we found that the new model performed comparatively worse in hard edge
anomalies but improved in the more challenging anomalies in the validation
set.

Table 1: Abdominal validation set. Ablations on including smoothed synthetic anomalies |  AP
---|---
Models trained on baseline anomalies
Baseline validation dataset ($N=200$) |  0.845
Full validation dataset ($N=260$, Baseline + Smoothed anomalies) |  0.740
Models trained on hard and smoothed anomalies
Full validation dataset ($N=260$, Baseline + Smoothed anomalies) |  0.681

Next, we attempted to assess whether training on complex anomaly shapes
improved the performance in the validation set. We evaluated a new method
trained with diverse shapes: cuboids, spheres and random blobs, chosen
randomly. We denote this method Complex shapes in Table 2 below. Our
experiments showed that using complex shapes improved AP in our validation set
substantially, from $0.740$ to $0.817$ for models trained using anomalies with
hard edges and from $0.681$ to $0.721$ on models trained using anomalies with
a combination of hard and smoothed edges. Furthermore we found that ensembling
models trained with a combination of hard and smoothed-edge anomalies improved
AP further to $0.881$.

Table 2: Ablations on complex shapes. Abdominal validation set. AP |  Hard anomalies |  \+ Smoothed |  Ensembled
---|---|---|---
Cuboid only anomaly shapes |  0.740 |  0.681 |  n/a
Complex anomaly shapes |  0.817 |  0.721 |  0.881

Working towards our MOOD 2022 challenge submission, qualitative analysis
showed that our method was performed comparatively worse in the bigger
synthetic anomalies. Consequently, we doubled the anomaly size so
$W_{FP}=H_{FP}=D_{FP}=128$. We also increased the patch overlap in the sliding
window inference from $0.25$ to $0.5$. Our final submission consisted of an
ensemble of both hard and smoothed anomaly trained model. These models
achieved AP of $0.874,0.894,0.940$, for hard, smoothed and ensembled models
respectively. In both abdominal and brain datasets we submitted an ensemble of
10 models, trained with hard and smoothed anomalies in 5 data folds. Figure 4
summarizes AP for the different experiments evaluated.

Figure 4: Summary of individual contributions to final validation AP.

Our team submission achieved 1st place in both Abdominal and Brain datasets
for the sample-wise task. We also achieved 1st and 4th place in Abdominal and
Brain pixel-wise tasks respectively, which resulted as well in a 1st place
overall in the pixel-wise task. Qualitative examples received from the
challenge are included in Figures 5 and 6.

Figure 5: Abdominal Qualitative results MOOD Challenge test set. Image, label
and pixel-wise anomaly score Figure 6: Brain Qualitative results MOOD
Challenge test set. Image, label and pixel-wise anomaly score

##  4 Conclusion

We build upon existing synthetic anomaly pretext tasks OOD detection methods
introducings improvements in the synthetic anomaly generation process: 1)
using random shapes instead of squares and 2) smoothing the interpolation edge
of anomalies so networks cannot rely on the high gradient between image -
foreign patch to identify anomalies. Results in a synthetic validation showed
a substantially performance improvement over baselines which was confirmed by
the first position in both sample and pixel-wise tasks in the MICCAI MOOD 2022
Challenge.

## References

  * [1] Baugh, M., Tan, J., Vlontzos, A., Müller, J.P., Kainz, B.: nnOOD: A Framework for Benchmarking Self-supervised Anomaly Localisation Methods. In: Sudre, C.H., Baumgartner, C.F., Dalca, A., Qin, C., Tanno, R., Van Leemput, K., Wells III, W.M. (eds.) Uncertainty for Safe Utilization of Machine Learning in Medical Imaging. pp. 103–112. Lecture Notes in Computer Science, Springer Nature Switzerland, Cham (2022)
  * [2] Baur, C., Denner, S., Wiestler, B., Albarqouni, S., Navab, N.: Autoencoders for Unsupervised Anomaly Segmentation in Brain MR Images: A Comparative Study. Medical image analysis (02 Jan 2021), 69:101952 (2021)
  * [3] Cardoso, M.J., et. al.: MONAI: An open-source framework for deep learning in healthcare (Nov 2022), arXiv:2211.02701 [cs]
  * [4] Cho, J., Kang, I., Park, J.: Self-supervised 3D Out-of-Distribution Detection via Pseudoanomaly Generation. In: Aubreville, M., Zimmerer, D., Heinrich, M. (eds.) Biomedical Image Registration, Domain Generalisation and Out-of-Distribution Analysis. pp. 95–103. Lecture Notes in Computer Science, Springer International Publishing, Cham (2022)
  * [5] Isensee, F., Jaeger, P.F., Kohl, S.A.A., Petersen, J., Maier-Hein, K.H.: nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods 18(2), 203–211 (2021), publisher: Nature Publishing Group
  * [6] Li, C.L., Sohn, K., Yoon, J., Pfister, T.: CutPaste: Self-Supervised Learning for Anomaly Detection and Localization. pp. 9664–9674 (2021), https://openaccess.thecvf.com/content/CVPR2021/html/Li˙CutPaste˙Self-Supervised˙Learning˙for˙Anomaly˙Detection˙and˙Localization˙CVPR˙2021˙paper.html
  * [7] Litjens, G., Kooi, T., Bejnordi, B.E., Setio, A.A.A., Ciompi, F., Ghafoorian, M., van der Laak, J.A.W.M., van Ginneken, B., Sánchez, C.I.: A Survey on Deep Learning in Medical Image Analysis. Medical image analysis vol. 42, 60–88 (2017)
  * [8] Naval Marimont, S., Tarroni, G.: Anomaly detection through latent space restoration using vector quantized variational autoencoders. In: 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI). pp. 1764–1767. IEEE (2021)
  * [9] Paszke, A., et. al: PyTorch: An Imperative Style, High-Performance Deep Learning Library. Advances in Neural Information Processing Systems pp. 8024–8035 (2019), publisher: Curran Associates, Inc.
  * [10] Pinaya, W.H.L., Tudosiu, P.D., Gray, R., Rees, G., Nachev, P., Ourselin, S., Cardoso, M.J.: Unsupervised brain imaging 3D anomaly detection and segmentation with transformers. Medical Image Analysis 79, 102475 (Jul 2022)
  * [11] Pérez, P., Gangnet, M., Blake, A.: Poisson image editing. In: ACM SIGGRAPH 2003 Papers, pp. 313–318 (2003)
  * [12] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). vol. 9351 (2015), iSSN: 16113349
  * [13] Schlegl, T., Seeböck, P., Waldstein, S.M., Langs, G., Schmidt-Erfurth, U.: f-AnoGAN: Fast unsupervised anomaly detection with generative adversarial networks. Medical image analysis 54, 30–44 (2019), publisher: Elsevier
  * [14] Schlüter, H.M., Tan, J., Hou, B., Kainz, B.: Natural Synthetic Anomalies for Self-supervised Anomaly Detection and Localization. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) Computer Vision – ECCV 2022. pp. 474–489. Lecture Notes in Computer Science, Springer Nature Switzerland, Cham (2022)
  * [15] Tan, J., Hou, B., Batten, J., Qiu, H., Kainz, B.: Detecting Outliers with Foreign Patch Interpolation. Machine Learning for Biomedical Imaging 1(April 2022 issue), 1–27 (Apr 2022), https://www.melba-journal.org/papers/2022:013.html
  * [16] Tan, J., Hou, B., Day, T., Simpson, J., Rueckert, D., Kainz, B.: Detecting Outliers with Poisson Image Interpolation. In: de Bruijne, M., Cattin, P.C., Cotin, S., Padoy, N., Speidel, S., Zheng, Y., Essert, C. (eds.) Medical Image Computing and Computer Assisted Intervention – MICCAI 2021. pp. 581–591. Lecture Notes in Computer Science, Springer International Publishing, Cham (2021)
  * [17] Wang, L., Zhang, D., Guo, J., Han, Y.: Image Anomaly Detection Using Normal Data Only by Latent Space Resampling. Applied Sciences 10(23), 8660 (Jan 2020), number: 23 Publisher: Multidisciplinary Digital Publishing Institute
  * [18] Yun, S., Han, D., Oh, S.J., Chun, S., Choe, J., Yoo, Y.: Cutmix: Regularization strategy to train strong classifiers with localizable features. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 6023–6032 (2019)
  * [19] Zimmerer, D., Full, P.M., Isensee, F., Jager, P., Adler, T., Petersen, J., Kohler, G., Ross, T., Reinke, A., Kascenas, A., Jensen, B.S., O’Neil, A.Q., Tan, J., Hou, B., Batten, J., Qiu, H., Kainz, B., Shvetsova, N., Fedulova, I., Dylov, D.V., Yu, B., Zhai, J., Hu, J., Si, R., Zhou, S., Wang, S., Li, X., Chen, X., Zhao, Y., Marimont, S.N., Tarroni, G., Saase, V., Maier-Hein, L., Maier-Hein, K.: MOOD 2020: A Public Benchmark for Out-of-Distribution Detection and Localization on Medical Images. IEEE transactions on medical imaging 41(10), 2728–2738 (Oct 2022)
  * [20] Zimmerer, D., Isensee, F., Petersen, J., Kohl, S., Maier-Hein, K.: Unsupervised Anomaly Localization using Variational Auto-Encoders. Medical Image Computing and Computer Assisted Intervention – MICCAI 2019. Lecture Notes in Computer Science vol 11767 (2019)