# Technical outlier detection via convolutional variational autoencoder for
the ADMANI breast mammogram dataset

Hui Li  Carlos A. Peña-Solorzano  Susan Wei  Davis J. McCarthy

###### Abstract

The ADMANI datasets (annotated digital mammograms and associated non-image
datasets) from the “Transforming Breast Cancer Screening with AI” programme
(BRAIx) run by BreastScreen Victoria in Australia are multi-centre, large
scale, clinically curated, real-world databases. The datasets are expected to
aid in the development of clinically relevant Artificial Intelligence (AI)
algorithms for breast cancer detection, early diagnosis, and other
applications. To ensure high data quality, technical outliers must be removed
before any downstream algorithm development. As a first step, we randomly
select 30,000 individual mammograms and use Convolutional Variational
Autoencoder (CVAE), a deep generative neural network, to detect outliers. CVAE
is expected to detect all sorts of outliers, although its detection
performance differs among different types of outliers. Traditional image
processing techniques such as erosion and pectoral muscle analysis can
compensate for CVAE’s poor performance in certain outlier types. We identify
seven types of technical outliers: implant, pacemaker, cardiac loop recorder,
improper radiography, atypical lesion/calcification, incorrect exposure
parameter and improper placement. The outlier recall rate for the test set is
$61\%$ if CVAE, erosion and pectoral muscle analysis each select the top $1\%$
images ranked in ascending or descending order according to image outlier
score under each detection method, and $83\%$ if each selects the top $5\%$
images. This study offers an overview of technical outliers in the ADMANI
dataset and suggests future directions to improve outlier detection
effectiveness.

###### keywords:

Anomaly detection ; Breast cancer mammogram ; Convolutional variational
autoencoder ; Image processing techniques ; Outlier detection

††journal: arXiv

[table]capposition=top capbtabboxtable[][]

[aff1]organization=School of Mathematics and Statistics, University of
Melbourne,addressline=813 Swanston Street, Parkville, city=Melbourne,
postcode=3010, state=VIC, country=Australia

[aff2]organization=Bioinformatics and Cellular Genomics, St Vincent’s
Institute of Medical Research,addressline=9 Princes St, Fitzroy,
city=Melbourne, postcode=3065, state=VIC, country=Australia

[aff3]organization=Melbourne Integrative Genomics, University of
Melbourne,addressline=Building 184/30 Royal Parade, Parkville, city=Melbourne,
postcode=3052, state=VIC, country=Australia

Convolutional variational autoencoder (CVAE), supplemented with traditional
image processing techniques (i.e. erosion, pectoral muscle analysis), can
detect a variety of technical outliers present in the ADMANI dataset.

##  1 Introduction

During breast cancer screening and diagnosis, mammogram images should meet
stringent technical image quality criteria [1, 2]. Technical artefacts reduce
image quality, imitate or mask clinical abnormalities, and cause
interpretation errors [3]. For example, blurring artefacts caused by patient
movement during mammogram scanning obscures microcalcifications [4]; some
types of antiperspirant and skin creams used by patients appear as high-
density particles on mammograms and have a similar appearance to
microcalcifications or unusual lesions [3]; implanted medical devices can
obscure part of mammogram, reduce contrast in craniocaudal (CC) and
mediolateral–oblique (MLO) views, and reduce projected breast tissue and
pectoral muscle projection [5]; whereas misread or dead pixels due to detector
problems appear as clusters of microcalcifications [3, 4]. As a result, it is
critical to detect and remove mammograms with technical artefacts prior to any
downstream diagnosis.

Mammograms containing technical artefacts are considered outliers or anomalies
when compared to mammograms without such artefacts. The primary purpose of
outlier detection is to identify data points that differ significantly from
the rest and are suspected to be the product of a separate cause [6]. Outlier
detection is challenging due to outliers’ irregular behaviour and the lack of
uniform criteria for what constitutes an outlier [7, 8, 9]. For example, in
general mammograms of malignant breast are considered as abnormal compared
with mammograms of healthy breast. However, both cancerous and healthy breast
mammograms are considered as normal when abnormal are defined as those with
technical artefacts in this research.

Breast mammogram datasets are large-scale and each image is high-dimensional,
which adds to the task of detecting outliers. Because of the size of the data
set, outlier detection algorithms are required to be both fast and scalable
[10, 11]. The “curse of dimensionality” issue for high-dimensional data causes
distance concentration and the creation of hubs and antihubs [12, 13], which
makes outlier discovery more difficult. To be more specific, it is widely
understood that as dimension grows, distances between pairwise points become
progressively indistinguishable (distance concentration) and every point is
detected as a nearly equally good outlier, even if there are inliers and
outliers. Some research challenges this viewpoint, claiming that particular
sites (such as antihubs) are more likely to be discovered as outliers even
when no actual outliers are expected.

Deep learning is particularly suited to large scale and high dimensional
tasks. During training, the large-scale dataset is split into mini-batches and
fed into the deep neural networks (DNN) for optimization. To tackle the “curse
of dimensionality” issue, the high dimensional data is reduced into low-
dimensional representations in the latent space of the DNN. Compared to
traditional dimensionality reduction algorithms such as principal component
analysis, DNN is able to learn more complex data patterns due to the
hierarchical and non-linear mapping from input to latent representations. The
ability to learn complex patterns is especially important for medical datasets
where the data points are inherently heterogeneous or noisy. In the ADMANI
databases [14], breast mammograms with various types of anomalous technical
artefacts are considered outliers, whereas inliers comprise both healthy and
malignant breasts. The continuous development from healthy to cancer state, as
well as the many cancer sub-types result in a highly heterogeneous composition
for inliers, making modelling the complex normal data distribution more
difficult [9]. The complex or computationally intractable data distributions
learnt by DNN [15] can thus better segregate anomalous data from normal ones.

Because outlier labels are rarely known in advance, unsupervised deep learning
is commonly utilized in outlier detection. Autoencoders (AEs), Variational
Autoencoders (VAEs), Vector Quantised-Variational Autoencoders (VQ-VAEs),
Generative Adversarial Networks (GANs), and one-class neural networks (OC-NNs)
are some of the most often utilized unsupervised DNNs. Convolutional
variational autoencoders (CVAE) are simply VAEs with convolutional layers
required to analyze image data.

AEs encode the inputs into latent representations, which are then
reconstructed to the original input. Autoencoders and its variants are widely
applied in outlier detection [16, 17, 18, 19]. VAEs are deep generative models
with latent variables of which the posterior distribution is parameterized
with neural networks. VAEs were proposed as anomaly detection algorithms by An
and Cho in 2015 and were demonstrated to outperform AEs in an experimental
setting, where anomaly was defined as one class in the MNIST/KDD dataset [20].
More examples of VAEs being used in anomaly identification can be found in
[21, 22, 23, 24]. The reasoning for AE and VAE to detect outliers is that when
AE or VAE is trained on a dataset with no or few outliers, during inference
time it will be more difficult to reconstruct or model an anomalous input,
resulting in larger reconstruction loss[20].

VQ-VAEs combine VAEs with vector quantization to obtain a discrete latent
representation; the prior of the latent representation is learnt to be auto-
regressive [25]. The expressive auto-regressive models of the latent
representation enables state-of-art density estimation in images, and has been
demonstrated to outperform VAEs in both sample-wise and pixel-wise anomaly
detection when applied to brain MR and abdominal CT datasets [26]. GAN-based
anomaly detectors are extensively summarized in the review by Xia et. al. in
2022 [27]. Generally speaking, GAN-based anomaly detection methods use normal
data points to train a feature extractor which maps the data to its latent
representations. During inference time, a latent representation is predicted
for the test data point and an anomaly is determined by comparing the
reconstructed and the test data point through the generator [28],
discriminator [29] or both [30]. In contrast with AE, VAE, VQ-VAE, and GAN
where the latent representations are tailored for reconstruction first and
then applied to traditional anomaly detection methods, OC-NN was proposed [31,
32] in which the latent representations are trained by maximising an objective
function tuned for anomaly detection directly.

The anomalies discovered in the mammography scans that comprise the ADMANI
dataset are the focus of this study. We chose CVAE as the model after
comprehensive considerations of outlier identification performance and neural
network architectural complexity. Although VAE can target different types of
outliers with varying success rates, the performance was supplemented using
simple hand-crafted outlier identification approaches that target specific
kinds of outliers.

##  2 Method

###  2.1 Dataset

The ADMANI (annotated digital mammograms and associated non-image) datasets
from the “Transforming Breast Cancer Screening with AI” programme (BRAIx) run
by BreastScreen Victoria in Australia are investigated in this research. As
the current largest breast mammogram dataset in the world [14], ADMANI
datasets are 1) large scale with specifically 4,410,212 breast mammogram
images from 629,700 clients, 2) longitudinal spanning from 2013 to 2019 with
intention to grow continually in subsequent years, 3) enhanced with associated
client demographic and clinical non-image data. The large clinically curated
real-world ADMANI datasets are established to aid in the development of
clinically relevant Artificial Intelligence (AI) algorithms for breast cancer
detection, early diagnosis, and other applications. Removing images with
technical artefacts is necessary to ensure high data quality for any
downstream AI algorithm development. We randomly choose 30,000 mammogram
images with their related meta data for the technical outlier detection.

###  2.2 CVAE

The image dataset can be represented by an $N\times C\times H\times W$ matrix
$\bm{X}$, where $N$ is the number of images, $C$ is the number of channels per
image, and $H$ and $W$ are the image height and width, respectively. Since the
mammograms present in the ADMANI dataset are gray-scale, $C=1$, and will be
omitted as matrix dimension on latter sections. For each image $\bm{x}_{i}$,
where $i=1,...,N$, the entry $\bm{x}_{i}^{h\times w}$ refers to the intensity
value of the pixel at row $h$ and column $w$, where $h=1,...,H$ and
$w=1,...,W$. The generative model for $\bm{x}_{i}$ is
$p(\bm{x}_{i}|\bm{z}_{i})$, where $p$ is a multivariate Gaussian distribution
with parameters $\bm{\mu}_{i}$ and $\bm{\sigma}_{i}$, and $\bm{z}_{i}$ is a
low-dimensional latent vector for $\bm{x}_{i}$ with a prior distribution
$p(\bm{z}_{i})=\mathcal{N}(\bm{0},\bm{I})$, where $I$ is the identity matrix.
The generative model is shown in Eq. (1).

$\begin{split}\bm{z}_{i}&\sim\mathcal{N}(\mathbf{0},\mathbf{I})\\\
\bm{\mu}_{i}&=f_{\bm{\theta}}(\bm{z}_{i})\\\
\bm{x}_{i}&\sim\mathcal{N}(\bm{\mu}_{i},\bm{\sigma}_{i}),\end{split}$ (1)

where $f_{\bm{\theta}}$ is a deconvolutional neural network (decoder) with
parameters $\bm{\theta}$, mean $\bm{\mu}_{i}$ ($H\times W$ vector), and
variance parameter $\bm{\sigma}$, which is assumed to be the $H\times W$
identity matrix. The spatial location of each pixel in image $\bm{x}_{i}$ is
recovered by the deconvolution process.

The distribution of the posteriors $p(\bm{z}|\bm{x})$ is intractable and could
be approximated by Variational inference (VI), a faster alternative to Markov
chain Monte Carlo (MCMC). Variational inference is a process to find a member
distribution $q$ in a predefined approximated distribution family $Q$ to
satisfy some criteria, and $q\in Q$. For mathematical convenience, the
approximated posterior family $Q$ is assumed to be a multivariate Gaussian
family with diagonal covariance matrix. Since the mean and variance parameters
of $q$ are learnt through an encoder convolutional neural network with weights
$\bm{\phi}$ and input $\bm{x}$, the distribution $q$ is further denoted as
$q_{\bm{\phi}}(\bm{z}|\bm{x})$. The criteria of VI is to find a member
$q_{\bm{\phi}}(\bm{z}|\bm{x})$ which minimizes the Kullback-Leibler (KL)
divergence (D) between $q_{\bm{\phi}}(\bm{z}|\bm{x})$ and the intractable true
posterior $p(\bm{z}|\bm{x})$.

$D(q_{\bm{\phi}}(\bm{z}|\bm{x})||p(\bm{z}|\bm{x}))=\operatorname{\mathbb{E}}_{q_{\bm{\phi}}(\bm{z}|\bm{x})}(\log
q_{\bm{\phi}}(\bm{z}|\bm{x})-\log p(\bm{z}|\bm{x})).$ (2)

After applying the Bayes rule and rearranging, Eq. (2) can be rewritten as:

$\begin{split}\log
p(\bm{x})-D(q_{\bm{\phi}}(\bm{z}|\bm{x})||p(\bm{z}|\bm{x}))&=\operatorname{\mathbb{E}}_{q_{\bm{\phi}}(\bm{z}|\bm{x})}\log
p_{\bm{\theta}}(\bm{x}|\bm{z})-D(q_{\bm{\phi}}(\bm{z}|\bm{x})||p(\bm{z})),\end{split}$
(3)

where $p(\bm{z})$ is the prior. Since
$D(q_{\bm{\phi}}(\bm{z}|\bm{x})||p(\bm{z}|\bm{x}))$ is non-negative, we have

$\log
p(\bm{x})\geq\operatorname{\mathbb{E}}_{q_{\bm{\phi}}(\bm{z}|\bm{x})}\log
p_{\bm{\theta}}(\bm{x}|\bm{z})-D(q_{\bm{\phi}}(\bm{z}|\bm{x})||p(\bm{z})).$
(4)

The right side of the Eq. (4) is called the Evidence Lower Bound (ELBO) for
the log evidence, $\log p(\bm{x})$. Minimizing the KL divergence in Eq. (2) is
equivalent to maximizing the ELBO. Given a training set
$\\{(\bm{x}_{i})\\}_{i=1}^{N}$, define

$\begin{split}\operatorname{ELBO}(\bm{\phi},\bm{\theta})=\frac{1}{N}\sum_{i=1}^{N}\operatorname{\mathbb{E}}_{q_{\bm{\phi}}(\bm{z}_{i}|\bm{x}_{i})}\log
p_{\bm{\theta}}(\bm{x}_{i}|\bm{z}_{i})-\frac{1}{N}\sum_{i=1}^{N}D(q_{\bm{\phi}}(\bm{z}_{i}|\bm{x}_{i})||p(\bm{z}_{i})).\end{split}$
(5)

The plug-in estimator of the expectation in the reconstruction term is for $L$
samples of $\bm{z}$ from the posterior distribution
$q_{\bm{\phi}}(\bm{z}_{i}|\bm{x}_{i})$ for each $\bm{x}_{i}$. As a standard
practice, the hyperparameter $L$ is set to be 1. Taking together that
$p_{\bm{\theta}}(\bm{x}_{i}|\bm{z}_{i})$ is a multivariate Gaussian with
identity matrix and the analytical form of KL-divergence is available when one
probability distribution is Gaussian with diagonal covariance and the other is
standard Gaussian, we have

$\begin{split}\operatorname{ELBO}(\bm{\phi},\bm{\theta})=&-\frac{H\times
W}{2}\log(2\pi)+\frac{1}{N}\sum_{i=1}^{N}\left\\{\sum_{t=1}^{H\times
W}-\frac{1}{2}\big{(}\bm{x}_{ijt}-(f_{\bm{\theta}}(\bm{z}_{ij})_{t}\big{)}^{2}\right\\}\\\
&+\frac{1}{N}\sum_{i=1}^{N}\left\\{-\frac{1}{2}\sum_{j=1}^{K}\big{(}\sigma_{ij}^{2}+\mu_{ij}^{2}-1-\ln(\sigma_{ij}^{2})\big{)}\right\\},\end{split}$
(6)

where $H$ and $W$ are image height and width, $K$ is the dimension for latent
embeddings, and $\mu_{ij}$ and $\sigma_{ij}$ are mean and variance for
dimension $j$ of latent vector $\bm{z}_{i}$ with posterior
$q(\bm{z}_{i}|\bm{x}_{i})$. The expression $-\frac{H\times W}{2}\log(2\pi)$ is
constant and can be omitted. The second term of ELBO in Eq. 6 is referred to
as negative Reconstruction loss, while the third term is referred to as
negative KLD loss.

###  2.3 CVAE at various depths

We compared two CVAE architectures, VanillaCVAE [33] and ResNetCVAE [34].
VanillaCVAE is the architecture when VAE was first introduced by Kingma et.al.
in 2013 and it only has 5 layers for both encoder and decoder. ResNetCVAE is
deeper than VanillaCVAE with 18 layers for both encoder and decoder. To avoid
degradation of training accuracy for the deep ResNetCVAE, the ordinary layers
are reformulated as learning residual functions with reference to the layer
inputs by adding identity shortcut connections [35].

For both VanillaCVAE and ResNetCVAE, we used grid-search among 24 alternatives
to get the best configuration. These 24 configuration options are determined
empirically. Because greater ELBO is not a consistent criterion across
different setups, the ideal hyper-parameters were chosen by visually
inspecting the generated output image. Table 1 shows the grid values and the
ideal configuration that was chosen. For all the 24 configurations, the ADMANI
dataset was split into train, valid and test at a ratio 0.6:0.1:0.3 and the
neural network was trained with Adam optimiser, a learning rate 0.0005, batch
size 64 for 100 epochs. The generated image is deemed to be with satisfactory
quality for the optimal VanillaCVAE and ResNetCVAE selections (see Figure S1
in supplementary file).

| resize height | resize width | number of output channels | latent dimension
---|---|---|---|---
Grid values | 256, 512 | 256 | 8, 16, 32 | 128, 256, 512, 1024
Optimal VanillaCVAE | 512 | 256 | 8 | 512
Optimal ResNetCVAE | 256 | 256 | 8 | 256

Table 1: Tune hyperparameters for VanillaCVAE and ResNetCVAE. Resize means the
resize transformation for the input image. The number of output channels is
for the output from the first hidden layer in encoder. The number of input
channels for the first layer in encoder is 1 since the breast mammogram is
grayscale. Latent dimension is the dimension of the latent representation for
each image.

###  2.4 Outlier scores

We defined 15 outlier scores, which are grouped into three categories: 1)
associated with the generative loss, 2) associated with the latent
representation, and 3) a mix of the generative loss and the latent
representation. For all of the 15 outlier scores, smaller values indicate
outliers and are assigned numerals for ease of reference, as shown in Table 2.
The following section provides a detailed explanation of the outlier scores.

First category | Second category | Third category
---|---|---
|  | Reconstruction loss | KLD | ELBO
Reconstruction loss(1) | latent IF(4) | Reconstruction latent IF(7) | KLD latent IF(10) | ELBO latent IF(13)
KLD(2) | latent LOF(5) | Reconstruction latent LOF(8) | KLD latent LOF(11) | ELBO latent LOF(14)
ELBO(3) | latent OCSVM(6) | Reconstruction latent OCSVM(9) | KLD latent OCSVM(12) | ELBO latent OCSVM(15)

Table 2: Abbreviations for outlier scores.There are a total of 15 outlier
scores, each with its own reference number.

In the first category, there are three outlier scores: Reconstruction loss,
KLD and ELBO (see Eq. 6). For Reconstruction loss and KLD loss, we actually
use their negative as the outlier scores such that outliers have smaller
values than inliers, consistent with ELBO. CVAE, trained on a dataset with no
or few outliers, is more difficult to model an anomalous input during
inference time, resulting in lower negative Reconstruction loss[20]. Recent
research compared the performance of Reconstruction loss, KLD loss, and ELBO
for outlier detection and discovered that Reconstruction loss presents lower
discriminative power than either KLD loss or ELBO [36].

In the second category, there are three outlier scores. We obtain a low-
dimensional latent vector for each image (sampled from the approximated
posterior distribution for that image). The latent vectors are then passed
into three traditional unsupervised outlier detection techniques, namely
Isolation Forest (IF), Local Outlier Factor (LOF), and One-Class Support
Vector Machine (OCSVM). We briefly summarize these three methods now.

The IF model [37] presupposes that anomalies are infrequent and distinct in
the feature space. It is based on decision trees, in which data points are
partitioned in each step by a random splitting value of a randomly selected
characteristic. Outliers, due to their distinct values away from the more
frequent inliers, are discovered to be partitioned closer to the root of the
tree (i.e. shorter average path length). LOF [38] computes the density
deviation of a given data point with respect to its local k-nearest neighbors,
where $k$ is a predefined hyperparameter. Outliers are samples that have a
significantly lower density compared to their surrounding neighbours. OCSVM
[39] calculates a decision boundary, and any data points that fall beyond of
that threshold are considered outliers. We use the IsolationForest,
LocalOutlierFactor and OneClassSVM functions in sklearn package. Except for
the one indicating the outlier ratio, which was set at $0.5\%$ based on prior
experience with the ADMANI dataset, the default hyperparameters were used.

In the third category, the Reconstruction loss, KLD loss, and ELBO are each
added as an extra dimension to the latent vector. The newly created vectors
are then fed into IF, LOF, and OCSVM, resulting in $3\times 3=9$ outlier
scores. The reasoning for this is that the generative loss and the latent
vector may provide complementary information for outlier detection [40].

###  2.5 Receiver operating characteristic curve and precision-recall curve

In this section we discuss the evaluation metrics for outlier detection
performance of various outlier scores. Based on the confusion matrix, a
contingency table that show the number of correct predictions and incorrect
predictions per class (see Table 3), the definitions for true positive rate
(TPR), false positive rate (FPR), precision, recall and F1-score are shown
below.

  1. 1.

$\textrm{TPR}=\textrm{Recall}=\frac{\textrm{true positives}}{\textrm{true
positives}\textrm{ + }\textrm{false negatives}}$;

  2. 2.

$\textrm{FPR}=\frac{\textrm{false positives}}{\textrm{false positives}\textrm{
+ }\textrm{true negatives}}$;

  3. 3.

$\textrm{Precision}=\frac{\textrm{true positives}}{\textrm{true
positives}\textrm{ + }\textrm{false positives}}$;

  4. 4.

$\textrm{F1-score}=\frac{2\times\textrm{precision}\times\textrm{recall}}{\textrm{precision}\textrm{
+ }\textrm{recall}}$.

|  | True outliers |  | True inliers
---|---|---|---|---
Predicted Outliers |  | Number of true positives |  | Number of false positives
Predicted inliers |  | Number of false negatives |  | Number of true negatives

Table 3: Contingency table between true and predicted outlier labels..

The receiver operating characteristic (ROC) curve is a graphical
representation of a binary classifier’s trade-off between TPR and FPR. Because
the ratio of outliers to inliers is so unbalanced, the ROC curve can provide
an excessively optimistic impression of the classifier’s effectiveness [41].
When there is a substantial skew in the class distribution, the precision-
recall curve is a more informative option. The area under the ROC curve is
referred to as AUROC, while the area under the precision-recall curve is
referred to as AUPRC. Higher values for both AUROC and AUPRC indicate greater
performance. The baseline for AUROC is 0.5 or random guessing, while for
AUPRC, it is the proportion of true outliers in the data set, which is
approximately 0.005. Note that the baseline for AUPRC of each outlier type is
also set to be approximately 0.005 since we get a random sample of inliers
(i.e. mammogram without any technical artefacts) to make the ratio of one
particular outlier type remain 0.005. The maximum of a list of
precision/recall/F1-score given a list of thresholds are also provided.

###  2.6 Preprocessing steps for the ADMANI dataset

Mammograms with breast implants differ significantly from other scans and are
regarded as outliers. Because we already know if a breast image contains
implants or not, we removed them from the ADMANI dataset. We then removed
images where implants can be inferred based on the image manufacturer’s
algorithm attribute, leaving just images with the most commonly used methods.
Following this sequence of filtering procedures, the ADMANI dataset’s 30,000
images were reduced to 29,248.

Because nuisance variables contribute to image dissimilarities, reducing
outlier identification systems’ effectiveness [9], we began by removing the
text artefacts from each image in the remaining ADMANI dataset. In each image,
a minimal bounding rectangle surrounding the breast was clipped off and padded
to a height/width ratio of two. This ratio has been chosen to minimize the
padding required after cropping for this particular dataset. To further
eliminate unwanted factors, left and right breast scans were normalized by
mirroring all right breast images(see Figure S2 in supplementary file).

###  2.7 Using true outliers in the ADMANI dataset for reference

To determine the reference true outliers in the ADMANI dataset, we combined
deep learning with a well-trained radiologist.and obtained two reference
lists, the union of which were confirmed as the final reference. To get the
first reference list,

  1. 1.

Train ResNetCVAE using the best configuration (see Section 2.3) on the
preprocessed ADMANI dataset (see Section 2.6).

  2. 2.

Choose 600 images with the lowest scores for each of the 15 outlier metrics
and obtain a union of the selected images.

  3. 3.

Have a non-radiologist choose true outliers from the union 111At this point,
the goal is to incorporate as many outliers as possible, even if they may be
false positives. and remove the selected true outliers from the ADMANI
dataset.

  4. 4.

Train ResNetCVAE again on the leftover ADMANI dataset and repeat steps 2-3.

  5. 5.

Collect any true outliers identified by the non-radiologist in the two
iterations and refer them to a professional radiologist for evaluation.

To get the second reference list,

  1. 1.

Using the preprocessed ADMANI dataset (see Section 2.6), train ResNetCVAE with
the best setup (see Section 2.3).

  2. 2.

Choose the 200 images with the lowest scores for each of the 15 outlier
metrics and get a union of all of the selected images.

  3. 3.

Remove the union of all selected images from the ADMANI dataset and train
ResNetCVAE again on the leftover dataset.

  4. 4.

Repeat step 2).

  5. 5.

Combine the two removed unions and determine potential true outliers by a non-
radiologist, which is further confirmed by a professional radiologist.

The union of the two reference lists contains 136 true outliers, and the
outlier ratio in the ADMANI dataset is $136/29248=0.465\%$.

The 136 true outliers are classified into seven categories by the professional
radiologist: implant, pacemaker, cardiac loop recorder, improper radiography,
atypical lesion/calcification, incorrect exposure parameter and improper
placement. It is worth noting that, despite the fact that mammograms with
implants were eliminated during the preprocessing step (see Section 2.6),
there were still mammograms with implants remaining since they were
incorrectly categorised as having no implants. Figure 1 depicts representative
outliers for each category, and Table. 4 displays the number and percentage of
outliers in each category given that that the total number of true outliers is
136.

Figure 1: Examples of true outlier subgroups in the ADMANI dataset. From left
to right and top to bottom, the outlier types are: implant, pacemaker, cardiac
loop recorder, improper radiography, improper radiography, atypical
lesions/calcification, incorrect exposure parameter, improper placement.
Improper radiography is classified into two subtypes: those with heterogeneous
pectoral muscle and the rest, respectively.

Outlier type | Quantity | Percentage
---|---|---
Implants | 36 | 0.265
Pacemaker | 25 | 0.184
Cardiac loop recorder | 6 | 0.044
Improper radiography subtype 1 | 38 | 0.279
Improper radiography subtype 2 | 18 | 0.132
Atypical lesions/calcifications | 9 | 0.066
Incorrect exposure parameter | 2 | 0.0147
Improper placement | 2 | 0.015

Table 4: Frequency table for all the outlier subgroups. Quantity and
percentage of images for each outlier category.

###  2.8 Hand-crafted outlier detection methods

#### Erosion

We designed hand-crafted techniques in 5 phases to discover outliers that
produce bright regions in the image, such as pacemakers, cardiac loop
recorders, unusual lesions, and implants: 1) preprocess images (see Section
2.6), 2) threshold all images by a predetermined value to produce binary
counterparts, 3) erode binary images to remove scattered signals, 4) for each
image, add the sums of all pixels, 5) Sort the image pixel sums in descending
order and choose the top $1\%$, $2\%$ and $5\%$ images.

In total 16 configurations of hyperparameters were searched, i.e. 4 thresholds
(180, 200, 220 and 240) cross 2 kernel sizes (5, 10) cross 2 iteration numbers
(5, 10) for erosion. Supplementary Table S1 displays the mean recall rate for
both training and testing set if $1\%$, $2\%$ and $5\%$ images are picked. The
training and testing set are split once at a ratio of 0.6:0.4. The mean recall
rate is averaged over 20 bootstraps of either training or testing set.

#### Pectoral Muscle Analysis

To discover outliers with highly heterogeneous pectoral muscle regions 222The
pectoral muscle can only be observed in the mediolateral oblique view position
(MLO).: 1) preprocess images (see Section 2.6), 2) extract the pectoral muscle
region, 3) count major lines in the muscle region, with greater line counts
indicating outliers with more heterogeneous pectoral muscle regions.

To extract the pectoral muscle region in step 2, we specifically followed the
procedures in [42]: a) Resize the preprocessed images (see Section 2.6) to
$256\times 256$, b) Use canny edge detection to obtain strong edges in the
image. Canny edge detection employs a Gaussian filter with $\sigma=5$ to
remove noise, $3\times 3$ Sobel kernels to calculate intensity gradient, non-
maximum suppression to suppress pixels with no maximum gradient along the edge
direction, and hysteresis thresholding to pick pixels with significant
intensity gradient (pixels on strong edges) 333Use the default lower and
higher hysteresis thresholds for the skimage.feature.canny function, which are
respectively $10\%$ and $20\%$ of the maximum gradient intensity for each
image., c) apply the Hough line transformation to the canny edges and identify
lines that are parameterized in angles and distances with respect to the
center of the image. The lines with distances in the range
$lowerdistance,upperdistance]$ and angles in the range
$(10^{\circ},70^{\circ})$ were recorded, and the line with the shortest
distance was chosen as the muscle border and guides the removal of the
pectoral muscle. We tried two choices for the lower distance, 5 and 20, and
one option for the upper distance, 182.

In step 3, we used canny edge detection and the Hough line transform to count
the principal lines in the pectoral muscle region. The aperture size for the
Sobel operator for canny edge detection was 3; we tested 160 and 170 for lower
thresholds, 180 and 220 for upper thresholds in hysteresis thresholding. We
utilised the OpenCV library for the Hough line transformation, with $\rho$ set
to 1, $\theta$ set to $\pi/180$ [42], and selection thresholds $40,50,60$. The
images were then ranked in descending order by the number of lines in their
pectoral muscle regions, with the top $1\%$, $2\%$, and $5\%$ images chosen as
probable outliers. Images with line numbers greater than 8 were not considered
in the ranking since the cut regions for these images from step 2 include not
only pectoral muscle but also breast.

In total 24 configurations of hyperparameters were searched, i.e. 2 lower
distances for muscle boundary selection in step 2 cross 2 lower thresholds
cross 2 upper thresholds cross 3 selection thresholds in step 3. The mean
recall rate is shown in Supplementary Table S2 if the top $1\%$, $2\%$ and
$5\%$ images are chosen when they are ranked in descending order by number of
lines in their pectoral muscle regions. The training and testing set are split
once at the ratio of 0.6 to 0.4. The mean is calculated over 20 bootstraps for
either the training or testing datasets.

##  3 Experiments

###  3.1 Outlier detection using CVAE

We used CVAE to find technical outliers in the preprocessed ADMANI dataset
(Section 2.6). Figure 2 depicts the performance of 15 outlier scores for both
training and testing sets using two distinct CVAE architectures (VanillaCVAE
and ResNetCVAE, see Section 2.3). Both neural networks were trained only once,
but the metrics (AUROC/AUPRC/Max Precision/Max Recall/Max F1 Score, see
Section 2.5) were derived by averaging over 20 bootstraps for both the
training and testing sets. To ensure consistent outlier composition (outlier
types and ratios), we sampled with replacement inliers and each outlier type
separately for both training and testing set before combining them. The
maximum precision/recall/F1 score is the highest value of a list of mean
precision/recall/F1 scores across a range of threshold ratios, which includes
0.06% to 0.2% with interval 0.02%, 0.2% to 0.8% with interval 0.1%, 0.8% to
12% with interval 0.2%, and 12% to 30% with interval 2%.

In Figure 2, VanillaCVAE and ResNetCVAE do not detect outliers statistically
differently for any of the outlier scores. Since ResNetCVAE is more complex
and takes longer to train, we employed VanillaCVAE in the following study.
Although the AUROC is relatively high, the maximum AUPRC across the 15 outlier
scores in the test set by VanillaCVAE is 0.166, suggesting the need for
outlier detection performance enhancement. We begin by investigating the
possible causes of low AUPRC and then assess the performance of each outlier
type.

Figure 2: Outlier detection performance for all 15 outlier scores. For both
VanillaCVAE and ResNetCVAE, the performance of all 15 outlier scores to detect
outliers in the ADMANI dataset is shown in metrics: AUROC, AUPRC, max
precision, max recall and max F1 score. To show the details of AUPRC and max
F1 score for each outlier score, the y-axis has been reduced to the range [0,
0.4]. The number and name for each outlier score can be referred to Table. 2.
There is not statistically significant difference between VanillaCVAE and
ResNetCVAE. Although latent OCSVM(number 6) is the outlier scores with top
AUPRC, overall speaking, the AUPRC is not high enough.

The AUPRC of the 15 outlier scores for each outlier type is shown in Figure 3
and supplementary Figure S3. As expected, VanillaCVAE performs differently in
different outlier types: it has higher maximum AUPRC (across all 15 outlier
scores in the test set) for outliers with implants (maximum AUPRC: 0.442),
pacemaker (maximum AUPRC: 0.349) (Figure 3), incorrect exposure parameter
(maximum AUPRC: 1.0) and improper placement (maximum AUPRC: 0.687)
(supplementary Figure S3); however lower maximum AUPRC in detecting outliers
with cardiac loop recorder (maximum AUPRC: 0.028), improper radiography
(maximum AUPRC: 0.066) (Figure 3) and atypical lesions/calcification (maximum
AUPRC: 0.155)(supplementary Figure S3).

For outlier types with high AUPRC, different outlier scores behave
differently: latent OCSVM has greater AUPRC than reconstruction loss for
outliers with implants and pacemakers (Figure 3), whereas reconstruction loss
is better for outliers with improper placement (supplementary Figure S3). KLD
has similar performance as latent OCSVM for outliers with implants (Figure 3)
and improper placement (supplementary Figure S3), but similar performance as
Reconstruction loss for outliers with pacemaker (Figure 3), which suggests
that KLD loss may contain extra information for outlier detection which is
also consistent with the fact that KLD loss is mathematically different from
Reconstruction loss and latent OCSVM (see Section 2.4). In this study, for
other outlier scores (numbers 7 to 15), adding Reconstruction loss, KLD loss,
and ELBO as extra dimensions to the latent vector that are then applied to
traditional outlier detection methods (e.g. IF, LOF, OCSVM, see Section 2.4)
does not help to improve outlier detection performance.

Figure 3: AUPRC of outlier scores for each specific outlier category. For each
outlier category, the AUPRC across all 15 outlier scores is displayed. The
number on the x-axis corresponds to outlier scores in Table 2. The ensb1 and
ensb2 values are the average and minimum ensemble of three best performing
outlier scores: Reconstruction loss, KLD and latent OCSVM. To show the AUPRC
details for each outlier score, the y-axis for the cardiac loop recorder and
improper radiology has been reduced to the range [0, 0.12] and [0, 0.25].
Implants and pacemakers are discovered at a higher rate than cardiac loop
recorder and improper radiography. Different outlier scores work differently
in recognising each outlier type. In outlier detection, the average ensemble
(ensb1) is comparable to or better than the minimal ensemble (ensb2).

Given that no outlier score excels for all outlier categories, we selected
Reconstruction loss, KLD, and latent OCSVM and combined the three into a
single indicator. The first ensemble takes the average of the three, whereas
the second takes the smallest of the three. Because the three outlier scores
are on different scales, they were first min-max normalised before being
ensembled. Figures 3 and S3 demonstrate how the two ensembles (ensb1 and
ensb2) perform for each outlier type. The average ensemble, in particular, is
either comparable or better than the minimal ensemble at findings outliers,
and will be utilised as the sole outlier indicator in the following study.

###  3.2 Improve outlier detection performance using CVAE

In the previous section, we examined CVAE’s outlier detection ability on the
ADMANI dataset. The benefit of CVAE is that it detects a variety of outlier
types without prior expert knowledge and has satisfactory performance in some
outlier kinds (Figures 3 and S3). Poorly detected outlier types can be easily
discovered using CVAE’s outlier type information and specific hand-crafted
image processing approaches.

CVAE, as previously observed, performs poorly in detecting outliers with
cardiac loop recorder, improper radiography and atypical
lesions/calcifications (Figures 3 and S3). The presence of bright regions
(unrelated to cancer tissue) is a common feature of these poorly recognized
outliers (Figure 1); as a result, it is natural to utilise the presence of
such regions as a signal of outliers. The major steps are shown in Figure 4
where an outlier image due to improper radiography (Figure 4A) was first
converted to a white-black binary image thresholded by some fixed pixel value
(Figure 4B), and then eroded such that only the bright region remains (Figure
4C). The image is more likely to be an outlier if the remaining region is
larger. Detailed methodology can be found in Section 2.8.

A

B

C

D

E

F

Figure 4: Image processing techniques for outlier detection. Panels A, B and C
show the image erosion process for outlier detection. A fixed threshold
transforms the original outlier in Panel A into the binary image in Panel B.
Panel B’ scattered signal gets eroded, as demonstrated in Panel C. Outliers
present a large bright region in Panel C. Panel D, E, and F show how to
perform pectoral muscle analysis for outlier detection. Panel E removes the
pectoral muscle region of the original outlier in Panel D, and Panel F
displays the major lines in the pectoral muscle region. A higher number
denotes an anomaly.

There is a unique subgroup of improper radiography outliers with very diverse
pectoral muscle region. Such an outlier is seen in Figure 4D, and it is found
by first cutting the pectoral muscle region out (Figure 4E) and then counting
the major line number in the pectoral muscle region (Figure 4F). The greater
the number, the greater the likelihood that the image is an outlier. Detailed
information can be found in Section 2.8.

Table 5 shows the true reference outlier number as well as the outlier
detection performance by VanillaCVAE, erosion and pectoral muscle analysis for
both the training and testing set. The results for the VanillaCVAE is the
outlier recall rate if the top $1\%$, $2\%$, or $5\%$ images ranked in
ascending order by the average ensemble outlier score (see Section 3.1)) are
selected; the results for erosion are the recall rate if VanillaCVAE is
complemented by erosion where erosion selects the top $1\%$, $2\%$ or $5\%$
images ranked in descending order by pixel sums (see Section 2.8); and the
results for the pectoral muscle analysis are the recall rate if VanillaCVAE
and erosion are complemented by pectoral muscle analysis where pectoral muscle
analysis selects the top $1\%$, $2\%$ and $5\%$ images ranked in descending
order by line numbers (see Section 2.8). VanillaCVAE is trained only once and
the train, valid and test split ratio is 0.6:0.1:0.4. All percentages are mean
values across 20 bootstraps for both the training and testing dataset
evaluated by the trained VanillaCVAE.

Erosion significantly increases the recall rate in addition to VanillaCVAE,
while pectoral muscle analysis moderately boosts the recall rate in addition
to VanillaCVAE and erosion. VanillaCVAE’s recall rate in the test set is
$0.31\pm 0.06$ if the top $1\%$ images are chosen. When VanillaCVAE is
complemented by erosion, the recall rate is $0.56\pm 0.07$, with $0.25$ recall
rate increase. When VanillaCVAE and erosion is complemented further by
pectoral muscle analysis, the recall rate is $0.61\pm 0.07$, with $0.05$
recall rate increase. If VanillaCVAE, erosion, and pectoral muscle analysis
each select $5\%$ images, the recall rate rises to $83\%$.

Reference Number | 1% | 2% | 5%
---|---|---|---
VanillaCVAE | Erosion | Pectoral Muscle Analysis | VanillaCVAE | Erosion | Pectoral Muscle Analysis | VanillaCVAE | Erosion | Pectoral Muscle Analysis
Train
87 | $0.35\pm 0.04$ | $0.55\pm 0.03$ | $0.55\pm 0.03$ | $0.49\pm 0.05$ | $0.7\pm 0.05$ | $0.72\pm 0.05$ | $0.63\pm 0.06$ | $0.84\pm 0.05$ | $0.86\pm 0.05$
Test
39 | $0.31\pm 0.06$ | $0.56\pm 0.07$ | $0.61\pm 0.07$ | $0.37\pm 0.07$ | $0.64\pm 0.07$ | $0.69\pm 0.07$ | $0.54\pm 0.06$ | $0.75\pm 0.06$ | $0.83\pm 0.06$

Table 5: Outlier detection performance using VanillaCVAE, erosion, and
pectoral muscle analysis. In this table, for both training and testing
dataset, we show the reference true outlier number, the recall rate by
VanillaCVAE if the top $1\%$, $2\%$ or $5\%$ images ranked in ascending order
by average ensembled outlier score are selected, the recall rate if
VanillaCVAE is complemented by erosion where erosion selects the top $1\%$,
$2\%$ or $5\%$ images ranked in descending order by pixel sums (see Section
2.8), and the recall rate if VanillaCVAE and erosion is complemented by
pectoral muscle analysis where the pectoral muscle analysis selects the top
$1\%$, $2\%$ or $5\%$ images ranked in descending order by line numbers (see
Section 2.8).

##  4 Conclusions and future work

As a first step, we use deep learning and traditional image processing
approaches to detect technical outliers in the ADMANI dataset, which contains
breast cancer mammograms. Because there are no outlier labels available, we
utilize CVAE, an unsupervised deep learning method, to scan the dataset first
and learn about the types of outliers. CVAE performs differently for different
outlier types, and classic image processing techniques like erosion and
pectoral muscle analysis supplement CVAE in recognizing specific outlier types
that it fails to detect. We recall $56\%$ of outliers if VanillaCVAE, erosion
and pectoral muscle analysis each select $1\%$ images of the test set, and
$81\%$ if each select $5\%$ images.

The reference true outliers in this research are obtained by deep learning
screening with relaxed criteria first, then selected by a unprofessional
research staff and finally confirmed by a professional radiologist. It only
includes outlier types with obvious technical artefacts such as implanted
medical devices but excludes those easily ignored by either deep learning or
the unprofessional research staff such as antiperspirant, dust artefacts due
to dust on compression paddle etc. Participation of professional radiologists
to examine each image in the whole dataset will provide a more inclusive
reference list. Professional expert knowledge and traditional image processing
techniques may have potential to detect technical outliers that are not
detected in this research.

For outlier types that could be discovered by deep learning, increasing the
detection performance is a remaining task in the future. One direction is to
try novel and efficient outlier score for VAE, such as likelihood regret which
measures the log likelihood improvement of VAE configuration that maximizes
the likelihood of an individual sample over the configuration that maximizes
the likelihood of the whole dataset [43]. A second direction is to obtain
better latent representations using algorithms such as VQ-VAE (see sec. 1), or
obtaining latent representations optimizing the objective of outlier detection
directly such as OC-NN (see sec. 1) or the combination of VQ-VAE and OC-NN.
With the accumulation of more outlier labels determined by unsupervised
learning, combined with data augmentation to increase outlier variation and
proportion, the technical outlier detection problem in the ADMANI dataset
could gradually change into supervised learning.

## References

  * [1] N. Perry, M. Broeders, C. de Wolf, S. Törnberg, R. Holland, L. von Karsa, European guidelines for quality assurance in breast cancer screening and diagnosis. -summary document, Oncology in Clinical Practice 4 (2) (2008) 74–86.
  * [2] C. van Landsveld-Verhoeven, The right focus: manual on mammography positioning technique, LRCB, 2015.
  * [3] W. R. Geiser, T. M. Haygood, L. Santiago, T. Stephens, D. Thames, G. J. Whitman, Challenges in mammography: part 1, artifacts in digital mammography, American Journal of Roentgenology 197 (6) (2011) W1023–W1030.
  * [4] R. S. Ayyala, M. Chorlton, R. H. Behrman, P. J. Kornguth, P. J. Slanetz, Digital mammographic artifacts on full-field systems: what are they and how do i fix them?, Radiographics 28 (7) (2008) 1999–2008.
  * [5] E. Paap, M. Witjes, C. van Landsveld-Verhoeven, R. M. Pijnappel, A. H. Maas, M. J. Broeders, Mammography in females with an implanted medical device: impact on image quality, pain and anxiety, The British Journal of Radiology 89 (1066) (2016) 20160142.
  * [6] D. M. Hawkins, Identification of outliers, Vol. 11, Springer, 1980.
  * [7] R. Chalapathy, S. Chawla, Deep learning for anomaly detection: A survey, arXiv preprint arXiv:1901.03407 (2019).
  * [8] T. Fernando, S. Denman, D. Ahmedt-Aristizabal, S. Sridharan, K. R. Laurens, P. Johnston, C. Fookes, Neural memory plasticity for medical anomaly detection, Neural Networks 127 (2020) 67–81.
  * [9] T. Fernando, H. Gammulle, S. Denman, S. Sridharan, C. Fookes, Deep learning for medical anomaly detection–a survey, ACM Computing Surveys (CSUR) 54 (7) (2021) 1–37.
  * [10] A. Koufakou, Scalable and efficient outlier detection in large distributed data sets with mixed-type attributes, University of Central Florida, 2009.
  * [11] S. Thudumu, P. Branch, J. Jin, J. J. Singh, A comprehensive survey of anomaly detection techniques for high dimensional big data, Journal of Big Data 7 (1) (2020) 1–30.
  * [12] M. Radovanovic, A. Nanopoulos, M. Ivanovic, Hubs in space: Popular nearest neighbors in high-dimensional data, Journal of Machine Learning Research 11 (sept) (2010) 2487–2531.
  * [13] M. Radovanović, A. Nanopoulos, M. Ivanović, Reverse nearest neighbors in unsupervised distance-based outlier detection, IEEE transactions on knowledge and data engineering 27 (5) (2014) 1369–1382.
  * [14] H. M. Frazer, J. S. Tang, M. S. Elliott, K. M. Kunicki, B. Hill, R. Karthik, C. F. Kwok, C. A. Peña-Solorzano, Y. Chen, C. Wang, et al., Admani: Annotated digital mammograms and associated non-image datasets, Radiology: Artificial Intelligence 5 (2) (2022) e220072.
  * [15] Y. Lu, J. Lu, A universal approximation theorem of deep neural networks for expressing probability distributions, Advances in neural information processing systems 33 (2020) 3094–3105.
  * [16] S. Hawkins, H. He, G. Williams, R. Baxter, Outlier detection using replicator neural networks, in: International Conference on Data Warehousing and Knowledge Discovery, Springer, 2002, pp. 170–180.
  * [17] T. Tagawa, Y. Tadokoro, T. Yairi, Structured denoising autoencoder for fault detection and analysis, in: Asian conference on machine learning, PMLR, 2015, pp. 96–111.
  * [18] R. Chalapathy, A. K. Menon, S. Chawla, Robust, deep and inductive anomaly detection, in: Joint European Conference on Machine Learning and Knowledge Discovery in Databases, Springer, 2017, pp. 36–51.
  * [19] Z. Chen, C. K. Yeo, B. S. Lee, C. T. Lau, Autoencoder-based network anomaly detection, in: 2018 Wireless telecommunications symposium (WTS), IEEE, 2018, pp. 1–5.
  * [20] J. An, S. Cho, Variational autoencoder based anomaly detection using reconstruction probability, Special Lecture on IE 2 (1) (2015) 1–18.
  * [21] Y. Lu, P. Xu, Anomaly detection for skin disease images using variational autoencoder, arXiv preprint arXiv:1807.01349 (2018).
  * [22] D. Zimmerer, S. A. Kohl, J. Petersen, F. Isensee, K. H. Maier-Hein, Context-encoding variational autoencoder for unsupervised anomaly detection, arXiv preprint arXiv:1812.05941 (2018).
  * [23] P. Matias, D. Folgado, H. Gamboa, A. V. Carreiro, Robust anomaly detection in time series through variational autoencoders and a local similarity score., in: Biosignals, 2021, pp. 91–102.
  * [24] H. Xu, W. Chen, N. Zhao, Z. Li, J. Bu, Z. Li, Y. Liu, Y. Zhao, D. Pei, Y. Feng, et al., Unsupervised anomaly detection via variational auto-encoder for seasonal kpis in web applications, in: Proceedings of the 2018 world wide web conference, 2018, pp. 187–196.
  * [25] A. Van Den Oord, O. Vinyals, et al., Neural discrete representation learning, Advances in neural information processing systems 30 (2017).
  * [26] S. N. Marimont, G. Tarroni, Anomaly detection through latent space restoration using vector quantized variational autoencoders, in: 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), IEEE, 2021, pp. 1764–1767.
  * [27] X. Xia, X. Pan, N. Li, X. He, L. Ma, X. Zhang, N. Ding, Gan-based anomaly detection: a review, Neurocomputing (2022).
  * [28] D. Li, D. Chen, J. Goh, S.-k. Ng, Anomaly detection with generative adversarial networks for multivariate time series, arXiv preprint arXiv:1809.04758 (2018).
  * [29] J. Donahue, P. Krähenbühl, T. Darrell, Adversarial feature learning, arXiv preprint arXiv:1605.09782 (2016).
  * [30] T. Schlegl, P. Seeböck, S. M. Waldstein, U. Schmidt-Erfurth, G. Langs, Unsupervised anomaly detection with generative adversarial networks to guide marker discovery, in: Information Processing in Medical Imaging: 25th International Conference, IPMI 2017, Boone, NC, USA, June 25-30, 2017, Proceedings, Springer, 2017, pp. 146–157.
  * [31] R. Chalapathy, A. K. Menon, S. Chawla, Anomaly detection using one-class neural networks, arXiv preprint arXiv:1802.06360 (2018).
  * [32] L. Ruff, R. Vandermeulen, N. Goernitz, L. Deecke, S. A. Siddiqui, A. Binder, E. Müller, M. Kloft, Deep one-class classification, in: International conference on machine learning, PMLR, 2018, pp. 4393–4402.
  * [33] D. P. Kingma, M. Welling, Auto-encoding variational bayes, arXiv preprint arXiv:1312.6114 (2013).
  * [34] Github, Vae-resnet18-pytorch (2019).
URL https://github.com/julianstastny/VAE-ResNet18-PyTorch

  * [35] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  * [36] D. Zimmerer, F. Isensee, J. Petersen, S. Kohl, K. Maier-Hein, Unsupervised anomaly localization using variational auto-encoders, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2019, pp. 289–297.
  * [37] F. T. Liu, K. M. Ting, Z.-H. Zhou, Isolation forest, in: 2008 eighth ieee international conference on data mining, IEEE, 2008, pp. 413–422.
  * [38] M. M. Breunig, H.-P. Kriegel, R. T. Ng, J. Sander, Lof: identifying density-based local outliers, in: Proceedings of the 2000 ACM SIGMOD international conference on Management of data, 2000, pp. 93–104.
  * [39] B. Schölkopf, J. C. Platt, J. Shawe-Taylor, A. J. Smola, R. C. Williamson, Estimating the support of a high-dimensional distribution, Neural computation 13 (7) (2001) 1443–1471.
  * [40] F. Angiulli, F. Fassetti, L. Ferragina, Improving deep unsupervised anomaly detection by exploiting vae latent space distribution, in: International Conference on Discovery Science, Springer, 2020, pp. 596–611.
  * [41] J. Davis, M. Goadrich, The relationship between precision-recall and roc curves, in: Proceedings of the 23rd international conference on Machine learning, 2006, pp. 233–240.
  * [42] A. R. Beeravolu, S. Azam, M. Jonkman, B. Shanmugam, K. Kannoorpatti, A. Anwar, Preprocessing of breast cancer images to create datasets for deep-cnn, IEEE Access 9 (2021) 33438–33463.
  * [43] Z. Xiao, Q. Yan, Y. Amit, Likelihood regret: An out-of-distribution detection score for variational auto-encoder, Advances in neural information processing systems 33 (2020) 20685–20696.