# Data refinement for fully unsupervised visual inspection using pre-trained
networks

Antoine Cordier  Scortex, 22 rue Berbier du Mets, Paris, France  Benjamin
Missaoui  Scortex, 22 rue Berbier du Mets, Paris, France  Pierre Gutierrez
Scortex, 22 rue Berbier du Mets, Paris, France

###### keywords:

quality control, visual inspection, deep learning, anomaly detection,
pollution, pre-training

Anomaly detection has recently seen great progress in the field of visual
inspection. More specifically, the use of classical outlier detection
techniques on features extracted by deep pre-trained neural networks have been
shown to deliver remarkable performances on the MVTec Anomaly Detection (MVTec
AD) dataset. However, like most other anomaly detection strategies, these pre-
trained methods assume all training data to be normal. As a consequence, they
cannot be considered as fully unsupervised. There exists to our knowledge no
work studying these pre-trained methods under fully unsupervised setting. In
this work, we first assess the robustness of these pre-trained methods to
fully unsupervised context, using polluted training sets (i.e. containing
defective samples), and show that these methods are more robust to pollution
compared to methods such as CutPaste. We then propose SROC, a Simple
Refinement strategy for One Class classification. SROC enables to remove most
of the polluted images from the training set, and to recover some of the lost
AUC. We further show that our simple heuristic competes with, and even
outperforms much more complex strategies from the existing literature.

##  1 Introduction

Availability of training data holds sway over the applicability of deep
learning. In the field of automated quality control, the scarcity of defective
parts and the high cost of annotation is heading the industry towards anomaly
detection, for which the methods learn to model normality from easily
acquired, healthy (i.e. non-defective) images only. However, implicitely
assuming the data to be healthy has the consequence that the approach cannot
be considered fully unsupervised. In practice, it is always possible for a
human operator to inadvertently gather a few defective parts in his
predominantly healthy training set, unless he spends time to thoroughly
inspect all training parts, which can be a cumbersome task. Indeed, proper
labelling may sometimes only be performed by well-trained operators. In this
work, we tackle the problem of truly unsupervised anomaly detection for visual
inspection, where no labels are available and where the training set may
contain defective samples.

We focus on anomaly detection methods that make use of pre-trained networks,
since these methods have shown remarkable results in recent works. In
particular, we lead comprehensive experiments with four of these methods
(KNN[1], Mahalanobis[2], PaDiM[3], and PatchCore[4]) on the publicly available
MVTec AD dataset[5]. Note that the same pre-trained feature extractor
(EfficientNetB4[6]) is used for all four outlier detectors for fair
comparison. Our work brings a twofold contribution:

  1. 1.

We first assess in section 4.3 the impact of defective training parts on the
performances of these detectors, and show that the latter are already quite
robust to pollution compared to other methods such as CutPaste[7].

  2. 2.

We then propose in section 4.4 a simple yet efficient refinement strategy to
remove polluted samples from the training data. Our method is inspired by
STOC[8], which applies such a refinement strategy to the CutPaste[7] anomaly
detection method. We apply our own simple strategy to the four aforementioned
outlier detection methods. We demonstrate the effectiveness of our method, and
show that it can compete with much more complex heuristics in section 4.5.
Interestingly, our refinement strategy is simple enough to be agnostic of the
chosen anomaly detection method, and could in theory be used with most anomaly
detection methods (e.g. reconstruction or self-supervision based).

Note that we also provide a comprehensive qualitative analysis in order to
better understand the robustness of the pre-trained methods to pollution in
section 4.6.

##  2 Related work

###  2.1 Visual inspection via deep learning

Visual inspection is a task which can be automated via supervised deep
learning [9, 10], in which a convolutional neural network (CNN) (such as U-Net
[11] or RetinaNet[12]) is trained to detect specific defects. Unfortunately,
supervised deep learning requires a high volume of annotated images, and
cannot handle types of defect which were not seen at training time (like
unusual, or theoretical defects). One can either solve the issue in a data-
centric manner by generating more data (for example, with the use of synthetic
approaches [13, 14]), or by shifting the defect detection paradigm to an
anomaly detection one, via the use of semi-supervised or unsupervised
learning. In this work, we focus on anomaly detection via unsupervised
learning.

###  2.2 “Unsupervised” learning for anomaly detection

Anomaly detection has become a very popular field of research lately,
especially with regards to computer vision. We focus this review on works
which are close to our task, and refer the reader to the Pang et al.[15]
review from 2020 for a broader look at the topic. We also refer the reader to
the review of our previous paper,[16] which also dealt with anomaly detection
for visual inspection using pre-trained networks. We group visual anomaly
detection approaches into three main categories: reconstruction of the input,
classification of anomalies and modelling of normality.

Reconstruction of the input is most of the time performed using generative
convolutional networks, such as auto-encoders (AEs). They have been among the
first methods to make use of deep learning for anomaly detection[5, 17]. The
main objective of these methods consists in learning to reconstruct input
images using only normal data for training. At inference time, the network is
expected to reconstruct over input anomalies, which allows to compute the
anomaly score as a simple L2 distance between the original input image and its
reconstruction. This approach has been extended via the use of more
appropriate distances for measuring the difference between the input and its
reconstruction, such as structural similarity[18], as well as the use of more
complex generative networks, like variational auto-encoders (VAEs)[19] and
generative adversarial networks (GANs).[20]. Since reconstruction based
methods have the disadvantage of occasionally reconstructing the anomalies,
inpainting techniques have been often used to avoid this phenomenon,[21, 22]
sometimes using vision transformers (ViT) to incorporate global context[23].
We refer the reader to Baur et al.[19] for an extensive overview of these
reconstruction-based methods.

Classification of anomalies is done by leveraging the classification
confidences of a CNN, which can be either trained in a direct supervised
manner via outlier exposure[24], or in a self-supervised fashion. In the first
case, outliers are selected from different distributions, i.e. datasets that
are unrelated to the normal training set. Alternatively, outliers can be
crafted synthetically from the original training images using local
augmentations[25]. Crafting synthetic outliers can be considered as self-
supervised learning, since no manual labelling is required. In CutPaste[7],
the authors propose to transform any normal training image into an anomalous
one by locally incorporating a transformed patch coming from any of the other
training images. This allows to learn finer features, which will be more
specific to the task-at-hand than outliers coming from a different dataset.
This idea has recently been extended by Schlüter et al.[26], who train a
segmenter network with blended synthetic outliers using soft labels. Note that
synthetic outlier exposure can also serve reconstruction-based approaches.[27]
A slightly different approach is for the CNN to learn an auxiliary task, such
as predicting random augmentations applied to the normal input images. In
GeoTrans[28], the chosen augmentations are for example rotations and
translations. The intuition is that at test time, the augmentations of an out-
of-distribution image will not be predicted properly by the model. To some
extent, this can be seen as a way to perform synthetic outlier exposure[24].
However, it should be noted that the performance of such approaches highly
depends on what can be considered as acceptable data augmentations with
regards to the dataset of interest, and thus must be designed by hand. This is
due to the fact that the trained classifiers are by design discriminative
networks, and consequently prone to overfitting to the chosen augmentations.
This was pointed out by Zavrtanik et al.[29], who leverage the predictions of
a discriminative network laid on top of a reconstruction one to avoid
overfitting, and trained via synthetic outlier exposure.

Modelling normality for anomaly detection was initially framed as a one-class
classification problem by the support vector method (SVM) paper in 1999[30].
Since then, the use of deep neural networks for learning compact latent
distributions has considerably improved these methods, notably Deep SVDD[31]
and its pixel-wise extension Patch SVDD [32]. Unfortunately, training such
networks does not lead to discriminative (or rich) enough features for anomaly
detection, and may result in mode collapses under certain conditions. One way
to fix this issue is by extending this method to the semi-supervised case,
like Deep SAD[33, 34], possibly using synthetic outliers [35]. Another way to
force the network to be descriptive is to start from a pre-trained feature
extractor, and to simultaneously learn the compact one-class objective as well
as an auxiliary general supervised task on an external dataset, in order to
maintain descriptiveness[36]. A now common way is to use the features of such
a pre-trained feature extractor in combination with traditional anomaly
detection methods, as we will see now.

###  2.3 Modelling normality with deep pre-trained features

Building on that idea to balance out descriptiveness and compactness, a recent
trend in modelling normality for anomaly detection is to directly take
advantage of deep feature representations from a frozen pre-trained network.
Using a pre-trained CNN, images are encoded in the embedding space of the
network where traditional anomaly detectors can then be used. This guarantees
descriptiveness of the features, and avoids mode collapses since the pre-
training is performed on a large, generalist dataset such as ImageNet[37].
Typical methods for computing the final anomaly score in the embedding space
are: distance to the k-nearest neighbors (KNN) [1], Mahalanobis distance [2],
and kernel density estimation [38]. Because these methods only provide an
anomaly score at the image level and do not necessarily offer anomaly
localization, they have been improved in several ways. SPADE[39] and
PatchCore[4] extend the KNN approach by looking at the nearest neighbors
locally, in the pixel-embedding space. Correspondingly, PaDiM [3] extends the
Mahanalobis approach by locally modeling patch-wise gaussian covariances.
Finally, in “Deep Feature Reconstruction” (DFR)[40], Yang et. al. use a
reconstruction autoencoder in the pixel-embedding space of the pre-trained
feature extractor. Note that the idea of feature reconstruction was first
proposed without the need of a frozen pre-trained network, in the “Uninformed
Students” paper [41], where an ensemble of CNNs is trained to mimic a teacher
network. The idea has since been extended by using several feature maps from a
single network [42, 43], as well as by blurring the input [44].

The methods benefiting from frozen pre-trained networks often perform very
well because they cannot forget the richness of the pre-trained feature
representations, which often happens when fine-tuning such networks on
different data due to catastrophic forgetting [45]. By taking advantage of
descriptive features, combined with the compactness that the traditional
anomaly detectors bring, these methods allow to get state-of-the-art
performance on many standard anomaly detection datasets for computer vision,
such as MVTec. It is worth noting that techniques such as KNN, Mahalanobis,
and PatchCore are also very fast to train and to test (seconds or minutes)
compared to other methods that require an additional CNN training, such as
autoencoders or self-supervised networks (hours or days). This makes the said
methods prime for small series in quality inspection use cases.

###  2.4 Fully unsupervised anomaly detection

These pre-trained embeddings based methods are also quite robust to the low
data regimes, as shown in our previous work.[16] However, like all of the
other presented methods (AEs, self-supervised and one-class approaches), they
implicitely assume that the entirety of the training set is normal. In that
regard, they cannot be considered as fully unsupervised, since some degree of
supervision is required to select the normal training samples. We define fully
unsupervised learning as a learning which does not require any kind of
labelling of the data. Put differently, a fully unsupervised approach differs
from a common unsupervised approach in the fact that it can work with a
polluted training set. Examples of fully unsupervised anomaly detection
approaches are isolation forest[46] and local outlier factor[47]; neither of
which relies on deep learning.

This remark is highlighted in STOC[8], which proposes a self-supervised one-
class framework for fully unsupervised anomaly detection, by refining the data
over multiple iterations of training. Refinement is performed by removing
training samples with high anomaly scores, which are computed via the
ensembling of classifiers trained on independent splits of the training data.
They also assess robustness of the CutPaste[7] self-supervised method on the
MVTec AD dataset[5], and find that a pollution rate of 10% leads to a decrease
in AUC of around 6 points. To our knowledge, robustness of the pre-trained
methods to non-zero anomaly ratios in the training data (i.e. pollution) has
not been systematically studied (the original Mahalanobis paper however
reports a decrease of AUC of less than 2 points under the same conditions[2]).
In this work, we assess robutness of these methods to pollution, and show that
they can be more robust than CutPaste, even without refinement. We also
outline a simple yet effective refinement algorithm to recover lost
performance. One drawback of data refinement is that it is based on a
percentage of data to remove. This percentage is an hyper-parameter that is
directly linked to the actual pollution rate, which is not precisely known in
practice, but can be estimated based on past observations.

##  3 Materials and methods

###  3.1 Pre-trained networks for anomaly detection

Anomaly detection with pre-trained networks attempts to model normality solely
from healthy samples. In this work, our goal is to assess and improve the
performances of these methods when the training set contains defective
samples. We will be comparing KNN[1], Mahalanobis[2], PaDiM[3] and the
recently introduced PatchCore [4] in the following experiments. As in-depth
explanations of KNN, Mahalanobis and PaDiM are already available in our
previous work [16], we refer to the latter for more details on these methods.

####  3.1.1 KNN

Introduced by Bergman et al., the KNN [1] uses a frozen feature extractor $f$
to embed the training set into a lower dimensional space. Global Average
Pooling (GAP) is then applied to these feature maps to obtain a final vector
representation for each image. At test time, the anomaly score of a new image
$y$ is calculated by taking the average euclidean distance of its associated
vector representation $f_{y}$ to its $k$ nearest neighbours in the training
set $N_{k}(f_{y})$:

$d(y)=\frac{1}{k}\sum_{f\in N_{k}(f_{y})}||f-f_{y}||^{2}$ (1)

In our experiments, we implement SPADE[39], a pixel-wise extension of the KNN
which allows localization of the anomalies. SPADE uses the exact same image-
wise anomaly score, but also gives an anomaly score for each patch $(i,j)$ by
calculating the average euclidean distance between the test feature vector
$f_{y_{i,j}}$ and the feature vectors retrieved by the KNN $f_{x^{k}_{i,j}}$.
Also, SPADE applies GAP to multiple feature levels and concatenate the results
to obtain a vector representation of the image with both coarse and fine-
grained features. The authors, as well as later work [2], show the
effectiveness of using multiple feature levels. We set the number of nearest
neighbors $k$ to 5.

####  3.1.2 Mahalanobis

Rippel et al.[2] propose to model the distribution of training embeddings with
a Multivariate Gaussian distribution (MVG) and use the Mahalanobis distance as
the anomaly score. More specifically, they estimate the mean and covariance of
the training embeddings at different feature levels. Then, they compute the
distance to the distribution of the test embeddings $f_{y}$ at each of these
levels:

$d(y)=\sqrt{(f_{y}-\mu)^{T}S^{-1}(f_{y}-\mu)}$ (2)

The distances are summed up across all feature levels to give the final
anomaly score. As our goal is to find a strategy to refine the polluted
training set (which will inevitably reduce the number of training samples), we
use Ledoit’s shrinkage[48] as the covariance estimator, which is known to
perform better when the number of datapoints becomes much lower than the
number of features[16]. In order to get localization for this method, we apply
Grad-CAM[49].

####  3.1.3 PaDiM

Proposed by Defard et al., PaDiM[3] can be seen as a pixel-wise extension of
the Mahalanobis approach. The features from the different levels are aligned
and concatenated, before fitting a MVG at every spatial location of the
training images. At test time, PaDiM calculates the Mahalanobis distance of
every patch $y_{i,j}$ of the test image $y$ to the corresponding MVG. The
image-wise anomaly score is the maximum of the scores at every location:

$d(y)=\max_{i,j}\sqrt{(f_{y_{i,j}}-\mu_{i,j})^{T}S_{i,j}^{-1}(f_{y_{i,j}}-\mu_{i,j})}$
(3)

Like for Mahalanobis (see 3.1.2), we also use Ledoit’s shrinkage as our
covariance estimator in order to get a more robust estimation when the number
of datapoints becomes much lower than the number of features.

####  3.1.4 PatchCore

Recently introduced by Roth et al, PatchCore[4] can be seen as another pixel-
wise extension of the KNN. PatchCore computes and stores a vector
representation $f_{x_{i,j}^{n}}$ for all patches $(i,j)$ of all training
images $x^{n},n\in\\{1,...,N\\}$. At test time, it computes a representation
$f_{y_{i,j}}$ for every patch location $(i,j)$ of the test image $y$, and
attempts to retrieve the $k$ nearest embeddings for every location. The patch-
level anomaly score is given by the average euclidean distance from
$f_{y_{i,j}}$ to its $k$ nearest embeddings $N_{k}(f_{y_{i,j}})$. The image-
level anomaly score is the maximum score across all spatial locations:

$d(y)=\max_{i,j}\frac{1}{k}\sum_{f_{i,j}\in
N_{k}(f_{y_{i,j}})}||f_{i,j}-f_{y_{i,j}}||^{2}$ (4)

In this approach, the $k$ nearest embeddings of $f_{y_{i,j}}$ can come from
any training image, at any location. This gives PatchCore a lot more
flexibility with regards to the data distribution (multimodality), and allows
for precise defect segmentation. In order to alleviate the time and memory
constraints of the nearest neighours search, the authors propose to use
coreset selection[50] to find a representative subsample of the training
embeddings, and perform nearest neighbors retrieval in this reduced space. In
our implementation of PatchCore, we do not use coreset but IVFFlat (Inverted
File Index Flat)[51] as an acceleration scheme, as it has showed better
results in our experimentations. We use the faiss library[52] for IVFFlat
implementation, and we set the number of nearest neighbors $k$ to 5. Note the
two main differences with SPADE: here the image-wise anomaly score is the
score of the highest patch instead of being the score of an image-wise KNN,
and the nearest neighbors of a test patch $y_{i,j}$ can come from any image in
the training set at any location, instead of being limited to patches
retrieved by the image-wise KNN.

###  3.2 SROC, a simple heuristic for data refinement

Figure 1: Overview of SROC. A frozen feature extractor is used to embed the
polluted training set, and a one-class classifier (or detector) is fitted on
these embeddings. The detector then predicts a score for each training
samples, and the samples with the highest scores are removed.

####  3.2.1 Methodology

We propose SROC, a Simple Refinement strategy for One Class anomaly detection.
The goal of our method is to leverage both the high proportion of healthy
samples in the training data and the good generalization capabilities of the
traditional one-class classifiers.

Here are the steps we follow. First, a pre-trained feature extractor is used
to extract a rich and meaningful representation of the polluted training set.
Then, an anomaly detector from the ones detailed in 3.1 is fitted on the
polluted embeddings. We evaluate the detector on the same training embeddings,
and we remove samples with the greatest anomaly scores. Finally, another
detector can then be fitted on the refined embeddings to reach better
performances. Figure 1 provides a visual representation of the proposed
method.

This strategy is based on the assumption that most of the training data is
healthy, because manufacturers optimize their production to yield as few
defective parts as possible. By leveraging this with the generalization
capabilities of the detectors, we aim at constructing a representation of
normality that is good enough for training defective samples to have a higher
score. In this paper, we validate the effectiveness of SROC with the 4 state-
of-the-art detectors presented in section 3.1, but our heuristic has the
advantage to be compatible with virtually any one-class classifier.

####  3.2.2 Hyperparameters

SROC is designed to be simple and easy to implement in a production setting,
where precise hyperparameter tuning for each part reference is not always
feasible. The only parameter of the method is the percentage of samples to
remove, which we call the refinement ratio. Ideally, the refinement ratio
should be close to the actual pollution rate, but can also be higher, as
already shown in STOC[8].

##  4 Experiments

###  4.1 Dataset and metrics

Figure 2: Creation of the MVTec polluted training sets. A set of defective
samples is extracted from the original validation set, and gradually injected
in the training set in place of healthy samples for pollution purposes. The
resulting training set has the same size as the original.

We conduct the experiments on MVTec AD[5], a dataset for real-world industrial
quality inspection comprised solely of healthy samples for training and of
both healthy and defective samples for validation. Images are divided into 15
classes or categories: some being textures (e.g. carpet, grid, leather) and
others being objects (e.g. bottle, cable, capsule).

Since no defects are available in the original training sets, we create
polluted training sets in a similar fashion as in the STOC paper[8]. For each
class, we first sample a “pollution set” from the original validation images
that is 20% the size of the training set $N$. This pollution set contains only
defective samples. Note that we ensure that the defects sampled in the
pollution set follow the same distribution as the defects in the original
validation set (i.e. there is the same proportion of each defect type in
both). Samples used in the pollution set are excluded from the validation set.
Then, we remove $l$ random images from the training set and replace them with
random images from the pollution set. $l$ varies from 0 to $0.2*N$, enabling
us to have training sets with 0 to 20% pollution. Figure 2 summarizes the
creation of the polluted training sets. Note that we exclude hazelnut and
transistor categories from our experiments, since their original validation
sets do not contain enough defective samples for polluting the training set
while keeping defective samples in the validation set. While this method
changes the size of the validation set, it has the advantage of preserving the
size of the training set. This is to allow measuring the impact of pollution
independently of the number of training samples.

We first assess the robustness of the detectors by measuring their AUC, AU-IoU
(area under the “IoU as a function of false positive rate” curve) and AU-PRO
(area under the “PRO as a function of false positive rate” curve) on the
validation set as a function of the pollution ratio. The PRO score (Per-Region
Overlap) has been used in previous works[5, 41] as an alternative to the
pixel-wise AUC and IoU metrics, which tend to be biased towards larger
anomalous regions. Here, we measure the AU-IoU and AU-PRO with up to a 30%
false positive rate (FPR) in order for our results to be comparable with
previous works [5, 41]. Note that this prevents in practice the metrics from
saturating between 30% and 100% FPR.

We then evaluate the efficiency of SROC as a data refinement strategy by
fixing the pollution ratio in the training set to 20%, and varying the
refinement ratio. We compute the same metrics, as well as the precision,
recall and F1 score of SROC on the training set. For example, the recall score
of SROC is defined as the number of defective samples removed, divided by the
total number of defective samples to remove in the training set.

###  4.2 Implementation details

In order to be fair, the same feature extractor is used for all methods,
namely EfficientNetB4[6], which showed the best results in previous works[2].
We use the outputs of the blocks 4, 6 and 7, as in PaDiM’s original paper[3].
These choices may be slightly detrimental to some methods and can explain the
performance differences with the ones reported by the original authors.

We implement all methods using Tensorflow / Keras for embedding the images
with EfficientNetB4, as well as Scikit-Learn and faiss[52] for the traditional
anomaly detection techniques. The hyperparameters chosen for each detector are
presented in section 3.1. Since we are using EfficientNetB4, we resize all
MVTec images to 380 x 380 pixels (the image size with which the network was
originally pre-trained). Note that we do not perform center cropping, as it is
common to do. This may also explain differences in performances.

We conduct the experiments with 5 different random seeds to provide both mean
results and standard deviations. The seed controls which healthy samples are
removed from the training set, and which defective samples are added in place.
Note that the 5 seeds choice is the same for all pollution and refinement
ratios.

###  4.3 Assessing pollution robustness

We start by assessing the robustness of each anomaly detection method to
increasingly polluted training sets. To that end, we fit the different
detectors on training sets with pollution ratios varying between 0 and 20%,
and evaluate them on the validation sets defined in 4.1. Table 1 compares the
performances of all detectors at exactly 0 and 20% pollution ratios. Figure 3
shows the evolution of the AUC as the pollution ratio increases. Note that the
AU-IoU and AU-PRO as functions of the pollution ratio are available in
Appendix A.1.

Table 1: Average performances of each detector across all MVTec AD categories, when the training set is polluted at 0 and 20%. Categories transistor and hazelnut were removed because of their low amount of available defective samples. | KNN | Mahalanobis | PaDiM | PatchCore
---|---|---|---|---
Pollution |  0% |  20% |  0% |  20% |  0% |  20% |  0% |  20%
carpet |  97.8 |  93.3±0.5 |  100 |  99.5±0.1 |  100 |  100±0.0 |  100 |  100±0.0
tile |  99.5 |  99.5±0.2 |  100 |  99.8±0.1 |  99.7 |  99.2±0.2 |  99.9 |  99.5±0.2
leather |  99.9 |  98.9±0.1 |  100 |  95.3±0.4 |  100 |  100±0.0 |  100 |  100±0.0
grid |  67.9 |  60.0±3.1 |  97.6 |  82.9±1.9 |  100 |  100±0.0 |  100 |  100±0.0
wood |  87.6 |  79.7±1.7 |  95.2 |  85.2±1.0 |  100 |  99.0±0.0 |  100 |  100±0.0
capsule |  92.7 |  91.9±1.6 |  96.7 |  91.4±1.2 |  95.5 |  81.7±2.8 |  98.2 |  95.6±1.0
cable |  95.3 |  94.6±0.2 |  97.9 |  94.2±0.4 |  97.7 |  94.2±0.5 |  97.4 |  94.2±0.5
pill |  80.4 |  78.4±0.9 |  88.8 |  84.0±0.4 |  94.6 |  88.5±0.7 |  93.4 |  90.4±0.4
metal nut |  88.6 |  84.5±1.5 |  97.5 |  93.9±0.1 |  95.5 |  88.4±0.7 |  100 |  99.3±0.2
toothbrush |  93.5 |  90.6±1.9 |  96.8 |  95.0±1.3 |  88.0 |  86.4±1.6 |  89.4 |  89.4±0.0
screw |  85.7 |  81.3±2.7 |  95.5 |  81.0±2.6 |  96.2 |  93.0±1.2 |  93.6 |  91.8±1.4
zipper |  95.3 |  93.8±0.3 |  98.1 |  93.1±1.0 |  96.3 |  90.8±1.2 |  96.3 |  94.0±0.1
bottle |  98.8 |  98.7±0.3 |  100 |  98.4±0.2 |  99.5 |  98.5±0.9 |  99.3 |  99.0±0.0
Mean AUC |  91.0 |  88.1±1.0 |  97.2 |  91.8±0.8 |  97.2 |  93.8±0.7 |  97.5 |  96.4±0.3
Mean AU-IoU |  0.164 |  0.135 |  0.142 |  0.121 |  0.147 |  0.128 |  0.167 |  0.141
Mean AU-PRO |  0.909 |  0.902 |  0.810 |  0.804 |  0.877 |  0.858 |  0.914 |  0.908
Figure 3: Average AUC across all MVTec AD categories as a function of the
pollution ratio, for each detector. Categories transistor and hazelnut were
removed because of their low amount of available defective samples.

Results show that detectors leveraging pre-trained feature extractors are
already quite robust to polluted training set, even at 20%. Mahalanobis and
PaDiM are the most impacted out of the four (-5.4 and -3.4 AUC points,
respectively), and PatchCore and KNN are the most robust (-1.1 and -2.9 AUC
points, respectively). In comparison, the authors of STOC report that
CutPaste[7] trained on a dataset with a 10% pollution ratio looses 5.97 AUC
points [8] on MVTec AD. As a consequence, anomaly detection methods leveraging
a pre-trained feature extractor appear as more robust to pollution than purely
self-supervised methods. We explan these discrepancies by the use of a frozen
feature extractor. The latter preserves the richness and descriptiveness of
the ImageNet features. The pollution only impacts the shallow anomaly detector
on top. Self-supervised methods, on the other hand, fit the feature extractor
direcly onto the polluted images, thus impacting the network representation.
It therefore becomes much harder for the detector on top to discriminate
between the healthy and the defective images.

Note however that our results are not directly comparable to the ones reported
for CutPaste, mainly for two reasons. First, even though the polluted training
set creation approach is the same as the one reported for CutPaste, the latter
only pollute at a 10% ratio at most, making the final validation sets
different from ours (not to mention that the sampling of the defective parts
from the original validation set is done at random, which could lead to
different results even if the maximum pollution ratio was the same). Second,
we had to exclude transistor and hazelnut categories because of their lack of
defective validation samples.

Interestingly, nearest neighbors based detectors (KNN, PatchCore) appear as
more robust than the ones relying on Mahalanobis distance (Mahalanobis,
PaDiM). In section 4.6, we conduct an in-depth qualitative analysis in order
to understand why.

###  4.4 Data refinement

Despite the relative robustness of the pre-trained methods, their loss of
performance could deter their use in a fully unsupervised setting. In this
section, we show the effectiveness of our method for data refinement to
alleviate this loss. We pollute all training sets with 20% of defective
samples and refine them with SROC. We compare the performances of each
detector when removing from 0% to 80% of the top predictions. Figure 4 and
Table 2 give an overview of the performances of the proposed method.

For all detectors, the highest F1-score is obtained when the refinement ratio
is approximately equal to the pollution ratio (here 20%). At this value, SROC
systematically removes more than 70% of the defective samples from the
training set, enabling Mahalanobis and KNN to gain back 2.6 and 0.8 AUC points
respectively when fitted on these refined embeddings. After this, the AUC
gradually decreases because we are removing too many images. PatchCore does
not really improve, but it was also the least impacted by pollution (see
section 4.3). More surpringly however, PaDiM does not improve. Indeed, it is
unclear why SROC does not manage to remove as many defective samples for that
method, as shown by the lower precision and recall. Additionally, we have
shown in previous works that PaDiM is the most data hungry method out of the
four tested for MVTec object categories. Thus, PaDiM is also expected to be
more sensible to data refinement than the other methods.

Because SROC works best with Mahalanobis (highest recall and precision), we
propose to use SROC+Mahalanobis to refine the embeddings, and to fit KNN,
PaDiM, and PatchCore on these refined embeddings. We show in Appendix B that
this enables to gain back 1.4 AUC points for PaDiM.

Figure 4: Average AUC for all MVTec classes as a function of the refinement ratio, for training sets polluted at a 20% ratio. Dashed lines represent maximum reachable AUC for a perfect strategy which would only removes the defective samples (average performances of each detector when fitted on 80% of the original healthy training set). Table 2: Average performances of SROC applied with 0%, 20% and 40% refinement ratios, for training sets polluted at a 20% ratio. Categories transistor and hazelnut were removed because of their low amount of available defective samples. Note that we do not display standard deviations here for clarity purposes. | KNN | Mahalanobis | PaDiM | PatchCore
---|---|---|---|---
Refinement |  0% |  20% |  40% |  0% |  20% |  40% |  0% |  20% |  40% |  0% |  20% |  40%
carpet |  93.3 |  97.8 |  97.8 |  99.5 |  99.8 |  100 |  100 |  100 |  100 |  100 |  100 |  100
tile |  99.5 |  99.5 |  99.2 |  99.8 |  100 |  100 |  99.2 |  98.9 |  98.9 |  99.5 |  99.7 |  99.5
leather |  98.9 |  99.9 |  99.9 |  95.3 |  100 |  100 |  100 |  100 |  100 |  100 |  100 |  100
grid |  60.0 |  55.5 |  48.8 |  82.9 |  85.5 |  84.0 |  100 |  100 |  100 |  100 |  100 |  100
wood |  79.7 |  86.8 |  86.4 |  85.2 |  94.4 |  95.9 |  99.0 |  100 |  100 |  100 |  100 |  100
capsule |  91.9 |  91.0 |  89.9 |  91.4 |  93.1 |  93.5 |  81.7 |  83.1 |  82.8 |  95.6 |  94.8 |  93.4
cable |  94.6 |  94.8 |  95.3 |  94.2 |  96.6 |  96.5 |  94.2 |  95.7 |  96.2 |  94.2 |  95.8 |  95.3
pill |  78.4 |  78.7 |  77.6 |  84.0 |  86.6 |  85.5 |  88.5 |  87.6 |  83.9 |  90.4 |  90.3 |  90.4
metal nut |  84.5 |  86.4 |  81.4 |  93.9 |  95.6 |  93.2 |  88.4 |  87.4 |  86.7 |  99.3 |  100 |  100
toothbrush |  90.6 |  90.6 |  89.4 |  95.0 |  96.2 |  94.6 |  86.4 |  85.8 |  86.8 |  89.4 |  89.3 |  89.3
screw |  81.3 |  81.6 |  78.0 |  81.0 |  83.2 |  84.0 |  93.0 |  90.3 |  84.2 |  91.8 |  91.6 |  90.5
zipper |  93.8 |  94.7 |  94.6 |  93.1 |  96.2 |  96.8 |  90.8 |  94.2 |  94.3 |  94.0 |  94.3 |  95.6
bottle |  98.7 |  98.7 |  98.5 |  98.4 |  99.4 |  99.5 |  98.5 |  97.8 |  99.8 |  99.0 |  99.0 |  99.3
Mean AUC |  88.1 |  88.9 |  87.4 |  91.8 |  94.4 |  94.1 |  93.8 |  93.9 |  93.3 |  96.4 |  96.5 |  96.4
Mean AU-IoU |  0.135 |  0.166 |  0.165 |  0.121 |  0.141 |  0.141 |  0.128 |  0.135 |  0.143 |  0.141 |  0.151 |  0.162
Mean AU-PRO |  0.902 |  0.906 |  0.906 |  0.804 |  0.794 |  0.797 |  0.858 |  0.862 |  0.865 |  0.908 |  0.907 |  0.907
Refin. Precision |  - |  78.6 |  45.5 |  - |  81.1 |  47.1 |  - |  71.5 |  46.4 |  - |  75.2 |  46.3
Refin. Recall |  - |  78.6 |  91.2 |  - |  81.1 |  94.3 |  - |  71.5 |  93.0 |  - |  75.2 |  92.7
Refin. F1 |  - |  78.6 |  60.7 |  - |  81.1 |  62.8 |  - |  71.5 |  61.9 |  - |  75.2 |  61.7

###  4.5 Comparison with other data refinement strategies

In this section, we provide a comprehensive comparison between our refinement
strategy and several others, from the literature and from our own experiments.

Random: The simplest heuristic for data refinement is to remove random
training samples, and hope to remove the defective ones. This idea is
motivated by the good performances obtained by pre-trained detectors in the
low data regime.[16] This will serve as baseline to evaluate the need for a
slightly more elaborate strategy like ours.

Cross-Validation: One criticism that could be made about our method is that
evaluating a detector on the training set is prone to overfitting. The
detector could learn the defects of the training set and therefore give them a
low score at test time. We thus compare our method with a cross-validation
strategy, where we split the training embeddings into $s$ non-overlapping
splits, we fit the detector on $s-1$ splits and evaluate on the remaining
split. By iterating over all splits, every training sample gets an anomaly
score. Then, we remove the images with the highest scores.

STOC[8]: While mainly designed for self-supervised methods, STOC could also be
used for pre-trained ones. Here, the training set is divided into $s$ splits,
a detector is fitted on each of them and evaluated on the whole training set.
Each training image is thus given $s$ scores, and we average them to get the
final one. The images with the highest scores are removed. This strategy has
the advantage of ensembling the decisions of multiple detectors, but each of
them is fitted on a fraction of the training data. Note that contrarily to the
original paper, we do not apply this strategy iteratively as our feature
extractor is frozen. We set the number of splits to $s=5$ for Cross-validation
and STOC (as in STOC’s original paper[8]). Figure 5 compares the different
strategies for all detectors.

Our simple heuristic manages to compete with, and most often outperform both
Cross-Validation and STOC. We explain the performance discrepancies by the
size of the training set. Our method, SROC, is fitted on the entire training
set, while Cross-Validation is trained with the images of $s-1$ splits (here
4/5 of the training data). For STOC, the training size is even smaller as each
detector is fitted on a single split $s$ (here 1/5 of the training set). Using
all of the available examples for training enables our method to build a more
robust and reliable representation of normality, making the polluted samples
stand out more from the rest. SROC is also much more computationally efficient
than the others. It only fits a single detector on the polluted embeddings.
The latter two, on the other hand fit as many detectors as the number of
splits, thus multiplying both the training and the inference time. The
simplicity, performance and efficiency of SROC make it a method of choice for
a deployment in a production setting.

(a) KNN (b) Mahalanobis

(c) PaDiM (d) PatchCore

Figure 5: AUC as a function of the refinement ratio for training sets polluted
at a 20% ratio, for all refinement strategies. Results are plot per detector.
Dashed lines are the average performances of each detector when fitted on 80%
of the original healthy training set (maximum reachable if the strategy only
removes defective samples). Our simple heuristic competes with much more
complex and costly strategies like STOC.

###  4.6 Qualitative analysis, robustness of pre-trained method

Finally, in this last section we aim at providing more intuition on the
results of section 4.3. In particular, our goal is to explain why KNN and
PatchCore are less impacted by pollution than Mahalanobis and PaDiM.

However, as showed in section 4.3, there are also discrepancies amongst pre-
trained methods. The $k$-Nearest Neighbors approaches (KNN and PatchCore)
proved more robust than the Mahalanobis-based ones (Mahalanobis and PaDiM). In
order to understand this phenomenon, we pollute two chosen MVTec training sets
(carpet and bottle) with 20% of defective samples (our maximum pollution
ratio), and we extract and process embeddings for these training sets. Then,
we fit Mahalanobis on the healthy part of the embedded training set and the
full, polluted embedded training set. We finally plot the contours of the
learned multi-variate gaussians (MVGs). We plot these contours amongst the
axis with the greatest variance change between the healthy and the polluted
MVG. Figure 6 shows these contours for the two aforementioned categories.

(a) carpet (b) bottle

Figure 6: Contours of the fitted multi-variate gaussian (MVG) when the
training set contains 0% (green) and 20% (red) of pollution for a texture
(carpet, left) and an object category (bottle, right).

Along the chosen axis, the MVG shifts significantly, thus giving an indication
of why Mahalanobis may not be the most robust to pollution. Some defective
samples are now much closer to the center of the MVG, and may thus be
incorrectly classified as healthy. These plots also give some intuition for
the robustness of the two nearest-neighbor approaches (KNN and PatchCore). In
these two dimensions, the healthy embeddings appear very close to each other,
while the defective embeddings are much more sparse.

To confirm this intuition, we calculate the average euclidean distance in the
embedding space from a polluted sample to another, from a polluted sample to a
healthy one, and from a healthy sample to another. We report these distances
for carpet and bottle in Table 3. We observe that defective samples are almost
as far from each other as they are from healthy samples in the embedding
space. This explains why the nearest neighbors approaches are so robust to
pollution. If all defective samples were close to each other, then many would
be classified as healthy, because the distance is used as the anomaly score.

Table 3: Average euclidean distance from healthy to healthy, healthy to defective and defective to defective in the embedding space for the two categories carpet and bottle. Defective embeddings are almost as far from each other than they they are from healthy embeddings, confirming the intuition on why nearest neighbors approaches are more robust to pollution. | carpet | bottle
---|---|---
|  Healthy |  Defective |  Healthy |  Defective
Healthy |  14.66 |  24.07 |  11.92 |  22.66
Defective |  24.07 |  24.61 |  22.66 |  22.90

##  5 Conclusion

In this work, we showed that anomaly detection methods leveraging pre-trained
feature extractors are quite robust to polluted datasets, notably compared to
self-supervised methods such as CutPaste. We introduced a simple strategy to
effectively refine the training data by fitting and evaluating the detectors
on the polluted embeddings. We also gave some intuition on why KNN-based
methods are more robust to pollution than the ones based on the Mahalanobis
distance. It is interesting to note that our refinement strategy is
theoretically agnostic of the chosen anomaly detection method, and could thus
be used with any. Note that it is also possible to use SROC in combination
with a method specifically chosen for refinement (e.g. Mahalanobis), and to
use a different method once refinement is performed.

To go further, it would be interesting to make use of the samples predicted as
defective, instead of simply removing them. Taking inspiration from the
literature[19, 18], one could make use of an generative network (such as an
autoencoder) trained with a reconstruction task on the refined data to erase
defects from the samples detected as pollution by SROC. This way, defective
samples could be transformed into new healthy data in order to improve
performance further. Another possibility could be to refine training patches
instead of images. This would benefit the pixel-wise methods and limit their
drops in performances when refining.

## References

  * [1] Bergman, L., Cohen, N., and Hoshen, Y., “Deep nearest neighbor anomaly detection,” arXiv preprint arXiv:2002.10445 (2020).
  * [2] Rippel, O., Mertens, P., and Merhof, D., “Modeling the distribution of normal data in pre-trained deep features for anomaly detection,” arXiv preprint arXiv:2005.14140 (2020).
  * [3] Defard, T., Setkov, A., Loesch, A., and Audigier, R., “Padim: a patch distribution modeling framework for anomaly detection and localization,” arXiv preprint arXiv:2011.08785 (2020).
  * [4] Roth, K., Pemula, L., Zepeda, J., Schölkopf, B., Brox, T., and Gehler, P., “Towards total recall in industrial anomaly detection,” (2021).
  * [5] Bergmann, P., Fauser, M., Sattlegger, D., and Steger, C., “Mvtec ad–a comprehensive real-world dataset for unsupervised anomaly detection,” in [Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition ], 9592–9600 (2019).
  * [6] Tan, M. and Le, Q. V., “Efficientnet: Rethinking model scaling for convolutional neural networks,” (2020).
  * [7] Li, C.-L., Sohn, K., Yoon, J., and Pfister, T., “Cutpaste: Self-supervised learning for anomaly detection and localization,” (2021).
  * [8] Yoon, J., Sohn, K., Li, C.-L., Arik, S. O., Lee, C.-Y., and Pfister, T., “Self-trained one-class classification for unsupervised anomaly detection,” (2021).
  * [9] Ren, R., Hung, T., and Tan, K. C., “A generic deep-learning-based approach for automated surface inspection,” IEEE transactions on cybernetics 48(3), 929–940 (2017).
  * [10] Cordier, A., Das, D., and Gutierrez, P., “Active learning using weakly supervised signals for quality inspection,” arXiv preprint arXiv:2104.02973 (2021).
  * [11] Ronneberger, O., Fischer, P., and Brox, T., “U-net: Convolutional networks for biomedical image segmentation,” in [International Conference on Medical image computing and computer-assisted intervention ], 234–241, Springer (2015).
  * [12] Lin, T.-Y., Goyal, P., Girshick, R., He, K., and Dollár, P., “Focal loss for dense object detection,” in [Proceedings of the IEEE international conference on computer vision ], 2980–2988 (2017).
  * [13] Zambal, S., Heindl, C., Eitzinger, C., and Scharinger, J., “End-to-end defect detection in automated fiber placement based on artificially generated data,” in [Fourteenth International Conference on Quality Control by Artificial Vision ], 11172, 111721G, International Society for Optics and Photonics (2019).
  * [14] Gutierrez, P., Luschkova, M., Cordier, A., Shukor, M., Schappert, M., and Dahmen, T., “Synthetic training data generation for deep learning based quality inspection,” arXiv preprint arXiv:2104.02980 (2021).
  * [15] Pang, G., Shen, C., Cao, L., and Hengel, A. v. d., “Deep learning for anomaly detection: A review,” arXiv preprint arXiv:2007.02500 (2020).
  * [16] Gutierrez, P., Cordier, A., Caldeira, T., and Sautory, T., “Data augmentation and pre-trained networks for extremely low data regimes unsupervised visual inspection,” (2021).
  * [17] Huang, C., Ye, F., Cao, J., Li, M., Zhang, Y., and Lu, C., “Attribute restoration framework for anomaly detection,” arXiv preprint arXiv:1911.10676 (2019).
  * [18] Bergmann, P., Löwe, S., Fauser, M., Sattlegger, D., and Steger, C., “Improving unsupervised defect segmentation by applying structural similarity to autoencoders,” arXiv preprint arXiv:1807.02011 (2018).
  * [19] Baur, C., Denner, S., Wiestler, B., Navab, N., and Albarqouni, S., “Autoencoders for unsupervised anomaly segmentation in brain mr images: A comparative study,” Medical Image Analysis , 101952 (2021).
  * [20] Schlegl, T., Seeböck, P., Waldstein, S. M., Schmidt-Erfurth, U., and Langs, G., “Unsupervised anomaly detection with generative adversarial networks to guide marker discovery,” in [International conference on information processing in medical imaging ], 146–157, Springer (2017).
  * [21] Zavrtanik, V., Kristan, M., and Skočaj, D., “Reconstruction by inpainting for visual anomaly detection,” Pattern Recognition 112, 107706 (2021).
  * [22] Nguyen, B., Feldman, A., Bethapudi, S., Jennings, A., and Willcocks, C. G., “Unsupervised region-based anomaly detection in brain mri with adversarial image inpainting,” (2020).
  * [23] Pirnay, J. and Chai, K., “Inpainting transformer for anomaly detection,” (2021).
  * [24] Hendrycks, D., Mazeika, M., and Dietterich, T., “Deep anomaly detection with outlier exposure,” arXiv preprint arXiv:1812.04606 (2018).
  * [25] daisukelab, “Spotting defects! — deep metric learning solution for mvtec anomaly detection dataset,” (Sep 2019).
  * [26] Schlüter, H. M., Tan, J., Hou, B., and Kainz, B., “Self-supervised out-of-distribution detection and localization with natural synthetic anomalies (nsa),” (2021).
  * [27] Collin, A.-S. and Vleeschouwer, C. D., “Improved anomaly detection by training an autoencoder with skip connections on images corrupted with stain-shaped noise,” (2020).
  * [28] Golan, I. and El-Yaniv, R., “Deep anomaly detection using geometric transformations,” arXiv preprint arXiv:1805.10917 (2018).
  * [29] Zavrtanik, V., Kristan, M., and Skočaj, D., “Draem – a discriminatively trained reconstruction embedding for surface anomaly detection,” (2021).
  * [30] Schölkopf, B., Williamson, R. C., Smola, A. J., Shawe-Taylor, J., Platt, J. C., et al., “Support vector method for novelty detection.,” in [NIPS ], 12, 582–588, Citeseer (1999).
  * [31] Ruff, L., Vandermeulen, R., Goernitz, N., Deecke, L., Siddiqui, S. A., Binder, A., Müller, E., and Kloft, M., “Deep one-class classification,” in [Proceedings of the 35th International Conference on Machine Learning ], Dy, J. and Krause, A., eds., Proceedings of Machine Learning Research 80, 4393–4402, PMLR (10–15 Jul 2018).
  * [32] Yi, J. and Yoon, S., “Patch svdd: Patch-level svdd for anomaly detection and segmentation,” (2020).
  * [33] Ruff, L., Vandermeulen, R. A., Görnitz, N., Binder, A., Müller, E., Müller, K.-R., and Kloft, M., “Deep semi-supervised anomaly detection,” arXiv preprint arXiv:1906.02694 (2019).
  * [34] Ruff, L., Vandermeulen, R. A., Franks, B. J., Müller, K.-R., and Kloft, M., “Rethinking assumptions in deep anomaly detection,” arXiv preprint arXiv:2006.00339 (2020).
  * [35] Liznerski, P., Ruff, L., Vandermeulen, R. A., Franks, B. J., Kloft, M., and Müller, K.-R., “Explainable deep one-class classification,” arXiv preprint arXiv:2007.01760 (2020).
  * [36] Perera, P. and Patel, V. M., “Learning deep features for one-class classification,” IEEE Transactions on Image Processing 28(11), 5450–5463 (2019).
  * [37] Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L., “Imagenet: A large-scale hierarchical image database,” in [2009 IEEE Conference on Computer Vision and Pattern Recognition ], 248–255 (2009).
  * [38] Erdil, E., Chaitanya, K., and Konukoglu, E., “Unsupervised out-of-distribution detection using kernel density estimation,” arXiv preprint arXiv:2006.10712 (2020).
  * [39] Cohen, N. and Hoshen, Y., “Sub-image anomaly detection with deep pyramid correspondences,” arXiv preprint arXiv:2005.02357 (2020).
  * [40] Yang, J., Shi, Y., and Qi, Z., “Dfr: Deep feature reconstruction for unsupervised anomaly segmentation,” arXiv preprint arXiv:2012.07122 (2020).
  * [41] Bergmann, P., Fauser, M., Sattlegger, D., and Steger, C., “Uninformed students: Student-teacher anomaly detection with discriminative latent embeddings,” in [Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition ], 4183–4192 (2020).
  * [42] Wang, G., Han, S., Ding, E., and Huang, D., “Student-teacher feature pyramid matching for unsupervised anomaly detection,” arXiv preprint arXiv:2103.04257 (2021).
  * [43] Salehi, M., Sadjadi, N., Baselizadeh, S., Rohban, M. H., and Rabiee, H. R., “Multiresolution knowledge distillation for anomaly detection,” arXiv preprint arXiv:2011.11108 (2020).
  * [44] Choi, S. and Chung, S.-Y., “Novelty detection via blurring,” arXiv preprint arXiv:1911.11943 (2019).
  * [45] Goodfellow, I. J., Mirza, M., Xiao, D., Courville, A., and Bengio, Y., “An empirical investigation of catastrophic forgetting in gradient-based neural networks,” arXiv preprint arXiv:1312.6211 (2013).
  * [46] Liu, F. T., Ting, K. M., and Zhou, Z.-H., “Isolation forest,” in [2008 Eighth IEEE International Conference on Data Mining ], 413–422 (2008).
  * [47] Breunig, M., Kriegel, H.-P., Ng, R., and Sander, J., “Lof: Identifying density-based local outliers.,” ACM Sigmod Record 29, 93–104 (06 2000).
  * [48] Ledoit, O. and Wolf, M., “A well-conditioned estimator for large-dimensional covariance matrices,” Journal of Multivariate Analysis 88(2), 365–411 (2004).
  * [49] Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D., “Grad-cam: Visual explanations from deep networks via gradient-based localization,” International Journal of Computer Vision 128, 336–359 (Oct 2019).
  * [50] Agarwal, P. K., Har-Peled, S., and Varadarajan, K. R., “Geometric approximation via coresets,” in [COMBINATORIAL AND COMPUTATIONAL GEOMETRY, MSRI ], 1–30, University Press (2005).
  * [51] Jégou, H., Douze, M., and Schmid, C., “Product quantization for nearest neighbor search,” IEEE Transactions on Pattern Analysis and Machine Intelligence 33(1), 117–128 (2011).
  * [52] Johnson, J., Douze, M., and Jégou, H., “Billion-scale similarity search with gpus,” arXiv preprint arXiv:1702.08734 (2017).

##  Appendix A AU-IoU and AU-PRO curves

###  A.1 Pollution robustness

Here we provide additional curves to supplement Table 1. In particular, we
plot the AU-IoU and the AU-PRO as a function of the pollution ratio.

Figure 7: Average AU-IoU across all MVTec AD categories as a function of the
pollution ratio, for each detector. Categories transistor and hazelnut were
removed because of their low amount of available defective samples. Figure 8:
Average AU-PRO across all MVTec AD categories as a function of the pollution
ratio, for each detector. Categories transistor and hazelnut were removed
because of their low amount of available defective samples.

###  A.2 SROC

Figure 9 and 10 complete Table 2 with AU-IoU and the AU-PRO as a function of
the refinement ratio.

Figure 9: Average AU-IoU across all MVTec AD categories as a function of the
refinement ratio, for training sets polluted at a 20% ratio. Categories
transistor and hazelnut were removed because of their low amount of available
defective samples. Figure 10: Average AU-PRO across all MVTec AD categories as
a function of the refinement ratio, for training sets polluted at a 20% ratio.
Categories transistor and hazelnut were removed because of their low amount of
available defective samples.

##  Appendix B Refinement with Mahalanobis

Finally, we showed in section 4.4 that amongst the 4 detectors, applying SROC
with Mahalanobis gave the best results. Mahalanobis indeed gets the best
precision, recall and F1 scores, meaning that more defectives samples are
removed (and thus more healthy samples are kept). In this section, we
therefore propose to refine the embeddings with Mahalanobis (i.e. we fit and
evaluate Mahalanobis on the polluted embeddings and remove the highest scores)
before training the other detectors with that refined training data. Table 4
shows the results. The main difference is for PaDiM, for which the AUC
increases by 1.4 points. This is probably due to the fact that SROC+PaDiM had
the lowest precision and recall scores (see Table 2). Now that we remove more
defective samples, we are able to gain more performance back.

Table 4: Comparison of SROC when refinement is done with the same detector (reference), and when the refinement is done with the Mahalanobis detector (Mahalanobis), for training sets polluted at a 20% ratio. | KNN | Mahalanobis | PaDiM | PatchCore
---|---|---|---|---
Refinement |  0% |  20% |  40% |  0% |  20% |  40% |  0% |  20% |  40% |  0% |  20% |  40%
Mean AUC (reference) |  88.1 |  88.9 |  87.4 |  91.8 |  94.4 |  94.1 |  93.8 |  93.9 |  93.3 |  96.4 |  96.5 |  96.4
Mean AUC (Mahalanobis) |  88.1 |  88.6 |  87.5 |  91.8 |  94.4 |  94.1 |  93.8 |  95.3 |  94.0 |  96.4 |  96.8 |  96.6
Mean AU-IoU (reference) |  0.135 |  0.166 |  0.165 |  0.121 |  0.141 |  0.141 |  0.128 |  0.135 |  0.143 |  0.141 |  0.151 |  0.162
Mean AU-IoU (Mahalanobis) |  0.135 |  0.167 |  0.166 |  0.121 |  0.141 |  0.141 |  0.128 |  0.149 |  0.149 |  0.141 |  0.165 |  0.165
Mean AU-PRO (reference) |  0.902 |  0.906 |  0.906 |  0.804 |  0.794 |  0.797 |  0.858 |  0.862 |  0.865 |  0.908 |  0.907 |  0.907
Mean AU-PRO (Mahalanobis) |  0.902 |  0.904 |  0.903 |  0.804 |  0.794 |  0.797 |  0.858 |  0.868 |  0.873 |  0.908 |  0.914 |  0.913
Figure 11: Average AUC of SROC+Mahalanobis across all MVTec AD categories as a
function of the refinement ratio, for training sets polluted at a 20% ratio.
Categories transistor and hazelnut were removed because of their low amount of
available defective samples. Figure 12: Average AU-IoU of SROC+Mahalanobis
across all MVTec AD categories as a function of the refinement ratio, for
training sets polluted at a 20% ratio. Categories transistor and hazelnut were
removed because of their low amount of available defective samples. Figure 13:
Average AU-PRO of SROC+Mahalanobis across all MVTec AD categories as a
function of the refinement ratio, for training sets polluted at a 20% ratio.
Categories transistor and hazelnut were removed because of their low amount of
available defective samples.