# Unsupervised anomaly detection algorithms on real-world data: how many do we
need?

Roel Bouman<EMAIL_ADDRESS>
Institute for Computing and Information Sciences
Radboud University
Toernooiveld 212, 6525 EC Nijmegen, The Netherlands Zaharah Bukhsh
<EMAIL_ADDRESS>
Industrial Engineering & Innovation Sciences, Information Systems
Eindhoven University of Technology
Groene Loper 3, 5612 AE Eindhoven, The Netherlands Tom Heskes<EMAIL_ADDRESS>
Institute for Computing and Information Sciences
Radboud University
Toernooiveld 212, 6525 EC Nijmegen, The Netherlands

###### Abstract

In this study we evaluate $32$ unsupervised anomaly detection algorithms on
$52$ real-world multivariate tabular datasets, performing the largest
comparison of unsupervised anomaly detection algorithms to date. On this
collection of datasets, the $k$-thNN (distance to the $k$-nearest neighbor)
algorithm significantly outperforms the most other algorithms. Visualizing and
then clustering the relative performance of the considered algorithms on all
datasets, we identify two clear clusters: one with “local” datasets, and
another with “global” datasets. “Local” anomalies occupy a region with low
density when compared to nearby samples, while “global” occupy an overall low
density region in the feature space. On the local datasets the $k$NN
($k$-nearest neighbor) algorithm comes out on top. On the global datasets, the
EIF (extended isolation forest) algorithm performs the best. Also taking into
consideration the algorithms’ computational complexity, a toolbox with these
three unsupervised anomaly detection algorithms suffices for finding anomalies
in this representative collection of multivariate datasets. By providing
access to code and datasets, our study can be easily reproduced and extended
with more algorithms and/or datasets.

Keywords: Unsupervised Anomaly Detection, Anomaly Analysis, Algorithm
Comparison

##  1 Introduction

Anomaly detection is the study of finding data points that do not fit the
expected structure of the data. Anomalies can be caused by unexpected
processes generating the data. In chemistry an anomaly might be caused by an
incorrectly performed experiment, in medicine a certain disease might induce
rare symptoms, and in predictive maintenance an anomaly can be indicative of
early system failure. Depending on the application domain, anomalies have
different properties, and may also be called by different names. Within the
domain of machine learning (and hence also in this paper), anomaly detection
is often used interchangeably with outlier detection.

Unsupervised, data-driven, detection of anomalies is a standard technique in
machine learning. Throughout the years, many methods, or algorithms, have been
developed in order to detect anomalies. Some of these algorithms aim to solve
specific issues, such as high dimensionality. Other methods try to detect
anomalies in the general sense, and focus on high performance or low
computational or memory complexity. Due to the many algorithms available, it
is hard to determine which algorithm is best suited for a particular use case,
especially for a user who is not intimately familiar with the field of anomaly
detection.

Several studies have been performed to provide guidelines on when to apply
which algorithm. Some review studies (Malik et al., 2014; Ruff et al., 2021),
give advice based on the theoretical properties of the algorithms. In recent
years, several studies have been conducted that empirically compare a number
of anomaly detection algorithms on a range of datasets.

Emmott et al. (2015) study 8 well-known algorithms on 19 datasets. They find
Isolation Forest to perform the best overall, but recommend using ABOD (Angle-
Based anomaly Detection) or LOF (Local anomaly Factor) when there are multiple
clusters present in the data.

Campos et al. (2016) compare 12 $k$-nearest neighbours based algorithms, on 11
base datasets. They find LOF to significantly outperform a number of other
methods, while KDEOS (Kernel Density Estimation anomaly Score) performs
significantly worse than most algorithms.

Goldstein and Uchida (2016) compare 19 algorithms on 10 datasets. Unlike
Campos et al., Goldstein and Uchida perform no explicit optimization or
selection, but rather evaluate the average performance over a range of
sensible hyperparameter settings. With methods based on $k$-nearest neighbours
generally giving stable results, Goldstein and Uchida recommend $k$NN
($k$-nearest neighbours) for global anomalies, LOF for local anomalies, and
HBOS (Histogram-Based anomaly Selection) in general (see 2.1 for an
explanation of global and local anomalies). Goldstein and Uchida compare on a
dataset basis, without any overall statistical analysis.

More recently, Domingues et al. (2018), apply 14 algorithms on 15 datasets,
some of which are categorical. They find IF (Isolation Forest) and robust KDE
(Kernel Density Estimation) to perform best, but note that robust KDE is often
too expensive too calculate for larger datasets.

Steinbuss and Böhm (2021) propose a novel strategy for synthesizing anomalies
in real-world datasets using several statistical distributions as a sampling
basis. They compare 4 algorithms across multiple datasets derived from 19 base
datasets, both using the original and synthesized anomalies. They find $k$NN
and IF to work best for detecting global anomalies, and LOF to work best for
local and dependency anomalies. In the same year, Soenen et al. (2021) study
the effect of hyperparameter optimization strategies on the evaluation and
propose to optimize hyperparameters on a small validation set, with evaluation
on a much larger test set. In their comparison of 6 algorithms on 16 datasets,
IF performs the best, closely followed by CBLOF/u-CBLOF ((unweighted-)Cluster-
Based Local Outlier Factor) and $k$NN, while OCSVM (One-Class Support Vector
Machine) performs worst unless optimized using a substantially larger
validation set than the other algorithms.

Han et al. (2022) performed an extensive comparison of anomaly detection
methods, including supervised and semi-supervised algorithms. They compare 14
unsupervised algorithms on 47 tabular datasets using out-of-the-box
hyperparameter settings. They subsample larger datasets to a maximum of 10.000
samples, duplicate samples for those datasets smaller than 1000 samples. They
find no significant differences between unsupervised algorithms. While real-
world datasets are being used, the anomalies in each dataset are generated
synthetically according to 4 different type definitions (see section 2.1), and
they compare the performance for each different type. Additionally, they have
analyzed more complex benchmark datasets used in CV and NLP, such as CIFAR10
(Krizhevsky et al., 2009) and the Amazon dataset (He and McAuley, 2016) by
performing neural-based feature extraction.

Other studies are of a more limited scope, and cover for example methods for
high-dimensional data (Xu et al., 2018), or consider only ensemble methods
(Zimek et al., 2014).

The studies done by Campos et al. (2016); Goldstein and Uchida (2016);
Domingues et al. (2018); Steinbuss and Böhm (2021); Soenen et al. (2021); Han
et al. (2022) have several limitations when used as a benchmark. Firstly, with
the exception of Han et al., all studies were done on a rather small
collection of datasets. Secondly, these studies cover only a small number of
methods. Campos et al. compare only $k$NN-based approaches, while Goldstein
and Uchida fail to cover many of the methods that have gained traction in the
last few years, such as IF (Liu et al., 2008) and variants thereof (Hariri et
al., 2019). Soenen et al. consider just 6 commonly used methods, Steinbuss and
Böhm cover 4 methods and Han et al. cover 14 unsupervised methods.

Some of these studies consider the performance on datasets containing specific
types of anomalies, such as global or local anomalies. Specifically, Steinbuss
and Böhm look at the performance of different algorithms on datasets
containing synthesized global, local, and dependency anomalies. Similarly, Han
et al. synthesize these three types of anomalies as well as cluster anomalies
for use in their comparison. Goldstein and Uchida’s study is, to the best of
our knowledge, the only one that analyzes real-world, i.e., non-synthesized
global and local anomalies. In particular, they analyze the ‘pen-local’ and
‘pen-global’ dataset, two variants of the same dataset where different classes
were selected to obtain local and global anomalies specifically.

In practice, very little is known regarding what types of anomalies are
present in commonly used benchmark datasets, and thus large scale comparisons
on real-world data for specifics types are still missing. In this study we
apply a large number of commonly used anomaly detection methods on a large
collection of multivariate datasets, to discover guidelines on when to apply
which algorithms. We explicitly choose to perform no optimization of
hyperparameters, so as to evaluate the performance of algorithms in a truly
unsupervised manner. Instead, we evaluate every algorithm over a range of
sensible hyperparameters, and compare average performances. This contrasts
with Soenen et al. (2021), who perform extensive optimization on a small
validation set and thereby supply guidelines for semi-supervised detection or
active learning. Our approach rather is similar to that used by Domingues et
al. (2018), who also compare off-the-shelf performance. To the best of our
knowledge, ours is the largest study of its kind performed so far.

##  2 Background

###  2.1 Types of anomalies

Many different definitions of anomalies and their properties exist, many of
these have been defined in an isolated context, not considering the
relationships with other definitions or properties (Breunig et al., 2000).
More recently, a review by Foorthuis (2021) tried to unify definitions across
multiple subdomains of anomaly detection in order to encompass all types of
anomalies. These definitions however do not encompass many properties or types
of anomalies, such as clustered or dependency anomalies.

Rather than aiming to redefine every distinct type of anomaly, we treat
anomalies as being able to have multiple, sometimes non-exclusive properties.
Instead, we define four scales of non-exclusive properties which, when
combined, encompass all types of anomalies found in multivariate tabular data
in literature known to us.

(a) Enclosed anomaly example

(b) Peripheral anomaly example

(c) Global anomaly example

(d) Local anomaly example

(e) Isolated anomaly example

(f) Clustered anomaly example

(g) Univariate anomaly example

(h) Multivariate anomaly example

Figure 1: 8 examples of different types of anomalies along the 4 defined
property axes. Normal data are visualized as blue points, while anomalies are
visualized as red crosses.

####  2.1.1 Enclosed and peripheral anomalies

Anomalies can be surrounded in the feature space by normal data. When this
occurs, we define them as enclosed anomalies. On the other end of this axis,
peripheral anomalies occupy the edges of the feature space, and have one or
more attribute scores either below the minimum or above the maximum of the
scores of the normal data region. Examples of both enclosed and peripheral
anomalies can be found in Figures 1(a) and 1(b) respectively.

####  2.1.2 Global and local density anomalies

The most commonly discussed types of anomalies are the global and local
anomalies. These definitions follow from the work of Breunig et al. (2000).
Global anomalies are points which can be isolated from normal data because
they occupy a lower density region of the feature space. Local anomalies
however, cannot be separated using just these criteria. Local anomalies rather
are located in regions with a density which is low compared to nearby normal
regions. This allows for well-defined anomalies even when multiple clusters
with differing density functions exist in the feature space. Examples of both
global and local density anomalies can be found in Figures 1(c) and 1(d)
respectively.

####  2.1.3 Isolated and clustered anomalies

Most often, anomalies are isolated, and consist of single datapoints without
any additional, normal or anomaly, datapoints nearby. In many practical cases,
anomalies are not that singular, and small groups of anomalies form clusters,
leading to clustered anomalies. Clustered anomalies are closely related to the
phenomenon known as “masking”, where similar anomalies mask each other’s
presence by forming a cluster (Liu et al., 2008). Examples of both isolated
and clustered anomalies can be found in Figures 1(e) and 1(f) respectively.

####  2.1.4 Univariate and multivariate anomalies

Some anomalies are clearly univariate in nature. That is, they can be
identified by just a single feature score in an anomalous range. Other
anomalies are multivariate in nature, requiring a specific combination of
feature scores to be identified as anomalies. These multivariate anomalies are
also often called dependency anomalies, as they differ from the normal
dependency, or causal, structure of the data. Examples of both isolated and
clustered anomalies can be found in Figures 1(g) and 1(h) respectively.

##  3 Materials and Methods

We evaluate the effectiveness of $32$ different algorithms, listed in Table 1.
We evaluate each algorithm multiple times for each dataset, each with a
different set of sensible hyperparameters. The results are then averaged
across hyperparameter settings, leading to a single average ROC-AUC score for
each method-dataset combination. We refrain from optimizing hyperparameters,
e.g., using cross-validation, to reflect the real-world situation in which no
labels are available for training the models. While unsupervised optimization
of hyperparameters has been studied Thomas et al. (2016) it has not been
applied to most algorithms considered in this study.

###  3.1 Algorithms

Of the $32$ methods, $27$ were used as implemented in the popular Python
library for anomaly detection, PyOD (Zhao et al., 2019). As part of this
research, we made several contributions to this open source library, such as a
memory-efficient implementation of the COF (Connectivity-based Outlier Factor)
method, as well as an implementation of the Birgé and Rozenholc (2006) method
for histogram size selection, which is used in HBOS and LODA (lightweight on-
line detector of anomalies). For EIF (extended isolation forest) we used the
implementation provided by the authors in the Python package “eif” by Hariri
et al. (2019). We implemented the ODIN (Outlier Detection using Indegree
Number) method in Python, and it is being prepared as a submission to the PyOD
package. The ensemble-LOF method was implemented using the base LOF algorithm
from PyOD. DeepSVDD is applied based on the publicly available code by its
author Lukas Ruff111https://github.com/lukasruff/Deep-SVDD-PyTorch, and we
modified it to work on general tabular datasets.

We left out several of the implemented methods in the PyOD package, such as
LOCI and ROD, because they have a time or memory complexity of
$\mathcal{O}(n^{3})$, with $n$ the number of data points. The PyOD SOS method
was also ignored, due to its $\mathcal{O}(n^{2})$ memory requirement. None of
these methods performed notably well compared to the other algorithms that we
included on the smaller datasets where evaluation was feasible.

[!htb]  Name Hyperparameters Publication ABOD FastABOD, $k=60$ Kriegel et al.
(2008) AE $n_{\textrm{\tiny layers}}={1,2,3}$, shrinkage
factor$={0.2,0.3,0.5}$ Japkowicz et al. (1995) ALAD $n_{\textrm{\tiny
layers}}=3$, shrinkage factor$={0.2,0.3,0.5}$ Zenati et al. (2018) CBLOF
$k={2,..,14}$, $\alpha={0.7,0.8,0.9}$, $\beta={3,5,7}$ He et al. (2003) COF
$k={5,10,15,20,25,30}$ Tang et al. (2002) COPOD Li et al. (2020) DeepSVDD
$n_{\textrm{\tiny layers}}={1,2,3}$, shrinkage factor$={0.2,0.3,0.5}$ Ruff et
al. (2019) ECOD Li et al. (2022) EIF $n_{\textrm{\tiny trees}}=1000$,
$n_{\textrm{\tiny samples}}={128,256,512,1024}$, no replacement, extension
levels: ${1,2,3}$ Hariri et al. (2019) ensemble-LOF maximum LOF score over
$k={5,...,30}$ Breunig et al. (2000) gen2out Lee et al. (2021) GMM
$n_{\textrm{\tiny gaussians}}={1,...,15}$ Agarwal (2007) HBOS
$n_{\textrm{\tiny bins}}$ based on Birgé-Rozenholc criterium Goldstein and
Dengel (2012) IF $n_{\textrm{\tiny trees}}=1000$, $n_{\textrm{\tiny
samples}}={128,256,512,1024}$, no replacement Liu et al. (2008) INNE 200
estimators Bandaragoda et al. (2018) KDE Gaussian kernel Latecki et al. (2007)
kNN $k={5,...30}$, mean distance Ramaswamy et al. (2000) kth-NN $k={5,...30}$,
largest distance Ramaswamy et al. (2000) LMDD $n_{\textrm{\tiny
shuffles}}=100$, MAD dissimilarity function Arning et al. (1996) LODA
$n_{\textrm{\tiny bins}}$ based on Birgé-Rozenblac criterium, 100 random
projections Pevnỳ (2016) LOF $k={5,...,30}$ Breunig et al. (2000) LUNAR
$k={5,10,15,20,25,30}$ Goodge et al. (2022) MCD subset
fraction$={0.6,0.7,0.8,0.9}$ Rousseeuw and Driessen (1999) OCSVM RBF kernel,
$\nu={0.5,0.6,0.7,0.8,0.9}$, $\gamma=1/d$ Schölkopf et al. (1999) ODIN
$k={5,...30}$ Hautamaki et al. (2004) PCA selected PCs explain
$>{30,50,70,90}$% of variance Shyu et al. (2003) sb-DeepSVDD $n_{\textrm{\tiny
layers}}={1,2,3}$, shrinkage factor$={0.2,0.3,0.5}$ Ruff et al. (2019) SOD
$k={20,25,30}$, $l={10,14,18}$, $\alpha={0.7,0.8,0.9}$ Kriegel et al. (2009)
SO-GAAL stop epochs$=50$ Liu et al. (2019) u-CBLOF $k={2,..,14}$,
$\alpha={0.7,0.8,0.9}$, $\beta={3,5,7}$ Amer and Goldstein (2012) VAE
$n_{\textrm{\tiny layers}}={1,2,3}$, shrinkage factor$={0.2,0.3,0.5}$ An and
Cho (2015) $\beta$-VAE $n_{\textrm{\tiny layers}}={1,2,3}$, shrinkage
factor$={0.2,0.3,0.5}$, $\gamma={10,20,50}$ Zhou et al. (2020)

Table 1: Overview of the algorithms, the setting of the hyperparameters, the
year of original publication, and the author(s). For the neural networks, the
“shrinkage factor” hyperparameter indicates that any subsequent layer in the
encoder is defined by: $\textrm{layer size}_{n+1}=\textrm{layer
size}_{n}\times\textrm{shrinkage factor}$.

###  3.2 Data

####  3.2.1 Datasets

In this study we consider a large collection of datasets from a variety of
sources. We focus on real-valued, multivariate, tabular data, comparable to
the datasets used by Fernández-Delgado et al. (2014); Campos et al. (2016);
Goldstein and Uchida (2016); Soenen et al. (2021); Domingues et al. (2018).
Table 2 contains a summary of the datasets, listing each dataset’s origin,
number of samples, number of variables, number and percentage of anomalies,
and how many duplicates were removed in the processing of the data.

Our collection consists for the most part of datasets from the ODDS data
collection (Rayana, 2016), specifically the multi-dimensional point datasets.
It is a collection of various datasets, mostly adapted from the UCI machine
learning repository (Dua and Graff, 2017). All datasets are real-valued,
without any categorical data. Curation of this collection is sadly not fully
up-to-date, causing some of the listed datasets to be unavailable. The
unavailable datasets were omitted from this comparison.

In addition to the ODDS dataset, we also incorporate publicly available
datasets used in earlier anomaly detection research: several datasets from the
comparison by Goldstein and Uchida (2016), from the comparison of Campos et
al. (2016) using ELKI (Schubert and Zimek, 2019), from a study on Generative
Adversial Active Learning, or GAAL (Liu et al., 2019), from a study on
extended Autoencoders (Shin and Kim, 2020), from the ADBench comparison (Han
et al., 2022), and from a study on Efficient Online Anomaly Detection (EOAD)
(Brandsæter et al., 2019). Datasets from these latter sources that are
(near-)duplicates of datasets present in the ODDS collection are left out. In
Table 2 we specify exactly where each dataset was downloaded or reconstructed
from.

Emmott et al. (2013, 2015) present a systematic methodology to construct
anomaly detection benchmarks, which is then also extensively applied by Pevnỳ
(2016). In this paper, we chose not to construct our own benchmark datasets,
which inevitably leads to some arbitrariness and possibly bias, but instead we
rely on a large collection of different datasets used in earlier comparison
studies. Synthetic datasets are not included in this study, as real-world
datasets are generally considered the best available tool for benchmarking
algorithms (Emmott et al., 2015; Domingues et al., 2018; Ruff et al., 2021).
While real-world datasets are preferred for benchmarking, we note the
usefulness of synthetic data in when studying specific properties of anomaly
detection algorithms.

Table 2: Summary of the $52$ multivariate datasets used in our anomaly
detection algorithm comparison: the colloquial name of the dataset, origin of
the dataset, the number of samples after removal of duplicates, variables,
anomalies and removed duplicates, as well as the percentage of anomalies.

| Origin | #samples | #variables | #outliers | %outliers | #duplicates | #removed variables
---|---|---|---|---|---|---|---
Name |  |  |  |  |  |  |
aloi | Goldstein | 49533 | 27 | 1507 | (3.04%) | 466 | 0
annthyroid | ODDS | 7062 | 6 | 534 | (7.56%) | 138 | 0
arrhythmia | ODDS | 452 | 257 | 66 | (14.6%) | 0 | 17
backdoor | ADBench | 87020 | 193 | 1879 | (2.16%) | 8309 | 3
breastw | ADBench | 449 | 9 | 236 | (52.56%) | 234 | 0
campaign | ADBench | 41176 | 62 | 4639 | (11.27%) | 12 | 0
cardio | ODDS | 1822 | 21 | 175 | (9.6%) | 9 | 0
celeba | ADRepository | 113983 | 39 | 2911 | (2.55%) | 88616 | 0
cover | ODDS | 286048 | 10 | 2747 | (0.96%) | 0 | 0
donors | ADBench | 35421 | 10 | 7667 | (21.65%) | 583905 | 0
fault | ADBench | 1941 | 27 | 673 | (34.67%) | 0 | 0
fraud | ADBench | 275661 | 29 | 473 | (0.17%) | 9146 | 0
glass | ODDS | 213 | 9 | 9 | (4.23%) | 1 | 0
hepatitis | ELKI | 80 | 19 | 13 | (16.25%) | 0 | 0
hrss_anomalous_optimized | ex-AE | 5874 | 18 | 1468 | (24.99%) | 13760 | 0
hrss_anomalous_standard | ex-AE | 7040 | 18 | 1830 | (25.99%) | 16605 | 0
http | ODDS | 221900 | 3 | 72 | (0.03%) | 345598 | 0
internetads | ELKI | 1966 | 1555 | 368 | (18.72%) | 0 | 0
ionosphere | ODDS | 350 | 33 | 125 | (35.71%) | 1 | 0
landsat | ADBench | 6435 | 36 | 1333 | (20.71%) | 0 | 0
letter | ODDS | 1598 | 32 | 100 | (6.26%) | 2 | 0
magic.gamma | ADBench | 18905 | 10 | 6573 | (34.77%) | 115 | 0
mammography | ODDS | 7848 | 6 | 253 | (3.22%) | 3335 | 0
mi-f | ex-AE | 24945 | 40 | 2048 | (8.21%) | 341 | 5
mi-v | ex-AE | 24945 | 40 | 3942 | (15.8%) | 341 | 5
mnist | ODDS | 7603 | 78 | 700 | (9.21%) | 0 | 22
musk | ODDS | 3062 | 166 | 97 | (3.17%) | 0 | 0
nasa | ex-AE | 4687 | 32 | 755 | (16.11%) | 0 | 0
optdigits | ODDS | 5198 | 62 | 132 | (2.54%) | 18 | 2
pageblocks | ELKI | 5393 | 10 | 510 | (9.46%) | 0 | 0
parkinson | ELKI | 195 | 22 | 147 | (75.38%) | 0 | 0
pen-global | Goldstein | 808 | 16 | 90 | (11.14%) | 0 | 0
pen-local | Goldstein | 6723 | 16 | 10 | (0.15%) | 0 | 0
pendigits | ODDS | 6870 | 16 | 156 | (2.27%) | 0 | 0
pima | ODDS | 768 | 8 | 268 | (34.9%) | 0 | 0
satellite | ODDS | 6435 | 36 | 2036 | (31.64%) | 0 | 0
satimage-2 | ODDS | 5801 | 36 | 69 | (1.19%) | 2 | 0
seismic-bumps | ODDS | 2578 | 21 | 170 | (6.59%) | 6 | 3
shuttle | ODDS | 49097 | 9 | 3511 | (7.15%) | 0 | 0
skin | ADBench | 51433 | 3 | 14654 | (28.49%) | 193624 | 0
smtp | ODDS | 71230 | 3 | 21 | (0.03%) | 23926 | 0
spambase | GAAL | 4206 | 57 | 1678 | (39.9%) | 0 | 0
speech | ODDS | 3686 | 400 | 61 | (1.65%) | 0 | 0
stamps | ELKI | 340 | 9 | 31 | (9.12%) | 0 | 0
thyroid | ODDS | 3656 | 6 | 93 | (2.54%) | 116 | 0
vertebral | ODDS | 240 | 6 | 30 | (12.5%) | 0 | 0
vowels | ODDS | 1452 | 12 | 46 | (3.17%) | 4 | 0
waveform | GAAL | 3442 | 21 | 99 | (2.88%) | 0 | 0
wbc | ODDS | 377 | 30 | 20 | (5.31%) | 1 | 0
wbc2 | ADBench | 223 | 9 | 10 | (4.48%) | 0 | 0
wilt | ELKI | 4819 | 5 | 257 | (5.33%) | 0 | 0
wine | ODDS | 129 | 13 | 10 | (7.75%) | 0 | 0
wpbc | ADBench | 198 | 33 | 47 | (23.74%) | 0 | 0
yeast | ADBench | 1453 | 8 | 481 | (33.1%) | 31 | 0
yeast6 | EOAD | 1453 | 8 | 35 | (2.41%) | 31 | 0

####  3.2.2 Preprocessing

Several steps have been undertaken to be able to compare the performance of
the various algorithms on the different datasets. Firstly, all duplicate
samples have been removed from each dataset, as many anomaly detection methods
cannot handle data with duplicates (Campos et al., 2016). Furthermore, all
variables in all datasets have been scaled and centered. Although some
algorithms, such as Isolation Forest, can implicitly handle variables with
different scales, methods that involve, for example, distance or cross-product
calculations are strongly affected by the scale of the variables. Centering is
done for each variable in a dataset by subtracting the median. Scaling is
performed by dividing each variable by its interquartile range. Our choice of
centering and scaling procedure is deliberate, as both the median and
interquartile range are influenced less by the presence of anomalies than the
mean and standard deviation. This procedure is generally considered to be more
stable than standardization when anomalies are known to be present (Rousseeuw
and Croux, 1993).

###  3.3 Evaluation Procedure

Due to the unsupervised nature of anomaly detection, it is generally more
useful to evaluate anomaly scores, rather than binary labels as also produced
by some algorithms. Using scores, samples can be ranked, providing insights
into the underlying nature of anomalies. For each dataset, we calculate
anomaly scores on all available data at once, without using any cross-
validation or train-test splits. The scores from this unsupervised analysis
are then used in conjunction with the ground truth labels to evaluate the
performance of the algorithm. In order to compare the different algorithms we
calculate the performance for each algorithm-dataset combination in terms of
the AUC (area under the curve) value resulting from the ROC (receiver
operating characteristic) curve. This is the most commonly used metric in
anomaly detection evaluations (Goldstein and Uchida, 2016; Campos et al.,
2016; Xu et al., 2018), which can be readily interpreted from a probabilistic
view. We considered using other metrics, such as the R-precision or average
precision and their chance-adjusted variants introduced by Campos et al.
(2016), but found these to be less stable, and harder to interpret.

For each dataset we rank the AUC scores calculated from the scores produced by
each algorithm. Following the recommendations for the comparison of
classifiers by Demšar (2006), we use the Iman-Davenport statistic (Iman and
Davenport, 1980) in order to determine whether there is any significant
difference between the algorithms. If this statistic falls below the desired
critical value corresponding to a $p$-value of 0.05, we apply the Nemenyi
post-hoc test (Nemenyi, 1963) to then assess which algorithms differ
significantly from each other.

In some of the visualizations in this paper we plot the percentage of maximum
AUC, defined as

$\widetilde{\textrm{AUC}}(a,d)=\frac{\textrm{AUC}(a,d)}{\max_{a^{\prime}\in
A}\textrm{AUC}(a^{\prime},d)}\times 100\>,$

with $a$ one of the $A$ algorithms and $d$ one of the $D$ datasets.

###  3.4 Reproducibility

In order to reproduce all our experiments, we have provided access to a public
GitHub repository222https://github.com/RoelBouman/outlierdetection containing
the code and datasets used for all experiments as well as for the production
of all figures and tables presented in this paper.

##  4 Results

###  4.1 Overall algorithm performance

In order to gauge the performance, we evaluated each algorithm on each dataset
using the AUC measure corresponding to a ROC curve. To evaluate the
performance across multiple sensible hyperparameters the AUC value for a given
method is the average of the AUC of each hyperparameter setting evaluated. In
our analysis, we found three datasets on which nearly every evaluated
algorithm performed subpar, i.e. with all AUC values between 0.4 and 0.6.
These datasets, the ‘hrss_anomalous_standard’, ‘wpbc’ and ‘yeast’ dataset,
were therefore excluded from further analysis. Some datasets showed no AUC
values above 0.6, but did show AUC values below 0.4. In these cases, the
detector performs better when the labels are inverted. This behaviour was
observed in the ‘skin’ and ‘vertebral’ datasets. The construction of these
datasets was done based on treating the largest group of samples as the normal
(0) class, and the smaller group as the anomaly (1) class. Yet, for both these
sets, the more heterogeneous group is chosen as the normal class, in constrast
to normal anomaly definitions. For these datasets, we inverted the labelling
and recalculated the AUC values.

Figure 2 shows the distribution of the performance for each method. In order
to compare the AUC across different datasets, which might have different
baseline performances, in a boxplot we express the AUC in terms of its
percentage of the maximum AUC value obtained by the best performing algorithm
on that particular dataset.

Figure 2: Boxplots of the performance of each algorithm on each dataset in
terms of percentage of maximum AUC. The maximum AUC is the highest AUC value
obtained by the best performing algorithm on that particular dataset. The
whiskers in the boxplots extend 1.5 times the interquartile range past the low
and high quartiles. Dataset-algorithm combinations outside of the whiskers are
marked as diamonds.

It can be seen that many of the algorithms perform comparably, with a median
percentage of maximum AUC around 90%. Several lower medians, as well as wider
quartiles, can be observed.

To determine whether any of the observed differences in performance from
Figure 2 are significant, we apply the Iman-Davenport test. This yielded a
test-statistic of $18.076$ , far above the critical value of $0.620$ , thus
refuting the null hypothesis. We then applied the Nemenyi post-hoc test to
establish which algorithms significantly outperform which other algorithms.
The results of this Nemenyi post-hoc test are summarized in Table 3.

Table 3: Significant differences between algorithms based on Nemenyi post-hoc
analysis. ++/+ denotes that the row algorithm outperforms the column algorithm
at $p=0.05$/$p=0.10$ respectively, while – denotes that the row algorithm is
outperformed by the column algorithm at $p=0.05$. Rows and columns are sorted
by descending and ascending mean performance, respectively. Columns are not
shown when the column algorithm is not outperformed by any other algorithms at
$p=0.05$ or $p=0.10$. The last column shows the mean AUC.

|  CBLOF |  sb-DeepSVDD |  SO-GAAL |  ALAD |  COF |  DeepSVDD |  ODIN |  LOF |  ensemble-LOF |  LMDD |  LODA |  SOD |  PCA |  VAE |  beta-VAE |  Mean AUC
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
EIF | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ |  |  | 0.773
kth-NN | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | + | ++ | 0.769
kNN | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ |  |  | 0.766
IF | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ |  | + |  | ++ |  |  | 0.764
MCD | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ |  |  |  |  |  |  |  | 0.756
INNE | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ |  |  |  |  |  |  |  | 0.749
KDE | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | + | ++ | + | ++ |  |  | 0.748
gen2out | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ |  |  |  |  |  |  |  | 0.748
ABOD | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | + |  |  |  |  |  |  | 0.747
u-CBLOF | ++ | ++ | ++ | ++ | ++ | ++ | ++ |  |  |  |  |  |  |  |  | 0.741
COPOD | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ |  |  |  |  |  |  |  | 0.738
HBOS | ++ | ++ | ++ | ++ | ++ | ++ | ++ |  |  |  |  |  |  |  |  | 0.725
OCSVM | ++ | ++ | ++ | ++ | ++ | ++ | ++ |  |  |  |  |  |  |  |  | 0.723
AE | ++ | ++ | ++ | ++ |  | ++ | + |  |  |  |  |  |  |  |  | 0.720
LUNAR | ++ | ++ | ++ | ++ |  | ++ | + |  |  |  |  |  |  |  |  | 0.711
beta-VAE | ++ | ++ | + | ++ |  | ++ |  |  |  |  |  |  |  |  |  | 0.710
VAE | ++ | ++ | ++ | ++ |  | ++ |  |  |  |  |  |  |  |  |  | 0.710
ECOD | ++ | ++ | ++ | ++ |  | ++ | + |  |  |  |  |  |  |  |  | 0.709
PCA |  | ++ |  |  |  |  |  |  |  |  |  |  |  |  |  | 0.692
SOD | ++ | ++ |  |  |  |  |  |  |  |  |  |  |  |  |  | 0.687
LODA | + | ++ |  |  |  |  |  |  |  |  |  |  |  |  |  | 0.685
LMDD | ++ | ++ |  |  |  |  |  |  |  |  |  |  |  |  |  | 0.684
GMM | ++ | ++ | ++ | ++ |  | ++ | + |  |  |  |  |  |  |  |  | 0.683
ensemble-LOF |  | ++ |  |  |  |  |  |  |  |  |  |  |  |  |  | 0.661
LOF |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 0.630
ODIN |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 0.595
DeepSVDD |  |  |  |  |  |  |  |  |  |  |  |  |  | – | – | 0.593
COF |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 0.592
ALAD |  |  |  |  |  |  |  |  |  |  |  |  |  | – | – | 0.569
SO-GAAL |  |  |  |  |  |  |  |  |  |  |  |  |  | – | - | 0.564
sb-DeepSVDD |  |  |  |  |  |  |  |  | – | – | – | – | – | – | – | 0.553
CBLOF |  |  |  |  |  |  |  |  |  | – | - | – |  | – | – | 0.519

Table 3 reveals that there are indeed several algorithms significantly
outperforming many other algorithms. Most notable here is $k$th-NN, which
significantly outperforms 14/15 algorithms evaluated in this study at the
$p=0.05/p=0.10$ significance level respectively. The popular Isolation Forest
method and its extended version show similar consistent performance. Since the
computational complexity of Isolation Forest and variants thereof scales
linearly with the number of samples $n$, this may give them a clear edge over
methods such as $k$NN and derivatives for large datasets, with a computational
complexity that scales quadratically or at best with $\mathcal{O}(n\log{n})$
when optimized.

From Figure 2 and Table 3 we can also observe that the original CBLOF method
is by far the worst performing method, being significantly outperformed by
21/22 algorithms at at the $p=0.05/p=0.10$ significance level respectively..
This corroborates the results of Goldstein and Uchida (2016), who also found
CBLOF to consistently underperform, while its unweighted variant, u-CBLOF,
performs comparably to other algorithms.

From these overall results it is clear that many of the neural networks do not
perform well. (Soft-boundary) DeepSVDD, ALAD, and SO-GAAL all occupy the lower
segment of overall method performance. We surmise that there are two likely
reasons for this phenomenon. Firstly, these methods were not designed with
tabular data in mind, and they can’t leverage the same feature extraction
capabilities that give them an edge on their typical computer vision tasks.
Secondly, these methods are relatively complex, making it exceedingly hard to
specify general hyperparameter settings and architectures which work on a
large variety of datasets. Not all neural networks suffer from this problem,
as the auto-encoder and variants, as well as LUNAR, perform about average.
This is likely caused by more straightforward optimisation criteria.

In addition to the neural networks, the local methods, such as LOF, ODIN, COF,
and CBLOF, are some of the most underperforming methods. This result for LOF
stands in stark contrast to the results of Campos et al. (2016), who found LOF
to be among the best performing methods. This is most likely caused by their
evaluation on a small number of datasets with a low percentage of anomalies,
which causes LOF to suffer less from swamping or masking (Liu et al., 2008).
We further study this finding in Section 4.3.

###  4.2 Clustering algorithms and datasets

To visualize the similarities between algorithms on one hand, and the datasets
on the other, Figure 3 shows a heatmap of the performance of each
dataset/algorithm combination and dendrograms of two hierarchical clusterings,
one on the datasets, and one on the algorithms. We applied average linkage
cluster analysis with Pearson correlation as distance measure and optimized
the leaf orderings for visualization with the method of Bar-Joseph et al.
(2001).

Figure 3: Clustered heatmap of the ROC/AUC performance of each algorithm. The
algorithms and datasets are each clustered using hierarchical clustering with
average linkage and the Pearson correlation as metric.

Figure 3 shows that many similar algorithms cluster together in an expected
way, with families of algorithms forming their own clusters. Some interesting
patterns can be observed at a larger level. For the algorithms, we obtain
several fairly distinct clusters. Firstly, CBLOF is distinct, as it
underperforms on nearly every dataset. Similarly, underperforming methods such
as SO-GAAL, GMM, and ALAD do not clearly belong to other emerging clusters.
The local methods, COF, ensemble-LOF, LOF and ODIN, form a separate cluster.
These algorithms, which are specifically designed to detect local anomalies,
work well on a few (approximately a quarter) of the datasets, but do not
perform well for most other datasets. We have a small cluster of $k$NN and
related methods such as ABOD and SOD, that performs decently for all datasets.
Lastly, a large cluster of methods seems to negatively correlate with the
local methods, performing well for most (approximately three-quarters) of the
datasets, but less so for the remainder.

The datasets split into two clearly distinct clusters: one cluster of datasets
on which the local algorithms perform well, and another cluster of datasets on
which the large cluster of algorithms performs well. Combining the two-way
clustering with knowledge of the algorithms suggests that approximately one
quarter of the datasets comprises so-called local problems, while the other
three-quarters comprises global problems. This is corroborated by specifically
constructed local and global sets ‘pen-local’ and ‘pen-global’, that clearly
belong to their expected clusters. This observation is corroborated by
research by Steinbuss and Böhm (2021) and Emmott et al. (2015), who similarly
find differences between what they categorize as local/dependency and multi-
cluster anomalies respectively, and global anomalies. We can not clearly
observe any other clear patterns of different anomaly properties arising from
our analysis.

To the best of our knowledge, no previous study on naturally occurring
anomalies in real-world data has looked, in detail, into the difference
between the performance of algorithms when specifically being applied to
either global or local anomaly detection problems.

###  4.3 Performance on global and local problems

In the previous section, we discovered a clear distinction between two
clusters of datasets: one with the “local” datasets ‘aloi’, ‘donors’, ‘fault’,
‘glass’, ‘ionosphere’, ‘landsat’, ‘letter’, ‘nasa’, ‘parkinson’, ‘pen-local’,
‘vowels’, and ‘wilt’, and another with the remaining “global” datasets.
Suspecting that different methods may do well on different types of datasets,
we repeated the significance testing procedure from Section 4.1 for both
clusters separately.

Figure 4: Boxplots of the performance of each algorithm on the “local”
datasets in terms of percentage of maximum AUC. The maximum AUC is the highest
AUC value obtained by the best performing algorithm on that particular
dataset. The whiskers in the boxplots extend 1.5 times the interquartile range
past the low and high quartiles. Dataset-algorithm combinations outside of the
whiskers are marked as diamonds.

Performance boxplots for all algorithms applied on the collection of local
datasets can be found in Figure 4. Figure 4 clearly shows the reversed
performance of some of the local methods for anomaly detection. Where COF,
ensemble-LOF, and LOF were among the worst performers over the entire
collection, they are among the best performers when applied to the problems
for which they were specifically developed. This phenomenon is a fine example
of Simpson’s paradox (Simpson, 1951). This also partially explains the
difference in findings of our overall comparison and the comparison of Campos
et al. (2016).

We then repeated the Nemeny-Friedman post hoc test on just the local datasets.
The results for this analysis are summarized in Table 4. $k$NN, and ABOD are
the top performers, and significantly outperform 16 other methods at $p=0.05$.

Table 4: Significant differences between algorithms on the collection of local
problems based on Nemenyi post-hoc analysis. ++/+ denotes that the row
algorithm outperforms the column algorithm at $p=0.05$/$p=0.10$. Rows and
columns are sorted by descending and ascending mean performance, respectively.
Columns are not shown when the column algorithm is not outperformed by any
other algorithms at $p=0.05$ or $p=0.10$. The last column shows the mean AUC.

|  SO-GAAL |  PCA |  ECOD |  beta-VAE |  ALAD |  LMDD |  COPOD |  AE |  LODA |  OCSVM |  sb-DeepSVDD |  DeepSVDD |  VAE |  HBOS |  CBLOF |  gen2out |  Mean AUC
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
kNN | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | 0.740
ABOD | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | 0.733
kth-NN | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | + |  |  |  |  | 0.722
ensemble-LOF | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ |  |  |  |  |  |  |  |  | 0.714
LOF | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | + | + | + |  |  |  |  |  | 0.713
COF | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ |  |  |  |  |  |  |  |  | 0.713
SOD | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ |  | + | + |  |  |  |  |  | 0.711
GMM | ++ | ++ | ++ | ++ |  | ++ |  |  |  |  |  |  |  |  |  |  | 0.701
KDE | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ |  |  | + |  |  |  |  |  | 0.693
MCD |  | ++ | + | + |  |  |  |  |  |  |  |  |  |  |  |  | 0.687
LUNAR | ++ | ++ | ++ | ++ |  | ++ |  |  |  |  |  |  |  |  |  |  | 0.677
ODIN |  | + |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 0.672
u-CBLOF |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 0.635
INNE |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 0.624
EIF |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 0.614
IF |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 0.607
gen2out |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 0.585
CBLOF |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 0.577
HBOS |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 0.575
VAE |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 0.563
DeepSVDD |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 0.557
sb-DeepSVDD |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 0.546
OCSVM |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 0.543
LODA |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 0.534
AE |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 0.528
COPOD |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 0.518
LMDD |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 0.516
ALAD |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 0.511
beta-VAE |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 0.499
ECOD |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 0.492
PCA |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 0.475
SO-GAAL |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 0.423

We then repeated the analysis for the global datasets, leading to the
performance boxplots in Figure 5 and the significance results in Table 5.

Figure 5: Boxplots of the performance of each algorithm on the global datasets
in terms of percentage of maximum AUC. The maximum AUC is the highest AUC
value obtained by the best performing algorithm on that particular dataset.
The whiskers in the boxplots extend 1.5 times the interquartile range past the
low and high quartiles. Dataset-algorithm combinations outside of the whiskers
are marked as diamonds. Table 5: Significant differences between algorithms on
the collection of global problems based on Nemenyi post-hoc analysis. ++/+
denotes that the row algorithm outperforms the column algorithm at
$p=0.05$/$p=0.10$. Rows and columns are sorted by descending and ascending
mean performance, respectively. Columns are not shown when the column
algorithm is not outperformed by any other algorithms at $p=0.05$ or $p=0.10$.
The last column shows the mean AUC.

|  CBLOF |  COF |  sb-DeepSVDD |  ODIN |  ALAD |  LOF |  DeepSVDD |  SO-GAAL |  ensemble-LOF |  GMM |  SOD |  LUNAR |  LODA |  LMDD |  PCA |  Mean AUC
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
EIF | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | + | + | 0.824
IF | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | + | ++ | ++ | + |  |  | 0.814
COPOD | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | + | ++ | ++ |  |  |  | 0.809
gen2out | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ |  | ++ |  |  |  |  | 0.801
INNE | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ |  | ++ |  |  |  |  | 0.790
kth-NN | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ |  | ++ |  |  |  |  | 0.785
AE | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ |  |  |  |  |  |  | 0.782
OCSVM | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ |  | ++ |  |  |  |  | 0.781
ECOD | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ |  | ++ |  |  |  |  | 0.780
MCD | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ |  | ++ |  |  |  |  | 0.779
beta-VAE | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | + |  |  |  |  |  |  | 0.779
u-CBLOF | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | + |  |  |  |  |  |  | 0.776
HBOS | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ |  | ++ |  |  |  |  | 0.774
kNN | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ |  | ++ |  |  |  |  | 0.774
KDE | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ |  | ++ |  |  |  |  | 0.766
PCA | ++ | ++ | ++ | ++ | ++ | ++ | ++ |  |  |  |  |  |  |  |  | 0.763
VAE | ++ | ++ | ++ | ++ | ++ | ++ | ++ | + |  |  |  |  |  |  |  | 0.758
ABOD | ++ | ++ | ++ | ++ | ++ | ++ | ++ | ++ |  |  |  |  |  |  |  | 0.752
LMDD | ++ | ++ | ++ | ++ | ++ | ++ | ++ |  |  |  |  |  |  |  |  | 0.738
LODA | ++ | ++ | ++ | ++ |  |  | + |  |  |  |  |  |  |  |  | 0.734
LUNAR | ++ | ++ | ++ | ++ |  |  |  |  |  |  |  |  |  |  |  | 0.722
SOD |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 0.679
GMM | ++ | ++ | ++ | ++ |  |  |  |  |  |  |  |  |  |  |  | 0.677
ensemble-LOF |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 0.643
SO-GAAL |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 0.609
DeepSVDD |  |  |  |  |  |  |  |  |  |  |  |  | - | – | – | 0.605
LOF |  |  |  |  |  |  |  |  |  |  |  |  |  | – | – | 0.603
ALAD |  |  |  |  |  |  |  |  |  |  |  |  |  | – | – | 0.588
ODIN |  |  |  |  |  |  |  |  |  | – |  | – | – | – | – | 0.571
sb-DeepSVDD |  |  |  |  |  |  |  |  |  | – |  | – | – | – | – | 0.556
COF |  |  |  |  |  |  |  |  |  | – |  | – | – | – | – | 0.553
CBLOF |  |  |  |  |  |  |  |  |  | – |  | – | – | – | – | 0.500

From Figure 5 and Table 5 we can see that the Extended Isolation Forest has
the highest mean performance, closely followed by the regular Isolation
Forest. The Extended Isolation Forest outperforms 13/15 methods at
$p=0.05/p=0.10$ respectively. Coincidentally, these methods also have the
lowest computational and memory requirement, leaving them as the most likely
choices for global anomalies.

##  5 Discussion

In our study we compared the performance of anomaly detection algorithms on
$52$ semantically meaningful real-world tabular datasets, more than any other
recent comparison studies (Campos et al., 2016; Goldstein and Uchida, 2016; Xu
et al., 2018; Soenen et al., 2021; Steinbuss and Böhm, 2021; Domingues et al.,
2018; Han et al., 2022). A somewhat comparable study by Fernández-Delgado et
al. (2014) on classification algorithms easily considered 121 datasets. The
main reason for this discrepancy is that datasets for comparing anomaly
detection algorithms rarely include categorical variables, which are not an
issue for comparing classification algorithms. It is certainly possible to
further extend the collection of dataset, e.g., through dataset modifications.
Campos et al. (2016), Emmott et al. (2013, 2015), and Steinbuss and Böhm
(2021) modified datasets in different ways to create similar datasets with
differing characteristics from a single base dataset. While such modifications
can be useful for targeted studies, near-duplicate datasets are far from
independent and then seem detrimental to a proper statistical comparison of
anomaly detection algorithms, such as can be observed in Emmott et al. (2015).

In this study we compared $32$ of the most commonly used algorithms for
anomaly detection. This collection is certainly not exhaustive: many more
methods exist (Schubert and Zimek, 2019; Goldstein and Uchida, 2016; Emmott et
al., 2015; Ruff et al., 2021; Domingues et al., 2018), and likely even more
will be invented. Also along this axis, there is a clear discrepancy with the
study by Fernández-Delgado et al. (2014) on classification algorithms, who
incorporated 179 classifiers from 17 different families. Apparently, the
number of classification algorithms largely exceeds the number of anomaly
detection algorithms. But perhaps more importantly, there are many more solid
and easy-to-use implementations of classification algorithms in many different
machine learning libraries than there are out-of-the-box implementations of
anomaly detection algorithms. Before being able to perform the comparison in
this study, we had to spend quite some effort to clean up and sometimes re-
implement (parts of) existing code.

In this research we chose not to cover meta-techniques for ensembling. While
ensembles are of great interest, a better understanding of the performance of
base learners is an essential prerequisite before moving on to a study of
ensemble methods.

While we evaluated neural networks in our comparison, no general guidelines
exist on how to construct a well-performing network for any given dataset,
which is essential for the unsupervised setup considered in this study.
Additionally, the strength of many of these methods comes from high-level
feature extraction implicitly performed by the network, which cannot be
leveraged on the smaller tabular datasets used in this benchmark. Like Ruff et
al. (2021), we recognize that there is a major discrepancy between the
availability of classification and anomaly detection benchmark datasets useful
for deep learning approaches. More anomaly detection benchmark datasets useful
for deep learning based anomaly detection would be a welcome addition to the
field.

Cross-comparing the performance of algorithms on datasets, we noticed a clear
separation between two clusters of datasets and roughly two clusters of
corresponding algorithms. We characterized these clusters as “local” and
“global” datasets and algorithms, in correspondence with common nomenclature
in the literature (Breunig et al., 2000; Goldstein and Uchida, 2016). However,
we are well aware that this characterization may turn out to be an
oversimplification when analyzing more datasets and more algorithms in closer
detail. For example, the local and global problems likely have quite some
overlap, but need not be fully equivalent with multimodal and unimodal anomaly
detection problems, respectively. Overlap between multimodal and local
problems occurs when the different modes start having different densities, so
that local algorithms that try to estimate these local densities fare better
than global algorithms that cannot make this distinction. Further theoretical
and empirical studies, e.g., on carefully simulated datasets are needed to
shed further light.

##  6 Conclusion

Based on our research we can establish general guidelines on when users should
apply which anomaly detection methods for their problem.

In general, when a user has no a priori knowledge on whether or not their
dataset contains local or global anomalies, $k$-thNN is the best choice. It
outperforms 14 out of $32$ other evaluated methods at $p=0.05$, and is one the
highest performing method based on its mean AUC score. In case the
$\mathcal{O}(n^{2})$ (or $\mathcal{O}(n\log{n})$ when optimized) computational
complexity of $k$NN is an issue, the slightly worse performing Isolation
Forest or its extended variant, are good alternatives.

When a dataset is known to contain local anomalies, the best performing method
is $k$NN, which outperforms 16 out of $32$ methods at $p=0.10$.

Datasets containing just global anomalies are best analyzed using IF
(isolation forest) or, preferably, its extended variant EIF, which is the top
performing algorithm on the datasets containing global anomalies. COPOD,
gen2out, INNE and $k$-thNN all perform comparably, and these methods all
outperform at least 10 other methods at $p=0.05$. IF and EIF are the
algorithms with the lowest computational complexity, which are usually
preferable in practice.

Contemplating the above considerations, we are tempted to answer the question
in the title of our paper with “three”: a toolbox with $k$-thNN, $k$NN, and
EIF seems sufficient to perform well on the type of multivariate datasets
considered in our study. These three algorithms are due to the scope of this
study likely to perform well on unseen real-world multivariate datasets. This
conclusion is open for further consideration when other algorithms and/or
datasets are added to the bag, which should be relatively easy to check when
extending the code and the dataset pre-processing procedures that we open-
sourced. Future work following this study may seek to extend our comparative
analysis with diverse types of data such as raw images, texts and time-series.

Acknowledgments and Disclosure of Funding

The research reported in this paper has been partly funded by the NWO grant
NWA.1160.18.238 (PrimaVera), as well as BMK, BMDW, and the State of Upper
Austria in the frame of the SCCH competence center INTEGRATE [(FFG grant no.
892418)] part of the FFG COMET Competence Centers for Excellent Technologies
Programme.

##  Appendix A AUC scores for each algorithm-dataset combination

See Table 6.

Table 6: The AUC values for each algorithm-dataset combination.

annthyroid wbc2 mnist cover donors mi-f cardio speech breastw arrhythmia
magic.gamma hrss pageblocks http vertebral wine seismic-bumps mammography
landsat thyroid wbc campaign pima mi-v glass fault parkinson vowels yeast6
wilt pendigits hepatitis satellite waveform shuttle satimage-2 pen-local
optdigits nasa letter skin spambase ionosphere smtp internetads musk aloi pen-
global stamps GMM 0.87 0.45 0.67 0.90 0.37 0.48 0.66 0.57 0.92 0.34 0.80 0.62
0.81 1.00 0.66 0.14 0.67 0.80 0.59 0.91 0.43 0.77 0.59 0.60 0.74 0.65 0.58
0.96 0.46 0.78 0.76 0.33 0.67 0.69 0.90 0.75 0.95 0.52 0.62 0.90 0.68 0.58
0.75 0.96 0.61 0.76 0.53 0.95 0.78 CBLOF 0.37 0.20 0.64 0.87 0.46 0.57 0.53
0.52 0.48 0.48 0.50 0.54 0.64 0.31 0.47 0.30 0.45 0.43 0.55 0.24 0.31 0.41
0.42 0.55 0.52 0.59 0.43 0.83 0.40 0.57 0.45 0.37 0.63 0.69 0.98 0.32 0.66
0.47 0.45 0.58 0.62 0.59 0.72 0.65 0.55 0.61 0.54 0.61 0.34 SO-GAAL 0.89 0.81
0.58 0.60 0.18 0.65 0.85 0.47 0.15 0.70 0.55 0.55 0.86 0.55 0.42 0.22 0.57
0.77 0.46 0.93 0.42 0.54 0.32 0.60 0.71 0.41 0.54 0.08 0.20 0.41 0.90 0.76
0.55 0.41 0.58 0.72 0.05 0.61 0.51 0.42 0.73 0.44 0.81 0.66 0.45 0.90 0.50
0.76 0.90 INNE 0.92 0.92 0.82 0.95 0.32 0.56 0.87 0.47 0.82 0.73 0.71 0.55
0.96 1.00 0.60 0.81 0.71 0.81 0.54 0.99 0.93 0.81 0.66 0.65 0.75 0.52 0.46
0.88 0.71 0.56 0.87 0.69 0.75 0.74 0.98 1.00 0.74 0.54 0.59 0.69 0.73 0.59
0.90 0.96 0.69 0.99 0.53 0.90 0.83 EIF 0.89 1.00 0.81 0.89 0.38 0.81 0.93 0.47
0.96 0.82 0.72 0.58 0.91 0.99 0.65 0.85 0.71 0.81 0.50 0.98 0.95 0.76 0.69
0.78 0.70 0.52 0.52 0.82 0.74 0.49 0.95 0.75 0.72 0.73 1.00 1.00 0.79 0.66
0.59 0.65 0.82 0.67 0.88 0.95 0.69 1.00 0.53 0.94 0.89 sb-DeepSVDD 0.50 0.91
0.60 0.46 0.50 0.33 0.73 0.50 0.58 0.63 0.45 0.49 0.58 0.59 0.52 0.58 0.55
0.45 0.54 0.71 0.82 0.61 0.52 0.37 0.57 0.55 0.48 0.53 0.49 0.45 0.47 0.63
0.52 0.47 0.46 0.60 0.63 0.45 0.52 0.52 0.52 0.41 0.75 0.57 0.63 0.64 0.51
0.59 0.62 ALAD 0.79 0.75 0.51 0.42 0.49 0.73 0.71 0.49 0.75 0.67 0.58 0.52
0.62 0.35 0.51 0.60 0.63 0.72 0.43 0.24 0.68 0.62 0.54 0.52 0.72 0.43 0.51
0.52 0.69 0.55 0.61 0.59 0.54 0.54 0.58 0.62 0.33 0.56 0.49 0.52 0.63 0.56
0.65 0.73 0.61 0.56 0.49 0.37 0.61 AE 0.64 0.98 0.85 0.92 0.36 0.84 0.93 0.47
0.89 0.77 0.69 0.56 0.92 1.00 0.62 0.74 0.67 0.81 0.37 0.93 0.94 0.73 0.66
0.75 0.63 0.51 0.35 0.79 0.69 0.33 0.93 0.73 0.60 0.64 0.99 0.98 0.55 0.51
0.50 0.56 0.71 0.55 0.82 0.89 0.61 1.00 0.55 0.89 0.88 MCD 0.93 0.99 0.83 0.84
0.37 0.40 0.81 0.50 0.94 0.72 0.74 0.59 0.93 1.00 0.61 0.77 0.73 0.76 0.56
0.99 0.94 0.78 0.68 0.58 0.77 0.50 0.62 0.92 0.67 0.85 0.84 0.76 0.76 0.59
1.00 0.99 0.74 0.41 0.61 0.81 0.82 0.49 0.95 0.96 0.75 1.00 0.53 0.91 0.84
kth-NN 0.92 0.99 0.82 0.79 0.48 0.41 0.84 0.48 0.95 0.80 0.79 0.57 0.95 1.00
0.66 0.90 0.73 0.79 0.61 0.99 0.95 0.81 0.71 0.59 0.79 0.64 0.60 0.95 0.68
0.69 0.82 0.81 0.71 0.74 0.84 0.97 0.98 0.48 0.65 0.81 0.70 0.67 0.90 0.96
0.71 0.63 0.58 0.98 0.89 COPOD 0.77 0.99 0.77 0.88 0.36 0.70 0.93 0.49 0.98
0.80 0.68 0.60 0.88 0.99 0.67 0.87 0.71 0.87 0.42 0.94 0.97 0.78 0.65 0.67
0.65 0.46 0.54 0.51 0.81 0.34 0.90 0.80 0.63 0.73 0.99 0.97 0.52 0.68 0.54
0.56 0.86 0.68 0.80 0.93 0.68 0.95 0.52 0.79 0.93 LOF 0.49 0.75 0.60 0.51 0.50
0.57 0.53 0.53 0.40 0.61 0.70 0.56 0.67 0.69 0.53 0.76 0.55 0.68 0.54 0.60
0.92 0.56 0.61 0.55 0.79 0.57 0.54 0.93 0.47 0.65 0.51 0.72 0.54 0.73 0.58
0.55 0.98 0.50 0.56 0.87 0.56 0.50 0.90 0.72 0.64 0.55 0.74 0.72 0.64 ODIN
0.50 0.80 0.62 0.52 0.50 0.49 0.57 0.65 0.51 0.63 0.65 0.58 0.58 0.53 0.50
0.66 0.55 0.61 0.49 0.57 0.79 0.56 0.56 0.53 0.64 0.59 0.43 0.87 0.44 0.66
0.51 0.67 0.49 0.66 0.50 0.55 0.88 0.51 0.56 0.86 0.51 0.53 0.85 0.45 0.57
0.55 0.74 0.66 0.59 ensemble-LOF 0.52 0.92 0.63 0.50 0.48 0.62 0.60 0.55 0.39
0.62 0.70 0.55 0.71 0.91 0.55 0.88 0.52 0.69 0.55 0.74 0.95 0.46 0.61 0.60
0.79 0.56 0.54 0.94 0.47 0.64 0.53 0.64 0.58 0.72 0.62 0.66 0.99 0.52 0.55
0.87 0.59 0.51 0.91 0.91 0.68 0.63 0.75 0.81 0.70 beta-VAE 0.67 0.99 0.85 0.93
0.40 0.83 0.95 0.47 0.73 0.77 0.67 0.55 0.91 1.00 0.62 0.82 0.66 0.83 0.39
0.95 0.94 0.73 0.66 0.75 0.59 0.49 0.35 0.64 0.77 0.33 0.94 0.75 0.60 0.65
0.99 0.98 0.44 0.51 0.49 0.52 0.71 0.55 0.80 0.89 0.61 0.86 0.55 0.81 0.91 COF
0.48 0.60 0.61 0.51 0.50 0.54 0.51 0.56 0.43 0.65 0.65 0.52 0.58 0.65 0.52
0.40 0.55 0.63 0.53 0.50 0.86 0.50 0.60 0.53 0.79 0.56 0.61 0.94 0.39 0.62
0.51 0.58 0.52 0.71 0.56 0.56 0.94 0.50 0.54 0.86 0.54 0.47 0.89 0.47 0.65
0.50 0.77 0.64 0.51 DeepSVDD 0.55 0.94 0.63 0.47 0.45 0.42 0.69 0.51 0.64 0.67
0.50 0.50 0.60 0.61 0.57 0.71 0.59 0.48 0.60 0.66 0.85 0.64 0.52 0.48 0.46
0.52 0.51 0.60 0.64 0.49 0.58 0.66 0.59 0.45 0.49 0.75 0.72 0.54 0.46 0.62
0.43 0.47 0.77 0.87 0.68 0.73 0.50 0.67 0.62 u-CBLOF 0.91 0.99 0.82 0.89 0.39
0.62 0.85 0.47 0.92 0.79 0.70 0.56 0.92 1.00 0.58 0.64 0.69 0.74 0.56 0.99
0.93 0.80 0.66 0.59 0.79 0.56 0.58 0.86 0.65 0.53 0.91 0.73 0.76 0.71 0.99
1.00 0.81 0.51 0.44 0.70 0.70 0.51 0.87 0.93 0.70 0.86 0.54 0.90 0.78 OCSVM
0.94 0.98 0.70 0.93 0.42 0.51 0.88 0.47 0.83 0.80 0.67 0.56 0.93 1.00 0.62
0.86 0.71 0.82 0.38 0.99 0.95 0.80 0.63 0.62 0.65 0.47 0.45 0.72 0.73 0.44
0.93 0.76 0.63 0.65 0.98 0.99 0.47 0.51 0.52 0.61 0.74 0.52 0.84 0.96 0.69
0.87 0.54 0.87 0.88 ABOD 0.91 0.99 0.81 0.80 0.50 0.40 0.76 0.57 0.95 0.82
0.80 0.52 0.94 0.99 0.64 0.76 0.74 0.79 0.58 0.98 0.94 0.79 0.70 0.59 0.80
0.68 0.63 0.96 0.62 0.69 0.77 0.79 0.66 0.70 0.79 0.95 0.96 0.47 0.63 0.82
0.69 0.52 0.93 0.95 0.74 0.18 0.61 0.98 0.85 PCA 0.69 0.99 0.85 0.93 0.33 0.84
0.95 0.47 0.51 0.78 0.65 0.55 0.90 1.00 0.51 0.81 0.64 0.75 0.40 0.96 0.93
0.74 0.59 0.74 0.53 0.50 0.31 0.65 0.71 0.31 0.92 0.76 0.63 0.60 0.99 0.97
0.34 0.48 0.46 0.52 0.64 0.55 0.80 0.94 0.61 1.00 0.55 0.78 0.85 kNN 0.91 0.99
0.81 0.79 0.49 0.41 0.81 0.51 0.95 0.80 0.80 0.57 0.95 1.00 0.65 0.81 0.73
0.79 0.60 0.99 0.95 0.80 0.71 0.58 0.82 0.66 0.64 0.97 0.64 0.69 0.79 0.79
0.70 0.74 0.82 0.95 0.98 0.48 0.65 0.85 0.70 0.65 0.92 0.96 0.73 0.54 0.60
0.98 0.86 VAE 0.65 0.99 0.84 0.94 0.37 0.64 0.87 0.47 0.78 0.77 0.69 0.55 0.91
1.00 0.62 0.79 0.67 0.80 0.51 0.94 0.90 0.72 0.66 0.65 0.62 0.60 0.45 0.69
0.77 0.33 0.92 0.72 0.63 0.66 0.79 0.87 0.63 0.53 0.54 0.63 0.71 0.56 0.84
0.90 0.61 0.80 0.55 0.86 0.86 LODA 0.72 0.98 0.82 0.91 0.94 0.84 0.76 0.55
0.92 0.51 0.69 0.49 0.78 0.98 0.65 0.62 0.57 0.83 0.39 0.85 0.91 0.51 0.56
0.69 0.18 0.40 0.75 0.58 0.78 0.37 0.91 0.61 0.62 0.70 0.67 0.98 0.64 0.53
0.57 0.55 0.76 0.59 0.53 0.83 0.60 0.99 0.52 0.58 0.85 IF 0.82 1.00 0.81 0.89
0.37 0.79 0.93 0.47 0.96 0.81 0.73 0.58 0.90 0.99 0.64 0.79 0.67 0.82 0.48
0.98 0.95 0.72 0.67 0.77 0.70 0.58 0.49 0.78 0.74 0.46 0.95 0.69 0.70 0.73
1.00 0.99 0.80 0.73 0.57 0.64 0.82 0.62 0.86 0.94 0.68 1.00 0.54 0.93 0.89
ECOD 0.78 0.99 0.75 0.93 0.41 0.60 0.95 0.49 0.82 0.81 0.64 0.59 0.91 0.99
0.58 0.71 0.68 0.78 0.37 0.98 0.91 0.77 0.52 0.65 0.68 0.47 0.41 0.41 0.71
0.42 0.91 0.79 0.75 0.72 1.00 0.97 0.45 0.60 0.44 0.53 0.72 0.64 0.77 0.91
0.68 0.96 0.53 0.79 0.92 LMDD 0.92 0.99 0.75 0.91 0.64 0.59 0.88 0.48 0.53
0.80 0.63 0.50 0.78 1.00 0.66 0.78 0.67 0.78 0.45 0.99 0.96 0.70 0.62 0.62
0.58 0.40 0.47 0.61 0.70 0.36 0.94 0.83 0.42 0.64 0.99 0.48 0.45 0.50 0.49
0.52 0.51 0.50 0.74 0.94 0.68 0.97 0.50 0.78 0.90 gen2out 0.86 1.00 0.78 0.92
0.41 0.70 0.95 0.46 0.97 0.78 0.72 0.58 0.91 1.00 0.63 0.87 0.71 0.81 0.48
0.98 0.95 0.66 0.66 0.70 0.74 0.50 0.52 0.75 0.68 0.47 0.95 0.72 0.72 0.63
0.99 0.99 0.66 0.57 0.53 0.68 0.81 0.66 0.76 0.96 0.56 1.00 0.51 0.86 0.91
LUNAR 0.69 0.97 0.76 0.73 0.46 0.42 0.66 0.46 0.95 0.79 0.81 0.60 0.72 0.99
0.63 0.69 0.73 0.79 0.59 0.92 0.92 0.67 0.69 0.62 0.80 0.70 0.47 0.83 0.64
0.48 0.72 0.66 0.66 0.74 0.65 0.89 0.90 0.43 0.54 0.74 0.67 0.51 0.91 0.95
0.65 0.73 0.71 0.81 0.80 KDE 0.94 0.97 0.73 0.92 0.39 0.37 0.81 0.43 0.96 0.67
0.68 0.55 0.94 1.00 0.67 0.77 0.74 0.81 0.60 0.98 0.94 0.84 0.70 0.59 0.77
0.63 0.62 0.89 0.73 0.50 0.96 0.78 0.78 0.76 0.93 0.99 0.87 0.51 0.64 0.88
0.75 0.60 0.94 0.97 0.66 0.08 0.59 0.96 0.87 HBOS 0.68 0.99 0.35 0.64 0.42
0.43 0.80 0.46 0.97 0.80 0.71 0.57 0.75 0.98 0.64 0.91 0.71 0.80 0.58 0.97
0.96 0.78 0.70 0.62 0.68 0.59 0.52 0.67 0.74 0.36 0.93 0.78 0.76 0.68 0.99
0.98 0.73 0.87 0.49 0.60 0.85 0.64 0.76 0.83 0.68 1.00 0.50 0.77 0.91 SOD 0.78
0.96 0.71 0.63 0.43 0.39 0.66 0.56 0.88 0.76 0.75 0.55 0.76 0.88 0.59 0.47
0.71 0.74 0.57 0.94 0.95 0.73 0.59 0.56 0.72 0.64 0.70 0.92 0.58 0.59 0.66
0.57 0.58 0.62 0.74 0.78 0.92 0.55 0.59 0.89 0.59 0.54 0.90 0.82 0.53 0.45
0.66 0.81 0.74

##  Appendix B Nemenyi post-hoc analysis results

Table 7 shows the results for all datasets, Table 8 shows the results for the
local datasets, and Table 9 shows the results for the global datasets.

Table 7: The p-values from Nemenyi post-hoc analysis on all algorithm pairs
based on all $52$ datasets. P-values below 0.05 have been printed bold.

| GMM | CBLOF | SO-GAAL | INNE | EIF | sb-DeepSVDD | ALAD | AE | MCD | kth-NN | COPOD | LOF | ODIN | ensemble-LOF | beta-VAE | COF | DeepSVDD | u-CBLOF | OCSVM | ABOD | PCA | kNN | VAE | LODA | IF | ECOD | LMDD | gen2out | LUNAR | KDE | HBOS | SOD
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
GMM | 1.0 | 0.001 | 0.016 | 0.9 | 0.272 | 0.001 | 0.002 | 0.9 | 0.9 | 0.138 | 0.9 | 0.878 | 0.072 | 0.9 | 0.9 | 0.147 | 0.007 | 0.9 | 0.9 | 0.9 | 0.9 | 0.272 | 0.9 | 0.9 | 0.827 | 0.9 | 0.9 | 0.9 | 0.9 | 0.666 | 0.9 | 0.9
CBLOF | 0.001 | 1.0 | 0.9 | 0.001 | 0.001 | 0.9 | 0.9 | 0.001 | 0.001 | 0.001 | 0.001 | 0.725 | 0.9 | 0.125 | 0.002 | 0.9 | 0.9 | 0.001 | 0.001 | 0.001 | 0.115 | 0.001 | 0.001 | 0.088 | 0.001 | 0.001 | 0.048 | 0.001 | 0.001 | 0.001 | 0.001 | 0.045
SO-GAAL | 0.016 | 0.9 | 1.0 | 0.001 | 0.001 | 0.9 | 0.9 | 0.021 | 0.001 | 0.001 | 0.001 | 0.9 | 0.9 | 0.64 | 0.058 | 0.9 | 0.9 | 0.001 | 0.001 | 0.001 | 0.622 | 0.001 | 0.034 | 0.563 | 0.001 | 0.019 | 0.433 | 0.001 | 0.021 | 0.001 | 0.001 | 0.416
INNE | 0.9 | 0.001 | 0.001 | 1.0 | 0.9 | 0.001 | 0.001 | 0.9 | 0.9 | 0.9 | 0.9 | 0.021 | 0.001 | 0.35 | 0.9 | 0.001 | 0.001 | 0.9 | 0.9 | 0.9 | 0.372 | 0.9 | 0.9 | 0.442 | 0.9 | 0.9 | 0.571 | 0.9 | 0.9 | 0.9 | 0.9 | 0.585
EIF | 0.272 | 0.001 | 0.001 | 0.9 | 1.0 | 0.001 | 0.001 | 0.226 | 0.9 | 0.9 | 0.9 | 0.001 | 0.001 | 0.002 | 0.104 | 0.001 | 0.001 | 0.9 | 0.754 | 0.9 | 0.002 | 0.9 | 0.161 | 0.003 | 0.9 | 0.241 | 0.007 | 0.9 | 0.226 | 0.9 | 0.857 | 0.007
sb-DeepSVDD | 0.001 | 0.9 | 0.9 | 0.001 | 0.001 | 1.0 | 0.9 | 0.001 | 0.001 | 0.001 | 0.001 | 0.265 | 0.9 | 0.013 | 0.001 | 0.9 | 0.9 | 0.001 | 0.001 | 0.001 | 0.012 | 0.001 | 0.001 | 0.008 | 0.001 | 0.001 | 0.004 | 0.001 | 0.001 | 0.001 | 0.001 | 0.003
ALAD | 0.002 | 0.9 | 0.9 | 0.001 | 0.001 | 0.9 | 1.0 | 0.002 | 0.001 | 0.001 | 0.001 | 0.9 | 0.9 | 0.268 | 0.008 | 0.9 | 0.9 | 0.001 | 0.001 | 0.001 | 0.251 | 0.001 | 0.004 | 0.203 | 0.001 | 0.002 | 0.123 | 0.001 | 0.002 | 0.001 | 0.001 | 0.115
AE | 0.9 | 0.001 | 0.021 | 0.9 | 0.226 | 0.001 | 0.002 | 1.0 | 0.9 | 0.112 | 0.9 | 0.9 | 0.091 | 0.9 | 0.9 | 0.181 | 0.01 | 0.9 | 0.9 | 0.9 | 0.9 | 0.226 | 0.9 | 0.9 | 0.776 | 0.9 | 0.9 | 0.9 | 0.9 | 0.615 | 0.9 | 0.9
MCD | 0.9 | 0.001 | 0.001 | 0.9 | 0.9 | 0.001 | 0.001 | 0.9 | 1.0 | 0.9 | 0.9 | 0.012 | 0.001 | 0.254 | 0.9 | 0.001 | 0.001 | 0.9 | 0.9 | 0.9 | 0.272 | 0.9 | 0.9 | 0.329 | 0.9 | 0.9 | 0.474 | 0.9 | 0.9 | 0.9 | 0.9 | 0.49
kth-NN | 0.138 | 0.001 | 0.001 | 0.9 | 0.9 | 0.001 | 0.001 | 0.112 | 0.9 | 1.0 | 0.9 | 0.001 | 0.001 | 0.001 | 0.045 | 0.001 | 0.001 | 0.776 | 0.578 | 0.9 | 0.001 | 0.9 | 0.075 | 0.001 | 0.9 | 0.121 | 0.002 | 0.9 | 0.112 | 0.9 | 0.681 | 0.002
COPOD | 0.9 | 0.001 | 0.001 | 0.9 | 0.9 | 0.001 | 0.001 | 0.9 | 0.9 | 0.9 | 1.0 | 0.042 | 0.001 | 0.494 | 0.9 | 0.001 | 0.001 | 0.9 | 0.9 | 0.9 | 0.512 | 0.9 | 0.9 | 0.571 | 0.9 | 0.9 | 0.695 | 0.9 | 0.9 | 0.9 | 0.9 | 0.71
LOF | 0.878 | 0.725 | 0.9 | 0.021 | 0.001 | 0.265 | 0.9 | 0.9 | 0.012 | 0.001 | 0.042 | 1.0 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.208 | 0.407 | 0.003 | 0.9 | 0.001 | 0.9 | 0.9 | 0.001 | 0.9 | 0.9 | 0.017 | 0.9 | 0.001 | 0.293 | 0.9
ODIN | 0.072 | 0.9 | 0.9 | 0.001 | 0.001 | 0.9 | 0.9 | 0.091 | 0.001 | 0.001 | 0.001 | 0.9 | 1.0 | 0.9 | 0.203 | 0.9 | 0.9 | 0.002 | 0.006 | 0.001 | 0.9 | 0.001 | 0.133 | 0.849 | 0.001 | 0.085 | 0.725 | 0.001 | 0.091 | 0.001 | 0.003 | 0.71
ensemble-LOF | 0.9 | 0.125 | 0.64 | 0.35 | 0.002 | 0.013 | 0.268 | 0.9 | 0.254 | 0.001 | 0.494 | 0.9 | 0.9 | 1.0 | 0.9 | 0.9 | 0.501 | 0.853 | 0.9 | 0.096 | 0.9 | 0.002 | 0.9 | 0.9 | 0.044 | 0.9 | 0.9 | 0.304 | 0.9 | 0.018 | 0.9 | 0.9
beta-VAE | 0.9 | 0.002 | 0.058 | 0.9 | 0.104 | 0.001 | 0.008 | 0.9 | 0.9 | 0.045 | 0.9 | 0.9 | 0.203 | 0.9 | 1.0 | 0.354 | 0.028 | 0.9 | 0.9 | 0.747 | 0.9 | 0.104 | 0.9 | 0.9 | 0.585 | 0.9 | 0.9 | 0.9 | 0.9 | 0.416 | 0.9 | 0.9
COF | 0.147 | 0.9 | 0.9 | 0.001 | 0.001 | 0.9 | 0.9 | 0.181 | 0.001 | 0.001 | 0.001 | 0.9 | 0.9 | 0.9 | 0.354 | 1.0 | 0.9 | 0.004 | 0.015 | 0.001 | 0.9 | 0.001 | 0.251 | 0.9 | 0.001 | 0.169 | 0.886 | 0.001 | 0.181 | 0.001 | 0.008 | 0.871
DeepSVDD | 0.007 | 0.9 | 0.9 | 0.001 | 0.001 | 0.9 | 0.9 | 0.01 | 0.001 | 0.001 | 0.001 | 0.9 | 0.9 | 0.501 | 0.028 | 0.9 | 1.0 | 0.001 | 0.001 | 0.001 | 0.482 | 0.001 | 0.016 | 0.416 | 0.001 | 0.009 | 0.279 | 0.001 | 0.01 | 0.001 | 0.001 | 0.265
u-CBLOF | 0.9 | 0.001 | 0.001 | 0.9 | 0.9 | 0.001 | 0.001 | 0.9 | 0.9 | 0.776 | 0.9 | 0.208 | 0.002 | 0.853 | 0.9 | 0.004 | 0.001 | 1.0 | 0.9 | 0.9 | 0.871 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9
OCSVM | 0.9 | 0.001 | 0.001 | 0.9 | 0.754 | 0.001 | 0.001 | 0.9 | 0.9 | 0.578 | 0.9 | 0.407 | 0.006 | 0.9 | 0.9 | 0.015 | 0.001 | 0.9 | 1.0 | 0.9 | 0.9 | 0.754 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9
ABOD | 0.9 | 0.001 | 0.001 | 0.9 | 0.9 | 0.001 | 0.001 | 0.9 | 0.9 | 0.9 | 0.9 | 0.003 | 0.001 | 0.096 | 0.747 | 0.001 | 0.001 | 0.9 | 0.9 | 1.0 | 0.104 | 0.9 | 0.849 | 0.133 | 0.9 | 0.9 | 0.22 | 0.9 | 0.9 | 0.9 | 0.9 | 0.232
PCA | 0.9 | 0.115 | 0.622 | 0.372 | 0.002 | 0.012 | 0.251 | 0.9 | 0.272 | 0.001 | 0.512 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.482 | 0.871 | 0.9 | 0.104 | 1.0 | 0.002 | 0.9 | 0.9 | 0.048 | 0.9 | 0.9 | 0.322 | 0.9 | 0.02 | 0.9 | 0.9
kNN | 0.272 | 0.001 | 0.001 | 0.9 | 0.9 | 0.001 | 0.001 | 0.226 | 0.9 | 0.9 | 0.9 | 0.001 | 0.001 | 0.002 | 0.104 | 0.001 | 0.001 | 0.9 | 0.754 | 0.9 | 0.002 | 1.0 | 0.161 | 0.003 | 0.9 | 0.241 | 0.007 | 0.9 | 0.226 | 0.9 | 0.857 | 0.007
VAE | 0.9 | 0.001 | 0.034 | 0.9 | 0.161 | 0.001 | 0.004 | 0.9 | 0.9 | 0.075 | 0.9 | 0.9 | 0.133 | 0.9 | 0.9 | 0.251 | 0.016 | 0.9 | 0.9 | 0.849 | 0.9 | 0.161 | 1.0 | 0.9 | 0.688 | 0.9 | 0.9 | 0.9 | 0.9 | 0.527 | 0.9 | 0.9
LODA | 0.9 | 0.088 | 0.563 | 0.442 | 0.003 | 0.008 | 0.203 | 0.9 | 0.329 | 0.001 | 0.571 | 0.9 | 0.849 | 0.9 | 0.9 | 0.9 | 0.416 | 0.9 | 0.9 | 0.133 | 0.9 | 0.003 | 0.9 | 1.0 | 0.065 | 0.9 | 0.9 | 0.389 | 0.9 | 0.028 | 0.9 | 0.9
IF | 0.827 | 0.001 | 0.001 | 0.9 | 0.9 | 0.001 | 0.001 | 0.776 | 0.9 | 0.9 | 0.9 | 0.001 | 0.001 | 0.044 | 0.585 | 0.001 | 0.001 | 0.9 | 0.9 | 0.9 | 0.048 | 0.9 | 0.688 | 0.065 | 1.0 | 0.794 | 0.115 | 0.9 | 0.776 | 0.9 | 0.9 | 0.123
ECOD | 0.9 | 0.001 | 0.019 | 0.9 | 0.241 | 0.001 | 0.002 | 0.9 | 0.9 | 0.121 | 0.9 | 0.9 | 0.085 | 0.9 | 0.9 | 0.169 | 0.009 | 0.9 | 0.9 | 0.9 | 0.9 | 0.241 | 0.9 | 0.9 | 0.794 | 1.0 | 0.9 | 0.9 | 0.9 | 0.633 | 0.9 | 0.9
LMDD | 0.9 | 0.048 | 0.433 | 0.571 | 0.007 | 0.004 | 0.123 | 0.9 | 0.474 | 0.002 | 0.695 | 0.9 | 0.725 | 0.9 | 0.9 | 0.886 | 0.279 | 0.9 | 0.9 | 0.22 | 0.9 | 0.007 | 0.9 | 0.9 | 0.115 | 0.9 | 1.0 | 0.527 | 0.9 | 0.054 | 0.9 | 0.9
gen2out | 0.9 | 0.001 | 0.001 | 0.9 | 0.9 | 0.001 | 0.001 | 0.9 | 0.9 | 0.9 | 0.9 | 0.017 | 0.001 | 0.304 | 0.9 | 0.001 | 0.001 | 0.9 | 0.9 | 0.9 | 0.322 | 0.9 | 0.9 | 0.389 | 0.9 | 0.9 | 0.527 | 1.0 | 0.9 | 0.9 | 0.9 | 0.541
LUNAR | 0.9 | 0.001 | 0.021 | 0.9 | 0.226 | 0.001 | 0.002 | 0.9 | 0.9 | 0.112 | 0.9 | 0.9 | 0.091 | 0.9 | 0.9 | 0.181 | 0.01 | 0.9 | 0.9 | 0.9 | 0.9 | 0.226 | 0.9 | 0.9 | 0.776 | 0.9 | 0.9 | 0.9 | 1.0 | 0.615 | 0.9 | 0.9
KDE | 0.666 | 0.001 | 0.001 | 0.9 | 0.9 | 0.001 | 0.001 | 0.615 | 0.9 | 0.9 | 0.9 | 0.001 | 0.001 | 0.018 | 0.416 | 0.001 | 0.001 | 0.9 | 0.9 | 0.9 | 0.02 | 0.9 | 0.527 | 0.028 | 0.9 | 0.633 | 0.054 | 0.9 | 0.615 | 1.0 | 0.9 | 0.058
HBOS | 0.9 | 0.001 | 0.001 | 0.9 | 0.857 | 0.001 | 0.001 | 0.9 | 0.9 | 0.681 | 0.9 | 0.293 | 0.003 | 0.9 | 0.9 | 0.008 | 0.001 | 0.9 | 0.9 | 0.9 | 0.9 | 0.857 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 1.0 | 0.9
SOD | 0.9 | 0.045 | 0.416 | 0.585 | 0.007 | 0.003 | 0.115 | 0.9 | 0.49 | 0.002 | 0.71 | 0.9 | 0.71 | 0.9 | 0.9 | 0.871 | 0.265 | 0.9 | 0.9 | 0.232 | 0.9 | 0.007 | 0.9 | 0.9 | 0.123 | 0.9 | 0.9 | 0.541 | 0.9 | 0.058 | 0.9 | 1.0

Table 8: The p-values from Nemenyi post-hoc analysis on all algorithm pairs
based on the $12$ local datasets. P-values below 0.05 have been printed bold.

| GMM | CBLOF | SO-GAAL | INNE | EIF | sb-DeepSVDD | ALAD | AE | MCD | kth-NN | COPOD | LOF | ODIN | ensemble-LOF | beta-VAE | COF | DeepSVDD | u-CBLOF | OCSVM | ABOD | PCA | kNN | VAE | LODA | IF | ECOD | LMDD | gen2out | LUNAR | KDE | HBOS | SOD
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
GMM | 1.0 | 0.855 | 0.039 | 0.9 | 0.9 | 0.434 | 0.128 | 0.155 | 0.9 | 0.9 | 0.125 | 0.9 | 0.9 | 0.9 | 0.034 | 0.9 | 0.647 | 0.9 | 0.452 | 0.9 | 0.01 | 0.9 | 0.81 | 0.499 | 0.9 | 0.026 | 0.039 | 0.869 | 0.9 | 0.9 | 0.855 | 0.9
CBLOF | 0.855 | 1.0 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.165 | 0.9 | 0.364 | 0.9 | 0.476 | 0.9 | 0.559 | 0.9 | 0.9 | 0.9 | 0.029 | 0.9 | 0.021 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.855 | 0.452 | 0.9 | 0.434
SO-GAAL | 0.039 | 0.9 | 1.0 | 0.795 | 0.9 | 0.9 | 0.9 | 0.9 | 0.107 | 0.001 | 0.9 | 0.002 | 0.232 | 0.004 | 0.9 | 0.007 | 0.9 | 0.633 | 0.9 | 0.001 | 0.9 | 0.001 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.039 | 0.004 | 0.9 | 0.003
INNE | 0.9 | 0.9 | 0.795 | 1.0 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.825 | 0.9 | 0.9 | 0.9 | 0.9 | 0.766 | 0.9 | 0.9 | 0.9 | 0.9 | 0.452 | 0.559 | 0.381 | 0.9 | 0.9 | 0.9 | 0.721 | 0.795 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9
EIF | 0.9 | 0.9 | 0.9 | 0.9 | 1.0 | 0.9 | 0.9 | 0.9 | 0.9 | 0.692 | 0.9 | 0.9 | 0.9 | 0.9 | 0.899 | 0.9 | 0.9 | 0.9 | 0.9 | 0.3 | 0.692 | 0.244 | 0.9 | 0.9 | 0.9 | 0.855 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9
sb-DeepSVDD | 0.434 | 0.9 | 0.9 | 0.9 | 0.9 | 1.0 | 0.9 | 0.9 | 0.647 | 0.022 | 0.9 | 0.071 | 0.84 | 0.11 | 0.9 | 0.155 | 0.9 | 0.9 | 0.9 | 0.002 | 0.9 | 0.002 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.434 | 0.099 | 0.9 | 0.093
ALAD | 0.128 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 1.0 | 0.9 | 0.285 | 0.003 | 0.9 | 0.011 | 0.499 | 0.02 | 0.9 | 0.031 | 0.9 | 0.884 | 0.9 | 0.001 | 0.9 | 0.001 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.128 | 0.017 | 0.9 | 0.016
AE | 0.155 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 1.0 | 0.33 | 0.004 | 0.9 | 0.015 | 0.544 | 0.025 | 0.9 | 0.039 | 0.9 | 0.9 | 0.9 | 0.001 | 0.9 | 0.001 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.155 | 0.022 | 0.9 | 0.021
MCD | 0.9 | 0.9 | 0.107 | 0.9 | 0.9 | 0.647 | 0.285 | 0.33 | 1.0 | 0.9 | 0.278 | 0.9 | 0.9 | 0.9 | 0.093 | 0.9 | 0.855 | 0.9 | 0.662 | 0.9 | 0.034 | 0.9 | 0.9 | 0.707 | 0.9 | 0.076 | 0.107 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9
kth-NN | 0.9 | 0.165 | 0.001 | 0.825 | 0.692 | 0.022 | 0.003 | 0.004 | 0.9 | 1.0 | 0.003 | 0.9 | 0.9 | 0.9 | 0.001 | 0.9 | 0.066 | 0.9 | 0.024 | 0.9 | 0.001 | 0.9 | 0.136 | 0.031 | 0.588 | 0.001 | 0.001 | 0.175 | 0.9 | 0.9 | 0.165 | 0.9
COPOD | 0.125 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.278 | 0.003 | 1.0 | 0.011 | 0.492 | 0.019 | 0.9 | 0.03 | 0.9 | 0.877 | 0.9 | 0.001 | 0.9 | 0.001 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.125 | 0.017 | 0.9 | 0.015
LOF | 0.9 | 0.364 | 0.002 | 0.9 | 0.9 | 0.071 | 0.011 | 0.015 | 0.9 | 0.9 | 0.011 | 1.0 | 0.9 | 0.9 | 0.002 | 0.9 | 0.175 | 0.9 | 0.076 | 0.9 | 0.001 | 0.9 | 0.314 | 0.093 | 0.81 | 0.001 | 0.002 | 0.381 | 0.9 | 0.9 | 0.364 | 0.9
ODIN | 0.9 | 0.9 | 0.232 | 0.9 | 0.9 | 0.84 | 0.499 | 0.544 | 0.9 | 0.9 | 0.492 | 0.9 | 1.0 | 0.9 | 0.208 | 0.9 | 0.9 | 0.9 | 0.855 | 0.9 | 0.087 | 0.9 | 0.9 | 0.899 | 0.9 | 0.175 | 0.232 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9
ensemble-LOF | 0.9 | 0.476 | 0.004 | 0.9 | 0.9 | 0.11 | 0.02 | 0.025 | 0.9 | 0.9 | 0.019 | 0.9 | 0.9 | 1.0 | 0.004 | 0.9 | 0.251 | 0.9 | 0.118 | 0.9 | 0.001 | 0.9 | 0.426 | 0.141 | 0.9 | 0.003 | 0.004 | 0.492 | 0.9 | 0.9 | 0.476 | 0.9
beta-VAE | 0.034 | 0.9 | 0.9 | 0.766 | 0.899 | 0.9 | 0.9 | 0.9 | 0.093 | 0.001 | 0.9 | 0.002 | 0.208 | 0.004 | 1.0 | 0.006 | 0.9 | 0.603 | 0.9 | 0.001 | 0.9 | 0.001 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.034 | 0.003 | 0.9 | 0.003
COF | 0.9 | 0.559 | 0.007 | 0.9 | 0.9 | 0.155 | 0.031 | 0.039 | 0.9 | 0.9 | 0.03 | 0.9 | 0.9 | 0.9 | 0.006 | 1.0 | 0.33 | 0.9 | 0.165 | 0.9 | 0.002 | 0.9 | 0.514 | 0.196 | 0.9 | 0.005 | 0.007 | 0.573 | 0.9 | 0.9 | 0.559 | 0.9
DeepSVDD | 0.647 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.855 | 0.066 | 0.9 | 0.175 | 0.9 | 0.251 | 0.9 | 0.33 | 1.0 | 0.9 | 0.9 | 0.009 | 0.9 | 0.006 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.647 | 0.232 | 0.9 | 0.219
u-CBLOF | 0.9 | 0.9 | 0.633 | 0.9 | 0.9 | 0.9 | 0.884 | 0.9 | 0.9 | 0.9 | 0.877 | 0.9 | 0.9 | 0.9 | 0.603 | 0.9 | 0.9 | 1.0 | 0.9 | 0.618 | 0.381 | 0.559 | 0.9 | 0.9 | 0.9 | 0.559 | 0.633 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9
OCSVM | 0.452 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.662 | 0.024 | 0.9 | 0.076 | 0.855 | 0.118 | 0.9 | 0.165 | 0.9 | 0.9 | 1.0 | 0.003 | 0.9 | 0.002 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.452 | 0.107 | 0.9 | 0.099
ABOD | 0.9 | 0.029 | 0.001 | 0.452 | 0.3 | 0.002 | 0.001 | 0.001 | 0.9 | 0.9 | 0.001 | 0.9 | 0.9 | 0.9 | 0.001 | 0.9 | 0.009 | 0.618 | 0.003 | 1.0 | 0.001 | 0.9 | 0.022 | 0.003 | 0.208 | 0.001 | 0.001 | 0.031 | 0.9 | 0.9 | 0.029 | 0.9
PCA | 0.01 | 0.9 | 0.9 | 0.559 | 0.692 | 0.9 | 0.9 | 0.9 | 0.034 | 0.001 | 0.9 | 0.001 | 0.087 | 0.001 | 0.9 | 0.002 | 0.9 | 0.381 | 0.9 | 0.001 | 1.0 | 0.001 | 0.9 | 0.9 | 0.795 | 0.9 | 0.9 | 0.9 | 0.01 | 0.001 | 0.9 | 0.001
kNN | 0.9 | 0.021 | 0.001 | 0.381 | 0.244 | 0.002 | 0.001 | 0.001 | 0.9 | 0.9 | 0.001 | 0.9 | 0.9 | 0.9 | 0.001 | 0.9 | 0.006 | 0.559 | 0.002 | 0.9 | 0.001 | 1.0 | 0.016 | 0.002 | 0.165 | 0.001 | 0.001 | 0.022 | 0.9 | 0.9 | 0.021 | 0.9
VAE | 0.81 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.136 | 0.9 | 0.314 | 0.9 | 0.426 | 0.9 | 0.514 | 0.9 | 0.9 | 0.9 | 0.022 | 0.9 | 0.016 | 1.0 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.81 | 0.399 | 0.9 | 0.381
LODA | 0.499 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.707 | 0.031 | 0.9 | 0.093 | 0.899 | 0.141 | 0.9 | 0.196 | 0.9 | 0.9 | 0.9 | 0.003 | 0.9 | 0.002 | 0.9 | 1.0 | 0.9 | 0.9 | 0.9 | 0.9 | 0.499 | 0.128 | 0.9 | 0.122
IF | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.588 | 0.9 | 0.81 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.208 | 0.795 | 0.165 | 0.9 | 0.9 | 1.0 | 0.9 | 0.9 | 0.9 | 0.9 | 0.884 | 0.9 | 0.869
ECOD | 0.026 | 0.9 | 0.9 | 0.721 | 0.855 | 0.9 | 0.9 | 0.9 | 0.076 | 0.001 | 0.9 | 0.001 | 0.175 | 0.003 | 0.9 | 0.005 | 0.9 | 0.559 | 0.9 | 0.001 | 0.9 | 0.001 | 0.9 | 0.9 | 0.9 | 1.0 | 0.9 | 0.9 | 0.026 | 0.002 | 0.9 | 0.002
LMDD | 0.039 | 0.9 | 0.9 | 0.795 | 0.9 | 0.9 | 0.9 | 0.9 | 0.107 | 0.001 | 0.9 | 0.002 | 0.232 | 0.004 | 0.9 | 0.007 | 0.9 | 0.633 | 0.9 | 0.001 | 0.9 | 0.001 | 0.9 | 0.9 | 0.9 | 0.9 | 1.0 | 0.9 | 0.039 | 0.004 | 0.9 | 0.003
gen2out | 0.869 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.175 | 0.9 | 0.381 | 0.9 | 0.492 | 0.9 | 0.573 | 0.9 | 0.9 | 0.9 | 0.031 | 0.9 | 0.022 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 1.0 | 0.869 | 0.468 | 0.9 | 0.452
LUNAR | 0.9 | 0.855 | 0.039 | 0.9 | 0.9 | 0.434 | 0.128 | 0.155 | 0.9 | 0.9 | 0.125 | 0.9 | 0.9 | 0.9 | 0.034 | 0.9 | 0.647 | 0.9 | 0.452 | 0.9 | 0.01 | 0.9 | 0.81 | 0.499 | 0.9 | 0.026 | 0.039 | 0.869 | 1.0 | 0.9 | 0.855 | 0.9
KDE | 0.9 | 0.452 | 0.004 | 0.9 | 0.9 | 0.099 | 0.017 | 0.022 | 0.9 | 0.9 | 0.017 | 0.9 | 0.9 | 0.9 | 0.003 | 0.9 | 0.232 | 0.9 | 0.107 | 0.9 | 0.001 | 0.9 | 0.399 | 0.128 | 0.884 | 0.002 | 0.004 | 0.468 | 0.9 | 1.0 | 0.452 | 0.9
HBOS | 0.855 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.165 | 0.9 | 0.364 | 0.9 | 0.476 | 0.9 | 0.559 | 0.9 | 0.9 | 0.9 | 0.029 | 0.9 | 0.021 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.855 | 0.452 | 1.0 | 0.434
SOD | 0.9 | 0.434 | 0.003 | 0.9 | 0.9 | 0.093 | 0.016 | 0.021 | 0.9 | 0.9 | 0.015 | 0.9 | 0.9 | 0.9 | 0.003 | 0.9 | 0.219 | 0.9 | 0.099 | 0.9 | 0.001 | 0.9 | 0.381 | 0.122 | 0.869 | 0.002 | 0.003 | 0.452 | 0.9 | 0.9 | 0.434 | 1.0

Table 9: The p-values from Nemenyi post-hoc analysis on all algorithm pairs
based on the $37$ global datasets. P-values below 0.05 have been printed bold.

| GMM | CBLOF | SO-GAAL | INNE | EIF | sb-DeepSVDD | ALAD | AE | MCD | kth-NN | COPOD | LOF | ODIN | ensemble-LOF | beta-VAE | COF | DeepSVDD | u-CBLOF | OCSVM | ABOD | PCA | kNN | VAE | LODA | IF | ECOD | LMDD | gen2out | LUNAR | KDE | HBOS | SOD
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
GMM | 1.0 | 0.007 | 0.757 | 0.613 | 0.002 | 0.001 | 0.2 | 0.9 | 0.9 | 0.165 | 0.063 | 0.35 | 0.034 | 0.9 | 0.9 | 0.01 | 0.154 | 0.9 | 0.791 | 0.9 | 0.9 | 0.605 | 0.9 | 0.9 | 0.052 | 0.9 | 0.9 | 0.187 | 0.9 | 0.622 | 0.9 | 0.9
CBLOF | 0.007 | 1.0 | 0.9 | 0.001 | 0.001 | 0.9 | 0.9 | 0.001 | 0.001 | 0.001 | 0.001 | 0.9 | 0.9 | 0.866 | 0.001 | 0.9 | 0.9 | 0.001 | 0.001 | 0.001 | 0.001 | 0.001 | 0.001 | 0.004 | 0.001 | 0.001 | 0.001 | 0.001 | 0.01 | 0.001 | 0.001 | 0.639
SO-GAAL | 0.757 | 0.9 | 1.0 | 0.001 | 0.001 | 0.9 | 0.9 | 0.005 | 0.001 | 0.001 | 0.001 | 0.9 | 0.9 | 0.9 | 0.006 | 0.9 | 0.9 | 0.007 | 0.001 | 0.014 | 0.2 | 0.001 | 0.073 | 0.647 | 0.001 | 0.001 | 0.148 | 0.001 | 0.816 | 0.001 | 0.001 | 0.9
INNE | 0.613 | 0.001 | 0.001 | 1.0 | 0.9 | 0.001 | 0.001 | 0.9 | 0.9 | 0.9 | 0.9 | 0.001 | 0.001 | 0.001 | 0.9 | 0.001 | 0.001 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.723 | 0.9 | 0.9 | 0.9 | 0.9 | 0.554 | 0.9 | 0.9 | 0.006
EIF | 0.002 | 0.001 | 0.001 | 0.9 | 1.0 | 0.001 | 0.001 | 0.613 | 0.892 | 0.9 | 0.9 | 0.001 | 0.001 | 0.001 | 0.58 | 0.001 | 0.001 | 0.554 | 0.9 | 0.44 | 0.05 | 0.9 | 0.148 | 0.004 | 0.9 | 0.845 | 0.073 | 0.9 | 0.001 | 0.9 | 0.9 | 0.001
sb-DeepSVDD | 0.001 | 0.9 | 0.9 | 0.001 | 0.001 | 1.0 | 0.9 | 0.001 | 0.001 | 0.001 | 0.001 | 0.9 | 0.9 | 0.597 | 0.001 | 0.9 | 0.9 | 0.001 | 0.001 | 0.001 | 0.001 | 0.001 | 0.001 | 0.001 | 0.001 | 0.001 | 0.001 | 0.001 | 0.002 | 0.001 | 0.001 | 0.35
ALAD | 0.2 | 0.9 | 0.9 | 0.001 | 0.001 | 0.9 | 1.0 | 0.001 | 0.001 | 0.001 | 0.001 | 0.9 | 0.9 | 0.9 | 0.001 | 0.9 | 0.9 | 0.001 | 0.001 | 0.001 | 0.014 | 0.001 | 0.003 | 0.128 | 0.001 | 0.001 | 0.009 | 0.001 | 0.249 | 0.001 | 0.001 | 0.9
AE | 0.9 | 0.001 | 0.005 | 0.9 | 0.613 | 0.001 | 0.001 | 1.0 | 0.9 | 0.9 | 0.9 | 0.001 | 0.001 | 0.048 | 0.9 | 0.001 | 0.001 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.137
MCD | 0.9 | 0.001 | 0.001 | 0.9 | 0.892 | 0.001 | 0.001 | 0.9 | 1.0 | 0.9 | 0.9 | 0.001 | 0.001 | 0.01 | 0.9 | 0.001 | 0.001 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.866 | 0.9 | 0.9 | 0.037
kth-NN | 0.165 | 0.001 | 0.001 | 0.9 | 0.9 | 0.001 | 0.001 | 0.9 | 0.9 | 1.0 | 0.9 | 0.001 | 0.001 | 0.001 | 0.9 | 0.001 | 0.001 | 0.9 | 0.9 | 0.9 | 0.706 | 0.9 | 0.9 | 0.249 | 0.9 | 0.9 | 0.782 | 0.9 | 0.128 | 0.9 | 0.9 | 0.001
COPOD | 0.063 | 0.001 | 0.001 | 0.9 | 0.9 | 0.001 | 0.001 | 0.9 | 0.9 | 0.9 | 1.0 | 0.001 | 0.001 | 0.001 | 0.9 | 0.001 | 0.001 | 0.9 | 0.9 | 0.9 | 0.491 | 0.9 | 0.727 | 0.105 | 0.9 | 0.9 | 0.567 | 0.9 | 0.047 | 0.9 | 0.9 | 0.001
LOF | 0.35 | 0.9 | 0.9 | 0.001 | 0.001 | 0.9 | 0.9 | 0.001 | 0.001 | 0.001 | 0.001 | 1.0 | 0.9 | 0.9 | 0.001 | 0.9 | 0.9 | 0.001 | 0.001 | 0.001 | 0.034 | 0.001 | 0.009 | 0.241 | 0.001 | 0.001 | 0.022 | 0.001 | 0.42 | 0.001 | 0.001 | 0.9
ODIN | 0.034 | 0.9 | 0.9 | 0.001 | 0.001 | 0.9 | 0.9 | 0.001 | 0.001 | 0.001 | 0.001 | 0.9 | 1.0 | 0.9 | 0.001 | 0.9 | 0.9 | 0.001 | 0.001 | 0.001 | 0.001 | 0.001 | 0.001 | 0.019 | 0.001 | 0.001 | 0.001 | 0.001 | 0.046 | 0.001 | 0.001 | 0.9
ensemble-LOF | 0.9 | 0.866 | 0.9 | 0.001 | 0.001 | 0.597 | 0.9 | 0.048 | 0.01 | 0.001 | 0.001 | 0.9 | 0.9 | 1.0 | 0.057 | 0.9 | 0.9 | 0.065 | 0.004 | 0.107 | 0.605 | 0.001 | 0.35 | 0.9 | 0.001 | 0.014 | 0.529 | 0.001 | 0.9 | 0.001 | 0.009 | 0.9
beta-VAE | 0.9 | 0.001 | 0.006 | 0.9 | 0.58 | 0.001 | 0.001 | 0.9 | 0.9 | 0.9 | 0.9 | 0.001 | 0.001 | 0.057 | 1.0 | 0.001 | 0.001 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.159
COF | 0.01 | 0.9 | 0.9 | 0.001 | 0.001 | 0.9 | 0.9 | 0.001 | 0.001 | 0.001 | 0.001 | 0.9 | 0.9 | 0.9 | 0.001 | 1.0 | 0.9 | 0.001 | 0.001 | 0.001 | 0.001 | 0.001 | 0.001 | 0.005 | 0.001 | 0.001 | 0.001 | 0.001 | 0.015 | 0.001 | 0.001 | 0.698
DeepSVDD | 0.154 | 0.9 | 0.9 | 0.001 | 0.001 | 0.9 | 0.9 | 0.001 | 0.001 | 0.001 | 0.001 | 0.9 | 0.9 | 0.9 | 0.001 | 0.9 | 1.0 | 0.001 | 0.001 | 0.001 | 0.009 | 0.001 | 0.002 | 0.095 | 0.001 | 0.001 | 0.006 | 0.001 | 0.194 | 0.001 | 0.001 | 0.9
u-CBLOF | 0.9 | 0.001 | 0.007 | 0.9 | 0.554 | 0.001 | 0.001 | 0.9 | 0.9 | 0.9 | 0.9 | 0.001 | 0.001 | 0.065 | 0.9 | 0.001 | 0.001 | 1.0 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.175
OCSVM | 0.791 | 0.001 | 0.001 | 0.9 | 0.9 | 0.001 | 0.001 | 0.9 | 0.9 | 0.9 | 0.9 | 0.001 | 0.001 | 0.004 | 0.9 | 0.001 | 0.001 | 0.9 | 1.0 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.731 | 0.9 | 0.9 | 0.018
ABOD | 0.9 | 0.001 | 0.014 | 0.9 | 0.44 | 0.001 | 0.001 | 0.9 | 0.9 | 0.9 | 0.9 | 0.001 | 0.001 | 0.107 | 0.9 | 0.001 | 0.001 | 0.9 | 0.9 | 1.0 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.264
PCA | 0.9 | 0.001 | 0.2 | 0.9 | 0.05 | 0.001 | 0.014 | 0.9 | 0.9 | 0.706 | 0.491 | 0.034 | 0.001 | 0.605 | 0.9 | 0.001 | 0.009 | 0.9 | 0.9 | 0.9 | 1.0 | 0.9 | 0.9 | 0.9 | 0.45 | 0.9 | 0.9 | 0.74 | 0.9 | 0.9 | 0.9 | 0.833
kNN | 0.605 | 0.001 | 0.001 | 0.9 | 0.9 | 0.001 | 0.001 | 0.9 | 0.9 | 0.9 | 0.9 | 0.001 | 0.001 | 0.001 | 0.9 | 0.001 | 0.001 | 0.9 | 0.9 | 0.9 | 0.9 | 1.0 | 0.9 | 0.715 | 0.9 | 0.9 | 0.9 | 0.9 | 0.546 | 0.9 | 0.9 | 0.006
VAE | 0.9 | 0.001 | 0.073 | 0.9 | 0.148 | 0.001 | 0.003 | 0.9 | 0.9 | 0.9 | 0.727 | 0.009 | 0.001 | 0.35 | 0.9 | 0.001 | 0.002 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 1.0 | 0.9 | 0.689 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.597
LODA | 0.9 | 0.004 | 0.647 | 0.723 | 0.004 | 0.001 | 0.128 | 0.9 | 0.9 | 0.249 | 0.105 | 0.241 | 0.019 | 0.9 | 0.9 | 0.005 | 0.095 | 0.9 | 0.9 | 0.9 | 0.9 | 0.715 | 0.9 | 1.0 | 0.088 | 0.9 | 0.9 | 0.28 | 0.9 | 0.731 | 0.9 | 0.9
IF | 0.052 | 0.001 | 0.001 | 0.9 | 0.9 | 0.001 | 0.001 | 0.9 | 0.9 | 0.9 | 0.9 | 0.001 | 0.001 | 0.001 | 0.9 | 0.001 | 0.001 | 0.9 | 0.9 | 0.9 | 0.45 | 0.9 | 0.689 | 0.088 | 1.0 | 0.9 | 0.529 | 0.9 | 0.039 | 0.9 | 0.9 | 0.001
ECOD | 0.9 | 0.001 | 0.001 | 0.9 | 0.845 | 0.001 | 0.001 | 0.9 | 0.9 | 0.9 | 0.9 | 0.001 | 0.001 | 0.014 | 0.9 | 0.001 | 0.001 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 1.0 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.047
LMDD | 0.9 | 0.001 | 0.148 | 0.9 | 0.073 | 0.001 | 0.009 | 0.9 | 0.9 | 0.782 | 0.567 | 0.022 | 0.001 | 0.529 | 0.9 | 0.001 | 0.006 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.529 | 0.9 | 1.0 | 0.816 | 0.9 | 0.9 | 0.9 | 0.757
gen2out | 0.187 | 0.001 | 0.001 | 0.9 | 0.9 | 0.001 | 0.001 | 0.9 | 0.9 | 0.9 | 0.9 | 0.001 | 0.001 | 0.001 | 0.9 | 0.001 | 0.001 | 0.9 | 0.9 | 0.9 | 0.74 | 0.9 | 0.9 | 0.28 | 0.9 | 0.9 | 0.816 | 1.0 | 0.148 | 0.9 | 0.9 | 0.001
LUNAR | 0.9 | 0.01 | 0.816 | 0.554 | 0.001 | 0.002 | 0.249 | 0.9 | 0.866 | 0.128 | 0.047 | 0.42 | 0.046 | 0.9 | 0.9 | 0.015 | 0.194 | 0.9 | 0.731 | 0.9 | 0.9 | 0.546 | 0.9 | 0.9 | 0.039 | 0.9 | 0.9 | 0.148 | 1.0 | 0.563 | 0.841 | 0.9
KDE | 0.622 | 0.001 | 0.001 | 0.9 | 0.9 | 0.001 | 0.001 | 0.9 | 0.9 | 0.9 | 0.9 | 0.001 | 0.001 | 0.001 | 0.9 | 0.001 | 0.001 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.731 | 0.9 | 0.9 | 0.9 | 0.9 | 0.563 | 1.0 | 0.9 | 0.007
HBOS | 0.9 | 0.001 | 0.001 | 0.9 | 0.9 | 0.001 | 0.001 | 0.9 | 0.9 | 0.9 | 0.9 | 0.001 | 0.001 | 0.009 | 0.9 | 0.001 | 0.001 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.841 | 0.9 | 1.0 | 0.032
SOD | 0.9 | 0.639 | 0.9 | 0.006 | 0.001 | 0.35 | 0.9 | 0.137 | 0.037 | 0.001 | 0.001 | 0.9 | 0.9 | 0.9 | 0.159 | 0.698 | 0.9 | 0.175 | 0.018 | 0.264 | 0.833 | 0.006 | 0.597 | 0.9 | 0.001 | 0.047 | 0.757 | 0.001 | 0.9 | 0.007 | 0.032 | 1.0

## References

  * Agarwal (2007) D. Agarwal.  Detecting anomalies in cross-classified streams: a bayesian approach.  _Knowledge and information systems_ , 11(1):29–44, 2007.
  * Amer and Goldstein (2012) M. Amer and M. Goldstein.  Nearest-neighbor and clustering based anomaly detection algorithms for Rapidminer.  In _Proceedings of the 3rd RapidMiner Community Meeting and Conference (RCOMM 2012)_ , pages 1–12, 2012.
  * An and Cho (2015) J. An and S. Cho.  Variational autoencoder based anomaly detection using reconstruction probability.  _Special Lecture on IE_ , 2(1):1–18, 2015.
  * Arning et al. (1996) A. Arning, R. Agrawal, and P. Raghavan.  A linear method for deviation detection in large databases.  _KDD_ , 1141(50):972–981, 1996.
  * Bandaragoda et al. (2018) T. R. Bandaragoda, K. M. Ting, D. Albrecht, F. T. Liu, Y. Zhu, and J. R. Wells.  Isolation-based anomaly detection using nearest-neighbor ensembles.  _Computational Intelligence_ , 34(4):968–998, 2018.
  * Bar-Joseph et al. (2001) Z. Bar-Joseph, D. K. Gifford, and T. S. Jaakkola.  Fast optimal leaf ordering for hierarchical clustering.  _Bioinformatics_ , 17(suppl_1):S22–S29, 2001\.
  * Birgé and Rozenholc (2006) L. Birgé and Y. Rozenholc.  How many bins should be put in a regular histogram.  _ESAIM: Probability and Statistics_ , 10:24–45, 2006.
  * Brandsæter et al. (2019) A. Brandsæter, E. Vanem, and I. K. Glad.  Efficient on-line anomaly detection for ship systems in operation.  _Expert Systems with Applications_ , 121:418–437, 2019\.
  * Breunig et al. (2000) M. M. Breunig, H.-P. Kriegel, R. T. Ng, and J. Sander.  LOF: identifying density-based local outliers.  In _Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data_ , pages 93–104, 2000.
  * Campos et al. (2016) G. O. Campos, A. Zimek, J. Sander, R. J. Campello, B. Micenková, E. Schubert, I. Assent, and M. E. Houle.  On the evaluation of unsupervised outlier detection: measures, datasets, and an empirical study.  _Data Mining and Knowledge Discovery_ , 30(4):891–927, 2016.
  * Demšar (2006) J. Demšar.  Statistical comparisons of classifiers over multiple data sets.  _The Journal of Machine Learning Research_ , 7:1–30, 2006\.
  * Domingues et al. (2018) R. Domingues, M. Filippone, P. Michiardi, and J. Zouaoui.  A comparative evaluation of outlier detection algorithms: Experiments and analyses.  _Pattern Recognition_ , 74:406–421, 2018.
  * Dua and Graff (2017) D. Dua and C. Graff.  UCI machine learning repository, 2017.  URL http://archive.ics.uci.edu/ml.
  * Emmott et al. (2015) A. Emmott, S. Das, T. Dietterich, A. Fern, and W.-K. Wong.  A meta-analysis of the anomaly detection problem.  _arXiv preprint arXiv:1503.01158_ , 2015.
  * Emmott et al. (2013) A. F. Emmott, S. Das, T. Dietterich, A. Fern, and W.-K. Wong.  Systematic construction of anomaly detection benchmarks from real data.  In _Proceedings of the ACM SIGKDD workshop on outlier detection and description_ , pages 16–21, 2013.
  * Fernández-Delgado et al. (2014) M. Fernández-Delgado, E. Cernadas, S. Barro, and D. Amorim.  Do we need hundreds of classifiers to solve real world classification problems?  _The Journal of Machine Learning Research_ , 15(1):3133–3181, 2014.
  * Foorthuis (2021) R. Foorthuis.  On the nature and types of anomalies: A review of deviations in data.  _International Journal of Data Science and Analytics_ , 12(4):297–331, 2021.
  * Goldstein and Dengel (2012) M. Goldstein and A. Dengel.  Histogram-based outlier score (HBOS): A fast unsupervised anomaly detection algorithm.  _KI-2012: Poster and Demo Track_ , pages 59–63, 2012.
  * Goldstein and Uchida (2016) M. Goldstein and S. Uchida.  A comparative evaluation of unsupervised anomaly detection algorithms for multivariate data.  _PloS one_ , 11(4):e0152173, 2016.
  * Goodge et al. (2022) A. Goodge, B. Hooi, S.-K. Ng, and W. S. Ng.  Lunar: Unifying local outlier detection methods via graph neural networks.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pages 6737–6745, 2022.
  * Han et al. (2022) S. Han, X. Hu, H. Huang, M. Jiang, and Y. Zhao.  Adbench: Anomaly detection benchmark.  In _Neural Information Processing Systems (NeurIPS)_ , 2022.
  * Hariri et al. (2019) S. Hariri, M. C. Kind, and R. J. Brunner.  Extended Isolation Forest.  _IEEE Transactions on Knowledge and Data Engineering_ , 33(4):1479–1489, 2019.
  * Hautamaki et al. (2004) V. Hautamaki, I. Karkkainen, and P. Franti.  Outlier detection using k-nearest neighbour graph.  In _Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004._ , volume 3, pages 430–433. IEEE, 2004.
  * He and McAuley (2016) R. He and J. McAuley.  Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering.  In _proceedings of the 25th international conference on world wide web_ , pages 507–517, 2016.
  * He et al. (2003) Z. He, X. Xu, and S. Deng.  Discovering cluster-based local outliers.  _Pattern Recognition Letters_ , 24(9-10):1641–1650, 2003.
  * Iman and Davenport (1980) R. L. Iman and J. M. Davenport.  Approximations of the critical region of the fbietkan statistic.  _Communications in Statistics - Theory and Methods_ , 9(6):571–595, 1980.
  * Japkowicz et al. (1995) N. Japkowicz, C. Myers, M. Gluck, et al.  A novelty detection approach to classification.  In _IJCAI_ , volume 1, pages 518–523. Citeseer, 1995.
  * Kriegel et al. (2008) H.-P. Kriegel, M. Schubert, and A. Zimek.  Angle-based outlier detection in high-dimensional data.  In _Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data mining_ , pages 444–452, 2008.
  * Kriegel et al. (2009) H.-P. Kriegel, P. Kröger, E. Schubert, and A. Zimek.  Outlier detection in axis-parallel subspaces of high dimensional data.  In _Pacific-Asia Conference on Knowledge Discovery and Data Mining_ , pages 831–838. Springer, 2009.
  * Krizhevsky et al. (2009) A. Krizhevsky, G. Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * Latecki et al. (2007) L. J. Latecki, A. Lazarevic, and D. Pokrajac.  Outlier detection with kernel density functions.  In _International Workshop on Machine Learning and Data Mining in Pattern Recognition_ , pages 61–75. Springer, 2007.
  * Lee et al. (2021) M.-C. Lee, S. Shekhar, C. Faloutsos, T. N. Hutson, and L. Iasemidis.  Gen 2 out: Detecting and ranking generalized anomalies.  In _2021 IEEE International Conference on Big Data (Big Data)_ , pages 801–811. IEEE, 2021.
  * Li et al. (2020) Z. Li, Y. Zhao, N. Botta, C. Ionescu, and X. Hu.  Copod: copula-based outlier detection.  In _2020 IEEE International Conference on Data Mining (ICDM)_ , pages 1118–1123. IEEE, 2020.
  * Li et al. (2022) Z. Li, Y. Zhao, X. Hu, N. Botta, C. Ionescu, and G. H. Chen.  Ecod: Unsupervised outlier detection using empirical cumulative distribution functions, 2022.
  * Liu et al. (2008) F. T. Liu, K. M. Ting, and Z.-H. Zhou.  Isolation forest.  In _2008 Eighth IEEE International Conference on Data Mining_ , pages 413–422. IEEE, 2008.
  * Liu et al. (2019) Y. Liu, Z. Li, C. Zhou, Y. Jiang, J. Sun, M. Wang, and X. He.  Generative adversarial active learning for unsupervised outlier detection.  _IEEE Transactions on Knowledge and Data Engineering_ , 32(8):1517–1528, 2019.
  * Malik et al. (2014) K. Malik, H. Sadawarti, and K. G S.  Comparative analysis of outlier detection techniques.  _International Journal of Computer Applications_ , 97(8):12–21, 2014.
  * Nemenyi (1963) P. B. Nemenyi.  _Distribution-free multiple comparisons._ Princeton University, 1963.
  * Pevnỳ (2016) T. Pevnỳ.  Loda: Lightweight on-line detector of anomalies.  _Machine Learning_ , 102(2):275–304, 2016.
  * Ramaswamy et al. (2000) S. Ramaswamy, R. Rastogi, and K. Shim.  Efficient algorithms for mining outliers from large data sets.  In _Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data_ , pages 427–438, 2000.
  * Rayana (2016) S. Rayana.  ODDS library, 2016.  URL http://odds.cs.stonybrook.edu.
  * Rousseeuw and Croux (1993) P. J. Rousseeuw and C. Croux.  Alternatives to the median absolute deviation.  _Journal of the American Statistical association_ , 88(424):1273–1283, 1993.
  * Rousseeuw and Driessen (1999) P. J. Rousseeuw and K. V. Driessen.  A fast algorithm for the minimum covariance determinant estimator.  _Technometrics_ , 41(3):212–223, 1999.
  * Ruff et al. (2019) L. Ruff, R. A. Vandermeulen, N. Görnitz, A. Binder, E. Müller, K.-R. Müller, and M. Kloft.  Deep semi-supervised anomaly detection.  _arXiv preprint arXiv:1906.02694_ , 2019.
  * Ruff et al. (2021) L. Ruff, J. R. Kauffmann, R. A. Vandermeulen, G. Montavon, W. Samek, M. Kloft, T. G. Dietterich, and K.-R. Müller.  A unifying review of deep and shallow anomaly detection.  _Proceedings of the IEEE_ , 2021.
  * Schölkopf et al. (1999) B. Schölkopf, R. C. Williamson, A. J. Smola, J. Shawe-Taylor, J. C. Platt, et al.  Support vector method for novelty detection.  In _NIPS_ , volume 12, pages 582–588, 1999.
  * Schubert and Zimek (2019) E. Schubert and A. Zimek.  Elki: A large open-source library for data analysis.  _CoRR_ , abs/1902.03616, 2019.  URL https://arxiv.org/abs/1902.03616.
  * Shin and Kim (2020) S. Y. Shin and H.-j. Kim.  Extended autoencoder for novelty detection with reconstruction along projection pathway.  _Applied Sciences_ , 10(13):4497, 2020.
  * Shyu et al. (2003) M.-L. Shyu, S.-C. Chen, K. Sarinnapakorn, and L. Chang.  A novel anomaly detection scheme based on principal component classifier.  Technical report, University of Miami, department of Eletrical and Computer Engineering, 2003.
  * Simpson (1951) E. H. Simpson.  The interpretation of interaction in contingency tables.  _Journal of the Royal Statistical Society: Series B (Methodological)_ , 13(2):238–241, 1951.
  * Soenen et al. (2021) J. Soenen, E. Van Wolputte, L. Perini, V. Vercruyssen, W. Meert, J. Davis, and H. Blockeel.  The effect of hyperparameter tuning on the comparative evaluation of unsupervised anomaly detection methods.  In _Proceedings of the KDD’21 Workshop on Outlier Detection and Description_ , pages 1–9. Outlier Detection and Description Organising Committee, 2021.
  * Steinbuss and Böhm (2021) G. Steinbuss and K. Böhm.  Benchmarking unsupervised outlier detection with realistic synthetic data.  _ACM Transactions on Knowledge Discovery from Data (TKDD)_ , 15(4):1–20, 2021.
  * Tang et al. (2002) J. Tang, Z. Chen, A. W.-C. Fu, and D. W. Cheung.  Enhancing effectiveness of outlier detections for low density patterns.  In _Pacific-Asia Conference on Knowledge Discovery and Data Mining_ , pages 535–548. Springer, 2002.
  * Thomas et al. (2016) A. Thomas, A. Gramfort, and S. Clémençon.  Learning hyperparameters for unsupervised anomaly detection.  In _Conférence sur L’apprentissage automatique-Cap 2016_ , 2016\.
  * Xu et al. (2018) X. Xu, H. Liu, L. Li, and M. Yao.  A comparison of outlier detection techniques for high-dimensional data.  _International Journal of Computational Intelligence Systems_ , 11(1):652–662, 2018.
  * Zenati et al. (2018) H. Zenati, M. Romain, C.-S. Foo, B. Lecouat, and V. Chandrasekhar.  Adversarially learned anomaly detection.  In _2018 IEEE International conference on data mining (ICDM)_ , pages 727–736. IEEE, 2018.
  * Zhao et al. (2019) Y. Zhao, Z. Nasrullah, and Z. Li.  Pyod: A python toolbox for scalable outlier detection, 2019.
  * Zhou et al. (2020) L. Zhou, W. Deng, and X. Wu.  Unsupervised anomaly localization using vae and beta-vae.  _arXiv preprint arXiv:2005.10686_ , 2020.
  * Zimek et al. (2014) A. Zimek, R. J. Campello, and J. Sander.  Ensembles for unsupervised outlier detection: challenges and research questions a position paper.  _ACM SIGKDD Explorations Newsletter_ , 15(1):11–22, 2014.