# Unsupervised Anomaly Detection in Time-series: An Extensive Evaluation and
Analysis of State-of-the-art Methods

Nesryne Mejri, Laura Lopez-Fuentes, Kankana Roy, Pavel Chernakov, Enjie
Ghorbel, and Djamila Aouada All authors are affiliated to the Computer Vision,
Imaging & Machine Intelligence Research Group (CVI²) at the Interdisciplinary
Centre for Security, Reliability and Trust (SnT), University of Luxembourg
E-mail<EMAIL_ADDRESS>
###### Abstract

Unsupervised anomaly detection in time-series has been extensively
investigated in the literature. Notwithstanding the relevance of this topic in
numerous application fields, a complete and extensive evaluation of recent
state-of-the-art techniques is still missing. Few efforts have been made to
compare existing unsupervised time-series anomaly detection methods
rigorously. However, only standard performance metrics, namely precision,
recall, and F1-score are usually considered. Essential aspects for assessing
their practical relevance are therefore neglected.

This paper proposes an original and in-depth evaluation study of recent
unsupervised anomaly detection techniques in time-series. Instead of relying
solely on standard performance metrics, additional yet informative metrics and
protocols are taken into account. In particular, (1) more elaborate
performance metrics specifically tailored for time-series are used; (2) the
model size and the model stability are studied; (3) an analysis of the tested
approaches with respect to the anomaly type is provided; and (4) a clear and
unique protocol is followed for all experiments. Overall, this extensive
analysis aims to assess the maturity of state-of-the-art time-series anomaly
detection, give insights regarding their applicability under real-world setups
and provide to the community a more complete evaluation protocol.

###### Index Terms:

Time-series, Unsupervised Anomaly Detection, Evaluation Study.

##  1 Introduction

AMultivariate time-series corresponds to a temporally ordered set of
variables. This mathematical representation has been used in countless domains
such as finance, health, and biomechanics. Designing methods for automatically
analyzing time-series (e.g. forecasting, classification, anomaly detection)
has been widely investigated by researchers [6, 9]. A particular focus is
given to anomaly detection in time-series [15]. In general, an anomaly or
outlier can be defined as an observation or sample that does not follow an
expected pattern. The popularity of anomaly detection in time-series is
probably due to its interest in numerous industrial contexts. As an example,
one can mention the detection of faulty sensors [23], fraudulent bank
transactions [7], and pathologies in medical data [22].

In the literature, some attempts have been made to develop supervised and
semi-supervised approaches [10, 8]. Nevertheless, the task of time-series
anomaly detection is usually formulated as an unsupervised problem [20]. In
fact, since anomalies occur rarely, annotating data becomes challenging and
costly. In this article, we focus on the topic of unsupervised anomaly
detection in time-series.

Paper |  | Standard performance
---
metrics
| Performance metrics
---
for time-series [59]
| Network
---
size
| Evaluation of
---
recent techniques
| Analysis w.r.t
---
anomaly type
| Unified experimental
---
protocol
| Model
---
stability
[3] | ✓ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗
[41] | ✓ | ✗ | ✗ | ✗ | ✓ | ✓ | ✗
[1] | ✓ | ✗ | ✗ | ✓ | ✗ | ✓ | ✗
This work | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓

TABLE I: Comparison of existing evaluation studies of anomaly detection in
time-series: we specify which of the following aspects were taken into
account: (1) standard performance metrics which correspond to the precision,
recall, and F1-score; (2) revisited performance metrics extending the
precision, recall, and F1-score to time-series introduced in [59] ; (3)
network size; (4) evaluation of recent deep learning techniques; (5) analysis
with respect to the types of anomalies; and (6) use of a unified experimental
protocol.

Earlier methods for anomaly detection in time-series have mostly employed
traditional machine learning [17, 18] and auto-regressive [16, 19] techniques.
However, as discussed in [3], these approaches are mainly subject to the curse
of dimensionality. In other words, their performance drops in the presence of
high-dimensional time-series.

To address this, motivated by the tremendous advances in Deep Learning (DL),
massive efforts have been recently made to design suitable Neural Network
(DNN) architectures [5, 2, 26]. These DL-based approaches have achieved
impressive performance in terms of standard performance metrics (precision,
recall, and F1-score). Nevertheless, despite their promising results, their
suitability in a realistic industrial context still needs further
investigation. For that purpose, it is timely to propose an extensive
comparison of recent DL techniques that considers the following aspects:
(1) Model size and model stability: Some important metrics for comparing DNNs
in a real-world context are usually not reported. Existing methods overlook
the model size and the model stability. By stable model, we mean a model which
has stable performance under different training trials.
(2) Unified experimental protocol: There is no clear experimental protocol for
evaluating state-of-the-art methods. As a consequence, it can be noted that
the reported experimental values vary considerably from one paper to another.
For instance, as highlighted in [1], a peculiar evaluation protocol called
point adjustment [2] is often used [24, 26], while it is ignored in other
cases [34, 33].
(3) Performance metrics for time-series: As discussed in [59], the used
standard performance metrics (precision, recall, and F1-score) might not be
entirely adequate for evaluating time-series anomaly detectors. These metrics
were initially designed for time-independent predictions and not for range-
based ones. As an alternative, Tatbul et al. [59] extended these metrics to
time-series. However, it can be noted that current state-of-the-art methods do
not consider these relatively novel evaluation criteria.
(4) Experimental analysis with respect to the anomaly type: a detailed
experimental evaluation with respect to the type of anomaly is missing in the
state-of-the-art. Significant efforts have been dedicated to rigorously
defining the different possible types of outliers in time-series[3, 41].
However, no concrete experimental analysis has been carried out in that
direction.

In the literature, few evaluation studies for comparing recent anomaly
detection algorithms [1, 3, 41] have been conducted. For instance, Choi et al.
[3] present a brief comparison of recent DL algorithms in terms of precision,
recall, and F1-score but neglect the model size and model stability. We can
also mention the work of Lai et al. [41], where a new taxonomy for time-series
outliers is proposed. Then, based on that, a methodology to generate synthetic
datasets is suggested. They finally compare 9 different algorithms according
to to the outlier types but do not include the latest DL algorithms. Moreover,
similar to [3], they only focus on classical evaluation criteria. Finally, Kim
et al. [1] present a rigorous evaluation of recent DL techniques by
questioning the point adjustment protocol. Nevertheless, the model size and
model stability as well as the performance metrics for time-series are not
considered.

Hence, in this survey, we provide a comprehensive evaluation study of state-
of-the-art algorithms by taking into account all the mentioned aspects (1) to
(4). As summarized in Table I, standard performance metrics, as well as the
novel performance metrics proposed in [59] are considered. In addition, the
number of parameters of DL-based approaches is reported as it directly impacts
the memory consumption and the model scalability. Moreover, experiments
according to the nature of anomalies are carried out using the taxonomy that
was recently introduced in [41]. Lastly, a unified experimental protocol is
used to compare existing methods. This in-depth evaluation study will help the
community understand the advantages and limitations of state-of-the-art
techniques from a broader applicative perspective and lay the foundations for
better experimental evaluation practices.

The remainder of this paper is organized as follows. Section 2 presents
preliminaries necessary for the understating of this paper. Section 3 reviews
state-of-the-art time-series anomaly detection methods. Section 4 describes
the used datasets and details the evaluation protocol considered in the
experiments. Section 5 presents and analyzes the results. Finally, Section 6
concludes this work.

##  2 Preliminaries

A time-series is a temporally ordered set of $n$ variables which can be
denoted by $X=\\{X_{t}\\}_{1\leq t\leq N}$ where $X_{t}\in\mathbb{R}^{n}$
refers to the $n$-dimensional vector of variables at an instant $t$. Note that
the time-series is univariate if $n=1$, and is multivariate otherwise ($n>1$).
This section reviews the necessary background for a better understanding of
this survey. Specifically, we start by recalling the different types of time-
series anomalies according to the taxonomy of [59]. Then, we present the usual
paradigms employed for anomaly detection in time-series.

###  2.1 Types of anomalies

As discussed in [3], anomalies in time-series can generally be classified into
three main categories, namely, point, contextual, and collective anomalies.
However, unlike point anomalies, the definitions of contextual and collective
ones are more ambiguous in the state-of-the-art, as stated in [41]. Indeed,
they are heavily impacted by the application context. For instance, Yu et al.
[29] define contextual anomalies as small temporal segments formed by
neighboring points, while Golmohammadi et at. [30] consider them as seasonal
points (occurring periodically). Lai et al. have recently refined the
definition of outlier types [41]. They distinguish between point-wise outliers
and pattern-wise outliers. The former is formed by global and contextual
outliers while the latter is composed of shapelet, seasonal, and trend
outliers. In the following, the taxonomy proposed in [41], which is central to
our analysis, is recalled.

(a) Global outlier

(b) Contextual outlier

(c) Shapelet outlier

(d) Seasonal outlier

(e) Trend outlier

Figure 1: Examples of the 5 different types of anomalies proposed in [41].

####  2.1.1 Point-wise anomalies

Point-wise outliers are local anomalies occurring on individual time stamps.
Let $X=\\{X_{t}\\}_{1\leq t\leq N}$ be a multivariate time-series and
${\hat{X}}_{t}$ the expected value of $X_{t}$ at an instant $t$ according to a
regression model. Given a well-chosen threshold $\delta>0$, an anomaly at an
instant $t$ can be formally defined by,

$\lVert X_{t}-{\hat{X}}_{t}\rVert>\delta,$ (1)

where $\lVert.\rVert$ defines an $L_{p}$ norm.
Global outliers. They can be seen as point-wise anomalies which importantly
deviate from the rest of the points in a time-series. They usually correspond
to spikes in the time-series, as shown in Fig. 1(a). In this case, the
threshold $\delta$ can be formulated as,

$\delta=\lambda\sigma({X}),$ (2)

where $\sigma(.)$ refers to the standard deviation operator and
$\lambda\in\mathbb{R}^{+*}$.

Contextual outliers. They refer to individual points which differ
significantly from their neighbors. The latter are often small glitches in the
time-series as illustrated in Fig. 1(b). The threshold can be defined as,

$\delta=\lambda\sigma(X_{t-k:t+k}),$ (3)

where $X_{t-k:t+k}=\\{X_{t-k},X_{t-k+1},...,X_{t+k}\\}$ is the signal
corresponding to the temporal window centered on $t$. The function $\sigma(.)$
refers to the standard deviation operator and $\lambda\in\mathbb{R}^{+*}$.

####  2.1.2 Pattern-wise anomalies

Pattern-wise anomalies refer to anomalous sub-sequences which typically
showcase discords or irregularities. These anomalies are defined in [41] by
modeling a time-series $X$ with spectral structural analysis [61] as follows,

$X=\rho(2\pi\omega T)+\tau(T),$ (4)

such that $\rho(2\pi\omega
T)=\sum_{k}[A\sin(2\pi\omega_{k}T)+B\cos(2\pi\omega_{k}T)]$ corresponds to the
base shapelet function which can be interpreted as the characteristic shape of
$X$. The seasonality, which describes a pattern occurring at specific regular
intervals in a time-series, is modeled with $w=\\{w_{1},w_{2},...,w_{k}\\}$.
Finally, a trend function denoted by $\tau$ defines the global direction of
$X$. In particular, a sub-sequence $X_{i:j}$ of a time-series $X$ with $1\leq
i<j\leq N$ can be formulated using a shapelet function such that,

${X}_{i:j}={\rho}(2\pi\omega T_{i,j})+\tau(T_{i,j}),$ (5)

with $\rho$, $\omega$, $\tau$, and $T_{i,j}$ respectively being the shape, the
seasonality, the trend, and the time-stamps of the sub-sequence. The analysis
of the shapelet, the seasonality as well as the trend functions allow
distinguishing the three following outliers:

Shapelet outliers. They represent the anomalous sub-sequences enclosing
shapelets that are different from the expected ones, as shown in Fig. 1(c).
The following condition can be used to define shapelet outliers as follows,

$d_{\rho}(\rho(.),\hat{\rho}(.))>\delta,$ (6)

with $d_{\rho}$ being a dissimilarity measure computed between two sets of
shapelets. $\hat{\rho}(.)$ corresponds to the expected shapelets in a given
sub-sequence and $\delta$ is the threshold.

Seasonal outliers. They can be defined as sub-sequences with unexpected
seasonalities with respect to the full sequence, as illustrated in Fig. 1(d).

$d_{\omega}(\omega,\hat{\omega})>\delta,$ (7)

with $d_{\omega}$ being a dissimilarity measure between two seasonality,
$\hat{\omega}$ being the expected seasonality in the sub-sequence, and
$\delta$ being the threshold.

Trend outliers. They refer to sub-sequences with an importantly altered trend.
Consequently, a shift in the mean data can be observed, as shown in Fig. 1(e).
Mathematically, trend outliers can be defined by,

$d_{\tau}(\tau,\hat{\tau})>\delta$ (8)

where $d_{\tau}$ is a dissimilarity measure computed between two trends,
$\hat{\tau}$ is the expected trend of the sub-sequence, and $\delta$ is the
threshold.

###  2.2 Paradigms for anomaly detection in times-series

Existing anomaly detection methods in time-series mainly employ five different
paradigms, namely, clustering-based, density estimation-based, distance-based,
reconstruction-based and forecasting-based methods.

Figure 2: Overview of the different paradigms for anomaly detection in time-
series: in contrast to clustering and probabilistic approaches, distance-
based, reconstruction-based and forecasting-based approaches take into account
the temporal aspect.

####  2.2.1 Clustering-based methods

Let $\mathcal{S}^{n}$ be the feature space of multivariate time-series of
dimension $n$. Let $\mathcal{N}^{n}$ be the estimated sub-space of normal
time-series of dimension $n$ such that
$\mathcal{N}^{n}\subset\mathcal{S}^{n}$. Let $f$ be a feature extractor
function which maps an input time-series ${X}\in\mathbb{R}^{n\times N}$ to
$\mathcal{S}^{n}$. An anomaly is detected if,

$f({X})\notin\mathcal{N}^{n}.$ (9)

Note that the classification of $X$ as an anomaly or not can also be
determined with the use of a distance that is compared to a threshold. This is
the case, for example, of Support Vector Data Description (SVDD) [54], which
measures the distance from the centroids.

####  2.2.2 Density estimation-based methods

Density estimation-based methods mainly aim to estimate the probability
density function of normal time-series denoted as $\mathbf{p}_{\theta}$. Given
a time-series ${X}$, the likelihood function $\mathcal{L}$ of $\theta$ and a
threshold $\tau$, an anomaly is detected if,

$\mathcal{L}(\theta|{X})>\tau.$ (10)

####  2.2.3 Distance-based methods

Distance-based methods rely on the definition of an adequate distance between
two temporal sequences. This distance should measure the dissimilarity between
them. Let ${X}$ and ${R}$ be, respectively, a given time-series and a
reference normal time-series. Let us denote by $D$ a distance for time-series.
Given a predefined threshold $\delta$, an anomaly is detected in ${X}$ if,

$D({X},{R})>\delta.$ (11)

####  2.2.4 Reconstruction-based methods

Reconstruction-based approaches aim at learning a model for the accurate and
full reconstruction of a normal time-series. The assumption is that the
learned model will fail when reconstructing abnormal sequences. Let ${X}$ and
${\hat{X}}$ be respectively the original and the reconstructed time-series.
Given a predefined threshold $\delta$, an anomaly is detected in ${X}$ if,

$\|{X}-{\hat{X}}\|>\delta.$ (12)

####  2.2.5 Forecasting-based methods

Forecasting-based approaches are based on the prediction of future states
given previous observations. Similar to reconstruction-based methods, they
assume that the prediction will be less accurate in the presence of an
anomaly. Let $X=\\{{X}_{0},{X}_{1},...,{X}_{N}\\}$ be a time-series where
${X}_{i}$ refers to an observation of ${X}$ at an instant $i$. Given a
threshold $\delta$, an anomaly is detected at an instant $i$ if,

$\|{\hat{X}}_{i}-{X}_{i}\|>\delta,$ (13)

where ${\hat{X}}_{t_{i}}$ corresponds to the predicted state given the
observation ${X}_{0:{i-1}}=\\{{X}_{0},{X}_{1},...,{X}_{{i-1}}\\}$.

##  3 State-of-the-art on time-series anomaly detection

Over the last two decades, the research community has widely explored the
field of anomaly detection in time-series. The latter can be addressed from
five different perspectives. As reported in Section 2, we distinguish between
clustering-based, density-estimation-based, distance-based, reconstruction-
based, and forecasting-based techniques. Earlier techniques have investigated
these five different paradigms by exploiting traditional machine learning [17,
18] and statistical tools [19]. Nevertheless, as mentioned in [3], these
approaches have shown a drop in performance when dealing with high-dimensional
time series. Given the recent advances in DL, DNNs have been considered as an
alternative [26, 34, 24, 33], mainly taking inspiration from traditional
methods. In the following, we review these five categories of approaches,
starting with conventional techniques, then moving to current DL methods.

###  3.1 Clustering-based methods

Clustering-based methods are discriminative approaches aiming to estimate
explicitly or implicitly decision boundaries for detecting anomalies [4, 49,
54] as depicted in Eq. 9. One-Class Support Vector Machine (OC-SVM) [4] is
probably one of the most popular algorithms for anomaly detection. Its goal is
to estimate the support of a high-dimensional distribution. This one-class
classification method has been mainly used for detecting time-independent
anomalies [56, 57] but has also been employed for isolating outliers in time-
series [58]. Inspired by Support Vector Machines (SVM), Support Vector Data
Description (SVDD) [54] is another well-known method that is often used in the
context of anomaly detection [55]. Similar to SVM, kernels that map data
representations to a higher dimensional space can be used. However, instead of
relying on the estimation of a hyperplane, SVDD computes spherically shaped
boundaries.

Shallow clustering-based approaches necessitate the hand-crafting of
discriminative features and often require the selection of an appropriate
kernel. Recently, with the advances in DL, there have been attempts to extend
these classical approaches. Most of these methods, such as Deep SVDD [48]
variants, extend traditional methods by learning a kernel that maps data to a
discriminative high-dimensional feature space. This is usually carried out by
optimizing a Neural Network. These approaches have shown promising results
when dealing with non-sequential data. Unfortunately, the temporal modeling of
time-series is often disregarded, mainly relying on a simple sliding window.
As a solution, Shen et al.[25] suggest fusing multi-scale temporal features
and employing a Recurrent Neural Networks (RNN) to model temporal
dependencies.

###  3.2 Density-estimation methods

As described in Section 2.2.2, these probabilistic approaches detect anomalies
by estimating the normal data density function. For example, Breunig et al.
[50] proposed a method referred to as Local Outlier Factor (LOF) to detect
anomalies by computing the local density. Tang et al. [53] calculate the local
connectivity for determining anomalies instead. In [51] and [52], a Gaussian
Mixture Model (GMM) and Kernel Density Estimation (KDE) are respectively used
for estimating the density of normal representations. Over the last years,
efforts have been made to introduce DNN-based probabilistic methods. For
instance, Zong et al. [35] proposed to train an auto-encoder for extracting
relevant representations before fitting a GMM. Nevertheless, as for
clustering-based methods, probabilistic approaches usually do not model the
temporal aspect restricting their effectiveness in the context of time-series
anomaly detection.

###  3.3 Distance-based methods

Distance-based methods usually define explicitly a distance between a time-
series and a reference to detect anomalies [37, 38, 39, 28], as described in
Section 2.2.3. Among the most used distance-based algorithm, one can refer to
Dynamic Time Warping [40], which aims at finding the optimal match between two
ordered sequences. Despite their relevance, distance-based methods have been
less explored given their relatively high complexity induced by optimal
matching and the need for defining a reference time-series [37].

###  3.4 Reconstruction-based methods

Reconstruction-based methods aim at reconstructing the entire time-series, as
presented in Section 2.2.4. Shallow reconstruction-based time-series anomaly
detection methods [42, 43, 44] have mainly adopted Principal Component
Analysis [46] (PCA) or its variants such as kernel PCA (kPCA) [47]. These
approaches estimate an orthogonal projection, then compute a reconstruction
error between the original and reconstructed time-series. Lately, Auto-
Encoders (AE) [45] have been introduced as the deep learning-based counterpart
of PCA. Unsurprisingly, the latter has been adopted in the context of anomaly
detection in time-series [32]. For example, Lin et al. [31] introduce a Long-
Short Term Memory Variation Auto-Encoder (LSTM-VAE) architecture. While the
Variation Auto-Encoder architecture (VAE) is used for learning robust
representations, a Long Short-Term Memory (LSTM) network allows modeling
temporal dependencies. Generative Adversarial Networks (GAN) have also been
proposed as a reconstruction-based method. In [24], Audibert et al. attempted
to take the best of both worlds. In particular, they introduced adversarially
trained autoencoders for detecting anomalies in time-series.

###  3.5 Forecasting-based methods

As discussed in Section 2.2.5, traditional forecasting-based anomaly detection
methods are primarily based on auto-regression-based models such as
AutoRegressive Integrated Moving Average (ARIMA) [36]. With the recent
advances in deep learning, LSTM has been used to replace auto-regression
models [34]. This architecture allows modeling short-term as well as long-term
temporal dependencies. Deng et al. [5] have recently proposed a graph-based
deep learning model with an attention mechanism for capturing multivariate
correlations.

###  3.6 Hybrid methods

As discussed in [14], reconstruction and forecasting-based approaches have
shown to be, so far, the best candidates for anomaly detection in time-series.
While reconstruction-based methods allow modeling inconsistencies within the
global distribution of time series, forecasting-based approaches are more
appropriate for capturing local anomalies. For that reason, Zhao et al. [14]
have introduced a hybrid method leveraging these two complementary paradigms.
Specifically, they design a two-stream attention-based graph network that
simultaneously optimizes forecasting and reconstruction losses.

##  4 Datasets and Evaluation protocol

In this section, the datasets, the evaluation criteria, the pre-processing and
post-processing algorithms as well as the considered methods for the
experiments are presented.

###  4.1 Datasets

A total of five datasets have been considered for evaluating recent methods
for anomaly detection in time-series. Table II details the different
characteristics of each dataset. The considered benchmarks are:

Secure Water Treatment (SWaT). It is a dataset collected from a testbed water
treatment for 11 days. During the last 4 days, 36 attacks of different
duration and natures have been introduced. The data collected over the 7 first
days have been used for training in all our experiments. During this period,
the water treatment was carried out under normal conditions. In contrast, the
data gathered during the last 4 days were exposed to multiple attacks. The
latter have been used for testing111Details of the SWaT dataset can be found
here .
Mars Science Laboratory (MSL). It is formed by 27 telemetry signals collected
from the Curiosity Rover spacecraft on Mars. Each signal consists of a
multivariate time-series of dimension 55. The first dimension encloses
telemetry data, while the remaining 54 correspond to a one-hot encoded
command. The publicly available dataset has been released by NASA [13]. The
training and testing data are separated, and anomalies are annotated.
Nevertheless, it can be noted that the experimental protocol varies from one
reference to another. In particular, some studies such as [11] ignore the one-
hot encoded vector considering only telemetry data. In addition, other
approaches such as [5] combine the telemetric data from 27 signals assuming
that it forms a unique dataset. Nevertheless, in most cases, authors do not
provide sufficient information about their experimental protocol, making a
direct comparison not straightforward. In this paper, we follow the
experimental protocol of [12]. Each signal is considered to be a separate and
independent multivariate sub-dataset. This means that the training and testing
phases are performed each time on one single sub-dataset. Finally, the average
performance is reported.
Soil Moisture Active Passive dataset (SMAP). This dataset contains telemetry
data and one-hot encoded vectors similar to the MSL dataset. It has also been
released by NASA [13]. However, in this case, the dataset is formed by 53
signals received from the Soil Moisture Active Passive satellite. The
annotated training and testing data are provided. Nevertheless, as for the MSL
dataset, similar inconsistencies regarding the experimental protocol can be
remarked. For that reason, we propose using the protocol of [12], where each
signal is considered to be a separate and independent multivariate sub-
dataset. This leads to train and test on 53 different sub-datasets and
reporting the obtained average performance.
UCR time series anomaly archive (UCR). It has been recently proposed by Renjie
and Keogh [21]. In [21], the authors claim that most of the existing anomaly
detection datasets are trivial. By trivial, they mean that an anomaly can be
detected with a single line of MATLAB code. They also criticize the lack of
realism and annotation precision in current datasets. As an alternative, they
introduce the UCR dataset, which gathers 250 realistic sub-datasets. This
dataset is collected from various fields, including medicine, sports, and
robotics. The training and test sets are well-defined222More information about
the UCR dataset can be found here.
Automated Time-series Outlier Detection System (TODS). It is a collection of 5
synthetically generated multivariate datasets. The dataset was generated using
the source code from [41], therefore producing different types of anomalies
following the taxonomy of [41]. The dimension of the generated time-series is
10. Training datasets contain only normal values, while testing datasets
incorporate 5 different types of anomalies. The annotation of the outlier
types is provided, therefore allowing a per-type analysis.

| SWaT | MSL | SMAP | UCR | TODS
---|---|---|---|---|---
Number of datasets | 1 | 27 | 55 | 250 | 5
Variables | 52 | 55 | 25 | 1 | 10
Percentage of anomalies | 12.14 | 10.48 | 12.82 | 0.38 | 5
Training data points | 495000 | 58317 | 138004 | 5302449 | 10000
Testing data points | 449919 | 73729 | 435826 | 12919799 | 10000
Type of data | Real | Real | Real | Real | Synthetic
Type of anomalies | Artificially forced | Natural | Natural | Natural/Synthetic | Synthetic
TABLE II: Summary of the 5 datasets used in the experiments. The percentage of
anomalies in the testing set is reported.

###  4.2 Evaluation criteria

In this section, we present the used evaluation criteria. Precision, recall
and F1-scores. The most common metrics used to evaluate the performance of
time-series anomaly detection algorithms are the precision computed as
follows,

$\mbox{Precision}=\frac{\mbox{True positives}}{\mbox{True
positives}+\mbox{False positives}},$ (14)

the recall is calculated as below,

$\mbox{Recall}=\frac{\mbox{True positives}}{\mbox{True Positives}+\mbox{False
Negatives}}$ (15)

and the F1-score corresponding to,

$\mbox{F1-score}=\frac{2\cdot\mbox{Precision}\cdot\mbox{Recall}}{\mbox{Precision}+\mbox{Recall}}.$
(16)

Revisited precision, recall and F1-scores for time-series. In addition to
conventional performance metrics, more recent and elaborate performance
metrics tailored to time-series introduced in [59] are considered. These
metrics extend classical precision, recall, and F1-score, from point-based to
range-based anomaly detection. Fig. 3 highlights the distinction between
point-based and range-based anomalies. Contrary to the case of point-based
approaches, a prediction in a time-series can be both a true positive (TP) and
a false negative (FN) due to partial overlap with the ground-truth as shown in
Fig. 3 b. Therefore, as discussed in [59], a more informative time-series
evaluation process should (1) quantify the size of the partial overlap; (2)
identify the overlap position, and; (3) take into account its cardinality,
i.e., with how many anomalous ground-truth sub-sequences it overlaps. More
specifically, given a set of real anomaly sequences $R=\\{R_{1},...,R_{Nr}\\}$
and a set of predicted anomaly sequences $P=\\{P_{1},...,P_{N_{p}}\\}$ the
recall is expressed with respect to the number of real anomalies $N_{r}$ in a
dataset [59]. It seeks to reward a detector when it predicts a TP and
penalizes it when the prediction is an FN as follows,

${\text{Recall}_{T}(R,P)}=\frac{1}{N_{r}}\sum_{i=1}^{N_{r}}{{\text{Recall}_{T}(R_{i},P)}},$
(17)

and,

$\text{Recall}_{T}(R_{i},P)=\alpha\cdot\mathbbm{1}_{\sum_{j=1}^{N_{p}}|R_{i}\cap
P_{j}|\geq 1}+\frac{1-\alpha}{\sum_{j=1}^{N_{p}}|R_{i}\cap
P_{j}|}\cdot\mathcal{S}_{c}(R_{i},P),$ (18)

where $0\leq\alpha\leq 1$ is a scaling factor that rewards the detector when
it detects the existence of the anomaly $R_{i}$ and $\mathbbm{1}$ is an
indicator function. Finally, $\mathcal{S}_{c}(R_{i},P)$ which quantifies the
overlap size is computed based on the cumulative overlap size $\omega$ as
follows,

$\mathcal{S}_{c}(R_{i},P)=\sum_{j=1}^{N_{p}}\omega(R_{i},R_{i}\cap
P_{j},\delta),$ (19)

where $\delta$ returns a score depending on the overlap location between
$R_{i}$ and a prediction $P_{j}$ (flat bias, front bias, middle bias, and back
bias). Further details could be found in the original manuscript [59]. The
precision is similarly defined. It seeks to assess the quality of the
predictions by rewarding a detector in the presence of a TP and penalizing it
when facing an FP. It is computed as follows,

$\text{Precision}_{T}=\frac{1}{N_{p}}\sum_{i=1}^{N_{p}}{\text{Precision}_{T}(R,P_{i})},$
(20)

and,

$\text{Precision}_{T}(R,P_{i})=\frac{1}{\sum_{j=1}^{N_{r}}|R_{j}\cap
P_{i}|}\cdot\mathcal{S}(R,P_{i}),$ (21)

where $\mathcal{S}_{c}(R,P_{i})$ quantifies the cumulative overlap between the
considered prediction $P_{i}$ and all the ground-truths in $R$ as explained in
Equation (19). It is expressed as,

$\mathcal{S}_{c}(R,P_{i})=\sum_{j=1}^{N_{r}}\omega(P_{i},P_{i}\cap
R_{j},\delta).$ (22)

Finally, the F1-score is redefined as follows,

$\text{F1-score}_{T}=\frac{2\cdot\text{Precision}_{T}\cdot\text{Recall}_{T}}{\text{Precision}_{T}+\text{Recall}_{T}}$
(23)

Figure 3: The evaluation process of (a) point-based anomalies versus (b)
range-based anomalies. Range-based anomalies are characterized by partial
overlap(s) with the ground-truth. A more accurate evaluation for time-series
should quantify the overlap in terms of size, position, and cardinality.

Model stability. We define model stability as the ability of a machine/deep
learning algorithm to reproduce similar results when retrained under the same
conditions. While OC-SVM ensures stability because of its deterministic
nature, most of the considered methods rely on a random parameter
initialization which may impact the final performance of the model. Ideally,
the model should achieve the same results regardless of this initialization.
To assess the stability, each experiment is carried out five times. Then, the
mean and standard deviation of those five runs are reported. A lower standard
deviation reflects higher stability. To the best of our knowledge, we are
among the first to analyze this aspect experimentally in the context of
anomaly detection in time-series.
Generalization to different types of anomalies. We propose reporting the
performance according to the anomaly type encountered. This analysis can help
identify the most suitable algorithm for a given application. To that aim, the
TODS benchmark, which encloses the annotation of 5 different types of outliers
depicted in Section 4.1 is used. Although the definition of different anomaly
types has been reported in several references, very few works have carried out
an experimental study with respect to the anomaly type. A rare example we can
mention is the work of [41]. Nevertheless, it can be noted that in this paper,
recent DL-based state-of-the-art approaches such as GDN [5], USAD [24], and
MTAD-GAT [14] are not evaluated. For each anomaly type, the percentage of
well-detected anomalies is reported.
Model size. In a real-world context, deploying algorithms on specific hardware
with a limited memory capacity can be challenging. Therefore, being aware of
the model size, which directly impacts the memory consumption, is a crucial
component often neglected. For that purpose, we report the number of
parameters and the size in MegaBytes (MB) of the trained deep learning models
considered for this evaluation.

###  4.3 Post-processing and Pre-processing

Data Normalization. Normalizing the data is a common practice in machine
learning, particularly in anomaly detection. Hence, for the sake of fairness,
a data normalization pre-processing was applied in all our experiments. More
specifically, the data are normalized using the maximum and minimum values in
the training data as in [14].
Point Adjustment. Point adjustment initially introduced in [2] is a protocol
that adjusts the predictions before computing performance metrics. It acts as
follows: if at least one point is classified as an anomaly in an outlier
segment, all the predictions in that segment are set to anomalous. The idea
behind this protocol is that an algorithm triggering an alert for any point in
a contiguous anomaly segment might be sufficient for a timely reaction. Fig. 4
illustrates the point adjustment protocol by showing the ground-truth, the
original predictions, and the predictions after point adjustment of a given
time-series. After applying the point adjustment protocol, the F1-Score goes
from $0.32$ to $0.85$. This significant gap has therefore raised some concerns
in the literature regarding the use of point adjustment. For example, Kim et
al. [1] claim that by using this protocol, a randomly generated anomaly score
might outperform several recently proposed time-series anomaly detection
algorithms. In this paper, we report the performance of existing methods with
and without point adjustment.

Figure 4: Application of the Point Adjustment (PA) on a given time-series: the
Ground-Truth (GT), the original prediction (Pred) and the prediction after
Point Adjustment (PA) are reported. In this example, the performance of the
algorithm without and with point adjustment is respectively: Precision
$=0.75$, Recall $=0.2$, F1-Score $=0.32$, and Precision $=0.92$, Recall
$=0.79$, F1-Score $=0.85$. Best viewed in colors.

###  4.4 Evaluated methods

We consider in total 8 anomaly detection methods. Table III summarizes the
characteristics of each evaluated method.

Method | Type of paradigm | Nature
---|---|---
OC-SVM [4] | Clustering | Shallow
iForest [49] | Clustering | Shallow
ARIMA [36] | Forecasting | Shallow
DA-GMM [35] | Density-estimation | Deep
THOC [25] | Clustering | Deep
USAD [24] | Reconstruction | Deep
GDN [5] | Forecasting | Deep
MTAD-GAT [14] | Hybrid (Forecasting & Reconstruction) | Deep

TABLE III: Paradigm type and nature of evaluated methods

SWaT MSL SMAP UCR TODS Avg. per method Point adjustment Without With Without
With Without With Without With Without With Without With Precision
0.28$\pm$0.015 0.32$\pm$0.019 0.15$\pm$0.010 0.22$\pm$0.009 0.18$\pm$0.008
0.26$\pm$0.013 0.01$\pm$0.000 0.02$\pm$0.000 0.05$\pm$0.002 0.07$\pm$0.001
Recall 0.74$\pm$0.014 0.89$\pm$0.029 0.57$\pm$0.051 0.99$\pm$0.015
0.49$\pm$0.008 0.95$\pm$0.014 0.48$\pm$0.002 0.95$\pm$0.003 0.54$\pm$0.017
0.71$\pm$0.016 USAD [24] F1-Score 0.41$\pm$0.019 0.47$\pm$0.024 0.21$\pm$0.015
0.33$\pm$0.013 0.21$\pm$0.006 0.34$\pm$0.015 0.02$\pm$0.000 0.04$\pm$0.000
0.10$\pm$0.003 0.13$\pm$0.003 0.19$\pm$0.131 0.26$\pm$0.155 Precision
0.34$\pm$0.034 0.40$\pm$0.047 0.31$\pm$0.012 0.39$\pm$0.016 0.25$\pm$0.002
0.36$\pm$0.011 0.12$\pm$0.003 0.31$\pm$0.009 0.07$\pm$0.007 0.10$\pm$0.018
Recall 0.72$\pm$0.037 0.72$\pm$0.037 0.64$\pm$0.017 1.00$\pm$0.002
0.55$\pm$0.042 1.00$\pm$0.001 0.42$\pm$0.001 0.99$\pm$0.002 0.59$\pm$0.160
0.75$\pm$0.124 GDN [5] F1-Score 0.46$\pm$0.030 0.57$\pm$0.048 0.35$\pm$0.006
0.50$\pm$0.015 0.33$\pm$0.013 0.46$\pm$0.013 0.12$\pm$0.003 0.39$\pm$0.011
0.11$\pm$0.003 0.16$\pm$0.018 0.27$\pm$0.141 0.42$\pm$0.140 Precision
0.62$\pm$0.160 0.77$\pm$0.083 0.22$\pm$0.009 0.31$\pm$0.013 0.16$\pm$0.007
0.26$\pm$0.007 0.01$\pm$0.005 0.06$\pm$0.017 0.05$\pm$0.008 0.09$\pm$0.014
Recall 0.46$\pm$0.130 0.86$\pm$0.020 0.46$\pm$0.017 0.87$\pm$0.015
0.27$\pm$0.009 0.84$\pm$0.017 0.00$\pm$0.009 0.06$\pm$0.020 0.19$\pm$0.033
0.35$\pm$0.059 THOC [25] F1-Score 0.52$\pm$0.140 0.81$\pm$ 0.054
0.25$\pm$0.009 0.41$\pm$0.015 0.12$\pm$0.008 0.34$\pm$0.008 0.00$\pm$0.001
0.06$\pm$0.019 0.08$\pm$0.14 0.14$\pm$0.023 0.19$\pm$0.181 0.35$\pm$0.262
Precision 0.85$\pm$0.037 0.86$\pm$0.036 0.57$\pm$0.037 0.60$\pm$0.040
0.58$\pm$0.034 0.59$\pm$0.034 0.10$\pm$0.004 0.17$\pm$0.003 0.16$\pm$0.078
0.16$\pm$0.078 Recall 0.90$\pm$0.033 0.96$\pm$0.032 0.79$\pm$0.030
0.86$\pm$0.034 0.87$\pm$0.029 0.91$\pm$0.030 0.28$\pm$0.008 0.57$\pm$0.017
0.01$\pm$0.017 0.01$\pm$0.017 MTAD-GAT [14] F1-Score 0.87$\pm$0.014
0.90$\pm$0.014 0.60$\pm$0.032 0.64$\pm$0.035 0.65$\pm$0.033 0.67$\pm$0.034
0.13$\pm$0.005 0.25$\pm$0.005 0.02$\pm$0.027 0.02$\pm$0.027 0.45$\pm$0.324
0.50$\pm$0.317 Precision 0.43$\pm$0.000 0.49$\pm$0.000 0.12$\pm$0.006*
0.20$\pm$0.002* 0.11$\pm$0.008* 0.16$\pm$0.003 0.01$\pm$0.000 0.03$\pm$0.000
0.12$\pm$0.000 0.15$\pm$0.000 Recall 0.71$\pm$0.000 0.90$\pm$0.002
0.19$\pm$0.004* 0.44$\pm$0.000* 0.17$\pm$0.010 0.41$\pm$0.000* 0.20$\pm$0.002
0.78$\pm$0.002 0.49$\pm$0.000 0.63$\pm$0.000 DAGMM [35] F1-Score
0.54$\pm$0.000 0.64$\pm$0.001 0.12$\pm$0.004* 0.25$\pm$0.002* 0.10$\pm$0.006*
0.19$\pm$0.002* 0.01$\pm$0.000 0.06$\pm$0.000 0.19$\pm$0.000 0.24$\pm$0.000
0.19$\pm$0.183 0.28$\pm$0.194 Precision 0.24$\pm$0.000 0.26$\pm$0.000
0.15$\pm$0.000 0.25$\pm$0.000 0.12$\pm$0.000 0.15$\pm$0.000 0.01$\pm$0.000
0.02$\pm$0.000 0.05$\pm$0.000 0.05$\pm$0.000 Recall 0.85$\pm$0.000
0.95$\pm$0.000 0.66$\pm$0.000 0.95$\pm$0.000 0.66$\pm$0.000 0.85$\pm$0.000
0.73$\pm$0.000 0.85$\pm$0.000 0.85$\pm$0.000 0.85$\pm$0.000 OCSVM [4] F1-Score
0.37$\pm$0.000 0.41$\pm$0.000 0.24$\pm$0.000 0.40$\pm$0.000 0.20$\pm$0.000
0.26$\pm$0.000 0.02$\pm$0.000 0.04$\pm$0.000 0.09$\pm$0.000 0.09$\pm$0.000
0.18$\pm$0.121 0.24$\pm$0.153 Precision 0.23$\pm$0.097 0.26$\pm$0.118
0.18$\pm$0.041 0.47$\pm$0.037 0.10$\pm$0.008 0.34$\pm$0.009 0.05$\pm$0.006
0.16$\pm$0.006 0.05$\pm$0.012 0.17$\pm$0.057 Recall 0.83$\pm$0.096
0.97$\pm$0.003 0.16$\pm$0.053 0.66$\pm$0.058 0.04$\pm$0.005 0.40$\pm$0.009
0.12$\pm$0.013 0.45$\pm$0.008 0.04$\pm$0.009 0.17$\pm$0.018 iForest [49]
F1-Score 0.36$\pm$0.099 0.40$\pm$0.134 0.17$\pm$0.034 0.55$\pm$0.039
0.08$\pm$0.004 0.36$\pm$0.009 0.07$\pm$0.003 0.24$\pm$0.005 0.04$\pm$0.007
0.17$\pm$0.023 0.14$\pm$0.116 0.34$\pm$0.131 Precision 0.13$\pm$0.000
0.13$\pm$0.000 0.28$\pm$0.000 0.31$\pm$0.000 0.17$\pm$0.000 0.18$\pm$0.000
0.01$\pm$0.000 0.01$\pm$0.000 0.05$\pm$0.000 0.06$\pm$0.000 Recall
0.99$\pm$0.000 1.00$\pm$0.000 0.83$\pm$0.000 1.00$\pm$0.000 0.82$\pm$0.000
0.96$\pm$0.000 0.85$\pm$0.000 0.97$\pm$0.000 0.69$\pm$0.000 0.87$\pm$0.000
ARIMA [36] F1-Score 0.23$\pm$0.000 0.23$\pm$0.000 0.28$\pm$0.000
0.39$\pm$0.000 0.19$\pm$0.000 0.24$\pm$0.000 0.02$\pm$0.000 0.02$\pm$0.000
0.09$\pm$0.000 0.11$\pm$0.000 0.16$\pm$0.950 0.20$\pm$0.126 Avg. F1-score per
dataset 0.47$\pm$0.177 0.55$\pm$0.209 0.28$\pm$0.138 0.43$\pm$0.117
0.23$\pm$0.173 0.36$\pm$0.141 0.05$\pm$0.048 0.14$\pm$0.128 0.09$\pm$0.047
0.13$\pm$0.060

TABLE IV: Results in terms of traditional performance metrics of evaluated
state-of-the-art methods (precision, recall, F1-score) on the 5 considered
datasets. The experiments have been performed 5 times for each algorithm and
dataset. The provided values correspond to the mean and standard deviation.
The bold and underlined results correspond to the first and second-best
F1-Score obtained for a particular dataset.
*DAGMM did not converge on sub-datasets from MSL and SMAP. Hence, the reported results for this particular method are only considering the sub-datasets on which DAGMM converged.

Three shallow standard methods are evaluated, namely, OC-SVM [4], iForest [49]
and ARIMA [36]. In addition, five recent DL-based methods have been
considered: DA-GMM [35], THOC [25] USAD [24], GDN [5] and MTAD-GAT [14]. The
latter has been selected according to the following criteria: (1) Relevance of
the topic: all the chosen anomaly detection algorithms are unsupervised and
have been specifically designed for detecting anomalies in time-series. (2)
Publication date: all the DL-based algorithms are recent. In particular, they
have been introduced between 2018 and early 2022. (3) Impact: the chosen
algorithms have been published in top-tier conferences and are highly cited
papers from the field. (4) Code availability: the official codes of the
selected algorithms are publicly available. (5) Diversity: methods from
different paradigms have been considered. The only paradigm that was ignored
is the distance-based since we were not able to find a deep learning approach
falling in this category.

##  5 Results

###  5.1 Performance using standard metrics

Best performing approach. Table IV reports the performance of the evaluated
methods using standard metrics (precision, recall, and F1-score) on the five
considered datasets. For a more intuitive visualization of the results, Fig. 5
shows the F1-score. In general, MTAD-GAT [14] is the best-performing approach,
surpassing other methods on four datasets, namely, SWaT, MSL, SMAP, and UCR.
This can be explained by the fact that this approach is hybrid as it is based
on forecasting and reconstruction losses. Indeed, this allows the simultaneous
detection of local and global anomalies. Nevertheless, it can be remarked that
the results obtained on TODS contradict this statement. Indeed, MTAD-GAT
registers inferior performance on this benchmark as compared to other methods,
including DL and conventional methods. Two hypotheses might justify this drop:
(1) the TODS dataset encloses complex anomalies that are moderately local and
are hardly captured by a simple forecasting and/or reconstruction approach,
favoring probabilistic modeling as in DAGMM [35]. However, the higher
performance obtained for GDN [5] and USAD [24] partly disprove this
assumption; and (2) the synthetic data in TODS are not realistic, making them
hardly predictable. Another observation that can be made is that GDN [5]
presents the second-best performance on three datasets, namely MSL, SMAP, and
UCR. This confirms the relevance of using graph representations for modeling
time-series. Surprisingly, graph-based approaches (GDN and MTAD-GAT) remain
relatively effective on a univariate dataset (UCR), although modeling the
connectivity between variables is unnecessary.
DL vs conventional methods. As reported in Table IV and Fig. 5, DL methods,
specifically MTAD-GAT and GDN, generally outperform conventional methods. For
example, the superiority of DL approaches is extremely noticeable when
comparing GDN and ARIMA, which are both forecasting techniques. This increase
in performance can be explained by the fact that ARIMA struggles to model the
dependencies between variables. However, the gap in performance between DL and
traditional methods is less visible in some cases, contradicting the
assumption of [3], which states that DL methods are more effective in the
presence of high-dimensional time-series. For instance, OC-SVM shows
comparable performance with several DL-based anomaly methods such as THOC and
USAD on high-dimensional datasets. More precisely, OC-SVM achieves an F1-score
of 0.24 and 0.2 on MSL and MSAP against 0.21 and 0.21 for USAD and 0.25 and
0.2 for THOC, respectively. Another observation that can be made is that
conventional approaches, except iForest, seem to be suitable for applications
where recall is more important than precision. An example of such an
application could be the detection of debris among other objects in space
[60]. On the contrary, DL approaches are overall more precise.

(a) SWAT

(b) MSL

(c) SMAP

(d) UCR

(e) TODS

Figure 5: Mean F1-Score on the five datasets. The non-hatched and hatched bars
correspond to the mean F1-Score with and without Point Adjustment (PA),
respectively. The vertical black line represents the standard deviation over
five runs.

Impact of point adjustment. From the results of Table IV, it can be noted that
the Point Adjustment (PA) process significantly boosts the performance. In
particular, the highest performance gain can be observed for iForest on the
MSL dataset, where the F1-score increases from 17% to 55%. This can be
explained by the fact that PA adjusts the predictions before computing the
metrics. The adjustment is made in a way that rewards a detector when
detecting at least one instance of an anomalous segment. The intuition behind
that is that finding one anomaly in a segment is sufficient for a timely
reaction. Such an intuition closely impacts the recall since it increases the
number of False Positives (FP). In addition, before applying PA, all DL
approaches seem to be in general, more effective than conventional approaches.
However, after PA, this is no longer the case. For example, iForest achieves
comparable performance with THOC. Overall, PA seems to bias the analysis as it
treats range-based data as punctual, neglecting the overlap size and the
location of anomalies. Consequently, this blurs the applicability of detectors
in real-life setups.

Benchmark complexity. All tested methods fail to detect effectively anomalies
in UCR, although it is a univariate dataset. This might be due to its low
ratio of anomalies. As discussed in Section 4.1, UCR is among the first
benchmark to mimic a more realistic configuration, highlighting the difficulty
of detecting rare anomalies. The rate of anomalies might have a significant
role in defining the complexity of a given dataset. For natural anomalies, two
observations could be made. First, the average performance on MSL and SMAP is
comparable despite having a significantly different number of variables.
Second, natural but induced/forced anomalies seem easy to detect, given that
all methods perform well on the SWaT dataset. Unfortunately, such a scenario
is unrealistic in most real-world scenarios as the anomalies are generally
infrequent.

Figure 6: The mean performance per method on all datasets using the range-
based metrics [59], with different location biases.

###  5.2 Performance using revisited metrics for time-series

Conventional metrics vs revisited metrics. Table VI shows the results of
evaluation using the revisited F1-score for time-series calculated using
RecallT and PrecisionT proposed in [59].

It can be remarked that there exists a significant gap in performance between
the results based on conventional and revisited metrics. One main reason is
that the revisited metrics consider the overlap size between the predicted
sequences and the ground-truth. In contrast, the traditional metrics do not
take into account the sequential aspect nor quantify the overlap between the
predictions and the ground-truth. Moreover, it can be noted that the results
of the revisited metrics are not in full accordance with the conventional
ones. On the one hand, GDN and USAD achieve more competitive results as
compared to other approaches. On the other hand, the assumption about the
superiority of the hybrid method no longer holds. Overall, both classical and
DL forecasting-based techniques give the highest performance. This perspective
suggests that the majority of anomalies in benchmarked datasets are local.
Finally, DAGMM seems to be among the least effective methods, suggesting its
inability to model the distribution of anomalies. This can be explained by the
fact that anomalies do not necessarily follow a multimodal Gaussian
distribution. Some observations can be made regarding the difficulty of each
dataset. First, MSL seems slightly less challenging than SWAT. Second, in line
with the results based on conventional metrics, the obtained performance
suggests that natural anomalies are more straightforward to detect than
synthetic ones. Two reasons could potentially explain that: (1) the datasets
with natural anomalies have a high percentage of anomalies, and (2) synthetic
datasets do not reliably reflect reality and do not include a sufficient
number of anomalies.

Location bias. Herein, we analyze the results with and without location bias.
Table VI, Table VII, Table VIII, and Table IX show the results using flat,
front, middle and back bias, respectively. Fig. 6 depicts the overall
performance for each method under different bias settings. Although the idea
of location bias seems theoretically interesting and flexible for different
domain-specific applications, it does not practically bring more information,
as the results do not change importantly. However, it can be noted that the
middle and back bias allowed the detection of more anomalies compared to flat
and front. This suggests that most detectors remain immature for applications
necessitating an early anomaly such as real-time intrusion detection [62] or
cyberattack attempts via network activity [63].

DL vs conventional methods. Although the top-three best-performing methods are
DL models (according to the conventional metrics), it can be seen that
classical approaches such as OC-SVM can achieve comparable performance with
its counterpart clustering DL approach, namely THOC. This suggests that
conventional methods are not necessarily obsolete and that, depending on the
application, they can be considered for anomaly detection.

Univariate vs multivariate. The results of Table IV and Table VI seem to be in
accordance. Indeed, all methods seem to have poor performance on UCR, which is
univariate, while on other multivariate datasets such as MSL, the performance
is relatively higher.

###  5.3 Model stability

Besides reporting the precision, recall, and F1-score, it is interesting to
observe the behavior of detectors when trained with different initializations.
Table IV and Fig. 5 report the performance average and standard deviation for
every approach after five runs. Undoubtedly the most stable methods are the
deterministic ones which are ARIMA and OC-SVM. Among DL approaches, DAGMM
seems to be the most stable. This could be explained by the fact that it is a
density-based approach that relies on estimating the density of normal data.
In contrast, THOC and iForest achieve less stable results, especially on SWAT.

Figure 7: Relation between the number of the parameters of the model and the
number of features in the considered dataset. Figure 8: The ratio of true
anomalies detected for each tested method when varying the anomaly types. All
methods succeeded in partially detecting each anomaly type, except MTAD-GAT
which was unable to detect any collective trend anomaly.

Model SWAT MSL SMAP UCR TODS Parameters Model size (Mb) Parameters Model size
(Mb) Parameters Model size (Mb) Parameters Model size (Mb) Parameters Model
size (Mb) USAD [24] 1.256.871 4,79 1.414.755 5,40 441.225 1,68 12.321 0,05
136.710 0,54 GDN [5] 4.225 0,02 4.481 0,02 2.561 0,01 1.025 0,01 1.601 0,01
THOC [25] 104.768 0,41 105.792 0,42 98.112 0,39 91.968 0,36 94.272 0,37 MTAD-
GAT [14] 373.637 1,62 384.145 1,66 314.695 1,39 274.687 1,05 288.070 1,05
DAGMM [35] 266.930 1,50 270.542 1,50 243.452 1,40 221.780 1,30 288.070 1,40

TABLE V: Number of parameters and model size in Mega Bytes (MB) of the trained
models on the different datasets

###  5.4 Model size and memory consumption

Table V depicts the number of parameters and the model size in Mega Bytes (MB)
of the tested DL architectures. GDN seems to have the lowest number of
parameters when tested on all the datasets. In fact, the number of parameters
in USAD is around 300 times higher than GDN. This is explained by the fact
that the architecture of USAD is complex and is composed of two adversarially
trained auto-encoders. In Fig. 7, it can also be seen that contrary to other
models, which vary almost linearly, the number of parameters increases at a
considerably higher rate. Additionally, despite the significant difference in
parameter number, GDN still achieves better results than USAD. This highlights
the relevance of using graph representations not only for modeling time-series
but also for building less complex model architectures.

###  5.5 Generalization to different types of anomalies

Fig. 8 shows the percentage of detected anomalies per type for all the tested
methods. In general, it can be noted that for the majority of tested
techniques, collective trend anomalies are probably the most challenging to
detect. ARIMA and GDN, which are predictive approaches, show the best
generalization capacity to different types of anomalies. OC-SVM easily detects
global point and collective shapelet anomalies but still presents decent
results for other anomaly types. The results obtained for USAD suggest that it
is more robust to collective outliers (e.g., collective shapelet and
seasonality), which can be explained by the fact that it is a reconstruction
approach that can essentially capture global inconsistencies. DAGMM
effectively detects collective seasonal anomalies but shows less impressive
results for collective shape outliers. Again, this might return to the
probabilistic nature of DAGMM, which is coupled with a sliding window.
Finally, MTAD-GAT fails in detecting collective anomalies, despite being
hybrid.

|  | USAD [24] | GDN [5] | THOC [25] | MTAD-GAT [14] | DAGMM [35] | OC-SVM [4] | iForest [49] | ARIMA [36] |  | Avg. F1-score per
---
dataset
SWaT | $\text{Precision}_{T}$ | 0.0583 $\pm$ 0.0210 | 0.0703 $\pm$ 0.0173 | 0.2905 $\pm$ 0.1113 | 0.0331 $\pm$ 0.0082 | 0.1032 $\pm$ 0.0022 | 0.0312 $\pm$ 0.0000 | 0.0038 $\pm$ 0.0014 | 0.0395 $\pm$ 0.0000 | 0.0726 $\pm$ 0.0376
$\text{Recall}_{T}$ | 0.3284 $\pm$ 0.0396 | 0.2505 $\pm$ 0.0290 | 0.0204 $\pm$ 0.0108 | 0.3946 $\pm$ 0.0385 | 0.1889 $\pm$ 0.0022 | 0.8566 $\pm$ 0.0000 | 0.9021 $\pm$ 0.0014 | 0.8085 $\pm$ 0.0000
F1-Score | 0.0976 $\pm$ 0.0296 | 0.1081 $\pm$ 0.0188 | 0.0381 $\pm$ 0.0196 | 0.0607 $\pm$ 0.0133 | 0.1334 $\pm$ 0.0026 | 0.0602 $\pm$ 0.0000 | 0.0077 $\pm$ 0.0029 | 0.0753 $\pm$ 0.0000
MSL | $\text{Precision}_{T}$ | 0.1210 $\pm$ 0.0079 | 0.2494 $\pm$ 0.0174 | 0.1592 $\pm$ 0.0143 | 0.0768 $\pm$ 0.0045 | 0.1109 $\pm$ 0.0025 | 0.0528 $\pm$ 0.0000 | 0.0389 $\pm$ 0.0123 | 0.0843 $\pm$ 0.0000 | 0.0734 $\pm$ 0.0390
$\text{Recall}_{T}$ | 0.2014 $\pm$ 0.0263 | 0.2293 $\pm$ 0.0217 | 0.2234 $\pm$ 0.0213 | 0.1846 $\pm$ 0.0045 | 0.0379 $\pm$ 0.0025 | 0.8316 $\pm$ 0.0000 | 0.5846 $\pm$ 0.0123 | 0.4106 $\pm$ 0.0000
F1-Score | 0.0666 $\pm$ 0.0030 | 0.1535 $\pm$ 0.0133 | 0.0770 $\pm$ 0.0044 | 0.0319 $\pm$ 0.0023 | 0.0161 $\pm$ 0.0004 | 0.0856 $\pm$ 0.0000 | 0.0625 $\pm$ 0.0138 | 0.0937 $\pm$ 0.0000
SMAP | $\text{Precision}_{T}$ | 0.1204 $\pm$ 0.0095 | 0.1500 $\pm$ 0.0032 | 0.0814 $\pm$ 0.0047 | 0.1401 $\pm$ 0.0000 | 0.0709 $\pm$ 0.0094 | 0.0656 $\pm$ 0.0000 | 0.0178 $\pm$ 0.0064 | 0.1401 $\pm$ 0.0000 | 0.0683 $\pm$ 0.0407
$\text{Recall}_{T}$ | 0.2501 $\pm$ 0.0070 | 0.2023 $\pm$ 0.0160 | 0.0807 $\pm$ 0.0081 | 0.5272 $\pm$ 0.0000 | 0.0379 $\pm$ 0.0094 | 0.8365 $\pm$ 0.0000 | 0.3728 $\pm$ 0.0064 | 0.5272 $\pm$ 0.0000
F1-Score | 0.0634 $\pm$ 0.0032 | 0.0793 $\pm$ 0.0033 | 0.0259 $\pm$ 0.0018 | 0.1179 $\pm$ 0.0000 | 0.0091 $\pm$ 0.0009 | 0.1040 $\pm$ 0.0000 | 0.0285 $\pm$ 0.0064 | 0.1179 $\pm$ 0.0000
UCR | $\text{Precision}_{T}$ | 0.0106 $\pm$ 0.0002 | 0.0749 $\pm$ 0.0032 | 0.0106 $\pm$ 0.0040 | 0.0113 $\pm$ 0.0017 | 0.0709 $\pm$ 0.0094 | 0.0083 $\pm$ 0.0000 | 0.0255 $\pm$ 0.0034 | 0.0074 $\pm$ 0.0000 | 0.0176 $\pm$ 0.0147
$\text{Recall}_{T}$ | 0.2320 $\pm$ 0.0023 | 0.2501 $\pm$ 0.0118 | 0.0011 $\pm$ 0.0040 | 0.0189 $\pm$ 0.0017 | 0.0379 $\pm$ 0.0094 | 0.8356 $\pm$ 0.0000 | 0.4089 $\pm$ 0.0034 | 0.2907 $\pm$ 0.0000
F1-Score | 0.0168 $\pm$ 0.0003 | 0.0473 $\pm$ 0.0013 | 0.0016 $\pm$ 0.0005 | 0.0054 $\pm$ 0.0005 | 0.0091 $\pm$ 0.0009 | 0.0134 $\pm$ 0.0000 | 0.0352 $\pm$ 0.0031 | 0.0119 $\pm$ 0.0000
TODS | $\text{Precision}_{T}$ | 0.0420 $\pm$ 0.0039 | 0.0537 $\pm$ 0.0130 | 0.0617 $\pm$ 0.0047 | 0.0806 $\pm$ 0.0970 | 0.0709 $\pm$ 0.0094 | 0.0197 $\pm$ 0.0000 | 0.0975 $\pm$ 0.0862 | 0.0685 $\pm$ 0.0000 | 0.0583 $\pm$ 0.0392
$\text{Recall}_{T}$ | 0.4796 $\pm$ 0.0015 | 0.5546 $\pm$ 0.1713 | 0.1766 $\pm$ 0.0047 | 0.0019 $\pm$ 0.0970 | 0.0379 $\pm$ 0.0094 | 0.8054 $\pm$ 0.0000 | 0.0645 $\pm$ 0.0862 | 0.6395 $\pm$ 0.0000
F1-Score | 0.0767 $\pm$ 0.0065 | 0.0843 $\pm$ 0.0147 | 0.0887 $\pm$ 0.0073 | 0.0027 $\pm$ 0.0035 | 0.0091 $\pm$ 0.0009 | 0.0382 $\pm$ 0.0000 | 0.0435 $\pm$ 0.0041 | 0.1233 $\pm$ 0.0000
Avg. F1-score per method | 0.0642 $\pm$ 0.0266 | 0.0945 $\pm$ 0.0353 | 0.0463 $\pm$ 0.0323 | 0.0437 $\pm$ 0.0426 | 0.0354 $\pm$ 0.0491 | 0.0603 $\pm$ 0.0324 | 0.0355 $\pm$ 0.0180 | 0.0844 $\pm$ 0.0402 |

TABLE VI: The flat-bias performance of the tested methods on the 5 benchmarks
using the metrics proposed by [59]. The average and the standard deviation of
five runs are reported.

|  | USAD [24] | GDN [5] | THOC [5] | MTAD-GAT [14] | DAGMM [35] | OC-SVM [4] | iForest [49] | ARIMA [36] |  | Avg. F1-score per
---
dataset
SWaT | $\text{Precision}_{T}$ | 0.0590 $\pm$ 0.0213 | 0.0705 $\pm$ 0.0173 | 0.2905 $\pm$ 0.1114 | 0.0312 $\pm$ 0.0076 | 0.1046 $\pm$ 0.0023 | 0.0331 $\pm$ 0.0000 | 0.0039 $\pm$ 0.0014 | 0.0404 $\pm$ 0.0000 | 0.0734 $\pm$ 0.0372
$\text{Recall}_{T}$ | 0.3201 $\pm$ 0.0348 | 0.2504 $\pm$ 0.0296 | 0.0221 $\pm$ 0.0113 | 0.4632 $\pm$ 0.0480 | 0.1816 $\pm$ 0.0023 | 0.8571 $\pm$ 0.0000 | 0.9028 $\pm$ 0.0014 | 0.8119 $\pm$ 0.0000
F1-Score | 0.0981 $\pm$ 0.0296 | 0.1084 $\pm$ 0.0189 | 0.0409 $\pm$ 0.0204 | 0.0580 $\pm$ 0.0129 | 0.1327 $\pm$ 0.0026 | 0.0638 $\pm$ 0.0000 | 0.0079 $\pm$ 0.0029 | 0.0771 $\pm$ 0.0000
MSL | $\text{Precision}_{T}$ | 0.1210 $\pm$ 0.0081 | 0.2494 $\pm$ 0.0176 | 0.1631 $\pm$ 0.0150 | 0.0758 $\pm$ 0.0046 | 0.1110 $\pm$ 0.0025 | 0.0527 $\pm$ 0.0000 | 0.0407 $\pm$ 0.0132 | 0.0690 $\pm$ 0.0000 | 0.0705 $\pm$ 0.0384
$\text{Recall}_{T}$ | 0.2114 $\pm$ 0.0262 | 0.2293 $\pm$ 0.0217 | 0.1996 $\pm$ 0.0192 | 0.1871 $\pm$ 0.0046 | 0.0291 $\pm$ 0.0025 | 0.8125 $\pm$ 0.0000 | 0.5542 $\pm$ 0.0132 | 0.4230 $\pm$ 0.0000
F1-Score | 0.0671 $\pm$ 0.0023 | 0.1535 $\pm$ 0.0133 | 0.0806 $\pm$ 0.0047 | 0.0308 $\pm$ 0.0029 | 0.0156 $\pm$ 0.0005 | 0.0813 $\pm$ 0.0000 | 0.0627 $\pm$ 0.0140 | 0.0727 $\pm$ 0.0000
SMAP | $\text{Precision}_{T}$ | 0.1202 $\pm$ 0.0095 | 0.1509 $\pm$ 0.0031 | 0.0817 $\pm$ 0.0048 | 0.1102 $\pm$ 0.0000 | 0.0708 $\pm$ 0.0094 | 0.0363 $\pm$ 0.0000 | 0.0192 $\pm$ 0.0071 | 0.1102 $\pm$ 0.0000 | 0.0544 $\pm$ 0.0277
$\text{Recall}_{T}$ | 0.2634 $\pm$ 0.0079 | 0.1913 $\pm$ 0.0164 | 0.0690 $\pm$ 0.0123 | 0.5410 $\pm$ 0.0000 | 0.0388 $\pm$ 0.0094 | 0.8238 $\pm$ 0.0000 | 0.3529 $\pm$ 0.0071 | 0.5410 $\pm$ 0.0000
F1-Score | 0.0642 $\pm$ 0.0025 | 0.0761 $\pm$ 0.0030 | 0.0227 $\pm$ 0.0013 | 0.0843 $\pm$ 0.0000 | 0.0090 $\pm$ 0.0011 | 0.0648 $\pm$ 0.0000 | 0.0296 $\pm$ 0.0061 | 0.0843 $\pm$ 0.0000
UCR | $\text{Precision}_{T}$ | 0.0106 $\pm$ 0.0002 | 0.0753 $\pm$ 0.0030 | 0.0106 $\pm$ 0.0040 | 0.0113 $\pm$ 0.0017 | 0.0708 $\pm$ 0.0094 | 0.0079 $\pm$ 0.0000 | 0.0270 $\pm$ 0.0035 | 0.0076 $\pm$ 0.0000 | 0.0181 $\pm$ 0.0154
$\text{Recall}_{T}$ | 0.2295 $\pm$ 0.0029 | 0.2510 $\pm$ 0.0109 | 0.0011 $\pm$ 0.0040 | 0.0190 $\pm$ 0.0017 | 0.0388 $\pm$ 0.0094 | 0.8286 $\pm$ 0.0000 | 0.3895 $\pm$ 0.0035 | 0.2911 $\pm$ 0.0000
F1-Score | 0.0167 $\pm$ 0.0002 | 0.0496 $\pm$ 0.0010 | 0.0015 $\pm$ 0.0005 | 0.0051 $\pm$ 0.0003 | 0.0090 $\pm$ 0.0011 | 0.0145 $\pm$ 0.0000 | 0.0362 $\pm$ 0.0032 | 0.0121 $\pm$ 0.0000
TODS | $\text{Precision}_{T}$ | 0.0418 $\pm$ 0.0042 | 0.0532 $\pm$ 0.0122 | 0.0620 $\pm$ 0.0042 | 0.0801 $\pm$ 0.097 | 0.0708 $\pm$ 0.0094 | 0.0247 $\pm$ 0.0000 | 0.0979 $\pm$ 0.0865 | 0.0679 $\pm$ 0.0000 | 0.0592 $\pm$ 0.0385
$\text{Recall}_{T}$ | 0.4795 $\pm$ 0.0115 | 0.5549 $\pm$ 0.1724 | 0.1775 $\pm$ 0.0042 | 0.0019 $\pm$ 0.0970 | 0.0388 $\pm$ 0.0094 | 0.8028 $\pm$ 0.0000 | 0.0656 $\pm$ 0.0865 | 0.6396 $\pm$ 0.0000
F1-Score | 0.0767 $\pm$ 0.0065 | 0.0837 $\pm$ 0.0136 | 0.0886 $\pm$ 0.0075 | 0.0023 $\pm$ 0.0035 | 0.0090 $\pm$ 0.0011 | 0.0472 $\pm$ 0.0000 | 0.0437$\pm$ 0.0035 | 0.1222 $\pm$ 0.0000
Avg. F1-score per method | 0.0646 $\pm$ 0.0267 | 0.0943 $\pm$ 0.0351 | 0.0469 $\pm$ 0.0333 | 0.0361 $\pm$ 0.0314 | 0.0351 $\pm$ 0.0489 | 0.0543 $\pm$ 0.0226 | 0.0360 $\pm$ 0.0179 | 0.0737 $\pm$ 0.0354 |

TABLE VII: The front-bias performance of the tested methods on the 5
benchmarks using the metrics proposed by [59]. The average and the standard
deviation of five runs are reported.

|  | USAD [24] | GDN [5] | THOC [25] | MTAD-GAT [14] | DAGMM [35] | OC-SVM [4] | iForest [49] | ARIMA [36] |  | Avg. F1-score per
---
dataset
SWaT | $\text{Precision}_{T}$ | 0.0592 $\pm$ 0.0214 | 0.0705 $\pm$ 0.0173 | 0.2905 $\pm$ 0.1113 | 0.0344 $\pm$ 0.0085 | 0.1040 $\pm$ 0.0022 | 0.0337 $\pm$ 0.0000 | 0.0040 $\pm$ 0.0015 | 0.0382 $\pm$ 0.0000 | 0.0743 $\pm$ 0.0379
$\text{Recall}_{T}$ | 0.3382 $\pm$ 0.0432 | 0.2523 $\pm$ 0.0301 | 0.0217 $\pm$ 0.0131 | 0.4391 $\pm$ 0.0431 | 0.2018 $\pm$ 0.0022 | 0.8571 $\pm$ 0.0000 | 0.9028 $\pm$ 0.0015 | 0.8057 $\pm$ 0.0000
F1-Score | 0.0993 $\pm$ 0.0303 | 0.1084 $\pm$ 0.0188 | 0.0401 $\pm$ 0.0236 | 0.0634 $\pm$ 0.0140 | 0.1372 $\pm$ 0.0025 | 0.0648 $\pm$ 0.0000 | 0.0080 $\pm$ 0.0030 | 0.0731 $\pm$ 0.0000
MSL | $\text{Precision}_{T}$ | 0.1208 $\pm$ 0.0081 | 0.2518 $\pm$ 0.0180 | 0.1590 $\pm$ 0.0141 | 0.0760 $\pm$ 0.0046 | 0.1109 $\pm$ 0.0025 | 0.0567 $\pm$ 0.0000 | 0.0410 $\pm$ 0.0133 | 0.0831 $\pm$ 0.0000 | 0.0748 $\pm$ 0.0424
$\text{Recall}_{T}$ | 0.2221 $\pm$ 0.0330 | 0.2547 $\pm$ 0.0222 | 0.2379 $\pm$ 0.0248 | 0.1846 $\pm$ 0.0046 | 0.0405 $\pm$ 0.0025 | 0.8491 $\pm$ 0.0000 | 0.6078 $\pm$ 0.0133 | 0.4078 $\pm$ 0.0000
F1-Score | 0.0666 $\pm$ 0.0037 | 0.1652 $\pm$ 0.0176 | 0.0741 $\pm$ 0.0043 | 0.0301 $\pm$ 0.0023 | 0.0152 $\pm$ 0.0004 | 0.0927 $\pm$ 0.0000 | 0.0655 $\pm$ 0.0149 | 0.0891 $\pm$ 0.0000
SMAP | $\text{Precision}_{T}$ | 0.1204 $\pm$ 0.0096 | 0.1506 $\pm$ 0.0032 | 0.0800 $\pm$ 0.0050 | 0.1363 $\pm$ 0.0000 | 0.0708 $\pm$ 0.0094 | 0.0582 $\pm$ 0.0000 | 0.0189 $\pm$ 0.0070 | 0.1363 $\pm$ 0.0000 | 0.0673 $\pm$ 0.0392
$\text{Recall}_{T}$ | 0.2608 $\pm$ 0.0096 | 0.2254 $\pm$ 0.0179 | 0.0813 $\pm$ 0.0047 | 0.5403 $\pm$ 0.0000 | 0.0416 $\pm$ 0.0094 | 0.8484 $\pm$ 0.0000 | 0.3890 $\pm$ 0.0070 | 0.5403 $\pm$ 0.0000
F1-Score | 0.0656 $\pm$ 0.0039 | 0.0847 $\pm$ 0.0025 | 0.0238 $\pm$ 0.0018 | 0.1146 $\pm$ 0.0000 | 0.0089 $\pm$ 0.0010 | 0.0962 $\pm$ 0.0000 | 0.0301 $\pm$ 0.0069 | 0.1146 $\pm$ 0.0000
UCR | $\text{Precision}_{T}$ | 0.0104 $\pm$ 0.0002 | 0.0759 $\pm$ 0.0029 | 0.0106 $\pm$ 0.0040 | 0.0114 $\pm$ 0.0017 | 0.0708 $\pm$ 0.0094 | 0.0090 $\pm$ 0.0000 | 0.0269 $\pm$ 0.0035 | 0.0075 $\pm$ 0.0000 | 0.0176 $\pm$ 0.0141
$\text{Recall}_{T}$ | 0.2375 $\pm$ 0.0024 | 0.2524 $\pm$ 0.0123 | 0.0015 $\pm$ 0.0040 | 0.0201 $\pm$ 0.0017 | 0.0416 $\pm$ 0.0094 | 0.8421 $\pm$ 0.0000 | 0.4258 $\pm$ 0.0035 | 0.2920 $\pm$ 0.0000
F1-Score | 0.0165 $\pm$ 0.0003 | 0.0439 $\pm$ 0.0019 | 0.0019 $\pm$ 0.0008 | 0.0054 $\pm$ 0.0006 | 0.0089 $\pm$ 0.0010 | 0.0151 $\pm$ 0.0000 | 0.0371 $\pm$ 0.0032 | 0.0121 $\pm$ 0.0000
TODS | $\text{Precision}_{T}$ | 0.0426 $\pm$ 0.0040 | 0.0541 $\pm$ 0.0131 | 0.0622 $\pm$ 0.0046 | 0.0802 $\pm$ 0.097 | 0.0708 $\pm$ 0.0094 | 0.0204 $\pm$ 0.0000 | 0.0972 $\pm$ 0.0862 | 0.0682 $\pm$ 0.0000 | 0.0586 $\pm$ 0.0393
$\text{Recall}_{T}$ | 0.4791 $\pm$ 0.0116 | 0.5557 $\pm$ 0.1702 | 0.1772 $\pm$ 0.0046 | 0.0018 $\pm$ 0.0970 | 0.0416 $\pm$ 0.0094 | 0.8065 $\pm$ 0.0000 | 0.0645 $\pm$ 0.0862 | 0.6405 $\pm$ 0.0000
F1-Score | 0.0777 $\pm$ 0.0066 | 0.0847 $\pm$ 0.0144 | 0.0894 $\pm$ 0.0073 | 0.0023 $\pm$ 0.0035 | 0.0089 $\pm$ 0.0010 | 0.0397 $\pm$ 0.0000 | 0.0434 $\pm$ 0.0039 | 0.1227 $\pm$ 0.0000
Avg. F1-score per method | 0.0651 $\pm$ 0.0272 | 0.0976 $\pm$ 0.0397 | 0.0459 $\pm$ 0.0321 | 0.0432 $\pm$ 0.0419 | 0.0358 $\pm$ 0.0507 | 0.0617 $\pm$ 0.0310 | 0.0368 $\pm$ 0.0187 | 0.0823 $\pm$ 0.0393 |

TABLE VIII: The middle-bias performance of the tested methods on the 5
benchmarks using the metrics proposed by [59]. The average and the standard
deviation of five runs are reported.

|  | USAD [24] | GDN [5] | THOC [25] | MTAD-GAT [14] | DAGMM [35] | OC-SVM [4] | iForest [49] | ARIMA [36] |  | Avg. F1-score per
---
dataset
SWaT | $\text{Precision}_{T}$ | 0.0576 $\pm$ 0.0208 | 0.0701 $\pm$ 0.0174 | 0.2905 $\pm$ 0.1113 | 0.0351 $\pm$ 0.0087 | 0.1018 $\pm$ 0.0022 | 0.0292 $\pm$ 0.000 | 0.0037 $\pm$ 0.0014 | 0.0385 $\pm$ 0.0000 | 0.0718 $\pm$ 0.0381
$\text{Recall}_{T}$ | 0.3367 $\pm$ 0.0449 | 0.2505 $\pm$ 0.0286 | 0.0188 $\pm$ 0.0103 | 0.3261 $\pm$ 0.0291 | 0.1961 $\pm$ 0.0022 | 0.8561 $\pm$ 0.0000 | 0.9014 $\pm$ 0.0014 | 0.8050 $\pm$ 0.0000
F1-Score | 0.0969 $\pm$ 0.0296 | 0.1078 $\pm$ 0.0187 | 0.0352 $\pm$ 0.0189 | 0.0628 $\pm$ 0.0135 | 0.1340 $\pm$ 0.0026 | 0.0565 $\pm$ 0.0000 | 0.0075 $\pm$ 0.0029 | 0.0734 $\pm$ 0.0000
MSL | $\text{Precision}_{T}$ | 0.1231 $\pm$ 0.0073 | 0.2494 $\pm$ 0.0172 | 0.1553 $\pm$ 0.0137 | 0.0778 $\pm$ 0.0044 | 0.1107 $\pm$ 0.0025 | 0.0528 $\pm$ 0.0000 | 0.0372 $\pm$ 0.0114 | 0.0996 $\pm$ 0.0000 | 0.0747 $\pm$ 0.0408
$\text{Recall}_{T}$ | 0.1914 $\pm$ 0.0264 | 0.2405 $\pm$ 0.0201 | 0.2471 $\pm$ 0.0245 | 0.1820 $\pm$ 0.0044 | 0.0467 $\pm$ 0.0025 | 0.8507 $\pm$ 0.0000 | 0.6151 $\pm$ 0.0114 | 0.3982 $\pm$ 0.0000
F1-Score | 0.0648 $\pm$ 0.0037 | 0.1558 $\pm$ 0.0130 | 0.0706 $\pm$ 0.0037 | 0.0325 $\pm$ 0.0016 | 0.0164 $\pm$ 0.0164 | 0.0887 $\pm$ 0.0000 | 0.0613 $\pm$ 0.0137 | 0.1076 $\pm$ 0.0000
SMAP | $\text{Precision}_{T}$ | 0.1206 $\pm$ 0.0025 | 0.1491 $\pm$ 0.0033 | 0.0811 $\pm$ 0.0047 | 0.1699 $\pm$ 0.0000 | 0.0711 $\pm$ 0.0094 | 0.0949 $\pm$ 0.0000 | 0.0164 $\pm$ 0.0057 | 0.1699 $\pm$ 0.0000 | 0.0767 $\pm$ 0.0503
$\text{Recall}_{T}$ | 0.2367 $\pm$ 0.0070 | 0.2133 $\pm$ 0.0157 | 0.0923 $\pm$ 0.0042 | 0.5133 $\pm$ 0.0000 | 0.0369 $\pm$ 0.0094 | 0.8491 $\pm$ 0.0000 | 0.3927 $\pm$ 0.0057 | 0.5133 $\pm$ 0.0000
F1-Score | 0.0612 $\pm$ 0.0025 | 0.0807 $\pm$ 0.0033 | 0.0275 $\pm$ 0.0022 | 0.1375 $\pm$ 0.0000 | 0.0092 $\pm$ 0.0008 | 0.1330 $\pm$ 0.0000 | 0.0270 $\pm$ 0.0065 | 0.1375 $\pm$ 0.0000
UCR | $\text{Precision}_{T}$ | 0.0106 $\pm$ 0.0002 | 0.0744 $\pm$ 0.0035 | 0.0106 $\pm$ 0.0040 | 0.0114 $\pm$ 0.0017 | 0.0711 $\pm$ 0.0094 | 0.0087 $\pm$ 0.0000 | 0.0239 $\pm$ 0.0032 | 0.0073 $\pm$ 0.0000 | 0.0164 $\pm$ 0.0129
$\text{Recall}_{T}$ | 0.2345 $\pm$ 0.0020 | 0.2492 $\pm$ 0.0126 | 0.0011 $\pm$ 0.0040 | 0.0188 $\pm$ 0.0017 | 0.0369 $\pm$ 0.0094 | 0.8426 $\pm$ 0.0000 | 0.4283 $\pm$ 0.0032 | 0.2903 $\pm$ 0.0000
F1-Score | 0.0167 $\pm$ 0.0003 | 0.0411 $\pm$ 0.0018 | 0.0017 $\pm$ 0.0006 | 0.0054 $\pm$ 0.0006 | 0.0092 $\pm$ 0.0008 | 0.0121 $\pm$ 0.0000 | 0.0337 $\pm$ 0.0031 | 0.0116 $\pm$ 0.0000
TODS | $\text{Precision}_{T}$ | 0.0421 $\pm$ 0.0039 | 0.0542 $\pm$ 0.0140 | 0.0614 $\pm$ 0.0063 | 0.0812 $\pm$ 0.0970 | 0.0711 $\pm$ 0.0094 | 0.0147 $\pm$ 0.0000 | 0.0970 $\pm$ 0.0859 | 0.0692 $\pm$ 0.0000 | 0.0573 $\pm$ 0.0401
$\text{Recall}_{T}$ | 0.4796 $\pm$ 0.0116 | 0.5543 $\pm$ 0.1702 | 0.1757 $\pm$ 0.0063 | 0.0018 $\pm$ 0.0970 | 0.0369 $\pm$ 0.0094 | 0.8080 $\pm$ 0.0000 | 0.0634 $\pm$ 0.0859 | 0.6393 $\pm$ 0.0000
F1-Score | 0.0769 $\pm$ 0.0065 | 0.0848 $\pm$ 0.0158 | 0.0882 $\pm$ 0.0084 | 0.0029 $\pm$ 0.0037 | 0.0092 $\pm$ 0.0008 | 0.0289 $\pm$ 0.0000 | 0.0432 $\pm$ 0.0046 | 0.1243 $\pm$ 0.0000
Avg. F1-score per method | 0.0633 $\pm$ 0.0264 | 0.0940 $\pm$ 0.0376 | 0.0446 $\pm$ 0.0310 | 0.0482 $\pm$ 0.0496 | 0.0356 $\pm$ 0.0493 | 0.0638 $\pm$ 0.0433 | 0.0345 $\pm$ 0.0178 | 0.0909 $\pm$ 0.0451 |

TABLE IX: The back-bias performance of the tested methods on the 5 benchmarks
using the metrics proposed by [59]. The average and the standard deviation of
five runs are reported.

##  6 Conclusion

This paper proposes an extensive evaluation study of recent time-series
anomaly detection methods. To the best of our knowledge, we are the first to
analyze these algorithms based on a more elaborate experimentation protocol.
In contrast to previous evaluation studies, which only consider the standard
performance metrics, we take into account revisited performance criteria
specifically designed for time series in our analysis. In addition, the model
stability, the model size as well as the robustness to different types of
anomalies are also investigated. All these additional elements give a more
complete picture of the current state-of-the-art. Moreover, the proposed
protocol is timely and could be beneficial for future investigations,
providing more insights regarding their applicability in a real-world context.

## Acknowledgment

This work is supported by the Luxembourg National Research Fund (FNR) under
the projects BRIDGES2021/IS/16353350/FaKeDeTeR, UNFAKE, ref.16763798 in
collaboration with POST Luxembourg and by the European Space Agency (ESA)
under the project SKYTRUST 4000133885/21/NL/MH/hm.

## References

  * [1] Kim, S., Choi, K., Choi, H. S., Lee, B., and Yoon, S. (2022, February). Towards a Rigorous Evaluation of Time-series Anomaly Detection. In Proceedings of the 2022 AAAI Conference on Artificial Intelligence.
  * [2] Xu, H., Chen, W., Zhao, N., Li, Z., Bu, J., Li, Z., Liu, Y., Zhao, Y. Pei, Dan Feng, Y., Chen, J, Wang, Z. and Qiao, H. (2018, April). Unsupervised anomaly detection via variational auto-encoder for seasonal kpis in web applications. In Proceedings of the 2018 world wide web conference (pp. 187-196).
  * [3] Choi, K., Yi, J., Park, C. and Yoon, S. (2021). Deep Learning for Anomaly Detection in Time-Series Data: Review, Analysis, and Guidelines. IEEE Access.
  * [4] Schölkopf, B., Williamson, R. C., Smola, A., Shawe-Taylor, J.and Platt, J. (1999). Support vector method for novelty detection. Advances in neural information processing systems, 12.
  * [5] Deng, A., and Hooi, B. (2021, February). Graph neural network-based anomaly detection in multivariate time series. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 35, No. 5, pp. 4027-4035).
  * [6] Zhao, B., Lu, H., Chen, S., Liu, J. and Wu, D. (2017). Convolutional neural networks for time series classification. Journal of Systems Engineering and Electronics, 28(1), 162-169.
  * [7] Devaki, R., Kathiresan, V., Gunasekaran, S. (2014). Credit card fraud detection using time series analysis. International Journal of Computer Applications, 3, 8-10.
  * [8] Jiang, J. R. and Kao, J. B. (2021). Semi-supervised time series anomaly detection based on statistics and deep learning. Applied Sciences, 11(15), 6698.
  * [9] Mahalakshmi, G., Sridevi, S. and Rajaram, S. (2016, January). A survey on forecasting of time series data. In 2016 International Conference on Computing Technologies and Intelligent Data Engineering (ICCTIDE’16) (pp. 1-8). IEEE.
  * [10] Golmohammadi, K. and Zaiane, O. R. (2017, August). Sentiment analysis on Twitter to improve time series contextual anomaly detection for detecting stock market manipulation. In International Conference on Big Data Analytics and Knowledge Discovery (pp. 327-342). Springer, Cham.
  * [11] Tinawi, I. (2019). Machine learning for time series anomaly detection (Doctoral dissertation, Massachusetts Institute of Technology).
  * [12] Hundman, K., Constantinou, V., Laporte, C., Colwell, I., and Soderstrom, T. (2018, July). Detecting spacecraft anomalies using lstms and nonparametric dynamic thresholding. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining (pp. 387-395).
  * [13] O’Neill, P., Entekhabi, D., Njoku, E., & Kellogg, K. (2010, July). The NASA soil moisture active passive (SMAP) mission: Overview. In 2010 IEEE International Geoscience and Remote Sensing Symposium (pp. 3236-3239). IEEE.
  * [14] Zhao, H., Wang, Y., Duan, J., Huang, C., Cao, D., Tong, Y., Xu, B., Bai, J., Tong, J., and Zhang, Q. (2020, November). Multivariate time-series anomaly detection via graph attention network. In 2020 IEEE International Conference on Data Mining (ICDM) (pp. 841-850). IEEE.
  * [15] Blázquez-García, A., Conde, A., Mori, U. and Lozano, J. A. (2021). A review on outlier/anomaly detection in time series data. ACM Computing Surveys (CSUR), 54(3), 1-33.
  * [16] Chen, X., Wang, H., Wei, Y., Li, J., and Gao, H. (2019). Autoregressive-Model-Based Methods for Online Time Series Prediction with Missing Values: an Experimental Evaluation. arXiv preprint arXiv:1908.06729.
  * [17] Jin, B., Chen, Y., Li, D., Poolla, K., & Sangiovanni-Vincentelli, A. (2019, June). A one-class support vector machine calibration method for time series change point detection. In 2019 IEEE International conference on prognostics and health management (ICPHM) (pp. 1-5). IEEE.
  * [18] Liu, F. T., Ting, K. M., and Zhou, Z. H. (2008, December). Isolation forest. In 2008 eighth ieee international conference on data mining (pp. 413-422). IEEE.
  * [19] Yaacob, A. H., Tan, I. K., Chien, S. F., and Tan, H. K. (2010, February). Arima based network anomaly detection. In 2010 Second International Conference on Communication Software and Networks (pp. 205-209). IEEE.
  * [20] Su, Y., Zhao, Y., Niu, C., Liu, R., Sun, W. and Pei, D. (2019, July). Robust anomaly detection for multivariate time series through stochastic recurrent neural network. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery and data mining (pp. 2828-2837).
  * [21] Wu, R. and Keogh, E. (2021). Current time series anomaly detection benchmarks are flawed and are creating the illusion of progress. IEEE Transactions on Knowledge and Data Engineering.
  * [22] Keogh, E., Lin, J., Fu, A. W., Van Herle, H. (2006). Finding unusual medical time-series subsequences: Algorithms and applications. IEEE Transactions on Information Technology in Biomedicine, 10(3), 429-439.
  * [23] Cook, A. A., Mısırlı, G., Fan, Z. (2019). Anomaly detection for IoT time-series data: A survey. IEEE Internet of Things Journal, 7(7), 6481-6494.
  * [24] Cook, A. A., Mısırlı, G., and Fan, Z. (2019). Anomaly detection for IoT time-series data: A survey. IEEE Internet of Things Journal, 7(7), 6481-6494.
  * [25] Shen, L., Li, Z., and Kwok, J. (2020). Timeseries anomaly detection using temporal hierarchical one-class network. Advances in Neural Information Processing Systems, 33, 13016-13026.
  * [26] Su, Y., Zhao, Y., Niu, C., Liu, R., Sun, W. and Pei, D. (2019, July). Robust anomaly detection for multivariate time series through stochastic recurrent neural network. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery and data mining (pp. 2828-2837).
  * [27] Geiger, A., Liu, D., Alnegheimish, S., Cuesta-Infante, A., and Veeramachaneni, K. (2020, December). TadGAN: Time series anomaly detection using generative adversarial networks. In 2020 IEEE International Conference on Big Data (Big Data) (pp. 33-43). IEEE.
  * [28] Baptista, R., Ghorbel, E., Moissenet, F., Aouada, D., Douchet, A., André, M., … & Bouilland, S. (2019). Home self-training: Visual feedback for assisting physical activity for stroke survivors. Computer methods and programs in biomedicine, 176, 111-120.
  * [29] Yu, Y., Zhu, Y., Li, S., & Wan, D. (2014). Time series outlier detection based on sliding window prediction. Mathematical problems in Engineering, 2014.
  * [30] Golmohammadi, K., & Zaiane, O. R. (2015, October). Time series contextual anomaly detection for detecting market manipulation in stock market. In 2015 IEEE international conference on data science and advanced analytics (DSAA) (pp. 1-10). IEEE.
  * [31] Lin, S., Clark, R., Birke, R., Schönborn, S., Trigoni, N., & Roberts, S. (2020, May). Anomaly detection for time series using vae-lstm hybrid model. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 4322-4326). Ieee.
  * [32] Reddy, K. K., Sarkar, S., Venugopalan, V., & Giering, M. (2016). Anomaly detection and fault disambiguation in large flight data: A multi-modal deep auto-encoder approach. In Annual Conference of the PHM Society (Vol. 8, No. 1).
  * [33] Li, D., Chen, D., Jin, B., Shi, L., Goh, J., and Ng, S. K. (2019, September). MAD-GAN: Multivariate anomaly detection for time series data with generative adversarial networks. In International Conference on Artificial Neural Networks (pp. 703-716). Springer, Cham.
  * [34] Malhotra, P., Vig, L., Shroff, G., & Agarwal, P. (2015, April). Long short term memory networks for anomaly detection in time series. In Proceedings (Vol. 89, pp. 89-94).
  * [35] Zong, B., Song, Q., Min, M. R., Cheng, W., Lumezanu, C., Cho, D., and Chen, H. (2018, February). Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In International conference on learning representations.
  * [36] Moayedi, H. Z., & Masnadi-Shirazi, M. A. (2008, August). Arima model for network traffic prediction and anomaly detection. In 2008 international symposium on information technology (Vol. 4, pp. 1-6). IEEE.
  * [37] Baptista, R., Demisse, G., Aouada, D., & Ottersten, B. (2018, November). Deformation-based abnormal motion detection using 3d skeletons. In 2018 Eighth International Conference on Image Processing Theory, Tools and Applications (IPTA) (pp. 1-6). IEEE.
  * [38] Diab, D. M., AsSadhan, B., Binsalleeh, H., Lambotharan, S., Kyriakopoulos, K. G., & Ghafir, I. (2019, August). Anomaly detection using dynamic time warping. In 2019 IEEE International Conference on Computational Science and Engineering (CSE) and IEEE International Conference on Embedded and Ubiquitous Computing (EUC) (pp. 193-198). IEEE.
  * [39] Benkabou, S. E., Benabdeslem, K., & Canitia, B. (2018). Unsupervised outlier detection for time series by entropy and dynamic time warping. Knowledge and Information Systems, 54(2), 463-486.
  * [40] Berndt, D. J., & Clifford, J. (1994, July). Using dynamic time warping to find patterns in time series. In KDD workshop (Vol. 10, No. 16, pp. 359-370).
  * [41] Lai, K. H., Zha, D., Xu, J., Zhao, Y., Wang, G. and Hu, X. (2021, June). Revisiting time series outlier detection: Definitions and benchmarks. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1).
  * [42] Hyndman, R. J., Wang, E., & Laptev, N. (2015, November). Large-scale unusual time series detection. In 2015 IEEE international conference on data mining workshop (ICDMW) (pp. 1616-1619). IEEE.
  * [43] Jin, Y., Qiu, C., Sun, L., Peng, X., & Zhou, J. (2017, September). Anomaly detection in time series via robust PCA. In 2017 2nd IEEE International Conference on Intelligent Transportation Engineering (ICITE) (pp. 352-355). IEEE.
  * [44] Wang, X., Miranda-Moreno, L., & Sun, L. (2021). Hankel-structured Tensor Robust PCA for Multivariate Traffic Time Series Anomaly Detection. arXiv preprint arXiv:2110.04352.
  * [45] Oja, E. (1982). Simplified neuron model as a principal component analyzer. Journal of mathematical biology, 15(3), 267-273.
  * [46] Kirby, M., & Sirovich, L. (1990). Application of the Karhunen-Loeve procedure for the characterization of human faces. IEEE Transactions on Pattern analysis and Machine intelligence, 12(1), 103-108.
  * [47] Schölkopf, B., Smola, A., & Müller, K. R. (1998). Nonlinear component analysis as a kernel eigenvalue problem. Neural computation, 10(5), 1299-1319.
  * [48] Ruff, L., Vandermeulen, R., Goernitz, N., Deecke, L., Siddiqui, S.A, Binder, A., Müller, E., and Kloft, M., Deep one-class classification. In International Conference on Machine Learning, pages 4393–4402, 2018.
  * [49] Liu, F. T., Ting, K. M., and Zhou, Z. H. (2008, December). Isolation forest. In 2008 eighth ieee international conference on data mining (pp. 413-422). IEEE.
  * [50] Breunig, M. M., Kriegel, H. P., Ng, R. T., & Sander, J. (2000, May). LOF: identifying density-based local outliers. In Proceedings of the 2000 ACM SIGMOD international conference on Management of data (pp. 93-104).
  * [51] Yairi, T., Takeishi, N., Oda, T., Nakajima, Y., Nishimura, N., & Takata, N. (2017). A data-driven health monitoring method for satellite housekeeping data based on probabilistic clustering and dimensionality reduction. IEEE Transactions on Aerospace and Electronic Systems, 53(3), 1384-1401.
  * [52] Lindstrom, M. R., Jung, H., & Larocque, D. (2020). Functional Kernel Density Estimation: Point and Fourier Approaches to Time Series Anomaly Detection. Entropy, 22(12), 1363.
  * [53] Tang, J., Chen, Z., Fu, A. W. C., & Cheung, D. W. (2002, May). Enhancing effectiveness of outlier detections for low density patterns. In Pacific-Asia conference on knowledge discovery and data mining (pp. 535-548). Springer, Berlin, Heidelberg.
  * [54] Tax, D. M., & Duin, R. P. (2004). Support vector data description. Machine learning, 54(1), 45-66.
  * [55] Huang, C., Min, G., Wu, Y., Ying, Y., Pei, K., & Xiang, Z. (2017). Time series anomaly detection for trustworthy services in cloud computing systems. IEEE Transactions on Big Data.
  * [56] Li, K. L., Huang, H. K., Tian, S. F., & Xu, W. (2003, November). Improving one-class SVM for anomaly detection. In Proceedings of the 2003 international conference on machine learning and cybernetics (IEEE Cat. No. 03EX693) (Vol. 5, pp. 3077-3081). IEEE.
  * [57] Wang, Y., Wong, J., & Miner, A. (2004, June). Anomaly intrusion detection using one class SVM. In Proceedings from the Fifth Annual IEEE SMC Information Assurance Workshop, 2004. (pp. 358-364). IEEE.
  * [58] Ma, J., & Perkins, S. (2003, July). Time-series novelty detection using one-class support vector machines. In Proceedings of the International Joint Conference on Neural Networks, 2003. (Vol. 3, pp. 1741-1745). IEEE.
  * [59] Tatbul, N., Lee, T. J., Zdonik, S., Alam, M., & Gottschlich, J. (2018). Precision and recall for time series. Advances in neural information processing systems, 31.
  * [60] Musallam, M. A., Gaudilliere, V., Ghorbel, E., Al Ismaeil, K., Perez, M. D., Poucet, M., & Aouada, D. (2021, September). Spacecraft Recognition Leveraging Knowledge of Space Environment: Simulator, Dataset, Competition Design and Analysis. In 2021 IEEE International Conference on Image Processing Challenges (ICIPC) (pp. 11-15). IEEE.
  * [61] Granger, C. W., & Watson, M. W. (1984). Time series and spectral methods in econometrics. Handbook of econometrics, 2, 979-1022.
  * [62] L. Zhang, R. Cushing, C. d. Laat and P. Grosso, ”A real-time intrusion detection system based on OC-SVM for containerized applications,” 2021 IEEE 24th International Conference on Computational Science and Engineering (CSE), 2021, pp. 138-145, doi: 10.1109/CSE53436.2021.00029.
  * [63] M. A. Siddiqui et al., ”Detecting Cyber Attacks Using Anomaly Detection with Explanations and Expert Feedback,” ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019, pp. 2872-2876, doi: 10.1109/ICASSP.2019.8683212.

|  Nesryne Mejri is currently a Ph.D. student at the Computer Vision, Imaging
and Machine Intelligence (CVI2) Research Group at the Interdisciplinary Centre
for Security, Reliability, and Trust (SnT), University of Luxembourg. She
received the M.Sc degree in Information and Computer Science from the
University of Luxembourg, in 2021. Her research interests are in Computer
Vision and Deep Learning.
---|---
|  Laura Lopez Fuentes holds a M.Sc in Computer Vision from the Autonomous
university of Barcelona. She received her Ph.D. in Computer Vision from the
University of the Balearic Islands (Spain) in 2021. Her Ph.D. was in
collaboration with the Computer Vision Center of Barcelona and the company
AnsuR Technologies based in Oslo (Norway). In 2021, Dr. Lopez joined the
University of Luxembourg in the Computer Vision, Imaging, and Machine
Intelligence Research Group.
---|---
|  Kankana Roy received the B.Tech. degree in Computer Science and Engineering
from the West Bengal University of Technology, Kolkata, India, in 2011, the
M.Tech. in Computer Science and Engineering from the National Institute of
Technology, Durgapur, India, in 2013 and Ph.D. degree from the Indian
Institute of Technology Kharagpur, India, in 2021. She was employed as a
software engineer at the NRI Financial Technologies from 2013 to 2015. She was
a research associate at University of Luxembourg, from 2020 to 2021. Her
research interests include computer vision, dictionary learning, human
computer interaction, and machine learning.
---|---
|  Pavel Chernakov is currently an B.Sci. student in Information and Computer
Science at the University of Luxembourg. He had several research internships
at the Computer Vision, Imaging and Machine Intelligence (CVI2) Research
Group. His interests include computer vision, cyber-security and software
development.
---|---
|  Enjie Ghorbel is Research Scientist at the Interdisciplinary Centre for
Security, Reliability, and Trust (SnT), University of Luxembourg. She received
her engineering diploma from the National Engineering Schoul of Sousse (ENISo)
in Applied Computer Science in 2014 and her PhD in Computer Science, from the
University of Normandie, in 2017. Her research interests include computer
vision and pattern recognition.
---|---
|  Djamila Aouada is Senior Research Scientist and Assistant Professor at the
Interdisciplinary Centre for Security, Reliability, and Trust (SnT),
University of Luxembourg. She is Head of the Computer Vision, Imaging and
Machine Intelligence (CVI2) Research Group at SnT, and Head of the SnT
Computer Vision Laboratory and co-Head of the SnT Zero-G Laboratory. Dr.
Aouada received the State Engineering degree in electronics in 2005, from the
École Nationale Polytechnique (ENP), Algiers, Algeria, and the Ph.D. degree in
electrical engineering in 2009 from North Carolina State University (NCSU),
Raleigh, NC, USA. Dr. Aouada has worked as a consultant for multiple renowned
laboratories (Los Alamos National Laboratory, Alcatel Lucent Bell Labs., and
Mitsubishi Electric Research Labs.). She has been leading the computer vision
activities at SnT since 2009. Her research interests span the areas of image
processing, computer vision, and data modelling. Dr. Aouada is Senior Member
of the IEEE, member of the IEEE Signal Processing Society, IEEE WIE, INSTICC
and the Eta Kappa Nu honor society (HKN). She has served as the Chair of the
IEEE Benelux Women in Engineering Affinity Group from 2014 to 2016, Chair of
the SHARP workshop at ECCV 2020, CVPR 2021 and CVPR 2022, Area Chair at 3DV
2020 and 2022 and Program Chair at 3DV 2021, General Chair at RTIP2R 2022, and
Program Chair at EUVIP 2023. She is the recipient of four IEEE best paper
awards.
---|---