# Two-Stage Deep Anomaly Detection with Heterogeneous Time Series Data

Kyeong-Joong Jeong, Jin-Duk Park, Kyusoon Hwang, Seong-Lyun Kim,
and Won-Yong Shin This work was supported by the Republic of Korea’s MSIT
(Ministry of Science and ICT), under the High-Potential Individuals Global
Training Program (No. 2020-0-01463) supervised by the IITP (Institute of
Information and Communications Technology Planning Evaluation) and by the
Yonsei University Research Fund of 2020 (2020-22-0101). (Corresponding author:
Won-Yong Shin.)K.-J. Jeong, J.-D. Park, and W.-Y. Shin are with the Department
of Computational Science and Engineering, Yonsei University, Seoul 03722,
Republic of Korea (e-mail: {jeongkj, jindeok6, wy.shin}@yonsei.ac.kr).K. Hwang
is with Wizcore Co., Ltd., Seoul 04209, Republic of Korea (e-mail:
hwkyso@wizcore.co.kr).S.-L. Kim is with the Department of Electrical and
Electronic Engineering, Yonsei University, Seoul 03722, Republic of Korea
(e-mail: slkim@yonsei.ac.kr).

###### Abstract

We introduce a data-driven anomaly detection framework using a manufacturing
dataset collected from a factory assembly line. Given _heterogeneous_ time
series data consisting of operation cycle signals and sensor signals, we aim
at discovering abnormal events. Motivated by our empirical findings that
conventional single-stage benchmark approaches may not exhibit satisfactory
performance under our challenging circumstances, we propose a _two-stage_ deep
anomaly detection (T-DAD) framework in which two different unsupervised
learning models are adopted depending on types of signals. In Stage I, we
select anomaly candidates by using a model trained by operation cycle signals;
in Stage II, we finally detect abnormal events out of the candidates by using
another model, which is suitable for taking advantage of temporal continuity,
trained by sensor signals. A distinguishable feature of our framework is that
operation cycle signals are exploited first to find likely anomalous points,
whereas sensor signals are leveraged to filter out unlikely anomalous points
afterward. Our experiments comprehensively demonstrate the superiority over
single-stage benchmark approaches, the model-agnostic property, and the
robustness to difficult situations.

###### Index Terms:

Anomaly detection, operation cycle, sensor, time series data, two-stage
detection.

##  I Introduction

###  I-A Background

With the rapid development of the fourth industrial revolution (Industry 4.0),
which encompasses Industrial Internet of Things (IIoT) as well as smart and
sustainable manufacturing systems [1], a vast amount of data are generated
every day [2]. Along with the growing importance of Big Data analysis, studies
on anomaly detection [3] have attracted increasing attention for modern
industry. Anomaly detection enables us to solve a variety of important
problems in smart industrial systems, e.g., security protection [4], fault
detection [5, 6, 7], and condition monitoring [8].

On the other hand, artificial intelligence (AI) or machine learning (ML)
models have been widely used in discovering anomalies on industrial datasets
(see, e.g., [4, 5, 9, 8, 6, 7, 10, 11, 12, 13, 14, 15] and references
therein). Nevertheless, real-world industrial datasets contain the following
challenging issues [16]: 1) the quality of data depends heavily on data
acquisition environments; 2) observed patterns of anomalies may not be readily
predictable; 3) and very few (trusted) labels are available. To tackle such
challenges, we should design AI/ML-based anomaly detection models in a data-
driven fashion, which are capable of capturing complex temporal or organic
relations and being robust to both label noise and class imbalance.

###  I-B Motivation and Main Contributions

In this paper, we present a novel data-driven unsupervised approach for
anomaly detection using a manufacturing dataset collected from a real-world
factory assembly line, which accompanies a variety of considerably challenging
problems such as severe data imbalance with very scarce labels and
intrinsically unpredictable patterns. In particular, given _heterogeneous_
multivariate time series data consisting of operation cycle signals and sensor
signals, we are interested in automatically discovering true abnormal events
in the factory assembly line, which are defined as anomalies in our context.

Our study is motivated by our empirical findings that conventional single-
stage benchmark detection approaches (even sophisticated) may not provide
satisfactory performance under our real-world manufacturing dataset; this is
because existing approaches are not inherently designed to suit our own
dataset composed of two different signals (i.e., operation cycle signals and
sensor signals), exhibiting significantly different patterns from each other.
More precisely, when there are two different signals recorded over a multi-
scale time span as in our dataset, existing anomaly detection methods are not
adequate for making use of such signals altogether. As a novel and effective
solution to resolve this issue, we propose a _two-stage_ deep anomaly
detection (T-DAD) framework in which two different unsupervised learning
models are adopted depending on types of signals. The two models are trained
without labels since unsupervised learning is rather appropriate for our
dataset in which very few abnormal events are only available, similarly as in
many studies in such industrial systems [8, 17]. More specifically, in Stage
I, we select anomaly candidates by using a model trained by using operation
cycle signals; in Stage II, we finally detect abnormal events (i.e., the most
likely anomalies) among the candidates by using another model, which is
suitable for capturing the temporal dependence, trained only by using sensor
signals. A distinguishable feature of our T-DAD framework is that _signal-
specific_ detection tasks are carried out in such a way that, along with the
two empirically optimized thresholds, operation cycle signals are exploited
first to find likely anomalous points whereas sensor signals are leveraged as
a secondary tool to filter out unlikely anomalous points afterward. In other
words, our T-DAD framework fully takes advantage of the characteristics of our
heterogeneous time series data in detecting anomalies, which basically differs
from prior attempts in the literature.

Through intensive experiments, we demonstrate the superiority of our T-DAD
framework over various single-stage benchmark methods using a certain assembly
process dataset in terms of the best $F_{1}$ score, representing the best
possible performance of a trained model on a test set given the optimal
threshold(s), where the $F_{1}$ score is the harmonic mean of Precision and
Recall. We also show the model-agnostic property of our T-DAD framework by
diversifying two appropriately chosen learning methods in Stages I and II and
then successfully integrating them into our framework. To validate the
robustness of the proposed T-DAD framework to other fringe environments, we
evaluate the performance of anomaly detection using other assembly process
datasets, which display consistent trends in terms of not only a balanced
number of features in each of operation cycle and sensor signals but also a
non-negligible fraction of anomalies by showing substantial gains of T-DAD
over benchmark methods.

Our methodology sheds crucial insights into a simple implementation of data-
driven anomaly detection methods when _heterogeneous_ multivariate signals
with unpredictable patterns are given as input and/or very few (or no) labels
are available. Various problems in real-world manufacturing domains include
but are not limited to fault detection, condition monitoring, manufacturing
inspection, functional failure detection, alarm management, and so forth. When
such practical problems deal with heterogeneous signals, using our distinctive
two-stage strategy can be promising with potential gains. Moreover, as
operational benefits in manufacturing domains, automatic discovery of abnormal
events with much higher accuracy through our approach would enable cost-
effective management in the Industry 4.0 era along with minimal or
complementary interventions by the operator with domain knowledge.

###  I-C Organization

The remainder of this paper is organized as follows. In Section II, we
summarize studies related to our work. Section III describes our dataset. The
overall methodology of T-DAD is presented in Section IV. Experimental results
are provided in Section V. In Section VI, we summarize the paper with some
concluding remarks.

Table I summarizes the notations used in this paper, which will be formally
defined in the following sections when we introduce our problem definition and
technical details.

TABLE I: Summary of notations Notation | Description
---|---
${{\bf x}_{t}^{(o)}}$ |  Operation cycle signal at timestamp $t$
${{\bf x}_{t}^{(s)}}$ |  Sensor signal at timestamp $t$
$a_{t}^{(o)}$ |  Anomaly score from ${{\bf x}_{t}^{(o)}}$
$a_{t}^{(s)}$ |  Anomaly score from ${{\bf x}_{t}^{(s)}}$
$\delta$ | Tolerance
$\mathcal{T}_{o}$ | Set of timestamps where operation cycle signals exist
$\hat{\mathcal{T}}_{\text{anomaly}}$ |  Set of detected abnormal events within the tolerance $\delta$
$\tau_{1}$ | Threshold on Stage I
$\tau_{2}$ | Threshold on Stage II
Figure 1: Heterogeneous features, including operation cycle signals and sensor
signals.

##  II Related Work

The framework that we propose in this paper is related to two broader fields
of research, namely 1) anomaly detection in real-world manufacturing domains
2) time series anomaly detection.

###  II-A Anomaly Detection in Manufacturing.

There has been a steady push to solve anomaly detection tasks using real-world
manufacturing datasets such as real-time motor fault detection using motor
current signals [6], fault detection using wind turbine datasets [7],
condition monitoring using a simulated bearing fault dataset [8],
manufacturing inspection using tile production data [9], and functional
failure detection using sensor data collected from a turbine syngas compressor
in global manufacturing plants [5]. Anomaly detection in manufacturing or
production lines (e.g., Intel’s CPU manufacturing [10], and semiconductor
manufacturing [11]) has also received significant attention.

###  II-B Time Series Anomaly Detection

Anomaly detection in a time series using stochastic time series models such as
autoregressive moving average (ARMA) [18] and autoregressive integrated moving
average (ARIMA) [19] has been actively studied in the past two decades. One-
class support vector machine (SVM) was also widely used for time series
novelty (i.e., out-of-distribution) detection [20]. Moreover, probabilistic
methods such as hidden Markov models [21], Bayesian networks [22], and Poisson
models [23] have been presented for time series anomaly detection.

With recent advances in deep learning across a wide range of domains, anomaly
detection tasks have also successfully employed various deep neural networks
such as recurrent neural network (RNN), convolutional neural network (CNN),
autoencoder (AE), variational autoencoder (VAE), generative adversarial
network (GAN) and graph neural network (GNN). First, RNN models have been
actively developed in time series anomaly detection to capture the temporal
dependence in time series sequential data. The strength of using a long short
term memory (LSTM) network for time series anomaly detection was demonstrated
in [24] by presenting a nonparametric and dynamic error thresholding approach
that does not depend on either scare labels or prior distributional
assumptions. A stochastic RNN for multivariate time series anomaly detection
to learn robust representations with a stochastic variable connection and
planar normalizing flow was presented in [25]. Second, with regard to CNN-
based anomaly detection models, a deep CNN approach was developed in [26] as a
time series predictor module. Moreover, an algorithm that combines spectral
residual and CNN was designed in [17] to achieve the state-of-the-art
performance on univariate time series anomaly detection. Third, AE-aided
approaches have also been widely employed by taking into account the
reconstruction error as an anomaly score. An AE-based algorithm was proposed
in [27] for anomaly detection in high-performance computing (HPC) systems. A
robust anomaly detection method integrating AE and robust principal component
analysis was presented in [28]. An encoder–decoder architecture-based anomaly
detection method using LSTM layers was proposed in [13]. As an end-to-end
learning solution, an unsupervised anomaly detection model that utilizes the
AE and Gaussian mixture model was proposed in [29]. In [15], a scalable
anomaly detection method for multivariate time series was built upon an
encoder–decoder architecture with an adversely training framework inspired by
GAN. Fourth, among generative models, VAE or GAN-based time series anomaly
detection approaches also received considerable attention. A VAE-based anomaly
detection algorithm was designed in [30] along with a solid theoretical
explanation for adopting VAE on univariate time series anomaly detection. An
approach combining LSTM and VAE was presented in [31] for time series anomaly
detection. An unsupervised anomaly detection method was also presented in [32]
based on GANs using LSTM networks for each generator and discriminator. Fifth,
increasing attention has recently been paid to time series anomaly detection
models [33, 34] based on GNNs [35] in order to capture complex relationships
between variables in multivariate time series data.

###  II-C Discussions

As shown above, various time series anomaly detection approaches from
traditional methods to current deep learning-based methods have been
extensively developed in the literature. Nonetheless, to the best of our
knowledge, there is no prior work on anomaly detection that makes use of the
characteristics of heterogeneous time series signals similarly as in our real-
world manufacturing dataset.

Figure 2: The schematic overview of our T-DAD framework.

##  III Dataset Description

We use a multivariate time series dataset collected from an assembly line of
automotive engine valves as one of the large-scale real-world manufacturing
datasets. The entire dataset consists of a set of features recorded from
December 2, 2019 to December 7, 2019, in which there are 518,400 timestamps
(i.e., temporal records). Let us describe the set of features, which are
partitioned into the following two types of signals.

  * •

_Sensor_ signals: Since sensor signals are recorded every second regardless of
the operation status, they tend to exhibit temporal continuity;

  * •

_Operation cycle_ signals: Due to the fact that operation cycle signals are
logged in units of operation cycles, they are less likely to follow temporal
continuity. Different from sensor signals, since such operation cycle signals
are purely event-driven, they occur irregularly.

We note that sensor signals are evenly spaced in time while operation cycle
signals are unevenly spaced in time; this comes from the fact that the
operation is inactive when either the planned short downtime or the irregular
(unplanned) long downtime occurs. The behavior of heterogeneous features is
illustrated in Fig. 1.

##  IV Methodology

In this section, to describe the proposed methodology, we first present the
problem definition. Then, we elaborate on our T-DAD framework.

###  IV-A Problem Definition

We define our problem in automatically detecting true abnormal events that
have occurred in the factory assembly line using the heterogeneous time series
manufacturing dataset. It would be senseless to perform point-wise anomaly
detection since it is rather crucial for the operator to be aware of whether a
possible abnormal situation has occurred within an acceptable time range [17].
Thus, we aim at conducting range-wise detection of abnormal events by
discovering a most likely anomalous point, which is defined as a nearby
timestamp manifesting an abnormal pattern from the point at which an abnormal
event has indeed occurred.

######  Problem 1.

Given a heterogeneous multivariate time series dataset, range-wise anomaly
detection is to find an anomalous point $t$ such that there exists an abnormal
event in $[t-\delta,t+\delta]$ for a certain tolerance $\delta>0$.

Note that the tolerance $\delta$ is assumed to be set appropriately depending
on the operator’s criterion beforehand.

###  IV-B Overall Architecture

To accomplish anomaly detection with our heterogeneous multivariate time
series data, we propose novel two-stage detection, termed T-DAD, built upon a
data-driven approach inspired by non-straightforward empirical findings. Fig.
2 illustrates the overall architecture of the proposed T-DAD framework in
which signal-specific detection tasks are carried out after training two
different unsupervised learning models, the so-called Stage I and Stage II
models, according to two types of signals (i.e., operation cycle signals and
sensor signals). To be specific, we start by adopting the Stage I model with
multivariate operation cycle signals and the Stage II model with multivariate
sensor signals that account for the temporal dynamics, where each model is
unsupervisedly trained without any labels. Then, we compute anomaly scores,
denoted by $a_{t}^{(o)}$ and $a_{t}^{(s)}$, from the output of the each model,
respectively, at each timestamp $t$. Thereafter, we perform two-stage anomaly
detection as follows. In Stage I, we select a set of very likely anomaly
candidates, denoted by $\mathcal{T}_{o,\text{anomaly}}$, at which the
resulting anomaly score $a_{t}^{(o)}$ is higher than a certain threshold
$\tau_{1}>0$, i.e.,

$\displaystyle\mathcal{T}_{o,\text{anomaly}}=\left\\{t|\forall
t\in\mathcal{T}_{o}:~{}a_{t}^{(o)}>\tau_{1}\right\\},$ (1)

where $\mathcal{T}_{o}$ is the set of timestamps at which there exist
operation cycle signals. On the contrary, in Stage II, we detect the most
likely anomalous points by filtering out very unlikely anomalous points among
the selected candidates. More precisely, we remove the set of points,
$\mathcal{T}_{s}$, such that the maximum of the resulting anomaly scores
$a_{t}^{(s)}$, $\forall t\in[t_{i}-\eta,t_{i}+\eta]$, is lower than another
threshold $\tau_{2}>0$, where $t_{i}\in\mathcal{T}_{o,\text{anomaly}}$ and
$\eta>0$ is a range setting parameter to be empirically found. That is, we
finally detect the set of anomalies, $\hat{\mathcal{T}}_{\text{anomaly}}$, as
in the following:

$\displaystyle\hat{\mathcal{T}}_{\text{anomaly}}=\mathcal{T}_{o,\text{anomaly}}\setminus\mathcal{T}_{s},$
(2)

where

$\displaystyle\mathcal{T}_{s}=\Big{\\{}t_{i}|\forall
t_{i}\in\mathcal{T}_{o,\text{anomaly}}:~{}\max_{t\in[t_{i}-\eta,t_{i}+\eta]}a_{t}^{(s)}<\tau_{2}\Big{\\}}.$
(3)

###  IV-C Implementation of the T-DAD Framework

In this subsection, we elaborate on the proposed T-DAD framework, which
constitutes two models along with signal-specific detection tasks. In our
study, we adopt the DAE model [27, 28] in Stage I with operation cycle signals
and the LSTM-DAE model [13] in Stage II with sensor signals. Note that, due to
the model-agnostic property of our proposed framework, any unsupervised
learning model with which the anomaly score is computed at each timestamp can
be employed in each stage. Nevertheless, we focus on describing our T-DAD
framework built upon the DAE and LSTM-DAE models since they are relatively
tractable.111In Section V-F3, we shall carry out comprehensive experimental
evaluations by diversifying unsupervised learning models in Stages I and II,
which demonstrates that the use of DAE and LSTM-DAE in Stages I and II,
respectively, indeed leads to the highest best $F_{1}$ score.

Let us describe the model training process. We first adopt a DAE model
employing a feed-forward multi-layer neural network in which the desired
output is the input itself, as stated in [27, 28]. Our DAE model contains a
multi-layer perceptron (MLP) encoder and an MLP decoder, denoted by
$E_{\text{MLP}}(\cdot)$ and $D_{\text{MLP}}(\cdot)$, respectively, where the
multivariate operation cycle signal ${\bf x}_{t}^{(o)}\in\mathbb{R}^{1\times
d_{o}}$ is used as input. Here, $d_{o}$ denotes the number of features in the
operation cycle signal. By training this DAE, we are capable of measuring the
reconstruction error at each timestamp $t\in\mathcal{T}_{o}$, which is
regarded as the anomaly score $a_{t}^{(o)}=\|{\bf x}_{t}^{(o)}-\hat{\bf
x}_{t}^{(o)}\|_{2}$, where $\hat{\bf
x}_{t}^{(o)}=D_{\text{MLP}}(E_{\text{MLP}}({\bf x}_{t}^{(o)}))$ and
$\|\cdot\|_{2}$ denotes the $\ell_{2}$ norm.

In the meantime, to take advantage of temporal continuity, we adopt an LSTM-
DAE model [13] along with the multivariate sensor signal ${\bf
x}_{t}^{(s)}\in\mathbb{R}^{1\times d_{s}}$, where $d_{s}$ denotes the number
of features in the sensor signal. This LSTM-DAE contains an encoder–decoder
with LSTM layers, denoted by $E_{\text{LSTM}}(\cdot)$ and
$D_{\text{LSTM}}(\cdot)$, respectively. We apply a sliding window technique
with the window size $w>0$ and the step size $\gamma>0$ to the sensor signal
${\bf x}_{t}^{(s)}$ so as to generate sequences $S_{1},S_{2},\cdots$, which
are used as input of LSTM-DAE, where $\gamma<w$ is assumed and $S_{j}=\\{{\bf
x}_{1+(j-1)\gamma}^{(s)},\cdots,{\bf x}_{w+(j-1)\gamma}^{(s)}\\}$ for
$j\in\\{1,2,\cdots\\}$. Since a sensor signal ${\bf x}_{t}^{(s)}$ at timestamp
$t$ is contained in multiple sequences, the reconstruction error at timestamp
$t$ from a sequence $S_{j}$ can be expressed as $\|{\bf x}_{t}^{(s)}-\hat{\bf
x}_{t;j}^{(s)}\|_{2}$, where $\hat{\bf x}_{t;j}^{(s)}$ is the entry of
$D_{\text{LSTM}}(E_{\text{LSTM}}(S_{j}))$ corresponding to ${\bf
x}_{t}^{(s)}$. We then take the maximum of those multiple reconstruction
errors calculated at each timestamp $t$,222Other values such as the minimum
and arithmetic mean can also be adopted. However, we have empirically verified
that taking the maximum leads to the highest performance. which is regarded as
the anomaly score $a_{t}^{(s)}=\max_{j\in J_{t}}\|{\bf x}_{t}^{(s)}-\hat{\bf
x}_{t;j}^{(s)}\|_{2}$, where $J_{t}$ is the set of sequences to which a sensor
signal ${\bf x}_{t}^{(s)}$ at timestamp $t$ belongs. In consequence,
heterogeneous multivariate time series signals {${\bf x}_{t}^{(o)},{\bf
x}_{t}^{(s)}\\}$ can be seen as univariate anomaly scores
$\\{a_{t}^{(o)},a_{t}^{(s)}\\}$ via two trained DAE models.

Next, we turn to describing our two-stage signal-specific detection tasks. We
optimally decide two thresholds $\tau_{1}$ and $\tau_{2}$ shown in (1) and
(3), respectively, via exhaustive search in the sense of maximizing the
performance (i.e., the $F_{1}$ score). To explain a rationale of our
distinctive two-stage strategy, we give a motivating example as follows. Fig.
3 illustrates the visualization of anomaly scores $a_{t}^{(o)}$ and
$a_{t}^{(s)}$ (top and middle, respectively, of Figs. 3(a) and 3(b)) as well
as actual abnormal events (bottom of Figs. 3(a) and 3(b)) within two different
20-minute fragments (05:30:22–05:50:22 on December 3, 2019 and
04:36:15–04:56:15 on December 7, 2019) in our time series dataset. From Fig.
3(a), we make the following observations: 1) there is an anomalous point
$t_{1}$ whose anomaly score $a_{t_{1}}^{(o)}$ is higher than the threshold
$\tau_{1}$; 2) the maximum of anomaly scores $a_{t}^{(s)},\forall
t\in[t_{1}-\eta,t_{1}+\eta]$ is higher than another threshold $\tau_{2}$,
which implies that the anomalous point $t_{1}$ will not be removed from the
candidate; and 3) since the difference between the timestamp at which an
actual abnormal event occurred and the candidate point $t_{1}$ is within the
tolerance $\delta$, it is possible to successfully detect an actual abnormal
event range-wisely. On the other hand, from Fig. 3(b), one can see that 1)
there is an anomalous point $t_{2}$ such that $a_{t_{2}}>\tau_{1}$; 2) the
anomalous point $t_{2}$ is filtered out due to the fact that
$\max_{t\in[t_{2}-\eta,t_{2}+\eta]}a_{t}^{(s)}<\tau_{2}$; and 3) no abnormal
event is finally discovered. We summarize the overall procedure of our T-DAD
framework in Algorithm 1.

(a) Fragement 1 (05:30:22–05:50:22 on December 3, 2019)

(b) Fragement 2 (04:36:15–04:56:15 on December 7, 2019)

Figure 3: An illustrative example of our T-DAD framework alongside two-stage
signal-specific tasks. Algorithm 1 : T-DAD

0: ${\bf x}_{t}^{(o)},{\bf x}_{t}^{(s)},\tau_{1},\tau_{2},\eta$

0: $\hat{\mathcal{T}}_{\text{anomaly}}$

1: Initialization:
$\mathcal{T}_{o,\text{anomaly}}\leftarrow\emptyset,\mathcal{T}_{s}\leftarrow\emptyset$

2: Train DAE with ${\bf x}_{t}^{(o)}$ on the training set

3: Train LSTM-DAE with ${\bf x}_{t}^{(s)}$ on the training set

4: for $t\in\mathcal{T}_{o}$ do

5: $a_{t}^{(o)}\leftarrow\|{\bf x}_{t}^{(o)}-\hat{\bf x}_{t}^{(o)}\|_{2}$

6: if $a_{t}^{(o)}>\tau_{1}$ then

7:
$\mathcal{T}_{o,\text{anomaly}}\leftarrow\mathcal{T}_{o,\text{anomaly}}\cup\\{t\\}$

8: end if

9: end for

10: for $t_{i}\in\mathcal{T}_{o,\text{anomaly}}$ do

11: for $t\in[t_{i}-\eta,t_{i}+\eta]$ do

12: $a_{t}^{(s)}\leftarrow\max_{j\in J_{t}}\|{\bf x}_{t}^{(s)}-\hat{\bf
x}_{t;j}^{(s)}\|_{2}$

13: end for

14: if $\max_{t\in[t_{i}-\eta,t_{i}+\eta]}a_{t}^{(s)}<\tau_{2}$ then

15: $\mathcal{T}_{s}\leftarrow\mathcal{T}_{s}\cup\\{t\\}$

16: end if

17: end for

18:
$\hat{\mathcal{T}}_{\text{anomaly}}\leftarrow\mathcal{T}_{o,\text{anomlay}}\setminus\mathcal{T}_{s}$

19: return $\hat{\mathcal{T}}_{\text{anomaly}}$

From the above empirical findings, we may conclude that 1) the DAE model based
on operation cycle signals, aimed at detecting highly likely anomalous points,
cannot solely solve the anomaly detection problem and 2) the LSTM-DAE model
based on sensor signals plays a supplementary but crucial role in eliminating
very unlikely anomalous points, thus resulting in much higher accuracy.

##  V Experimental Evaluation

In this section, we first describe the set of labels in our manufacturing
dataset and data preprocessing techniques. Then, we introduce several
performance metrics and five benchmark methods for anomaly detection. We also
present experimental settings. As our main contributions, we comprehensively
perform experimental evaluations to validate the superiority of our T-DAD
framework over single-stage benchmark methods.

###  V-A Label Description

True breakdown events caused by the assembly line are unavailable in our
collected dataset; thus, in our experiments, the set of alarm points is used
as a ground truth instead. In fact, many industrial plants include a key
component for not only efficient and safe operation but also control of
plants, the so-called alarm system [36, 37], which alerts an operator about
abnormal conditions or malfunctions of the process that might be anomaly
candidates. Timely intervention by the operator alongside alarm logs in
industrial datasets can be very useful in detecting possible malicious
anomalies (i.e., abnormal events to possibly catastrophic levels).

In our dataset, an alarm point indicates the timestamp at which an alarm event
arises and is recorded as a binary format. That is, if the value at a
timestamp in a sequential time series of binary signals is 1, then it
signifies that an alarm has occurred.

###  V-B Data Preprocessing

We start by addressing that our dataset is collected from a total of 39
assembly processes. Among them, we focus primarily on the 2 assembly process
dataset for the analysis since it contains not only a number of features in
each of heterogeneous signals but also a non-negligible fraction of alarm
events used as a ground truth. There are 32 alarm points in the second
assembly process dataset, which however contribute only to 0.06% of all the
timestamps. Additionally, the effectiveness of the proposed T-DAD framework
will be empirically validated using other assembly process datasets.

Next, we apply the following three data preprocessing techniques into our
selected assembly process dataset for successful analysis: data imputation,
feature extraction, and data normalization. Specifically, we perform data
imputation in such a way that the missing values in each signal are imputed
with those observed at the previous timestamp. After the imputation of missing
data, we exclude some features such that their values remain unchanged over
time. This is because static features over time would be useless for data
analysis. Finally, min-max normalization is adopted for each signal so that
all the values in each signal are transformed into $[0,1]$, which enables us
to alleviate the impact of different scales among the signals.

###  V-C Experimental Setup

In this subsection, We first describe the settings of the neural network
models used in our proposed T-DAD framework. Unless otherwise stated, as
default settings, we run experiments using the DAE and LSTM-DAE models in
Stages I and II, respectively, of T-DAD since this combination among the
unsupervised learning models under consideration in Section V-E leads to the
best performance (which will be specified in Section V-F).

In our experiments, both DAE models are trained using the Adam optimizer [38]
with an initial learning rate of $10^{-3}$, where the batch size is set to 32.
Loss functions based on the mean square error and the mean absolute error are
used for the DAE and LSTM-DAE models, respectively. As for the DAE model, the
MLP encoder–decoder has 2 hidden layers with the size of 6 each; the standard
tanh and ReLU activation functions are applied alternately in every layer
between input and output layers; an $\ell_{1}$-norm regularizer is adopted in
the first layer of the encoder. As for the LSTM-DAE model, the encoder–decoder
has 2 hidden LSTM layers with 64 hidden units each; dropout [39] is applied to
each layer with probability 0.2; the window size $w$ and the step size
$\gamma$ are set to 180 and 60, respectively.

Additionally, in our experiments, the tolerance $\delta$ is set to 600
(seconds) since such a tolerance level may be acceptable in the sense of
triggering an alert without causing a long delay. The range setting parameter
$\eta$ used in Stage II of T-DAD is set to 14 (seconds) due to the fact that
one operation cycle tends to be mostly from 13 to 15 seconds. Note that the
parameters $\delta$ and $\eta$ can be set to different values according to the
operator’s criteria as well as assembly process environments.

We split our time series dataset into training (records measured from 00:00:00
on December 2, 2019 to 00:00:00 on December 3, 2019) and test (remaining
records) sets in chronological order. During training, the two DAE models are
learned; and the signal-specific detection tasks with threshold selection are
conducted on the test set.

###  V-D Performance Metrics

We adopt a modification of the well-known Precision and Recall so that we
evaluate the performance of range-wise alarm detection appropriately since
operators generally do not care about point-wise metrics, similarly as in [30,
40]. More specifically, in our study, the Precision means the fraction of
finally detected anomalous points that find actual alarms within the tolerance
$\delta$; and the Recall means the fraction of alarm points that are actually
found by finally detected anomalous points within $\delta$. Furthermore,
instead of the original $F_{1}$ score, which is the harmonic mean of Precision
and Recall, given a particular threshold, we adopt the best $F_{1}$ score in
[30, 15, 25] that indicates the best possible performance of a model on the
test set given the globally optimal threshold.333We remark that, under our
dataset, it is hardly possible to properly tune the threshold values during
training/validation since very few alarm points (used as labels) are available
therein. In our T-DAD framework, two thresholds are chosen in the sense of
maximizing the $F_{1}$ score via exhaustive search, which would still be
valuable in understanding the fundamental limits of unsupervised alarm
detection in our manufacturing dataset.

###  V-E Benchmark Approaches

We describe five benchmark methods for anomaly detection. They are used not
only as backbone models in our two-stage framework but also as state-of-the-
art single-stage benchmark approaches for comparison.

  * •

DAE [27, 28]: This model is the same as the Stage I model of our T-DAD
framework in Section IV-C.

  * •

LSTM-DAE [13]: This model is the same as the Stage II model of our T-DAD
framework in Section IV-C.

  * •

DAGMM [29]: Deep autoencoding Gaussian mixture model (DAGMM) aims at detecting
anomalies by combining the AE and the Gaussian mixture model in an end-to-end
fashion.

  * •

LSTM-VAE [31]: This model uses a connected variational AE (VAE) with LSTM
layers.

  * •

USAD [15]: Unsupervised anomaly detection (USAD) utilizes adversely trained
AEs inspired by GANs.

Figure 4: The $F_{1}$ score of T-DAD over two thresholds $\tau_{1}$ and
$\tau_{2}$. TABLE II: Performance comparison between the T-DAD framework and
single-stage benchmark methods.

Single-stage methods Two-stage method Methods DAE DAGMM USAD LSTM-DAE LSTM-VAE
T-DAD Cases C1 C2 C3 C4 C1 C2 C3 C4 C1 C2 C3 C4 C1 C2 C3 C4 C1 C2 C3 C4 - P
0.60 0.04 0.04 0.53 0.67 0.08 0.06 0.23 0.58 0.01 0.01 0.42 0.05 0.02 0.02
0.05 0.07 0.03 0.03 0.04 0.71 R 0.84 0.69 0.44 0.66 0.56 0.34 0.97 0.41 0.78
0.06 0.06 0.72 0.25 0.59 0.5 0.25 0.18 0.56 0.63 0.13 0.84 $F_{1}$ 0.70 0.08
0.07 0.59 0.61 0.13 0.10 0.29 0.67 0.01 0.02 0.53 0.09 0.04 0.03 0.09 0.10
0.06 0.06 0.06 0.77

TABLE III: Performance comparison among the model combinations $(\cdot,\cdot)$
in (Stage I, Stage II) of T-DAD

Stage II Stage I DAE DAGMM USAD P R $F_{1}$ P R $F_{1}$ P R $F_{1}$ DAE 0.60
0.84 0.7 0.67 0.56 0.61 0.58 0.78 0.67 DAGMM 0.63 0.84 0.72 0.75 0.56 0.64
0.61 0.78 0.69 USAD 0.63 0.84 0.72 0.75 0.56 0.64 0.63 0.78 0.69 LSTM-DAE 0.71
0.84 0.77 0.86 0.56 0.68 0.65 0.78 0.71 LSTM-VAE 0.71 0.84 0.77 0.86 0.56 0.68
0.65 0.78 0.71

###  V-F Experimental Results

In this subsection, our empirical study is designed to answer the following
four key research questions.

  * •

RQ1. How sensitive is the T-DAD framework to the selection of two thresholds
$\tau_{1}$ and $\tau_{2}$?

  * •

RQ2. How much does the performance of T-DAD improve the detection accuracy
over single-stage benchmark approaches?

  * •

RQ3. Which combination of backbone models in T-DAD yields the best
performance?

  * •

RQ4. How robust is the T-DAD framework to other assembly process datasets with
fewer features?

To answer these questions, we carry out four comprehensive experiments as
follows.

####  V-F1 Sensitivity Analysis with Respect to Thresholds (RQ1)

To analyze the parameter sensitivity, we show the performance of the proposed
T-DAD framework in Section IV with respect to the $F_{1}$ score according to
the values of thresholds $\tau_{1}$ and $\tau_{2}$. From Fig. 4, it is
observed that the performance tends to be more sensitive to the values of
$\tau_{1}$ since a small variation in $\tau_{1}$ leads to a non-negligible
difference in the $F_{1}$ score. One can also see that the best $F_{1}$ score
is found at $(\tau_{1},\tau_{2})=(2.55,0.08)$, which will be used in the
subsequent experiments unless otherwise specified.

TABLE IV: Performance comparison on other assembly process datasets

Single-stage methods Two-stage method Methods DAE DAGMM USAD T-DAD Cases C1 C2
C3 C4 C1 C2 C3 C4 C1 C2 C3 C4 - P 0.46 0.04 0.04 0.47 0.41 0.05 0.04 0.57 0.47
0.29 0.01 0.47 0.67 R 0.71 0.35 0.35 0.41 0.71 0.65 0.53 0.38 0.71 0.15 0.24
0.71 0.62 $F_{1}$ 0.56 0.06 0.06 0.44 0.52 0.08 0.08 0.46 0.57 0.19 0.02 0.56
0.64

(a) The 5 assembly process dataset

Single-stage methods Two-stage method Methods DAE DAGMM USAD T-DAD Cases C1 C2
C3 C4 C1 C2 C3 C4 C1 C2 C3 C4 - P 0.14 0.06 0.10 0.13 0.10 0.09 0.08 0.08 0.13
0.15 0.15 0.12 0.43 R 0.39 0.81 0.89 0.61 0.69 0.81 0.89 0.42 0.39 0.11 0.11
0.56 0.25 $F_{1}$ 0.20 0.11 0.18 0.21 0.11 0.17 0.16 0.15 0.19 0.13 0.13 0.19
0.32

(b) The 12 assembly process dataset

####  V-F2 Comparison with Single-Stage Benchmark Approaches (RQ2)

Due to the fact that, to the best of our knowledge, anomaly detection
exploiting all the heterogeneous signals in our real-world manufacturing
dataset has never been studied yet, there is no method that works directly
under our setting. For this reason, we demonstrate the superiority of our
proposed two-stage framework over single-stage benchmark methods, where the
DAE and LSTM-DAE models are used in Stages I and II, respectively, of our
T-DAD framework. We take into account the following four cases as feasible
single-stage approaches.

  * •

Case 1 (C1): Each method only uses operation cycle signals ${\bf
x}_{t}^{(o)},\forall t\in\mathcal{T}_{o}$.

  * •

Case 2 (C2): Each method only uses sensor signals ${\bf x}_{t}^{(s)}$ for all
timestamps.

  * •

Case 3 (C3): We first impute missing values with zeros when operation cycle
signals are not recorded. Then, each method uses the vector concatenation of
both signals as input for all timestamps.

  * •

Case 4 (C4): Each method uses the vector concatenation of operation cycle
signals and sensor signals as input only when both signals are recorded, i.e.,
$[{\bf x}_{t}^{(o)};{\bf
x}_{t}^{(s)}]\in\mathbb{R}^{1\times(d_{o}+d_{s})},\forall
t\in\mathcal{T}_{o}$.

We present the performance comparison between the T-DAD framework and five
single-stage benchmark methods (including DAE, DAGMM, USAD, LSTM-DAE, and
LSTM-VAE) for the above four cases in Table II with respect to the Precision,
Recall, and best $F_{1}$ score (P, R, and $F_{1}$, respectively, for short in
the table). We would like to provide the following insightful observations:

  * •

T-DAD is much superior to all single-stage benchmark approaches in terms of
the best $F_{1}$ score;

  * •

Only using operation cycle signals (i.e., C1) among four cases exhibits the
best performance regardless of performance metrics, which reveals that
operation cycle signals (rather than sensor signals) play a significant role
in offering satisfactory performance;

  * •

While the Recall of T-DAD is identical to that of C1 with DAE, the Precision
of T-DAD is remarkably enhanced compared to the second best performer by
virtue of judicious utilization of sensor signals in Stage II of T-DAD.
Namely, sensor signals serve as a secondary tool in further improving the best
$F_{1}$ score. This implies that our strategy in Stage II indeed leads to the
removal of very unlikely anomalous points (i.e., reduction in false
positives).

  * •

Performance comparison between C1 and (C3, C4) indicates that the naïve vector
concatenation of two signals even degrades the detection accuracy;

  * •

From the comparison among five benchmark models in C1, it is seen that the
best performer in terms of the best $F_{1}$ score is DAE while the second and
third best performers are USAD and DAGMM, respectively;

  * •

The performance of the LSTM-DAE and LSTM-VAE models is quite poor, which
implies that simply using models with LSTM layers originally designed for
learning temporal dependence rather deteriorates the detection accuracy.

####  V-F3 Model-Agnostic Property (RQ3)

To show the model-agnostic property of our T-DAD framework, we diversify
underlying models in Stages I and II (i.e., the Stage I and II models) by
leveraging five benchmark methods shown in Section V-E. In Stage I where
operation cycle signals are used, we choose DAE, DAGMM, and USAD for model
candidates since using the other two (i.e., LSTM-DAE and LSTM-VAE) do not
perform appropriately as stated in Section V-F2). On the other hand, in Stage
II where sensor signals are used, we choose all five methods as model
candidates.

The performance comparison among the model combinations $(\cdot,\cdot)$ in
(Stage I, Stage II) of T-DAD is presented in Table III with respect to the
Precision, Recall, and best $F_{1}$ score (P, R, and $F_{1}$ for short in the
table). Our findings are as follows. First, experimental results clearly
exhibit the model-agnostic property of our T-DAD framework while showing that
the adoption of two stages indeed enhances the performance over its
counterpart (i.e., single-stage benchmark methods). From Tables II and III,
note that the performance of the single-stage approach showing the highest
best $F_{1}$ score among four cases (i.e., C1) is identical to that of using
DAE in Stage II of T-DAD regardless of models in Stage I. Second, it is
examined that RNN-based models such as LSTM-DAE and LSTM-VAE are more
appropriate in Stage II than other benchmark methods due to the fact that
sensor signals tend to show strong temporal dependence. From Table III, we may
conclude that either (DAE, LSTM-DAE) or (DAE, LSTM-VAE) is the best
combination of models in our T-DAD framework.

####  V-F4 Robustness to Fringe Conditions (RQ4)

To validate the robustness of our T-DAD framework, we perform anomaly
detection on two other assembly process datasets whose characteristics are
appropriate for our experimental study in terms of not only a balanced number
of features in each of heterogeneous signals but also a non-negligible
fraction of alarms. Specifically, we note that 32 out of 39 assembly processes
contain only a very few operation cycle or sensor signals, which means that
our framework exploiting heterogeneous time series signals may be unnecessary
(i.e., adoption of an existing single-stage detection model rather than multi-
stage methods would be satisfactory in such cases). Among the remaining 7
assembly process datasets, assembly process datasets that have less than 30
alarms are excluded from our experiments in order to avoid unreliable
verification caused by the very scarce labels. In consequence, we choose the
following two assembly process datasets:

  * •

The 5 assembly process dataset having 22 features in operation cycle signals
and 7 features in sensor signals after feature extraction;

  * •

The 12 assembly process dataset having 11 features in operation cycle signals
and 6 features in sensor signals after feature extraction.

To demonstrate the validity of our T-DAD framework on other datasets, we
present the performance comparison between T-DAD and three single-stage
benchmark approaches (including DAE, DAGMM, and USAD) in Table IV, where the
DAE and LSTM-DAE models are used in Stages I and II, respectively, of our
T-DAD framework. We do not adopt the other two benchmark methods (i.e., LSTM-
DAE and LSTM-VAE) since they do not perform appropriately similarly as in
Section VF2). It is clear to see that the results in the table follow similar
trends to those in Table II. On both assembly process datasets, T-DAD shows
superior performance over single-stage benchmark approaches, which
demonstrates that our T-TAD framework is robust to other assembly process
datasets.

##  VI Concluding Remarks

In this paper, we introduced a novel two-stage framework, termed T-DAD, for
automatically detecting anomalies using a real-world manufacturing dataset,
collected from an assembly line of automotive engine valves, containing many
challenging problems. More precisely, we developed our T-DAD framework in such
a way that signal-specific detection tasks are carried out in a data-driven
manner after unsupervisedly training a model in stage I with operation cycle
signals and another model in stage II with sensor signals. Through
comprehensive experiments, we demonstrated that the proposed framework not
only substantially outperforms various single-stage benchmark methods but also
is robust to fringe situations. Moreover, we found that the gains over single-
stage benchmark methods come from our two-stage detection strategy depending
on types of signals by judiciously exploiting sensor signals as a secondary
tool in further improving the performance. Potential avenues of future
research include how to solve a general setting where we undergo more
heterogeneity along with more than three types of signals in highly
sophisticated real-world manufacturing. Early detection of anomalies for
preventative maintenance also remains for future work.

## References

  * [1] A. Khakbaz and E. Babaee Tirkolaee, “A sustainable hybrid manufacturing/remanufacturing system with two-way substitution and WEEE directive under different market conditions,” _Optimization_ , pp. 1–24, 2021.
  * [2] J. Wan, S. Tang, D. Li, S. Wang, C. Liu, H. Abbas, and A. V. Vasilakos, “A manufacturing big data solution for active preventive maintenance,” _IEEE Trans. Ind. Informat._ , vol. 13, no. 4, pp. 2039–2047, Aug. 2017.
  * [3] V. Chandola, A. Banerjee, and V. Kumar, “Anomaly detection: A survey,” _ACM Computing Surveys (CSUR)_ , vol. 41, no. 3, pp. 1–58, Jul. 2009.
  * [4] J. Yang, C. Zhou, S. Yang, H. Xu, and B. Hu, “Anomaly detection based on zone partition for security protection of industrial cyber-physical systems,” _IEEE Trans. Ind. Electron._ , vol. 65, no. 5, pp. 4257–4267, May 2018.
  * [5] W. Yu, T. Dillon, F. Mostafa, W. Rahayu, and Y. Liu, “A global manufacturing big data ecosystem for fault detection in predictive maintenance,” _IEEE Trans. Ind. Informat._ , vol. 16, no. 1, pp. 183–192, Jan. 2020.
  * [6] T. Ince, S. Kiranyaz, L. Eren, M. Askar, and M. Gabbouj, “Real-time motor fault detection by 1-D convolutional neural networks,” _IEEE Trans. Ind. Informat._ , vol. 63, no. 11, pp. 7067–7075, Nov. 2016.
  * [7] J. Liu, F. Qu, X. Hong, and H. Zhang, “A small-sample wind turbine fault detection method with synthetic fault data using generative adversarial nets,” _IEEE Trans. Ind. Informat._ , vol. 15, no. 7, pp. 3877–3888, Jul. 2019.
  * [8] J. Wu, Z. Zhao, C. Sun, R. Yan, and X. Chen, “Fault-attention generative probabilistic adversarial autoencoder for machine anomaly detection,” _IEEE Trans. Ind. Informat._ , vol. 16, no. 12, pp. 7479–7488, Dec. 2020.
  * [9] L. Li, K. Ota, and M. Dong, “Deep learning for smart industry: Efficient manufacture inspection system with fog computing,” _IEEE Trans. Ind. Informat._ , vol. 14, no. 10, pp. 4665–4673, Oct. 2018.
  * [10] X. Wang, J. Lin, N. Patel, and M. Braun, “A self-learning and online algorithm for time series anomaly detection, with application in CPU manufacturing,” in _Proc. ACM Int. Conf. Inf. Knowl. Manage. (CIKM)_ , Indianapolis, IN, Oct. 2016, pp. 1823–1832.
  * [11] T. Nakazawa and D. V. Kulkarni, “Anomaly detection and segmentation for wafer defect patterns using deep convolutional encoder–decoder neural network architectures in semiconductor manufacturing,” _IEEE Trans. Semicond. Manuf._ , vol. 32, no. 2, pp. 250–256, May 2019.
  * [12] C. Yin, S. Zhang, J. Wang, and N. N. Xiong, “Anomaly detection based on convolutional recurrent autoencoder for IoT time series,” _IEEE Trans. Syst., Man, Cybern., Syst._ , Feb. 2020.
  * [13] P. Malhotra, A. Ramakrishnan, G. Anand, L. Vig, P. Agarwal, and G. Shroff, “LSTM-based encoder-decoder for multi-sensor anomaly detection,” in _Proc. Int. Conf. Mach. Learn. (ICML) Anomaly Detect. Workshop_ , New York, NY, Jul. 2016, pp. 1–5.
  * [14] D. Wu, Z. Jiang, X. Xie, X. Wei, W. Yu, and R. Li, “LSTM learning with Bayesian and Gaussian processing for anomaly detection in industrial IoT,” _IEEE Trans. Ind. Informat._ , vol. 16, no. 8, pp. 5244–5253, Aug. 2020.
  * [15] J. Audibert, P. Michiardi, F. Guyard, S. Marti, and M. A. Zuluaga, “USAD: Unsupervised anomaly detection on multivariate time series,” in _Proc. ACM SIGKDD Int. Conf. Knowl. Disc. Data Min. (KDD)_ , Virtual Event, Aug. 2020, pp. 3395–3404.
  * [16] S. R. Saufi, Z. A. B. Ahmad, M. S. Leong, and M. H. Lim, “Challenges and opportunities of deep learning models for machinery fault detection and diagnosis: A review,” _IEEE Access_ , vol. 7, pp. 122 644–122 662, Aug. 2019.
  * [17] H. Ren, B. Xu, Y. Wang, C. Yi, C. Huang, X. Kou, T. Xing, M. Yang, J. Tong, and Q. Zhang, “Time-series anomaly detection service at Microsoft,” in _Proc. ACM SIGKDD Int. Conf. Knowl. Disc. Data Min. (KDD)_ , Anchorage, AK, Aug. 2019, pp. 3009–3017.
  * [18] P. Galeano, D. Peña, and R. S. Tsay, “Outlier detection in multivariate time series by projection pursuit,” _J. Amer. Statist. Assoc._ , vol. 101, no. 474, pp. 654–669, Jun. 2006.
  * [19] Y. Zhang, Z. Ge, A. Greenberg, and M. Roughan, “Network anomography,” in _Proc. 5th ACM SIGCOMM Internet Measurement Conf. (IMC)_ , Berkeley, CA, Oct. 2005, pp. 317–330.
  * [20] J. Ma and S. Perkins, “Time-series novelty detection using one-class support vector machines,” in _Proc. Int. Joint Conf. Neural Netw. (IJCNN)_ , Portland, OR, Jul. 2003, pp. 1741–1745.
  * [21] T. Fuse and K. Kamiya, “Statistical anomaly detection in human dynamics monitoring using a hierarchical dirichlet process hidden markov model,” _IEEE Trans. Intell. Transp. Syst._ , vol. 18, no. 11, pp. 3083–3092, Nov. 2017.
  * [22] A. G. Tartakovsky, A. S. Polunchenko, and G. Sokolov, “Efficient computer network anomaly detection by changepoint detection methods,” _IEEE J. Sel. Topics Sig. Process._ , vol. 7, no. 1, pp. 4–11, Feb. 2012.
  * [23] S.-E. Benkabou, K. Benabdeslem, V. Kraus, K. Bourhis, and B. Canitia, “Local anomaly detection for multivariate time series by temporal dependency based on poisson model,” _IEEE Trans. Neural Netw. Learn. Syst._ , pp. 1–11, Jun. 2021.
  * [24] K. Hundman, V. Constantinou, C. Laporte, I. Colwell, and T. Soderstrom, “Detecting spacecraft anomalies using lstms and nonparametric dynamic thresholding,” in _Proc. ACM SIGKDD Int. Conf. Knowl. Disc. Data Min. (KDD)_ , London, United Kingdom, Aug. 2018, pp. 387–395.
  * [25] Y. Su, Y. Zhao, C. Niu, R. Liu, W. Sun, and D. Pei, “Robust anomaly detection for multivariate time series through stochastic recurrent neural network,” in _Proc. ACM SIGKDD Int. Conf. Knowl. Disc. Data Min. (KDD)_ , Anchorage, AK, Aug. 2019, pp. 2828–2837.
  * [26] M. Munir, S. A. Siddiqui, A. Dengel, and S. Ahmed, “DeepAnt: A deep learning approach for unsupervised anomaly detection in time series,” _IEEE Access_ , vol. 7, pp. 1991–2005, Jan. 2019.
  * [27] A. Borghesi, A. Bartolini, M. Lombardi, M. Milano, and L. Benini, “Anomaly detection using autoencoders in high performance computing systems,” in _Proc. AAAI Conf. Artif. Intell. (AAAI)_ , Honolulu, HI, Jan. 2019, pp. 9428–9433.
  * [28] C. Zhou and R. C. Paffenroth, “Anomaly detection with robust deep autoencoders,” in _Proc. ACM SIGKDD Int. Conf. Knowl. Disc. Data Min. (KDD)_ , Halifax, Canada, Aug. 2017, pp. 665–674.
  * [29] B. Zong, Q. Song, M. R. Min, W. Cheng, C. Lumezanu, D. Cho, and H. Chen, “Deep autoencoding gaussian mixture model for unsupervised anomaly detection,” in _Proc. Int. Conf. Learn. Rep. (ICLR)_ , Vancouver, Canada, Apr. 2018.
  * [30] H. Xu, W. Chen, N. Zhao, Z. Li, J. Bu, Z. Li, Y. Liu, Y. Zhao, D. Pei, Y. Feng, J. Chen, Z. Wang, and H. Qiao, “Unsupervised anomaly detection via variational auto-encoder for seasonal KPIs in web applications,” in _Proc. World Wide Web Conf. (WWW)_ , Lyon, France, Apr. 2018, pp. 187–196.
  * [31] D. Park, Y. Hoshi, and C. C. Kemp, “A multimodal anomaly detector for robot-assisted feeding using an LSTM-based variational autoencoder,” _IEEE Robot. Autom. Lett._ , vol. 3, no. 3, pp. 1544–1551, 2018.
  * [32] D. Li, D. Chen, B. Jin, L. Shi, J. Goh, and S.-K. Ng, “MAD-GAN: Multivariate anomaly detection for time series data with generative adversarial networks,” in _Proc. Int. Conf. Artif. Nerual Netw. (ICANN)_ , Munich, Germany, Sep. 2019, pp. 703–716.
  * [33] H. Zhao, Y. Wang, J. Duan, C. Huang, D. Cao, Y. Tong, B. Xu, J. Bai, J. Tong, and Q. Zhang, “Multivariate time-series anomaly detection via graph attention network,” in _Proc. IEEE Int. Conf. Data Mining (ICDM)_ , Virtual Event, Sep. 2020, pp. 841–850.
  * [34] A. Deng and B. Hooi, “Graph neural network-based anomaly detection in multivariate time series,” in _Proc. AAAI Conf. Artif. Intell. (AAAI)_ , Virtual Event, May 2021, pp. 4027–4035.
  * [35] Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang, and P. S. Yu, “A comprehensive survey on graph neural networks,” _IEEE Trans. Neural Netw. Learn. Syst._ , vol. 32, no. 1, pp. 4–24, Jan. 2021.
  * [36] J. Wang, F. Yang, T. Chen, and S. L. Shah, “An overview of industrial alarm systems: Main causes for alarm overloading, research status, and open problems,” _IEEE Trans. Autom. Sci. Eng._ , vol. 13, no. 2, pp. 1045–1061, Apr. 2016.
  * [37] P. Goel, A. Datta, and M. S. Mannan, “Industrial alarm systems: Challenges and opportunities,” _J. Loss Prevention Process Ind._ , vol. 50, pp. 23–36, Nov. 2017.
  * [38] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _Proc. Int. Conf. Learn. Rep. (ICLR)_ , San Diego, CA, May 2015, pp. 1–13.
  * [39] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: A simple way to prevent neural networks from overfitting,” _J. Mach. Learn. Res._ , vol. 15, no. 56, pp. 1929–1958, Jun. 2014.
  * [40] N. Tatbul, T. J. Lee, S. Zdonik, M. Alam, and J. Gottschlich, “Precision and recall for time series,” in _Proc. Adv. Neural. Inf. Process. Syst. (NIPS)_ , Montréal, Canada, Dec. 2018, pp. 1924–1934.