2023

[1,2]Seung-Ik Lee

1]Department of Artificial Intelligence, University of Science and Technology,
Daejeon, 34113, South Korea

2]Field Robotics Research Section, Electronics and Telecommunications Research
Institute, Daejeon, 34129, South Korea

3]Interdisciplinary Centre for Security, Reliability and Trust, University of
Luxembourg, Luxembourg, 1855, Luxembourg

4]Department of Computer Vision, Mohamed Bin Zayed University of Artificial
Intelligence, Abu Dhabi, United Arab Emirates

# Exploiting Autoencoder’s Weakness to Generate Pseudo Anomalies

Marcella Astrid<EMAIL_ADDRESS>Muhammad Zaigham Zaheer
<EMAIL_ADDRESS>Djamila Aouada<EMAIL_ADDRESS><EMAIL_ADDRESS>[  [  [  [

###### Abstract

Due to the rare occurrence of anomalous events, a typical approach to anomaly
detection is to train an autoencoder (AE) with normal data only so that it
learns the patterns or representations of the normal training data. At test
time, the trained AE is expected to well reconstruct normal but to poorly
reconstruct anomalous data. However, contrary to the expectation, anomalous
data is often well reconstructed as well. In order to further separate the
reconstruction quality between normal and anomalous data, we propose creating
pseudo anomalies from learned adaptive noise by exploiting the aforementioned
weakness of AE, i.e., reconstructing anomalies too well. The generated noise
is added to the normal data to create pseudo anomalies. Extensive experiments
on Ped2, Avenue, ShanghaiTech, CIFAR-10, and KDDCUP datasets demonstrate the
effectiveness and generic applicability of our approach in improving the
discriminative capability of AEs for anomaly detection.

###### keywords:

anomaly detection, one-class classification, generative model, autoencoder

##  1 Introduction

Anomaly detection has recently garnered significant attention from numerous
researchers due to its pivotal role in various applications, such as automatic
surveillance systems zaheer2020old ; gong2019memorizing ; li2022anomaly ;
astrid2021learning ; sultani2018real , medical data analysis wahid2021nanod ;
abhaya2022rdpod ; vafaeiflexible , network intrusion prevention zavrak2023flow
; shukla2021detection ; saeed2022real . By definition, anomalous events are
rare and can be cumbersome to collect. Therefore, anomaly detection is
commonly approached as a one-class classification (OCC) problem, in which only
normal data is utilized to train a detection model.

Typically, an autoencoder (AE) model is employed to address the OCC problem
luo2017revisit ; luo2017remembering ; gong2019memorizing ; park2020learning ;
astrid2021learning ; astrid2021synthetic ; liu2023csiamese . By learning to
reconstruct the normal training data, an AE encodes representations of
normalcy in its latent space. During testing, the model is expected to
reconstruct normal data well but exhibit poor reconstruction in anomalous
data. However, as observed by munawar2017limiting ; gong2019memorizing ;
zaheer2020old ; astrid2021learning ; astrid2021synthetic , AEs suffer from
reconstructing anomalous data too well, leading to reduced discrimination
between normal and anomalous data, as illustrated in Figure 1(a).

Figure 1: Illustration on how our method limits the reconstruction capability
of an AE across training iterations: (a) AE can reconstruct both normal data
and anomalous data, (b) The noise generator generates a noise $\Delta X$ to
produce pseudo anomalies within the reconstruction boundary of AE, (c) AE
learns to poorly reconstruct pseudo anomalies, (d) Pseudo anomalies are
generated to adapt to the new reconstruction boundary, and (e) AE learns to
poorly reconstruct the new pseudo anomalies.

To address this phenomenon, gong2019memorizing ; park2020learning  employ
memory-based networks to memorize normal patterns in the latent space. This
compels an AE to utilize the memorized latent codes for input reconstruction.
These approaches have proven successful in ensuring poor reconstruction for
anomalous data. However, a potential issue arises in that it may also restrict
the reconstruction of normal data, depending on the memory size, as depicted
in Figure 6 of gong2019memorizing .

Figure 2:  Comparison of pseudo anomaly generation using (a) skipping frames
astrid2021learning ; astrid2021synthetic , (b) patching astrid2021learning ,
and (c) our method. Our method is learnable and does not impose any strong
inductive bias.

To enable an AE to learn more appropriate reconstruction boundary, recently,
astrid2021learning ; astrid2021synthetic  proposed the utilization of pseudo
anomalies to assist the training of an AE. Pseudo anomalies are fake anomalies
generated from normal data in the training set to emulate anomalous data. The
AE is subsequently trained using both normal and pseudo anomalous data. While
the model is trained with normal data to minimize the reconstruction loss, in
the case of pseudo anomalous data, the AE is trained to inadequately
reconstruct the input. Despite these methods outperform memory-based networks,
one notable drawback is that pseudo anomalies are synthesized on the
assumptions, such as the speed of anomalous movements astrid2021learning ;
astrid2021synthetic  (Figure 2(a)) or anomalous objects astrid2021learning
(Figure 2(b)), significantly limiting their applicability.

To avoid making such assumptions and, more importantly, to achieve greater
generality across diverse areas, we propose constructing pseudo anomalies by
incorporating learned adaptive noise to the normal data (Figure 2(c)). To
implement this, we simultaneously train another network that learns to
generate noise based on a normal input. Pseudo anomaly data are subsequently
created by adding the generated noise to the input (Figure 1(b) & Figure
1(d)). An AE is then trained to poorly reconstruct the generated pseudo
anomalies (Figure 1(c) & Figure 1(e)). This approach ensures that the
reconstruction boundary of the AE may evolve towards a significantly improved
one as the training progresses, as illustrated in Figure 1.

There has been a controversy among machine learning researchers regarding
whether it limits the idea of artificial general intelligence
voss2007essentials  to impose a strong inductive bias or in other words, to
make use of strong assumptions such as in astrid2021learning ;
astrid2021synthetic  for an improved performance. Undoubtedly, inductive bias
can be beneficial for highly specific practical anomaly detection solutions.
However, methods relying on such bias can be vulnerable in cases where the
underlying assumptions do not hold. Additionally, their applicability is
extremely limited to specific applications, e.g., video. On the other hand,
our work refrains from employing any strong inductive bias, making it generic
and applicable across diverse domains, from visual (videos and images) to
network intrusion data.

In summary, the contributions of our work are as follows: 1) Our work is among
the first few to explore the possibility of generating pseudo anomalies in
anomaly detection; 2) We leverage the well-known weakness of AE, i.e., their
tendency to reconstruct anomalies too well, to our advantage. We achieve this
by learning to generate pseudo anomalies that can hinder the successful
reconstruction of anomalies; 3) Our noise-based pseudo anomaly generation
assists in training without relying on any strong inductive bias; 4) We
extensively evaluate our method, demonstrating its broad applicability across
highly complex video, image, and network intrusion datasets: Ped2
li2013anomaly , Avenue lu2013abnormal , ShanghaiTech luo2017revisit , CIFAR-10
krizhevsky2009learning , and KDDCUP Dua:2019 .

##  2 Related work

###  2.1 Limiting the reconstruction of AE

To address the issue of AEs reconstructing anomalies too well,
gong2019memorizing ; park2020learning  introduce memory mechanisms into the
latent space to constrain the reconstruction capability of an AE. However,
this approach may inadvertently restrict normal data reconstructions as well.
Another strategy proposed by astrid2021learning ; astrid2021synthetic ;
astrid2023pseudobound  involves utilizing data-heuristic pseudo anomalies.
When pseudo anomalies are input, astrid2021synthetic ; astrid2023pseudobound
aim to maximize the reconstruction loss, while astrid2021learning  minimizes
the reconstruction loss with respect to the normal data used for generating
the pseudo anomalies. Essentially, our training configuration is similar to
that of astrid2021learning , as we also minimize the reconstruction loss with
respect to the normal data. However, we distinguish our method by not
explicitly imposing any inductive bias during pseudo anomaly generation,
thereby extending the applicability of our approach to a broader range of
applications.

###  2.2 Pseudo anomalies

Several previous studies take advantage of inductive bias to generate pseudo
anomalies. astrid2021learning ; astrid2021synthetic  propose a skipping frame
technique that considers the relatively fast movements of objects as
anomalies. Similarly, astrid2021learning ; zhong2022cascade  suggest putting a
patch from another dataset into an image to mimic the existence of anomalous
objects. In addition, to create pseudo anomalies, PseudoBound
astrid2023pseudobound  explores several hand-crafted augmentation techniques,
such as skipping frames, adding patches, and incorporating noise into the
normal data. In contrast, our work does not impose any such inductive bias or
assumption (e.g., fast movements, existence of anomalous objects) nor use
hand-crafted augmentation techniques. Instead, we allow the model to learn to
generate pseudo anomalies based on the reconstruction boundary of AEs.

###  2.3 Adversarial training

From an architectural point-of-view, our method of learning to generate pseudo
anomalies may also be viewed as related to adversarial training
goodfellow2014generative . In this context, OGNet zaheer2020old ;
zaheer2022stabilizing  and G2D pourreza2021g2d  also employ adversarial
training, where an AE serves as the generator and a binary classifier
functions as the discriminator. In the second phase of training, pseudo
anomalies obtained from an undertrained generator in the first phase are used
to train the binary classifier. In contrast, we utilize the AE itself as
discriminator, eliminating the need for a separate discriminator. Furthermore,
unlike existing methods, ours learns to generate pseudo anomalies, thereby
freeing us from the burden of designing any hand-crafted schemes for pseudo
anomalies.

###  2.4 Non-reconstruction methods

Different from our method, non-reconstruction methods do not utilize
reconstruction as the training objective and/or as the sole decision factor of
the anomaly score. Several approaches fall into this category, such as
predicting future frames liu2018future ; park2020learning , utilizing object
detection under the assumption that anomalous events are always related to
objects ionescu2019object ; georgescu2021background , adding optical flow
components ji2020tam ; lee2019bman , and using a binary classifier to predict
anomaly scores zaheer2020old ; pourreza2021g2d . Since some of these methods
utilize AE georgescu2021background ; park2020learning ; liu2018future , our
work can also potentially be incorporated into these methods.

###  2.5 Non-OCC methods

To enhance the discrimination capability of an AE, several researchers
incorporate real anomalies during training munawar2017limiting ;
yamanaka2019autoencoding . Recently, video-level weakly supervised
sultani2018real ; zaheer2020claws ; zaheer2023clustering ; karim2024real ;
majhi2024oe  or fully unsupervised zaheer2022generative  training
configurations have also been introduced. However, due to the limited variety
of anomalous data, these approaches are prone to overfitting and are
restricted to specific anomalous cases appearing in the training data. On the
other hand, our method utilizes only normal training data, offering greater
potential to build a highly generic model capable of working with diverse
anomalous data that may deviate from the normal patterns in the training data.

###  2.6 With vs. without inductive bias

In existing literature, several anomaly detection approaches impose inductive
bias. Object-centric methods ionescu2019detecting ; georgescu2021background
assume that anomalous events are related to objects. However, such methods
encounter difficulties in detecting non-object-based anomalous events, such as
unattended fires or blasts. Several other pseudo-anomaly-based methods also
make strong assumptions, such as fast movements astrid2021learning ;
astrid2021synthetic  or out-of-distribution objects as anomalies
astrid2021learning ; zhong2022cascade . Consequently, their applicability may
be severely limited, for example, in video data. On the other hand, there have
been anomaly detection methods that do not impose inductive bias, such as
memory-based networks gong2019memorizing ; park2020learning , whose
limitations are discussed in Section 2.1.

###  2.7 Denoising AE

Our work is also related to denoising AE salehi2021arae ; jewell2022one ;
vincent2008extracting . However, the usage and purpose of the noise are where
differ. A typical denoising AE is trained with random noise added to the input
to capture robust features and to prevent the network from merely duplicating
the input at the output. In contrast, we learn to generate noise to create
pseudo anomalies.

###  2.8 Data augmentation

Creating pseudo anomalies can be seen as data augmentation since we
incorporate the generated pseudo anomalies during training, and they
contribute to the loss. However, instead of adding data from the same classes
as in the typical data augmentation techniques bengio2011deep ;
krizhevsky2012imagenet , creating pseudo anomalies introduces a new class,
i.e., the anomaly class, into the normal-only training set. Several data
augmentation techniques also employ adversarial training to generate augmented
examples that are adversarial for the model zhang2020adversarial ;
tang2020onlineaugment . In such approaches, the goal is to discover augmented
data that can fool the classification network into predicting the wrong
category. Our method also utilizes adversarial training to generate pseudo
anomalies. However, different from adversarial-based augmentation techniques,
our method generates augmented data that the AE can reconstruct well.

###  2.9 Generation of out-of-distribution data

Our method of generating pseudo anomalies is also related to the generation of
out-of-distribution (OOD) data. In semi-supervised learning dai2017good ;
dong2019margingan , OOD data are utilized to assist in training a classifier
with a small amount of labeled data and a large amount of unlabeled data. Bad
GAN dai2017good  obtains OOD data by employing PixelCNN++
salimans2017pixelcnn++ , an external separate network that predicts data
density. Margin GAN dong2019margingan  proposes generating real-looking fake
data that fools both the discriminator and the classifier. In object
detection, VOS du2022towards  models object features in each class using a
Gaussian distribution and then generate outliers from the Gaussian, allowing
the classifier to learn to recognize objects outside the provided object
classes. In deepfake detection, FaceXRay fxray  and SBI sbis  assume the
existence of mask boundary artifacts in the deepfakes to create OOD data as
pseudo-fake. In contrast, our method does not assume any particular family of
density functions or the appearance of anomalous data to generate OOD samples.
Instead, our model learns to generate OOD data near the reconstruction
boundary without making assumptions about the nature of anomalous data.

Figure 3: Our method consists of a main autoencoder $\mathcal{F}$ and a noise
generator $\mathcal{G}$ that are trained alternately: (a) A pseudo anomaly
instance is constructed by adding noise to the normal data, where the noise is
generated by $\mathcal{G}$. $\mathcal{G}$ learns to generate as much noise as
$\mathcal{F}$ is able to reconstruct the pseudo anomalies. In other words,
$\mathcal{G}$ is trained to generate anomalies (maximizing noise) that are
within the reconstruction boundary of $\mathcal{F}$ (minimizing reconstruction
loss). (b) $\mathcal{F}$ is trained to not reconstruct anomalies when the
inputs are generated pseudo anomalies and trained to reconstruct normal data
when the inputs are normal. During test time, only $\mathcal{F}$ is used. The
contrast of $\Delta X$ has been adjusted for visualization clarity.

##  3 Methodology

In this section, we discuss our proposed approach of learning to generate
pseudo anomalies. The overall configuration is illustrated in Figure 3, where
an autoencoder $\mathcal{F}$ (Figure 3(b)) and a noise generator $\mathcal{G}$
(Figure 3(a)) are trained alternately.

###  3.1 Learning not to reconstruct anomalies ($\mathcal{F}$)

In the OCC setting, an autoencoder (AE) is typically employed as a
reconstruction model hasan2016learning ; zhao2017spatio ; luo2017revisit ;
luo2017remembering ; gong2019memorizing ; park2020learning . To enhance the
discrimination between normal and anomalous reconstruction, we train the AE
$\mathcal{F}$ using pseudo anomaly input $X^{P}$ with a probability $p$ and
normal input $X^{N}$ with a probability $1-p$. The AE then outputs the
reconstruction of normal data and pseudo anomalous data as:

$\hat{X}^{N}=\mathcal{F}(X^{N})\text{,}$ (1)
$\hat{X}^{P}=\mathcal{F}(X^{P})\text{,}$ (2)

respectively.

A pseudo anomaly $X^{P}$ is supposedly anomalous data generated from the
normal training data. It is considered anomalous as it does not conform to the
normalcy defined in the training data. Additionally, it is pseudo as it is not
a real anomaly. In this work, we propose to generate noise $\Delta X$ from a
normal input $X^{N}$ to construct a pseudo anomaly $X^{P}$ by adding $\Delta
X$ to $X^{N}$:

$X^{P}=X^{N}+\Delta X\text{.}$ (3)

To generate $\Delta X$, an additional autoencoder $\mathcal{G}$ is employed
as:

$\Delta X=\mathcal{G}(X^{N})\text{.}$ (4)

The intuition behind generating $X^{P}$ from $X^{N}$ is illustrated in Figure
1(b) & (d).

In order to train $\mathcal{F}$ to well reconstruct normal input while poorly
reconstructing anomalies, we aim to train $\mathcal{F}$ to reconstruct any
given input (whether $X^{N}$ or $X^{P}$) as its normal data $X^{N}$.
Therefore, during normal input, the training involves minimizing the
reconstruction loss using mean squared error between $X^{N}$ and
$\hat{X}^{N}$:

$\min_{\mathcal{F}}\frac{1}{d}\left\|\hat{X}^{N}-X^{N}\right\|_{F}^{2}\text{,}$
(5)

where $\left\|.\right\|_{F}$ denotes Frobenius norm and $d$ is the number of
elements in $X^{N}$. Even when the input is a pseudo anomaly, the
reconstruction loss is calculated with the corresponding normal data $X^{N}$
as the target:

$\min_{\mathcal{F}}\frac{1}{d}\left\|\hat{X}^{P}-X^{N}\right\|_{F}^{2}\text{.}$
(6)

This way, $\mathcal{F}$ is learned not to reconstruct pseudo anomalies, as
illustrated in Figure 1(c) & (e). The training mechanism of $\mathcal{F}$ can
be seen in Figure 3(b).

###  3.2 Learning to generate pseudo anomalies ($\mathcal{G}$)

$\mathcal{G}$ is trained by exploiting what we may term as the weakness of
$\mathcal{F}$ in anomaly detection, i.e., reconstructing too well on anomalous
data. Therefore, we propose a loss consisting of two parts, including
reconstruction and noise amplitude:

$\min_{\mathcal{G}}\frac{1}{d}\left(\left\|\hat{X}^{P}-X^{P}\right\|_{F}^{2}-\lambda\left\|\Delta
X\right\|_{F}^{2}\right)\text{,}$ (7)

where $\lambda$ is a balancing hyperparameter. The first part of the training
objective is to reduce the reconstruction loss of $\mathcal{F}$ for the pseudo
anomalous input. Notice that, different from Eq. (6), the target for the
reconstruction loss in Eq. (7) is $X^{P}$. In this way, $\mathcal{G}$ is
trained to create noise in such a way that the resultant pseudo anomaly is
within the reconstruction boundary of $\mathcal{F}$. On the other hand, the
second part of the loss encourages the norm of the noise $\Delta X$ to be high
which makes the resultant pseudo anomaly to be far from the normal. In
summary, the overall loss encourages $\mathcal{G}$ to generate highly noisy
pseudo anomalies that can be reconstructed by $\mathcal{F}$ (Figure 1(b) &
(d)). The training of $\mathcal{G}$ is illustrated in Figure 3(a).
$\mathcal{G}$ is alternately trained with $\mathcal{F}$ and adapts to the
reconstruction boundary of $\mathcal{F}$, as illustrated in Figure 1. It is
pertinent to note that learning to generate noise for pseudo anomalies does
not impose any inductive bias, which allows our model to have generic
applicability.

###  3.3 Test time

At inference, we compute the anomaly score using the reconstruction error of
$\mathcal{F}$, where a higher reconstruction error corresponds to a higher
anomaly score. Without using $\mathcal{G}$, we directly input the test data
into $\mathcal{F}$ and calculate the reconstruction error by comparing the
output with the input. In this way, our final model does not incur any
additional computational cost compared to the baseline (i.e., AE trained only
with normal data).

Table 1: Default hyperparameter values used in this method for each setup and
dataset.

Probability $p$ |  Weighting factor $\lambda$ | Batch size |  $\mathcal{F}$ learning rate
---|---|---|---
Video | Image | Network | Video | Image | Network | Video | Image | Network | Video | Image | Network
0.5 | 0.1 | 0.5 | 0.1 | 10 | 1 | 4 | 256 | 1024 | $10^{-4}$ | $10^{-3}$ | $10^{-4}$

##  4 Experiments

In this section, we first define our datasets (Section 4.1) and experiment
setup (Section 4.2) used in the experiments. To evaluate the effectiveness of
our approach in generating pseudo anomalies with learnable noise, we
extensively compare the performances and properties between the baseline and
our method in Section 4.3. We then compare the performance and generic
applicability of our method with the state-of-the-art (SOTA) on various
applications in Section 4.4. As an in-depth analysis, we explore the effects
of the hyperparameters introduced in our method and discuss their selection in
Section 4.5, and the design choices and training progression are discussed in
Section 4.6.

###  4.1 Datasets

We evaluate our method on several highly complex datasets including three
surveillance video datasets (i.e., Ped2 li2013anomaly , Avenue lu2013abnormal
, and ShanghaiTech luo2017revisit ) as well as an image dataset CIFAR-10
krizhevsky2009learning . To further show the wide applicability of our
proposed method, we also assess its performance on non-visual datasets such as
the network security dataset, KDDCUP99 Dua:2019 . Details of each dataset used
in our experiments are as follows:

####  4.1.1 Ped2

It comprises 16 training and 12 test videos. The normal scenes consist of
pedestrians only, whereas anomalous scenes include bikes, carts, or
skateboards along with pedestrians.

####  4.1.2 Avenue

It consists of 16 training and 21 test videos. Examples of anomalies are
abnormal objects, such as bikes, and abnormal actions of humans, such as
unusual walking directions, running, or throwing objects.

####  4.1.3 ShanghaiTech

This is by far the largest one-class anomaly detection dataset consisting of
330 training and 107 test videos. The dataset is recorded at 13 different
locations of complex lighting conditions and camera angles. Within the test
videos, there are 130 anomalous events including running, riding bicycle, and
fighting.

####  4.1.4 CIFAR-10

It is originally a 10 classes image classification dataset. For anomaly
detection experiments, we adopt the setup from abati2019latent  and set one
class as normal while the others as anomaly. To train and evaluate our model,
we adhere to the original split of the CIFAR-10 dataset into training and test
sets. However, during training, we exclude the selected anomaly classes.
Similar to abati2019latent , we also separate 10% of the original training
split for validation. We choose the best validation model as our final model
for the test.

####  4.1.5 KDDCUP

Following the setup employed in other methods gong2019memorizing ;
zong2018deep , we designate the “attack” samples in KDDCUP99 10 percent
dataset Dua:2019  as normal data due to the abundance of “attack” samples
compared to “non-attack” data. To create our dataset, 50% of the randomly
selected normal data is used for training, while the remaining is reserved for
test data. Additionally, 50% of the anomalous (“non-attack”) data is allocated
for the test set.

###  4.2 Experiment setup

####  4.2.1 Evaluation criteria

##### Video datasets

We adhere to the widely popular frame-level area under the ROC curve (AUC)
metric zaheer2020old  for Ped2, Avenue, and ShanghaiTech datasets. A higher
AUC value indicates more accurate results. Unless otherwise specified, we
report the maximum AUC achieved over five repeated experiments.

##### Image dataset

Given an experiment setup in which one image category is selected as normal
and the others as anomalous class, AUC is calculated. The setup is repeated
for each of all categories as normal, then the AUC values are averaged for the
final result.

##### Network intrusion dataset

Following the protocol in gong2019memorizing ; zong2018deep , F1, precision,
and recall averaged out of 20 repeated runs are reported. Anomaly class is
considered as positive class.

Table 2: Architectures used in our video datasets experiments. Each number in
the tuple represents time, height, and width dimensions, respectively.

(a) $\mathcal{F}$ architecture. Also used in astrid2021learning .

| Layer | Out Channels | Filter | Stride | Padding | Neg. Slope
---|---|---|---|---|---|---
Encoder | Conv3D | 96 | (3, 3, 3) | (1, 2, 2) | (1, 1, 1) | -
BatchNorm3D | - | - | - | - | -
LeakyReLU | - | - | - | - | 0.2
Conv3D | 128 | (3, 3, 3) | (2, 2, 2) | (1, 1, 1) | -
BatchNorm3D | - | - | - | - | -
LeakyReLU | - | - | - | - | 0.2
Conv3D | 256 | (3, 3, 3) | (2, 2, 2) | (1, 1, 1) | -
BatchNorm3D | - | - | - | - | -
LeakyReLU | - | - | - | - | 0.2
Conv3D | 256 | (3, 3, 3) | (2, 2, 2) | (1, 1, 1) | -
BatchNorm3D | - | - | - | - | -
LeakyReLU | - | - | - | - | 0.2
Decoder | ConvTranspose3D | 256 | (3, 3, 3) | (2, 2, 2) | (1, 1, 1) | -
BatchNorm3D | - | - | - | - | -
LeakyReLU | - | - | - | - | 0.2
ConvTranspose3D | 128 | (3, 3, 3) | (2, 2, 2) | (1, 1, 1) | -
BatchNorm3D | - | - | - | - | -
LeakyReLU | - | - | - | - | 0.2
ConvTranspose3D | 96 | (3, 3, 3) | (2, 2, 2) | (1, 1, 1) | -
BatchNorm3D | - | - | - | - | -
LeakyReLU | - | - | - | - | 0.2
ConvTranspose3D | 1 | (3, 3, 3) | (1, 2, 2) | (1, 1, 1) | -
Tanh | - | - | - | - | -

(b) $\mathcal{G}$ architecture.

| Layer | Out Channels | Filter | Stride | Padding | Neg. Slope
---|---|---|---|---|---|---
Encoder | Conv3D | 96 | (3, 3, 3) | (1, 2, 2) | (1, 1, 1) | -
BatchNorm3D | - | - | - | - | -
LeakyReLU | - | - | - | - | 0.2
Conv3D | 128 | (3, 3, 3) | (2, 2, 2) | (1, 1, 1) | -
BatchNorm3D | - | - | - | - | -
LeakyReLU | - | - | - | - | 0.2
Conv3D | 256 | (3, 3, 3) | (2, 2, 2) | (1, 1, 1) | -
BatchNorm3D | - | - | - | - | -
LeakyReLU | - | - | - | - | 0.2
Decoder | ConvTranspose3D | 128 | (3, 3, 3) | (2, 2, 2) | (1, 1, 1) | -
BatchNorm3D | - | - | - | - | -
LeakyReLU | - | - | - | - | 0.2
ConvTranspose3D | 96 | (3, 3, 3) | (2, 2, 2) | (1, 1, 1) | -
BatchNorm3D | - | - | - | - | -
LeakyReLU | - | - | - | - | 0.2
ConvTranspose3D | 1 | (3, 3, 3) | (1, 2, 2) | (1, 1, 1) | -
Tanh * 2 | - | - | - | - | -

####  4.2.2 Input preprocessing

##### Video datasets

Following gong2019memorizing ; astrid2021learning , all frames are converted
to grayscale images and then resized to $256\times 256$. The pixel values
ranging from 0 to 255 are normalized into a range from -1 to 1. We take a
segment of 16 frames as an input to our model. Therefore, each input is of
size $16\times 1\times 256\times 256$, each corresponds to the number of
frames, channel size, height, and width.

##### Image dataset

Images are resized and normalized from a range from 0 to 255 into a range from
0 to 1. Finally, each input image has a size of $3\times 32\times 32$
respectively corresponding to the channel size, height, and width.

##### Network intrusion dataset

We preprocess each data into features of size 118 dimensions. All of the
features, both in the training and test sets, are normalized with the minimum
and maximum feature values of the training set. Consequently, the feature
values in the training set has a range of 0 to 1 while the test feature values
may not be limited to this range.

####  4.2.3 Hyperparameters and implementation details

On all the datasets, we train both $\mathcal{F}$ and $\mathcal{G}$ using Adam
optimizer kingma2014adam . The learning rate of $\mathcal{G}$ is set to
$10^{-4}$. For other hyperparameter values, refer to Table 1. We also provide
hyperparameter robustness analysis in Section 4.5.

Furthermore, to ensure that $X^{P}$ (generated pseudo anomly) remains
consistent with the input value range of $\mathcal{F}$, e.g., $[-1,1]$ in
video datasets and $[0,1]$ in image dataset, we clip $X^{P}$ (Eq. (3)) into
the same range. To integrate the clipped values into the noise amplitude in
Eq. (7), $\Delta X$ is recalculated as $X^{P}-X^{N}$.

Table 3: Architectures used in our CIFAR10 experiments.

(a) $\mathcal{F}$ architecture.

| Layer | Out Channels | Filter | Stride | Padding
---|---|---|---|---|---
Encoder | Conv2D | 64 | 3 | 2 | 0
BatchNorm2D | - | - | - | -
ReLU | - | - | - | -
Conv2D | 128 | 3 | 2 | 0
BatchNorm2D | - | - | - | -
ReLU | - | - | - | -
Conv2D | 128 | 3 | 2 | 0
BatchNorm2D | - | - | - | -
ReLU | - | - | - | -
Conv2D | 256 | 3 | 2 | 0
BatchNorm2D | - | - | - | -
ReLU | - | - | - | -
Decoder | ConvTranspose2D | 256 | 3 | 2 | 0
BatchNorm2D | - | - | - | -
ReLU | - | - | - | -
ConvTranspose2D | 128 | 3 | 2 | 0
BatchNorm2D | - | - | - | -
ReLU | - | - | - | -
ConvTranspose2D | 128 | 3 | 2 | 0
BatchNorm2D | - | - | - | -
ReLU | - | - | - | -
ConvTranspose2D | 3 | 4 | 2 | 0
Sigmoid | - | - | - | -

(b) $\mathcal{G}$ architecture.

| Layer | Out Channels | Filter | Stride | Padding
---|---|---|---|---|---
Encoder | Conv2D | 64 | 3 | 2 | 0
BatchNorm2D | - | - | - | -
ReLU | - | - | - | -
Conv2D | 128 | 3 | 2 | 0
BatchNorm2D | - | - | - | -
ReLU | - | - | - | -
Conv2D | 128 | 3 | 2 | 0
BatchNorm2D | - | - | - | -
ReLU | - | - | - | -
Decoder | ConvTranspose2D | 128 | 3 | 2 | 0
BatchNorm2D | - | - | - | -
ReLU | - | - | - | -
ConvTranspose2D | 128 | 3 | 2 | 0
BatchNorm2D | - | - | - | -
ReLU | - | - | - | -
ConvTranspose2D | 3 | 4 | 2 | 0
Tanh | - | - | - | -

Table 4: Architectures used in our KDDCUP experiments.

(a) $\mathcal{F}$ architecture. Similar to gong2019memorizing .

| Layer | Out Channels |  | Layer | Out Channels
---|---|---|---|---|---
Encoder | Linear | 60 |  Decoder | Linear | 10
Tanh | - | Tanh | -
Linear | 30 | Linear | 30
Tanh | - | Tanh | -
Linear | 10 | Linear | 60
Tanh | - | Tanh | -
Linear | 3 | Linear | 118
Tanh | - |  |  |

(b) $\mathcal{G}$ architecture.

| Layer | Out Channels |  | Layer | Out Channels
---|---|---|---|---|---
Encoder | Linear | 60 |  Decoder | Linear | 30
Tanh | - | Tanh | -
Linear | 30 | Linear | 60
Tanh | - | Tanh | -
Linear | 10 | Linear | 118
Tanh | - |  |  |

####  4.2.4 Architectures

The neural network architectures used on the video, image, and network
intrusion datasets are provided in Table 2(b), 3(b), and 4(b), respectively.
Encoder of $\mathcal{F}$ produces a latent code of size $T\times C\times
H\times W=2\times 256\times 16\times 16$ for video datasets, $C\times H\times
W=256\times 1\times 1$ for image dataset, and $C=3$ for network intrusion
dataset.

The final layer of $\mathcal{F}$ aligns with the input range. Specifically, a
tanh final layer is applied for video data within the range of $[-1,1]$ (Table
2(b)(a)), a sigmoid final layer for image data within the range of $[0,1]$
(Table 3(b)(a)), and a linear final layer for network intrusion dataset (Table
4(b)(a)). It may also be noted that for bounded data (i.e., video and image),
the final activation function of $\mathcal{G}$ is twice of the corresponding
output layer activation of $\mathcal{F}$ (Table 2(b)(b), 3(b)(b)). For
instance, tanh * 2 of $\mathcal{G}$ for tanh of $\mathcal{F}$. This
configuration allows the generated noise to modify an input from one extreme
value to the other, enabling, for example, a pixel with a value $-1$ to be
changed to $+1$ by a noise value of $+2$.

Figure 4: Visualizations of (a) pseudo anomalies constructed from a normal
frame by adding Gaussian noise with various $\sigma$ values, where the random
noise amplitude is affected by $\sigma$; and (b) noise generated by
$\mathcal{G}$ and the respective pseudo anomalies generated using our proposed
learning to generate pseudo anomalies mechanism across different training
iterations. Compared to the random noise in (a), the noise generated in our
propose mechanism changes with training iterations as $\mathcal{G}$ adapts to
the reconstruction boundary of $\mathcal{F}$.

####  4.2.5 Anomaly score calculation

##### Video datasets

Concurrent to park2020learning ; liu2018future ; astrid2021learning ;
astrid2021synthetic , for a given test video comprising frames
$\\{X_{1},X_{2},...,X_{n}\\}$, we feed $\mathcal{F}$ with a 16-frame input
sequence $\\{X_{t},X_{t+1},...,X_{t+15}\\}$, for $t=1$ to $t=n-15$. We then
compute the Peak Signal-to-Noise Ratio (PSNR) value $\mathcal{P}_{t+8}$
between the 9th frame of the input $X_{t+8}$ and its reconstruction
$\hat{X}_{t+8}$:

$\mathcal{P}_{t+8}=10\text{
log}_{10}\frac{M_{\hat{X}_{t+8}}^{2}}{\frac{1}{R}\left\|\hat{X}_{t+8}-X_{t+8}\right\|_{F}^{2}}\text{,}$
(8)

where $R$ represents the total number of pixels in $\hat{X}_{t+8}$, and
$M_{\hat{X}_{t+8}}$ is the maximum possible pixel value of $\hat{X}_{t+8}$,
i.e., $M_{\hat{X}_{t+8}}=1$ since the video frames are normalized to $[-1,1]$.
Min-max normalization is applied on the PSNR values across $t=1$ to $t=n-15$
of a test video to obtain the normalcy score $\mathcal{Q}_{t+8}$ within the
range of $[0,1]$. Finally, we calculate the anomaly score $\mathcal{A}_{t+8}$
for each input sequence as:

$\mathcal{A}_{t+8}=1-\mathcal{Q}_{t+8}\text{.}$ (9)

Table 5: Ablation studies comparing $\mathcal{F}$ trained without pseudo
anomalies, $\mathcal{F}$ trained with non-learnable Gaussian noise, and
$\mathcal{F}$ trained with learnable noise. The best performances are marked
as bold.

Pseudo anomaly | Ped2 | Avenue | ShanghaiTech | CIFAR-10 | KDDCUP
---|---|---|---|---|---
AUC | AUC | AUC | AUC | F1 | Precision | Recall
None (baseline) | 92.49 | 81.47 | 71.28 | 60.10 | 94.53 | 93.94 | 95.13
Gaussian noise $\sigma=0.1$ | 93.32 | 81.56 | 71.24 | 63.20 | 94.52 | 93.78 | 95.28
Gaussian noise $\sigma=0.5$ | 93.12 | 82.10 | 71.73 | 61.80 | 94.78 | 94.04 | 95.53
Gaussian noise $\sigma=1$ | 93.03 | 82.09 | 71.92 | 61.91 | 95.52 | 94.78 | 96.27
Learnable noise (ours) | 94.57 | 83.23 | 73.23 | 67.76 | 95.58 | 94.84 | 96.34

##### Image dataset

Given a test input image $X$ and its reconstruction $\hat{X}$, we calculate
the anomaly score using the reconstruction loss as:

$\mathcal{A}=\left\|\hat{X}-X\right\|_{F}^{2}\text{.}$ (10)

We then min-max normalize $\mathcal{A}$ across the test data to get scores
ranging from $0$ to $1$.

##### Network intrusion dataset

We first calculate the anomaly score $\mathcal{A}$ similarly to the
calculation on image dataset. However, as the evaluation metrics (i.e., F1,
precision, recall) on the network intrusion dataset necessitate a threshold to
distinguish normal from anomaly, unlike AUC for the image dataset, we select
the predictions with the top 20% anomaly scores across the test set as
anomaly/positive, while the rest as normal/negative, following the procedure
outlined in zong2018deep .

###  4.3 Ablation studies

To evaluate the effectiveness of utilizing learnable noise to generate pseudo
anomalies, we compare our method with a model wherein the main autoencoder
$\mathcal{F}$ is trained solely on normal data (baseline) and another model
wherein $\mathcal{F}$ is trained with non-learnable noise-based pseudo
anomalies. Training using only normal data is equivalent to setting the
probability $p$ to $0$. For experiments involving non-learnable noise-based
pseudo anomalies, we generate these pseudo anomalies by adding non-learnable
Gaussian noise augmentation to the normal data $X^{N}$. The noise at the
$i$-th position $\Delta X_{i}$ (Eq. 3) is defined as:

$\Delta X_{i}=\mathcal{N}(0,\sigma)\text{,}$ (11)

where $\mathcal{N}(0,\sigma)$ is Gaussian noise with mean $0$ and standard
deviation $\sigma$. Examples of pseudo anomalies generated with different
$\sigma$ values can be seen in Figure 4(a).

Figure 5: The distribution of reconstruction errors of the baseline and our
model for normal (blue) and anomalous (red) data in several videos. It is
evident that the reconstruction error distribution becomes more discriminative
with our model compared to the baseline. Table 6: Mean anomaly scores of
anomalous data ($\mu^{\mathcal{A}}_{A}$) and normal data
($\mu^{\mathcal{A}}_{N}$). A higher difference
$\mu^{\mathcal{A}}_{A}-\mu^{\mathcal{A}}_{N}$ (highlighted in bold) indicates
better discrimination between anomalous and normal data.

| Baseline | Ours
---|---|---
Dataset | $\mu^{\mathcal{A}}_{A}$ | $\mu^{\mathcal{A}}_{N}$ | $\mu^{\mathcal{A}}_{A}-\mu^{\mathcal{A}}_{N}$ | $\mu^{\mathcal{A}}_{A}$ | $\mu^{\mathcal{A}}_{N}$ | $\mu^{\mathcal{A}}_{A}-\mu^{\mathcal{A}}_{N}$
Ped2 | 0.6150 | 0.1929 | 0.4220 | 0.6169 | 0.1620 | 0.4549
Avenue | 0.5122 | 0.2351 | 0.2771 | 0.5230 | 0.2345 | 0.2885
ShanghaiTech | 0.5686 | 0.3763 | 0.1923 | 0.5692 | 0.3753 | 0.1939
CIFAR-10 | 0.7416 | 0.7133 | 0.0282 | 0.7765 | 0.7125 | 0.0640
KDDCUP | 0.9998 | 0.9913 | 0.0084 | 0.9997 | 0.9866 | 0.0131

The comparisons between $\mathcal{F}$ trained without pseudo anomalies,
$\mathcal{F}$ trained with Gaussian noise with various $\sigma$ values, and
$\mathcal{F}$ trained with our learnable noise mechanism can be seen in Table
5. As seen, adding Gaussian noise to create pseudo anomalies can generally
improve the model’s performance, highlighting the importance of noise-based
pseudo anomalies. However, using learnable noise can further enhance the
model’s performances which shows the significance of our proposed learning
component. We can also observe in Figure 4(b) that noise generation progresses
as the training goes. This learning contributes to the performance gain and
high discrimination capability for our anomaly detection model that utilizes
learnable noise over the non-learnable Gaussian noise.

To gain further insights into how our pseudo anomalies affect the
reconstruction capability of the model on normal and anomalous data, Figure 5
displays the distribution of reconstruction errors on several videos. As
observed, our model exhibits a more prominent separation between the normal
and anomalous data distributions compared to the baseline, indicating a better
discrimination capability. It is worth noting that what is important is the
reconstruction error distribution difference between the normal and anomalous
data, rather than the magnitude of the reconstruction errors. Similarly, as
for the anomaly score reported in Table 6, our model observes a higher
difference between anomalous and normal data scores indicating a better
discrimination capability.

###  4.4 Comparisons with SOTA

To evaluate the performance and generic applicability of our method, in Table
4.4, we compare our approach with other state-of-the-arts (SOTA) methods on
video datasets (i.e., Ped2, Avenue, and ShanghaiTech), an image dataset (i.e.,
CIFAR-10), and a network intrusion dataset (i.e., KDDCUP). Following
gong2019memorizing , we categorize the methods into Reconstruction and Non-
Reconstruction. Our method belongs to reconstruction-based methods which use
reconstruction quality to measure anomaly scores.

While most SOTA methods demonstrate superior performance on video datasets,
they often introduce inductive biases, such as assuming movement during
anomalous events or the presence of anomalous objects. Consequently, these
methods are not applicable to all three types of task (see results with
‘$\times$’ marks in Table 4.4). Specifically, skip frame-based methods
astrid2021learning ; astrid2021synthetic ; astrid2023pseudobound ;
georgescu2021anomaly , methods altering the order of frames astrid2022limiting
; georgescu2021anomaly , those strictly employing Recurrent Neural Network
(RNN), Long Short-Term Memory (LSTM), or 3D-convolution lee2019bman ;
zhao2017spatio ; luo2017remembering ; luo2017revisit , prediction-based
methods predicting missing frames georgescu2021anomaly ; lu2020few ;
liu2018future ; park2020learning ; zhao2017spatio , and methods requiring
optical flow wang2023memory ; vu2019robust ; georgescu2021background ;
liu2018future ; hasan2016learning  are incompatible with data without
movements, such as image and network intrusion data. Furthermore, methods
relying on an object detector georgescu2021background ; georgescu2021anomaly ;
sun2023hierarchical  are dependent on object properties available only in
video datasets. Patch-based methods, which inject patches from another dataset
to synthesize pseudo anomalies astrid2021learning ; astrid2021synthetic  may
work in both image and video domains but are not suitable for the network
intrusion dataset. Additionally, their performances on image datasets remain
undetermined as the results have not been reported.

Several others that do not impose the aforementioned limitations, such as
OGNet zaheer2020old , Pseudobound-Noise astrid2023pseudobound , and MNAD-
Reconstruction park2020learning , theoretically can be generic and work with
non-video domains, however nothing has been reported about their performance
on non-video datasets. One exceptional SOTA model is a memory-based method,
MemAE gong2019memorizing , that is evaluated on all video, image, and network
intrusion datasets. In comparison, our method outperforms MemAE on video and
image datasets while competitive on the KDDCUP network intrusion dataset. This
underscores the generic applicability and superiority of our approach compared
to other SOTA methods.

Table 7: Performance comparisons of our approach and the existing state-of-
the-art methods on video datasets (i.e., Ped2 li2013anomaly , Avenue
lu2013abnormal , and ShanghaiTech luo2017revisit ), image dataset (CIFAR-10
krizhevsky2009learning ), and network intrusion dataset (i.e., KDDCUP Dua:2019
). Best and second best in each category and dataset are marked as bold and
underlined. Results that are impossible to obtain due to inductive bias are
marked as ‘$\times$’, whereas results that are not reported in the original
paper but in principle possible to get are marked as ‘-’.

Methods | Ped2 | Avenue | ShanghaiTech | CIFAR-10 | KDDCUP
---|---|---|---|---|---
AUC | AUC | AUC | AUC | F1 | Precision | Recall
Non-Reconstruction | MAAM-Net wang2023memory  | 97.7 | 90.9 | 71.3 | $\times$ | $\times$ | $\times$ | $\times$
BMAN lee2019bman  | 96.6 | 90.0 | 76.2 | $\times$ | $\times$ | $\times$ | $\times$
Vu et al.vu2019robust  | 99.21 | 71.54 | - | $\times$ | $\times$ | $\times$ | $\times$
OGNet zaheer2020old  | 98.1 | - | - | - | - | - | -
Georgescu et al.georgescu2021background  | 98.7 | 92.3 | 82.7 | $\times$ | $\times$ | $\times$ | $\times$
Georgescu et al.georgescu2021anomaly  | 99.8 | 92.8 | 90.2 | $\times$ | $\times$ | $\times$ | $\times$
HSC sun2023hierarchical  | 98.1 | 93.7 | 83.4 | $\times$ | $\times$ | $\times$ | $\times$
Frame-Pred liu2018future  | 95.4 | 85.1 | 72.8 | $\times$ | $\times$ | $\times$ | $\times$
Lu et al.lu2020few  | 96.2 | 85.8 | 77.9 | $\times$ | $\times$ | $\times$ | $\times$
MNAD-Prediction park2020learning  | 97.0 | 88.5 | 70.5 | $\times$ | $\times$ | $\times$ | $\times$
Reconstruction | AE-Conv2D hasan2016learning  | 90.0 | 70.2 | 60.85 | $\times$ | $\times$ | $\times$ | $\times$
AE-Conv3D zhao2017spatio  | 91.2 | 71.1 | - | $\times$ | $\times$ | $\times$ | $\times$
AE-ConvLSTM luo2017remembering  | 88.10 | 77.00 | - | $\times$ | $\times$ | $\times$ | $\times$
TSC luo2017revisit  | 91.03 | 80.56 | 67.94 | $\times$ | $\times$ | $\times$ | $\times$
StackRNN luo2017revisit  | 92.21 | 81.71 | 68.00 | $\times$ | $\times$ | $\times$ | $\times$
MemAE gong2019memorizing  | 94.1 | 83.3 | 71.2 | 60.88 | 96.41 | 96.27 | 96.55
MNAD-Reconstruction park2020learning  | 90.2 | 82.8 | 69.8 | - | - | - | -
Astrid et al.astrid2022limiting  | 93.46 | 81.78 | - | $\times$ | $\times$ | $\times$ | $\times$
PseudoBound-Noise astrid2023pseudobound  | 97.78 | 82.11 | 72.02 | - | - | - | -
PseudoBound-Patch astrid2023pseudobound  | 95.33 | 85.36 | 72.77 | - | $\times$ | $\times$ | $\times$
STEAL Net astrid2021synthetic ; astrid2023pseudobound  | 98.44 | 87.10 | 73.66 | $\times$ | $\times$ | $\times$ | $\times$
LNTRA-Patch astrid2021learning  | 94.77 | 84.91 | 72.46 | - | $\times$ | $\times$ | $\times$
LNTRA-Skip frame astrid2021learning  | 96.50 | 84.67 | 75.97 | $\times$ | $\times$ | $\times$ | $\times$
2-9 | Baseline | 92.49 | 81.47 | 71.28 | 60.10 | 94.53 | 93.94 | 95.13
| Ours | 94.57 | 83.23 | 73.23 | 67.76 | 95.58 | 94.84 | 96.34

###  4.5 Hyperparameter evaluation

In this work, we introduce two new hyperparameters, $p$ and $\lambda$. Here,
$p$ represents the probability of using pseudo anomaly (Eq. (2)), while
$\lambda$ is the weighting factor that controls the amount of noise relative
to generating pseudo anomaly within the reconstruction boundary (Eq. (7)).
This section delves into the effects of hyperparameters values on the model’s
performance, specifically on Ped2, CIFAR-10, and KDDCUP. On Ped2 and KDDCUP,
we repeat each setup five and twenty times respectively and report the average
and maximum performance. For CIFAR-10, we conduct the experiment once and
report the AUC averaged out of the ten normal classes setup (refer to Section
4.1).

####  4.5.1 Pseudo anomaly ratio $p$

Figure 6 shows the results on different values of $p$ on Ped2, CIFAR-10, and
KDDCUP. The baseline is equivalent to $p=0$. It can be observed that any $p$
values other than zero outperforms the baseline demonstrating that our model
is robust to diverse $p$ values.

Figure 6: Probability $p$ hyperparameter evaluation on (a) Ped2, (b) CIFAR-10,
and (c) KDDCUP. The baseline (training without pseudo anomaly) is equivalent
to $p=0$. Our approach is robust across different $p$ values, shown by better
performances against the baseline.  Figure 7: Weighting factor $\lambda$
hyperparameter evaluation on (a) Ped2, (b) CIFAR-10, and (c) KDDCUP. In the
case of Ped2 and KDDCUP, too high $\lambda$ values can be harmful to the
model. For CIFAR-10, while higher $\lambda$ values are superior, smaller
values are also acceptable (i.e., better than the baseline). Therefore,
setting a smaller $\lambda$ value is relatively robust to any problems. Figure
8: Visualization of pseudo anomalies generated using different $\lambda$
values on (a) Ped2 and (b) CIFAR-10 (‘cat’ class as the normal class). Higher
$\lambda$ values produce pseudo anomalies quite different from the normal
data. Figure 9: During test time, anomaly scores are obtained from the
reconstruction loss, meaning high loss values correspond to high anomaly
scores. (a) As our default configuration, we utilize only the trained AE
(either with input-level or feature-level pseudo anomalies). When the noise
generator $\mathcal{G}$ is incorporated into the test pipeline, then we may
have two options in computing the reconstruction loss: (b) with the clean
input or (c) noisy input.

####  4.5.2 Noise weighting factor $\lambda$

In Eq. (7), a higher noise weighting factor $\lambda$ value gives more
importance to noise in relative to generating pseudo anomalies with low
reconstruction error of the AE. Figure 7 shows the results for different
values of $\lambda$ on Ped2, CIFAR-10, and KDDCUP. As seen in Figure 7(a) &
(c), putting too much importance on high noise can be harmful to the model, as
observed in our experiments on Ped2 and KDDCUP. Visualization of pseudo
anomalies generated by the model at the end of training on Ped2 can be seen in
Figure 8(a), which shows that when $\lambda\geq 0.2$, highly noisy pseudo
anomalies are generated. However, unlike the video and network intrusion
datasets, stronger noise works better in case of CIFAR-10 dataset, as seen in
Figure 7(b). This may be attributed to the characteristics of the image
anomalies that are far from normal data distribution, i.e., image anomalies
are different image categories from normal ones. As such, as visualized in
Figure 8(b), highly distorted pseudo anomalies are learned to help the model
to detect images of categories different from the normal category.
Nevertheless, as seen in 7(b), using smaller noise also works comparably
better than the baseline. Therefore, from this experiment, we can conclude
that setting smaller $\lambda$ values, e.g., $0<\lambda\leq 0.1$, are robust
across different types of datasets such as video, image, or network intrusion,
demonstrating the higher importance of generating pseudo anomalies inside the
reconstruction boundary in comparison to generating pseudo anomalies way far
away from the normal data for generic applicability.

Table 8: Comparisons of various test setups. Without $\mathcal{G}$, With
$\mathcal{G}$ (loss target: $X$), and With $\mathcal{G}$ (loss target:
$X^{\prime}$) are illustrated in Figure 9(a), (b), and (c), respectively.

Test method | Ped2 | Avenue | ShanghaiTech | CIFAR-10 | KDDCUP
---|---|---|---|---|---
AUC | AUC | AUC | AUC | F1 | Precision | Recall
Without $\mathcal{G}$ | 94.57 | 83.23 | 73.22 | 67.76 | 95.58 | 94.84 | 96.34
With $\mathcal{G}$ (loss target: $X$) | 91.47 | 79.99 | 65.05 | 59.69 | 33.70 | 33.11 | 33.63
With $\mathcal{G}$ (loss target: $X^{\prime}$) | 91.24 | 77.94 | 68.67 | 39.69 | 28.44 | 19.25 | 68.28

###  4.6 Additional discussions

In this subsection, we discuss our design choice of not using generator during
test time (Section 4.6.1) and provide comparison with adversarial denoising AE
in the term of training progression (Section 4.6.2).

####  4.6.1 Is generator necessary at test time?

In our default configuration, we use only $\mathcal{F}$ during test time
(Figure 9(a)). $\mathcal{G}$ is used only at training to generate pseudo
anomalies. However, one may wonder how things will go when the trained
$\mathcal{G}$ is incorporated into the test pipeline. As seen in Figure 9(b)
and (c), we put the trained $\mathcal{G}$ and feed the input to $\mathcal{G}$
to create noisy input $X^{\prime}$ that will be given to $\mathcal{F}$. In
this setting, we can have two options on calculating the reconstruction error:
with clean input $X$ (Figure 9(b)) or noisy input $X^{\prime}$ (Figure 9(c)).

AUC comparisons on Ped2, Avenue, ShanghaiTech, and CIFAR-10 of different test
configurations can be seen in Table 8. Testing without $\mathcal{G}$
consistently achieves the highest performances, supporting that the trained
$\mathcal{F}$ based on our approach is capable of well-reconstructing normal
data while poorly-reconstructing anomalous data. Testing with $\mathcal{G}$ is
not only harmful to the model performance but also add more inference time.

Figure 10: Training progression on Ped2 dataset seen in (a) $\mathcal{F}$ loss
(Eq. (5) & (6)) and $\mathcal{G}$ loss (Eq. (7)), (b) generated noise norm,
(c) AUC, and (d) difference of mean anomaly scores between anomalous and
normal data ($\mu^{\mathcal{A}}_{A}-\mu^{\mathcal{A}}_{N}$). Stable
progression during the training shows the stability of our approach in
enhancing the reconstruction boundary of the AE.

####  4.6.2 Training progression and comparisons with adversarial denoising AE

Despite the architectural similarity to adversarial denoising autoencoder
approaches salehi2021arae ; jewell2022one , our method is rather a cooperative
learning between the generator $\mathcal{G}$ and $\mathcal{F}$
(‘discriminator’), which can be supported by the mutual loss decrease and
convergence (Figure 10 (a)). Additionally, the convergence can also be seen in
the AUC trend over the training epochs in Figure 10(c). As our method can
converge in a cooperative way, it is evidently more stable compared to the
adversarial training methods salehi2021arae ; jewell2022one  in which the loss
tends to fluctuate due to adversarial training objectives and therefore
difficult to converge. Furthermore, there is a hint of instability in
jewell2022one  due to its delicate selection of L1/L2 loss and hyperparameters
while not providing any hyperparameter sensitivity evaluation. Moreover, both
these methods only report the results on simple datasets and their performance
on highly complex datasets such as ShanghaiTech is unknown.

The cooperation between $\mathcal{G}$ and $\mathcal{F}$ throughout training
can also be observed in Figure 4(b). At the beginning, $\mathcal{G}$ randomly
generates the noise. As $\mathcal{F}$ starts to learn to reconstruct noise-
free normal, $\mathcal{G}$ then starts to generate noises around moving
objects, where there are many movements so that $\mathcal{F}$ cannot easily
remove the noise. In this way, $\mathcal{G}$ presents diverse (i.e., from high
to low as seen in Figure 10(b)) noises to $\mathcal{F}$ across the training
iterations and $\mathcal{F}$ finally learns to reconstruct normal regardless
of all these kinds of variations given from $\mathcal{G}$.

We can also observe the enhancement of the reconstruction boundary in Figure
10(c) and (d). The rising AUC in Figure 10(c) indicates the increased
discriminative ability of the AE between normal and anomalous data, reflecting
improved reconstruction boundaries. This improvement is further substantiated
by the growing disparity between the mean anomaly score of anomalous data,
denoted as $\mu^{\mathcal{A}}_{A}$, and the mean anomaly score of normal data,
denoted as $\mu^{\mathcal{A}}_{N}$, as depicted in Figure 10(d).

##  5 Conclusion

In this work, we introduced a novel approach for generating pseudo anomalies
by incorporating noise into the input without imposing inductive bias. Rather
than using a non-learnable noise to generate pseudo anomalies, we proposed to
utilize an additional autoencoder that learns to generate this noise. We
provided ablation studies and evaluations using Ped2, Avenue, ShanghaiTech,
CIFAR-10, KDDCUP datasets to demonstrate the importance of the noise-based
pseudo anomalies and the training mechanism to generate noise. Even without
inductive bias, our approach demonstrated superiority and generic
applicability to diverse application domains spanning videos, images, and
network intrusion while achieving comparable performance to the existing
state-of-the-art methods.

## Acknowledgments

This work was supported by Institute of Information & communications
Technology Planning & Evaluation (IITP) grant funded by the Korea government
(MSIT) (RS-2023-00215760, Guide Dog: Development of Navigation AI Technology
of a Guidance Robot for the Visually Impaired Person (90%) and by the
Luxembourg National Research Fund (FNR) under the project
BRIDGES2021/IS/16353350/FaKeDeTeR (10%).

## Data availability

The data that support the findings of this study are publicly available: Ped2
li2013anomaly  at http://www.svcl.ucsd.edu/projects/anomaly/dataset.htm;
Avenue lu2013abnormal  at
http://www.cse.cuhk.edu.hk/leojia/projects/detectabnormal/dataset.html;
ShanghaiTech luo2017revisit  at https://svip-
lab.github.io/dataset/campus_dataset.html; CIFAR-10 krizhevsky2009learning  at
https://www.cs.toronto.edu/~kriz/cifar.html; and KDDCUP99 Dua:2019  at
http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html.

## Conflict of interest

Marcella Astrid and Muhammad Zaigham Zaheer were recently affiliated and
having a close collaboration with the University of Science and Technology,
South Korea and the Electronics and Telecommunications Research Institute,
South Korea. Marcella Astrid and Djamila Aouada are currently employed at the
University of Luxembourg, Luxembourg. Muhammad Zaigham Zaheer is currently
employed at the Mohamed Bin Zayed University of Artificial Intelligence,
United Arab Emirates. Seung-Ik Lee is currently employed at the University of
Science and Technology, South Korea and the Electronics and Telecommunications
Research Institute, South Korea.

## References

  *   * (1) Zaheer, M.Z., Lee, J.-h., Astrid, M., Lee, S.-I.: Old is gold: Redefining the adversarially learned one-class classifier training paradigm. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14183–14193 (2020)
  * (2) Gong, D., Liu, L., Le, V., Saha, B., Mansour, M.R., Venkatesh, S., Hengel, A.v.d.: Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1705–1714 (2019)
  * (3) Li, S., Cheng, Y., Tian, Y., Liu, Y.: Anomaly detection based on superpixels in videos. Neural Computing and Applications 34(15), 12617–12631 (2022)
  * (4) Astrid, M., Zaheer, M.Z., Lee, J.-Y., Lee, S.-I.: Learning not to reconstruct anomalies. In: British Machine Vision Conference (2021)
  * (5) Sultani, W., Chen, C., Shah, M.: Real-world anomaly detection in surveillance videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6479–6488 (2018)
  * (6) Wahid, A., Annavarapu, C.S.R.: Nanod: A natural neighbour-based outlier detection algorithm. Neural Computing and Applications 33, 2107–2123 (2021)
  * (7) Abhaya, A., Patra, B.K.: Rdpod: an unsupervised approach for outlier detection. Neural Computing and Applications 34(2), 1065–1077 (2022)
  * (8) Vafaei Sadr, A., Bassett, B.A., Kunz, M.: A flexible framework for anomaly detection via dimensionality reduction. Neural Computing and Applications, 1–11
  * (9) Zavrak, S., Iskefiyeli, M.: Flow-based intrusion detection on software-defined networks: a multivariate time series anomaly detection approach. Neural Computing and Applications 35(16), 12175–12193 (2023)
  * (10) Shukla, A.K.: Detection of anomaly intrusion utilizing self-adaptive grasshopper optimization algorithm. Neural Computing and Applications 33(13), 7541–7561 (2021)
  * (11) Saeed, M.M.: A real-time adaptive network intrusion detection for streaming data: a hybrid approach. Neural Computing and Applications 34(8), 6227–6240 (2022)
  * (12) Luo, W., Liu, W., Gao, S.: A revisit of sparse coding based anomaly detection in stacked rnn framework. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 341–349 (2017)
  * (13) Luo, W., Liu, W., Gao, S.: Remembering history with convolutional lstm for anomaly detection. In: IEEE International Conference on Multimedia and Expo, pp. 439–444 (2017). IEEE
  * (14) Park, H., Noh, J., Ham, B.: Learning memory-guided normality for anomaly detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14372–14381 (2020)
  * (15) Astrid, M., Zaheer, M.Z., Lee, S.-I.: Synthetic temporal anomaly guided end-to-end video anomaly detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pp. 207–214 (2021)
  * (16) Liu, D., Zhong, S., Lin, L., Zhao, M., Fu, X., Liu, X.: Csiamese: a novel semi-supervised anomaly detection framework for gas turbines via reconstruction similarity. Neural Computing and Applications, 1–25 (2023)
  * (17) Munawar, A., Vinayavekhin, P., De Magistris, G.: Limiting the reconstruction capability of generative neural network using negative learning. In: 2017 IEEE 27th International Workshop on Machine Learning for Signal Processing (MLSP), pp. 1–6 (2017). IEEE
  * (18) Voss, P.: Essentials of general intelligence: The direct path to artificial general intelligence. Artificial general intelligence, 131–157 (2007)
  * (19) Li, W., Mahadevan, V., Vasconcelos, N.: Anomaly detection and localization in crowded scenes. IEEE transactions on pattern analysis and machine intelligence 36(1), 18–32 (2013)
  * (20) Lu, C., Shi, J., Jia, J.: Abnormal event detection at 150 fps in matlab. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2720–2727 (2013)
  * (21) Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009)
  * (22) Dua, D., Graff, C.: UCI Machine Learning Repository (2017). http://archive.ics.uci.edu/ml
  * (23) Astrid, M., Zaheer, M.Z., Lee, S.-I.: Pseudobound: Limiting the anomaly reconstruction capability of one-class classifiers using pseudo anomalies. Neurocomputing 534, 147–160 (2023)
  * (24) Zhong, Y., Chen, X., Jiang, J., Ren, F.: A cascade reconstruction model with generalization ability evaluation for anomaly detection in videos. Pattern Recognition 122, 108336 (2022)
  * (25) Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. Advances in neural information processing systems 27 (2014)
  * (26) Zaheer, M.Z., Lee, J.H., Mahmood, A., Astrid, M., Lee, S.-I.: Stabilizing adversarially learned one-class novelty detection using pseudo anomalies. arXiv preprint arXiv:2203.13716 (2022)
  * (27) Pourreza, M., Mohammadi, B., Khaki, M., Bouindour, S., Snoussi, H., Sabokrou, M.: G2d: Generate to detect anomaly. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2003–2012 (2021)
  * (28) Liu, W., Luo, W., Lian, D., Gao, S.: Future frame prediction for anomaly detection–a new baseline. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6536–6545 (2018)
  * (29) Ionescu, R.T., Khan, F.S., Georgescu, M.-I., Shao, L.: Object-centric auto-encoders and dummy anomalies for abnormal event detection in video. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7842–7851 (2019)
  * (30) Georgescu, M.I., Ionescu, R., Khan, F.S., Popescu, M., Shah, M.: A background-agnostic framework with adversarial training for abnormal event detection in video. IEEE Transactions on Pattern Analysis & Machine Intelligence (01), 1–1 (2021)
  * (31) Ji, X., Li, B., Zhu, Y.: Tam-net: Temporal enhanced appearance-to-motion generative network for video anomaly detection. In: 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1–8 (2020). IEEE
  * (32) Lee, S., Kim, H.G., Ro, Y.M.: Bman: Bidirectional multi-scale aggregation networks for abnormal event detection. IEEE Transactions on Image Processing 29, 2395–2408 (2019)
  * (33) Yamanaka, Y., Iwata, T., Takahashi, H., Yamada, M., Kanai, S.: Autoencoding binary classifiers for supervised anomaly detection. In: Pacific Rim International Conference on Artificial Intelligence, pp. 647–659 (2019). Springer
  * (34) Zaheer, M.Z., Mahmood, A., Astrid, M., Lee, S.-I.: Claws: Clustering assisted weakly supervised learning with normalcy suppression for anomalous event detection. In: Proceedings of the European Conference on Computer Vision (ECCV) (2020)
  * (35) Zaheer, M.Z., Mahmood, A., Astrid, M., Lee, S.-I.: Clustering aided weakly supervised training to detect anomalous events in surveillance videos. IEEE Transactions on Neural Networks and Learning Systems (2023)
  * (36) Karim, H., Doshi, K., Yilmaz, Y.: Real-time weakly supervised video anomaly detection. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 6848–6856 (2024)
  * (37) Majhi, S., Dai, R., Kong, Q., Garattoni, L., Francesca, G., Brémond, F.: Oe-ctst: Outlier-embedded cross temporal scale transformer for weakly-supervised video anomaly detection. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 8574–8583 (2024)
  * (38) Zaheer, M.Z., Mahmood, A., Khan, M.H., Segu, M., Yu, F., Lee, S.-I.: Generative cooperative learning for unsupervised video anomaly detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14744–14754 (2022)
  * (39) Ionescu, R.T., Smeureanu, S., Popescu, M., Alexe, B.: Detecting abnormal events in video using narrowed normality clusters. In: 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1951–1960 (2019). IEEE
  * (40) Salehi, M., Arya, A., Pajoum, B., Otoofi, M., Shaeiri, A., Rohban, M.H., Rabiee, H.R.: Arae: Adversarially robust training of autoencoders improves novelty detection. Neural Networks 144, 726–736 (2021)
  * (41) Jewell, J.T., Khazaie, V.R., Mohsenzadeh, Y.: One-class learned encoder-decoder network with adversarial context masking for novelty detection. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3591–3601 (2022)
  * (42) Vincent, P., Larochelle, H., Bengio, Y., Manzagol, P.-A.: Extracting and composing robust features with denoising autoencoders. In: Proceedings of the 25th International Conference on Machine Learning, pp. 1096–1103 (2008)
  * (43) Bengio, Y., Bastien, F., Bergeron, A., Boulanger–Lewandowski, N., Breuel, T., Chherawala, Y., Cisse, M., Côté, M., Erhan, D., Eustache, J., et al.: Deep learners benefit more from out-of-distribution examples. In: Proceedings of the 14th International Conference on Artificial Intelligence and Statistics, pp. 164–172 (2011). JMLR Workshop and Conference Proceedings
  * (44) Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems 25, 1097–1105 (2012)
  * (45) Zhang, X., Wang, Q., Zhang, J., Zhong, Z.: Adversarial autoaugment. In: International Conference on Learning Representations (2020). https://openreview.net/forum?id=ByxdUySKvS
  * (46) Tang, Z., Gao, Y., Karlinsky, L., Sattigeri, P., Feris, R., Metaxas, D.: Onlineaugment: Online data augmentation with less domain knowledge. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) Computer Vision – ECCV 2020, pp. 313–329. Springer, Cham (2020)
  * (47) Dai, Z., Yang, Z., Yang, F., Cohen, W.W., Salakhutdinov, R.R.: Good semi-supervised learning that requires a bad gan. Advances in neural information processing systems 30 (2017)
  * (48) Dong, J., Lin, T.: Margingan: Adversarial training in semi-supervised learning. Advances in neural information processing systems 32 (2019)
  * (49) Salimans, T., Karpathy, A., Chen, X., Kingma, D.P.: Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv:1701.05517 (2017)
  * (50) Du, X., Wang, Z., Cai, M., Li, S.: Towards unknown-aware learning with virtual outlier synthesis. In: International Conference on Learning Representations (2022). https://openreview.net/forum?id=TW7d65uYu5M
  * (51) Li, L., Bao, J., Zhang, T., Yang, H., Chen, D., Wen, F., Guo, B.: Face x-ray for more general face forgery detection. CoRR abs/1912.13458 (2019) 1912.13458
  * (52) Shiohara, K., Yamasaki, T.: Detecting deepfakes with self-blended images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18720–18729 (2022)
  * (53) Hasan, M., Choi, J., Neumann, J., Roy-Chowdhury, A.K., Davis, L.S.: Learning temporal regularity in video sequences. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 733–742 (2016)
  * (54) Zhao, Y., Deng, B., Shen, C., Liu, Y., Lu, H., Hua, X.-S.: Spatio-temporal autoencoder for video anomaly detection. In: Proceedings of the 25th ACM International Conference on Multimedia, pp. 1933–1941 (2017)
  * (55) Abati, D., Porrello, A., Calderara, S., Cucchiara, R.: Latent space autoregression for novelty detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 481–490 (2019)
  * (56) Zong, B., Song, Q., Min, M.R., Cheng, W., Lumezanu, C., Cho, D., Chen, H.: Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In: International Conference on Learning Representations (2018)
  * (57) Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  * (58) Georgescu, M.-I., Barbalau, A., Ionescu, R.T., Khan, F.S., Popescu, M., Shah, M.: Anomaly detection in video via self-supervised and multi-task learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12742–12752 (2021)
  * (59) Astrid, M., Zaheer, M.Z., Lee, S.-I.: Limiting reconstruction capability of autoencoders using moving backward pseudo anomalies. In: 2022 19th International Conference on Ubiquitous Robots (UR), pp. 248–251 (2022). IEEE
  * (60) Lu, Y., Yu, F., Reddy, M.K.K., Wang, Y.: Few-shot scene-adaptive anomaly detection. In: European Conference on Computer Vision, pp. 125–141 (2020). Springer
  * (61) Wang, L., Tian, J., Zhou, S., Shi, H., Hua, G.: Memory-augmented appearance-motion network for video anomaly detection. Pattern Recognition, 109335 (2023)
  * (62) Vu, H., Nguyen, T.D., Le, T., Luo, W., Phung, D.: Robust anomaly detection in videos using multilevel representations. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 5216–5223 (2019)
  * (63) Sun, S., Gong, X.: Hierarchical semantic contrast for scene-aware video anomaly detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22846–22856 (2023)