# Continual Learning Approaches for Anomaly Detection

Davide Dalle Pezze
University of Padova
<EMAIL_ADDRESS>
&Eugenia Anello
University of Padova
<EMAIL_ADDRESS>
&Chiara Masiero
Statwolf Data Science Srl
<EMAIL_ADDRESS>
&Gian Antonio Susto
University of Padova
<EMAIL_ADDRESS>

###### Abstract

Anomaly Detection is a relevant problem that arises in numerous real-world
applications, especially when dealing with images. However, there has been
little research for this task in the Continual Learning setting. In this work,
we introduce a novel approach called SCALE (SCALing is Enough) to perform
Compressed Replay in a framework for Anomaly Detection in Continual Learning
setting. The proposed technique scales and compresses the original images
using a Super Resolution model which, to the best of our knowledge, is studied
for the first time in the Continual Learning setting. SCALE can achieve a high
level of compression while maintaining a high level of image reconstruction
quality. In conjunction with other Anomaly Detection approaches, it can
achieve optimal results. To validate the proposed approach, we use a real-
world dataset of images with pixel-based anomalies, with the scope to provide
a reliable benchmark for Anomaly Detection in the context of Continual
Learning, serving as a foundation for further advancements in the field.

_Keywords_ Anomaly Detection $\cdot$ Continual Learning $\cdot$ Catastrophic
Forgetting $\cdot$ Incremental Learning

##  1 Introduction

Anomaly Detection (AD) is an important and challenging problem in the field of
Machine Learning (ML) and Computer Vision. Anomalies are patterns
characterized by a noticeable deviation from the so-called normal data, where
normal means compliance with some typical or expected features [1]. In order
to effectively detect anomalies, it is important to have a clear understanding
of what constitutes normal behavior and how to differentiate it from abnormal
patterns.

Current state-of-the-art neural networks have achieved exceptional performance
in solving a wide range of classification and detection tasks, especially in
Computer Vision. However, these algorithms struggle to learn multiple tasks in
succession. As a result, they are prone to Catastrophic Forgetting (CF): when
learning a new task, artificial neural networks frequently forget the previous
ones [2]; this significantly hinders the adoption of these approaches in real-
world scenarios. To address this issue, a new branch of machine learning has
been introduced, known as Continual Learning (CL) [3]. CL focuses on learning
new tasks without forgetting the previous ones.
In this work, we define a general framework for Anomaly Detection in the
Continual Learning (ADCL) setting, within which we investigate various
architectures and CL strategies.

Several CL strategies have been developed in the literature in recent years to
prevent catastrophic forgetting. Replay is the most well-known and effective
method for preventing forgetting [4, 5]. It consists in storing some samples
in raw format and passing them during the subsequent tasks. One of the main
issues with such an approach is that memory size is limited. To address this
issue, some Compressed Replay (CR) methods have recently been developed with
the idea of compressing the samples to increase the number of stored images
with the same memory size. In the same vein, we consider Compressed Replay for
Anomaly Detection in the Continual Learning setting. The proposed framework
consists of two modules, that may coincide in some cases: (i) a Memory module
that enables Compressed Replay; (ii) an Anomaly Detection module that is able
to learn how to distinguish anomalies for new classes without forgetting how
to deal with previously learned ones.
Within the proposed framework for ADCL, we evaluate different approaches. In
particular, as for the Memory Module we propose using a cGAN [6] with the
Super Resolution (SR) objective to obtain compressed representations.

Concerning the AD module, we study both generative and features-based [7]
approaches. While generative models can achieve both CR and AD, feature-based
models can only perform AD but do not enable CR. In this work, we propose
using SCALE for CR, in conjunction with feature-based AD models, demonstrating
that such a combination can achieve optimal performance. To investigate the
performance of different choices for the Memory and the AD module in the ADCL
framework, we propose to use MVTec dataset as a benchmark [8]. It is designed
to be representative of many challenges of real-world pixel-level AD.
Furthermore, this dataset provides 10 classes of objects that can be used as
consecutive tasks to be learned in a CL setup.
To recap, the main contribution of this work is three-fold:

  1. 1.

We design a general framework to solve an AD problem in a CL setting.

  2. 2.

We propose a novel approach for ADCL where SR models are used to achieve
Compressed Replay for the first time, to the best of our knowledge.

  3. 3.

We propose to use the MVTec AD dataset as a new Continual Learning benchmark
for Anomaly Detection [8].

The outline of the paper is the following. In Section 2, we describe related
work in terms of AD, AD for CL, and Compressed Replay. In Section 3, we
introduce the framework proposed for AD and describe the CL strategies and
architectures considered. In Section 3.3, we describe our novel approach for
Compressed Replay based on an SR model to retain the knowledge of old tasks.
In Section 5, we describe the experimental setup that implements our approach,
including details about the evaluation metrics used, the dataset, and the CL
setting proposed. Finally, in Section 6, we present our findings before
concluding in Section 7.

Figure 1: Scheme of the proposed approach for compressed Replay called SCALE.
To perform Super Resolution (SR), we use a Pix2Pix architecture. First, the
current task images are scaled (compressed) and saved in memory (yellow area).
Then, they are rescaled ($input$), and the model is trained to reconstruct the
original image ($target$). Instead, for old tasks, the images are taken from
memory, rescaled, and reconstructed using the SR model from the previous task,
in order to retain old knowledge (light blue area). After that, the rescaled
blurry image ($input_{2}$) is fed into the current SR model, with the target
corresponding to the image reconstructed by SR in Task $i-1$ (i.e.,
$target_{2}$).

##  2 Related Work

In this section, we first summarize the challenges of Continual Learning with
a focus on approaches based on Replay. Then, we review some contributions
devoted to Anomaly Detection in the Continual Learning scenario. Finally, we
mention some non-continual Anomaly Detection approaches evaluated on the MVTec
dataset.

###  2.1 Compressed Replay for Continual Learning

Continual learning provides ML models with the ability to update and expand
their knowledge over time, with minimal computation and memory overhead [3].
Thus, an effective CL solution is expected to have low forgetting, require low
memory consumption and be computationally efficient. CL strategies are usually
classified into three categories: replay-based [9, 10, 11] methods,
regularization-based [12, 13, 14] methods and architectural-based [15, 16]
methods. The related literature suggests that the Replay (also known as
Rehearsal) approach appears to be the most effective and practical solution to
reduce CF [17, 18, 19, 20].

The simplest implementation, known as Experience Replay (ER) [10, 21] consists
in storing some samples in the raw format selected randomly from the current
task and passing them during the subsequent tasks. Therefore, during training,
each batch of data from the current task is combined with a batch of data from
the previous tasks (sampled among them with the same probability). However,
the memory storage limit is a significant constraint, the smaller the memory
size, the further we are from the original training distribution of old tasks.

Methods for reducing the storage cost of old training samples have recently
been developed for this problem. We refer to them as Compressed Replay, where
the goal is to learn a compressed data representation. The most important and
vast family of this field is Latent Replay. Latent Replay methods propose to
store activations from an intermediate layer of a neural network and Replay
these representations to prevent forgetting. Typically, in a classification
task, these encodings are used as Replay starting from a specific layer, as
done in [18], where lower layers are trained at a slower pace to avoid latent
representation shift. In [22] authors propose to use pre-trained models to
extract features and use them for training a classification model. They study
the performance of Replay and Latent Replay using several pre-trained models.
In [23], the lower layers are frozen, and only the upper layers are trained.
Moreover, to store more samples, they propose to use Product Quantization (PQ)
to compress and save the features efficiently, obtaining a compression factor
of around 50. In [24] authors propose to apply feature adaptation so that
features learned for the old task remain consistent with the new feature
space.

The previous works were done for the classification problem, and the
compression factor depends on the size of the layer chosen to extract the
features to store in memory, so different compression factors can be obtained
depending on the chosen layer. However, because we are interested in
generative problems, we want to be able to compress the original samples while
retaining the ability to reconstruct the original image in the output. For
instance, in [25] authors propose EEC, a Generative Replay approach that also
memorizes the centroid and covariance of each class to make the training more
stable. In [26] authors introduce the use of Adaptive Quantization
Modules(AQM) to Continual Compression on the LiDAR dataset and also allow the
selection of the trade-off between the compression factor and quality of
reconstructed images.

Many generative models, as described in Section 5, can be used for Latent
Replay. However, no research on generative models regarding Latent Replay was
conducted. Therefore, as an additional contribution, we also provide an
accurate comparison of several generative models for Latent Replay, examining
their efficacy in terms of compression factor and image quality. Moreover, we
propose a novel approach for Compressed Replay called SCALE and described in
Section 3.3. This approach shows a high level of compression and at the same
time, a high level of quality in the generated images.

###  2.2 Anomaly Detection in the Continual Learning setting

Despite the fact that it is extremely relevant in practice, only a few works
in the literature addressed the AD problem in the CL setting.

In [1], the authors propose using the Variational Autoencoder (VAE) with the
Generative Replay approach to learn continually new classes while also
performing AD. The datasets used in the evaluation are KDD Cup 1999 and MNIST,
which have been artificially adapted to AD by marking one of the classes as
anomalous. In [27], using the same dataset KDD Cup ’99 and other datasets
belonging to Network Intrusion Detection, the authors study the problem of AD
formulated as a binary classification supervised learning. The authors of [28]
propose a regularization-based approach for continual AD in manufacturing, and
evaluate it on a real industrial metal forming dataset.

As for AD for images, the majority of work in this field focuses on predicting
whether an image is normal or abnormal. However, in practice, anomaly
localization is frequently required. Not only is it necessary to explain the
predictions, but it is also necessary to perform root cause analysis. As a
result, we investigate a complex image dataset such as MVTec, which contains
pixel-based anomalies. We also investigate a wide range of models and assess
their performance.

###  2.3 Approaches for MVTec AD Dataset

In the literature, many non-continual approaches for pixel-level Anomaly
Detection are tested on the MVTEc dataset. We can split most of them into
reconstruction-based and embedding similarity-based methods.

Reconstruction-based methods learn to reconstruct normal images during
training. However, during the evaluation, images with defects are not expected
to be reconstructed well. Consequently, they should exhibit a higher
reconstruction error than normal images. Reconstruction-based methods include
neural network architectures like autoencoders [8, 29, 30], variational
autoencoders [31] and generative adversarial networks [32, 33]. The idea is
that autoencoders will ignore minor details that don’t belong to the training
distribution and reconstruct the image without the defects. However, they can
sometimes reconstruct defects in output, leading to a failure to distinguish
between abnormal and normal images.
In [34] an approach based on inpainting is proposed called RIAD. The idea is
to hide a portion of the input image and train the model to reconstruct it.
In the same spirit, we consider Super Resolution to solve Anomaly Detection.
In this method, the image is scaled to a lower resolution and then rescaled to
the original size, resulting in a loss of information about the original
image. The concept is similar to inpainting because a portion of the
information in the images is hidden. Then the model is trained to reconstruct
the image while also repairing the defects in the original image.

Embedding similarity-based methods, on the other hand, use pre-trained neural
networks to extract meaningful features from images. These kinds of techniques
proved to be the state of the art of the field. SPADE [35] first performs
image feature extraction based on a WideResNet50 [36] trained on ImageNet
[37], then retrieves the K-nearest normal images, and finally achieves pixel
alignment with deep feature pyramid correspondences. Similarly to SPADE, PaDiM
[38] relies on ImageNet pre-trained feature extractor with multi-scale pyramid
pooling. However, instead of k-nearest neighbor clustering, it computes an
anomaly score based on the Mahalanobis distance. Normalizing Flows (NF) [39,
40] are a powerful tool for modeling complex distributions, such as those that
arise in Anomaly Detection tasks. By embedding normal data into a high-
dimensional standard normal distribution, it becomes possible to use the
probability of the data under the distribution as a measure of its normality.
This can be useful for identifying and localizing anomalies in data. For
example, in the case of image data, normalizing flows can be used to model the
distribution of image features, and then use the probability of a given image
under the distribution to identify and locate anomalies in the image. FastFlow
and CFLOW-AD are two examples of methods that use normalizing flows for
anomaly detection, with FastFlow focusing on improving anomaly localization by
using a 2D flow model.

##  3 Proposed Framework for Anomaly Detection in Continual Learning setting

###  3.1 Problem Definition

Anomaly detection is a challenging task, particularly when the data contains
distinct objects. In this case, abnormal samples are often much closer to
normal ones than to samples from other classes. This is illustrated in Figure
2, where different colors represent different object classes, normal samples
are depicted as circles, and anomalies are depicted as crosses. The closeness
of the anomalies to the normal samples makes it difficult to identify and
distinguish them using traditional methods. The task becomes even more
challenging when we want to identify if the single pixel of an image is an
anomaly or not in the CL setup where new objects must be learned subsequently.

Figure 2: We show the PCA on the original images where the first two
dimensions capture 73% of the total variance. Each class is depicted in a
different colour, with circles representing normal images and crosses
representing abnormal images. As can be seen, the separation between normal
and abnormal samples coming from the same class is much lower than the
distance between classes. The actual scenario is even more challenging since
we aim at detecting pixel-level instead of image-level anomalies.

The goal is to train a neural network model that can detect anomalies while
also retaining the knowledge it has learned from previous tasks. The model is
trained on a sequence of tasks, each of which corresponds to one or more
object classes. During training, the model learns the distribution of normal
(i.e., anomaly-free) images for each task. Then, during evaluation, the model
is tested on a dataset that contains data from all tasks, both normal and
abnormal. This allows the model to detect anomalies in the test data by
comparing it to the learned distribution of normal data.

In more formal terms, ADCL is a method for learning a model that can detect
anomalies in data. The model is trained on a sequence of tasks with a total of
$T$ tasks. Each Anomaly Detection task $t$ corresponds to a dataset $D_{t}$.
Each dataset $D_{t}$ consists of pairs of data $(X_{t},Y_{t})$, where $X_{t}$
is a set of images or more formally $X_{t}\subset\mathbb{N}^{H\times W\times
3}$ where $H$ and $W$ denote the spatial dimensions and $t=1\cdots T$. $Y_{t}$
is a set of labels for each pixel of the image, indicating whether the pixel
is normal (0) or anomalous (1). The goal of the model is to learn a mapping
$f_{\theta}:X\rightarrow R^{H\times W}$ from the space of images $X$ to a
probability vector. This allows the model to assign a probability to each
pixel in an image, indicating how likely it is to be normal or anomalous.

In the field of continual learning (CL), there are three main scenarios that
are commonly studied: Task Incremental Learning (TIL), Domain Incremental
Learning (DIL), and Class Incremental Learning (CIL) [2, 41]. In our problem,
we are dealing with the DIL scenario because the classes of output remain
fixed independently by the change of task (i.e. there are always only two
classes, normal and abnormal, for each pixel of the image), while the input
distribution changes between tasks i.e. $P(X)\neq P(X^{\prime})$.

###  3.2 Proposed ADCL Framework

We introduce a general framework for ADCL. We present a general framework for
the task of Anomaly Detection in Continual Learning (ADCL). The goal of this
framework is to enable incremental learning of new objects while also
detecting anomalies in images from previous tasks. The proposed framework,
depicted in Fig. 3, is broken down into two modules:

  1. 1.

Memory Module: This component’s goal is to retain the memory of previous
tasks. Depending on the type of CL strategy considered, such a module may take
various forms. In the case of Replay, for example, such a module consists of a
limited memory containing some of the images of previous tasks. In the case of
Latent Replay, the module will consist of an autoencoder (AE) and a set of
compressed samples memorized through the latent space. In the case of
regularization-based strategies, such as EWC [42], the Memory Module will be
composed of the importance values associated with each weight of the
architecture used in the AD Module.

  2. 2.

AD Module: It is the architecture used to detect anomalous pixels. The chosen
architecture can be any previously proposed approaches, like AE, VAE, and GANs
for reconstruction-based approaches or FastFlow, and CFlow for embedding
similarity-based.

It is important to note that the Memory Module and the AD Module are typically
described as separate functional entities, but the same model can serve both
functions. For example, many reconstruction-based methods can generate images
of old tasks for use during the training of a new task, the ability to
reconstruct images can also be used for anomaly detection by comparing the
reconstructed image to the original image.

The Memory Module is a key component of a Continual Learning (CL) system,
responsible for storing and updating information about previous tasks. This
module is updated using a combination of data from the current task and a
batch of old data from the previous task, as depicted in Fig. 3. In the case
of classic Replay, the Memory Module simply stores images. With Latent Replay
and generative models such as CAE or VAE, the Memory Module stores the model
of the previous task and samples in the latent space, which allows for more
efficient storage of images compared to Replay. It is also possible to use
Generative Replay with VAE and GAN models, but it has been shown to result in
a high degeneration [43].

###  3.3  SCALE: A novel approach for Compressed Replay

Within the above-introduced framework for Visual Anomaly Detection in
Continual Learning, we proposed SCALE (SCALing is Enough), a Super Resolution-
based approach, as a Compressed Replay technique. Super Resolution aims to
convert a low-resolution image with coarse details into a high-resolution
image with improved visual quality and refined details [44]. In practice, as
SR model, we consider the Pix2Pix [6], a general-purpose model for image-to-
image translation composed of a conditional GAN [45]. The cGAN, unlike the
classic GAN [45], is conditioned not only by random noise but also by an input
image. The GANs have the flaw of having a high forgetting with Generative
Replay (where the images are generated using random noise) [43]. We show how
conditioning to an input image plus the noise can result in a method that can
still generate images with a low forgetting even though the output still has a
random component. Though such architecture is not the state-of-the-art of the
SR field, it is sufficient to obtain good results and justify its use in this
context, as shown in Section 6. Therefore, in our work, the SR model is
studied with three final objectives:

  1. 1.

Since it is the first time that the SR problem is studied in CL, we provide an
evaluation in terms of reconstruction of the original images and how this
approach behaves in terms of forgetting.

  2. 2.

We also investigate the use of memory in combination with AD approaches that
are by nature not able to perform the function of memory.

  3. 3.

We consider the efficacy of SR in terms of AD.

The proposed schema for using the SR model in Continual Learning is shown in
Fig. 1 and can be summarized in the following steps:

  1. a)

When a new task is received, the images are resized to a lower resolution and
a copy is saved in memory. The objects saved for a task will remain unchanged
during training, as opposed to Replay and Latent Replay.

  2. b)

The process for handling the images of the new task is depicted in the yellow
portion of Fig. 1. The other image is resized again (which reduces its
quality) and sent to the SR model to learn how to reconstruct the original
image.

  3. c)

The procedure to obtain images of old tasks to be used in the training of
current tasks is depicted in Fig. 1 as the light blue part. To obtain images
from old tasks to use in training the current tasks, the scaled image saved in
memory of shape (32,32,3) is rescaled to its original size (256,256,3) and
labeled as $input_{2}$ in Fig. 1. Then it is passed to the SR model to obtain
an image quality similar to the original, such output is labeled as
$target_{2}$ in Fig. 1. The SR model of the current task is then trained to
reconstruct the rescaled image ($input_{2}$) using the reconstructed image
($target_{2}$).

By memorizing the scaled images and increasing the resolution when needed, it
is possible to achieve a compression factor. The value of such compression
depends on the final quality that is desired, and it will depend on the final
scope.
We are going to present its results in terms of performance for Anomaly
Detection in Section 6.1 and in terms of the quality of reconstructed images
in the Section 6.2. Because this is the first time that SR has been studied in
the context of CL, we present additional results for such a model in Section
6.3 where we explain why it works compared to classic GANs and its advantages.
Among the results, it is shown to be advantageous to keep the memory of a task
fixed over time rather than changing continuously as is typical in Latent
Replay.

Figure 3: The proposed framework is composed of 2 modules: memory and Anomaly
Detection. The Memory Module will be updated using the data from the current
task as well as a batch of old data retrieved using the Memory Module of
previous task.

##  4 Considered Benchmark Dataset

We propose to use the MVTec Dataset [8] as a benchmark to study the
performance of architectures and CL strategies for Anomaly Detection. The
MVTec AD dataset is a recent and extensive industrial dataset that includes
5354 high-resolution images divided into 15 categories: 5 types of textures
and 10 types of objects. The training set only contains normal images, whereas
the test set includes both defect and defect-free images. The image
resolutions range from (700,700,3) to (1024,1024,3) as resolution. In our ADCL
framework, we will consider a sequence of 10 tasks. Each task corresponds to
one of the 10 classes associated with objects. In this study, the model is
presented with a series of tasks, and must be able to identify which pixels in
the image are abnormal for each object. The MVTec dataset used in this study
is more challenging and complex than the commonly used datasets MNIST and
CIFAR-10, which are often used in Continual Learning literature [46, 47, 48].
These datasets have smaller image sizes, (28,28) for MNIST and (32,32) for
CIFAR-10, and fewer images, making them not as representative of real-world
scenarios as the MVTec dataset.

Figure 4: Image examples from the MVTec Dataset AD. For each object is shown a
normal sample (in green) and an anomalous sample (in red). For each sample,
the entire object is displayed, as well as a zoomed-in view of the region
containing the defect. The samples illustrate the range of variations and
abnormalities that can be found in the dataset.

(a) Reconstruction based on VAE

(b) Reconstruction based on SR

Figure 5:  The images reported here show examples of the VAE (Variational
Autoencoder) architecture in (a) and the SR (Super-Resolution) architecture in
(b). The first image in each row is the original image, and the second column
is the model’s reconstructed image. The third column displays the ground truth
for each pixel, which can be used to evaluate the accuracy of the
reconstructed image. The fourth column shows the anomaly map, which indicates
the probability that each pixel is abnormal (i.e., significantly different
from the original pixel). This can help identify corrupted areas of the image.
The fifth column displays the predicted class for each pixel, which provides
more information about the model’s final prediction on what is considered
anomalous. The sixth column highlights any potential issues with the image by
showing the anomalous parts of the image, making it easier for users to
identify and address any problems.

##  5 Experimental Settings

###  5.1 Considered CL Strategies

We are going to evaluate the different architectures in terms of performance
for both the Memory Module and the AD Module. Such evaluation will be
performed considering different CL strategies that will be applied to the
models.

  * •

Single Model: As an upper bound, we train a different model for each task.

  * •

Fine-tuning: As a lower bound, we consider the fine-tuning approach, in which
a model is presented sequentially only with data from the current task.

  * •

Ideal Replay: As a benchmark Continual Learning strategy, we consider an
approach where we randomly select $n$ images from a task and create a training
batch with half data from the current task and half data from previous tasks.
The constraint on processing power is respected, but not the constraint on
memory. Indeed, this approach takes into memory all samples seen so far. This
implies that the memory can capture the original distribution completely. We
refer to this approach as Ideal Replay, and we propose to consider it as an
additional upper bound for CL approaches.

  * •

Replay: We consider a constrained memory size of $n$ images such that:
$n<<|D|$ where $D$ represents the entire dataset. In the experiments, $n$ is
set to $40$ images, representing less than $2\%$ of the total dataset. This
means that the total memory size is limited to $40$ images, i.e., four images
for each class at the end of the training.

  * •

Compressed Replay: When dealing with Replay, we should keep in mind that the
risk of overfitting is high when only a few samples represent a task. Given
that memory size can influence final results, the ability to save more samples
has the potential to reduce forgetting because the distribution in memory of
an old task is more similar to the original one (assuming that the quality of
compressed samples is high enough). So in Compressed Replay, the images are
compressed. For example, in Latent Replay, we consider their representation in
the latent space. Similar to Replay, we consider the constraints on processing
power and memory size, but with Compressive Replay, we try to keep more images
in memory by utilizing some compression method.
When considering Latent Replay, because we will be studying architecture such
as autoencoders, it is more natural to choose the latent space that
corresponds to the Encoder’s output and the Decoder’s input, i.e. the
bottleneck layer, which has the lowest dimensional space.

  * •

Generative Replay: In the case of Generative Replay [9], we use models such as
GANs to generate images without the need to store any additional information
other than the model itself. This can be very useful because it eliminates the
need to save any data in memory. However, in previous experiments [43], the
Generative Replay had a high level of forgetting and much of the information
was quickly lost.

To ensure a fair comparison between Compressed Replay and Replay, we use the
same amount of memory in terms of bytes for both approaches. We define the
memory size as $C$, which is the number of bytes needed to store $n$ images
using the Replay strategy. For Compressed Replay, a number of compressed
samples will be stored, taking up a maximum of $C$ bytes of storage.

In general, we will compare the different methods, with upper bounds included,
taking into account that, on the practical side, the only valid CL approaches
are Replay, Compressed Replay, and Generative Replay, since they are the only
ones with a memory size and processing power constraint.

We will be comparing the Continual Learning strategies introduced above in
terms of their performance in two key areas: Anomaly Detection and the quality
of the reconstructed images. For the Compressed Replay strategy, we will also
be considering the compression factor as a key metric. A higher compression
factor means that more samples can be stored, but we must also take into
account the quality of the reconstructed images. If the compression factor is
high, but the quality of the reconstructed images is poor, then the
distribution of the reconstructed images will be significantly different from
the original distribution. It is important to find the right balance between
these two factors to achieve optimal performance.

###  5.2 Considered Architectures in Memory and Anomaly Detection modules

The following models were considered as both Memory and AD modules:

  * •

Convolutional Auto-Encoder(CAE): It has a latent space of shape (512,4,4) i.e.
of dimension 8192, implying a compression factor value equal to 6 (assuming
that we are working with 4 bytes for each value). Taking into account an input
shape (256,256,3) we can simply calculate the compression factor as
$\frac{196608}{8192\cdot 4}=6$. We will memorize the activations of the CAE
obtained in the latent space in the case of Latent Replay.

  * •

Variational Auto-Encoder(VAE): Using the VAE (Variational Autoencoder)
architecture, we were able to achieve good results with a smaller latent space
compared to the CAE (Convolutional Autoencoder). Specifically, we obtained a
latent space dimension of 256 and a compression factor of 96. It is worth
noting that the VAE architecture can be used not only with the Latent Replay
strategy but also with Generative Replay, which allows the model to store
information without any actual samples. However, as previously mentioned and
shown by our results, the quality of the images generated by VAE with
Generative Replay degrades quickly.

As for the approaches that can be used as AD Module only, we consider:

  * •

Inpaint: We tried to use the same architecture of RIAD [30] in our study, but
found that the architecture was very sensitive to the continual learning
setting and we were unable to obtain satisfactory results. As an alternative,
we adopted the spirit of the RIAD approach (i.e., inpainting) and proposed a
different architecture, a Pix2Pix model [6], for the inpainting task. In our
version, training is done by masking random areas of the images. During
reconstruction, the same image is fed into the model multiple times (with
different masks each time), and averaging is used if the same pixel is masked
multiple times.

  * •

FastFlow: We use FastFlow [49] in combination with the WideResNet50 [36]. This
approach, described in Section 2, learns a Normalizing Flow that models normal
images. It should be noted that this is one of the few studies on Normalizing
Flow architectures in CL setting. To the best of our knowledge, only two other
studies were done [50, 51].

###  5.3 Evaluation Metrics for Anomaly Detection

There are several metrics commonly used to evaluate the performance of Anomaly
Detection models on the MVTec AD dataset. The ROC AUC score at the pixel and
image levels is the most widely used, but other metrics such as f1 score and
IoU are also taken into account. In the Anomaly Detection literature, it is
generally believed that metrics such as the f1 score are more fair when
dealing with imbalanced datasets, which is often the case with Anomaly
Detection [52]. Therefore, in this study, we use the f1 score to assess a
model’s ability to detect anomalies. A well-performing model should have a
high f1 score and a low FID score.

###  5.4 Evaluation of image reconstruction quality

To evaluate the quality of the reconstruction performed by the models used in
the Memory Module, we will use the Fréchet Inception Distance (FID) [53],
which is a commonly used metric for evaluating generative models [43]. This
type of evaluation is useful for analyzing the performance of all
architectures that can function as Memory Modules under various CL strategies.
The FID is calculated as:

$FID=||\mu_{r}-\mu_{g}||^{2}+Tr(\Sigma_{r}+\Sigma_{g}-2(\Sigma_{r}\Sigma_{g})^{\frac{1}{2}})$
(1)

The statistics $(\mu_{r},\Sigma_{r})$ and $(\mu_{g},\Sigma_{g})$ are the
activations of a specific layer of a discriminative neural network trained on
ImageNet for real and generated samples, respectively. A lower FID indicates
that real and generated samples are more similar, as measured by the distance
between their activation distributions.
Beyond the models that can be used in the Memory module, we will evaluate the
image reconstruction quality also for Inpaint, even if it lacks the ability to
compress them and to act as a Memory Module.

###  5.5 CL Metrics

Following the convention of Continual Learning we are going to considered the
average value at the end of the training and the forgetting. Let $s_{i,j}$ be
the performance f1 of the model on the test set of task $j$ after training the
model on task $i$. To measure performance in the CL setting, we introduce the
following metrics:

Average f1

The average f1 score $S_{T}\in[0,1]$ at task T is defined as:

$S_{T}=\frac{1}{T}\sum_{j=1}^{T}s_{T,j}$ (2)

Average Forgetting

$F_{T}\in[-1,1]$, the average forgetting measure at task T, is defined as:

$F_{T}=\frac{1}{T-1}\sum_{j=1}^{T-1}max_{l\in\\{1,\cdots,T-1\\}}\frac{s_{l,j}-s_{T,j}}{s_{l,j}}.$
(3)

With respect to the original definition used in [54], we are scaling respect
to the maximum f1 score, as done in [20, 55]; this is done to compare the
forgetting among tasks with very different scores. Notice that the closer the
metric $F_{T}$ is to 1, the higher the forgetting is. What it was defined
above for the metric f1 will be redefined in a very similar way for the metric
FID, with the consideration that the goal of FID is to be minimized while with
f1 we want to maximize it.

##  6 Experimental results

We will compare all models in terms of Anomaly Detection performance using the
Average f1 score as a metric in Subsection 6.1. Then, in Section 6.2, we will
try to motivate the results by evaluating the quality of reconstructed images
produced by different models and CL strategies. Finally, because this is the
first time (to the best of our knowledge) that a Super Resolution model has
been studied in a Continual Learning setting, we investigate the results at a
greater level in subsection 6.3. Each test is repeated ten times for each cell
in the result tables, and only the average is shown. The plots shown in the
Section Results will display the mean and variance of the runs. For the sake
of reproducibility, the code used in the experiments is available in a public
repository online111https://github.com/dallepezze/adcl_scale.

###  6.1 Quality of the Anomaly Detection

Table 1: Summary table for the AD performance under different CL strategies and architectures for the Memory and the AD Module. Since Inpaint and FastFlow cannot achieve Compressed Replay, but only act as AD modules, we consider them in combination with CAE, VAE and SR as Memory Modules. We report the Average f1 value ($\uparrow$ stands for "the higher the better") and, within round brackets, the forgetting associated to the metric, for each combination. We highlight in bold the best combination. We can observe that, among the viable Replay-based strategies, the best combination is using Inpaint as AD Module with SR as Memory Module adopting the Compressed Replay strategy. The best combination is in bold while the best architecture for each strategy is in italic.  Strategy |  Memory Module |  AD Module (Average $\mathrm{f1}$ metric $\uparrow$ )
---|---|---
CAE | VAE | SR | Inpaint | FastFlow
| Single
---
Model
- | 0.28 | 0.36 | 0.33 | 0.38 | 0.57
FT | - |  | 0.09
---
(66.35%)
| 0.11
---
(68.19%)
| 0.14
---
(59.39%)
| 0.11
---
(72.73%)
| 0.19
---
(69.79%)
| Ideal
---
Replay
- |  | 0.28
---
(13.22%)
| 0.33
---
(6.38%)
| 0.36
---
(5.72%)
| 0.34
---
(11.25%)
| 0.22
---
(62.53%)
Replay | - |  | 0.26
---
(19.88%)
| 0.30
---
(21.2%)
| 0.29
---
(22.23%)
| 0.34
---
(13.27%)
| 0.21
---
(64.57%)
| Generative
---
Replay
- | - |  | 0.11
---
(63.68%)
- | - | -
Compressed Replay | CAE |  | 0.23
---
(25.61%)
- | - |  | 0.25
---
(34.00%)
| 0.17
---
(53%)
VAE | - |  | 0.25
---
(21.87%)
- |  | 0.26
---
(30.27%)
| 0.2
---
(48%)
SR | - | - |  | 0.34
---
(7.99%)
| 0.35
---
(10.28%)
| 0.30
---
(41.27%)

(a) Single Model

(b) Ideal Replay

(c) Replay

(d) Compressed Replay

Figure 6: Each plot shows the performance for a strategy comparing different
architectures in terms of AD performance i.e. Average f1 score. The Average f1
metric is shown on axis y, and the index of the current training task is shown
on axis x.

Regarding the AD results, it should be kept in mind that CAE, VAE, SR, and
Inpaint are all reconstruction-based methods, whereas FastFlow is an embedding
similarity-based method that uses pre-trained models for feature extraction
and is considered state-of-the-art in the field. Embedding similarity-based
methods, such as FastFlow, are incapable of acting as Memory Module because
they produce only the anomaly map as output and not the reconstructed image.
As a result, when we use the Compressed Replay strategy on FastFlow and
Inpaint, we use the SR architecture as the Memory Module (which turns out to
be the best among the generative models as shown in Section 6.2) and FastFlow
and Inpaint as the AD Module.

In Fig. 5 we show the results of AD obtained for a sample with two different
architectures VAE and SR. The first image in the row shows the original image,
while the second column displays the reconstructed image produced by the
model. The third column presents the ground truth for each pixel, which
provides a reference for evaluating the accuracy of the reconstructed image.
The fourth column displays the anomaly map, which indicates the probability
that each pixel is anomalous (i.e., differs significantly from the ground
truth). This can help identify areas of the image that may be corrupted. The
fifth column shows the predicted class for each pixel, which can provide
additional information about the final prediction of the model on what is
considered anomalous. Finally, the sixth column presents the portion of the
image that contains anomalous parts, highlighting any potential issues with
the image. This can help users quickly identify and address any problems with
the image.

The final values for AD performance are showed in Table 1 varying strategy and
architecture.

In the Single Model (classic setting), FastFlow performs better than the other
models, as expected. It is significantly better than the next best model, with
a margin of 0.57. However, in the CL setting, FastFlow performs poorly for
both the Replay and Ideal Replay strategies. When using the Compressed Replay
strategy, FastFlow performs better than the CAE and VAE models, but still
worse than SR and Inpaint. Interestingly, when FastFlow is evaluated using the
Compressed Replay strategy, it outperforms Replay in terms of AD performance.
We can observe that, among the feasible Replay-based strategies, the best
combination is Inpaint as AD Module with SR as Memory Module trained according
to the Compressed Replay Strategy.

In addition, it should be observed from Table 1 that the fine-tuning strategy
appears to have better values in FastFlow than in other models.

In terms of forgetting, as shown in Table 1, FastFlow has a significantly
higher forgetting than the other approaches. Instead, in terms of forgetting,
SR with Compressed Replay has a very low forgetting.

Fig. 6 presents the results in greater detail, displaying the performance
progressively as each task is presented in sequence. The Figure shows four
different plots, each representing the performance of a specific strategy. The
x-axis of each plot indicates the task ID of the training, while the y-axis
represents the value of the chosen metric. Each line in the plot represents a
different model. The value at the task $i$ include only the performance on the
tasks seen so far.

###  6.2 Quality of the reconstructed images

Table 2: Summary table about the quality of reconstructed images for different models. For each model and strategy, we report the FID value. Please note that the lower FID, the better the reconstruction is. In round brackets we report also the forgetting value in percentual. Note that for each strategy the best architecture is SR. Moreover, the best strategy results to be Compressed Replay. The best combination is in bold while the best architecture for each strategy is in italic.  | Average FID $\downarrow$
---|---
Strategy | CAE | VAE | SR | Inpaint
Single Model |  | 266.52
---
| 219.40
---
| 125.15
---
| 204.29
---
FT |  | 376.64
---
(73.65%)
| 399.44
---
(91.33%)
| 244.25
---
(194.31%)
| 363.20
---
(114.41%)
Ideal Replay |  | 216.82
---
(7.50%)
| 255.84
---
(11.93%)
| 89.53
---
(7.74%)
| 192.12
---
(5.24%)
Replay |  | 227.88
---
(14.70%)
| 245.87
---
(14.54%)
| 111.87
---
(40.6%)
| 181.74
---
(7.90%)
Compressed Replay |  | 248.42
---
(20.33%)
| 316.48
---
(36.17%)
| 105.57
---
(14.66%)
-
Generative Replay | - |  | 405.34
---
(77.84%)
- | -

(a) Single Model

(b) Ideal Replay

(c) Replay

(d) Compressed Replay

Figure 7: The results show the quality of the constructed images for different
continual learning (CL) strategies and architectures. Each plot represents a
different CL strategy, and each line represents a different architecture. The
y-axis shows the FID metric, and the x-axis shows the index of the current
training task. Each value represents the average FID across all tasks seen so
far, with a lower value indicating a higher quality of reconstruction. The
plots show that the best architecture for each CL strategy is SR, our
approach.

Table 2 shows a summary of the results, which corresponds to the average FID
on all tasks evaluated at the end of the training. Table 2 also displays the
associated percentual forgetting value in round brackets.

Moreover, it is possible to examine average FID on the tasks seen so far in
the plots of Fig. 7. For each strategy, the performance of each model is
shown, where a lower value means a better quality of the reconstructed image.

From Fig. 7, we can clearly see that the SR model is the best architecture for
each strategy considered. For each of the strategies considered, the gap
between SR and other architectures is significant. In other words, regardless
of the strategy, the SR model is the best for learning a good representation
of the original images. Moreover, its output is conditoned by the scaled image
and random noise which means a certain level of randomness in output, while in
the classic GANs imply a very quickly forgetting [43], that it doesn’t happen
in our case. Despite all these aspects, the SR is able to obtain a good
representation of the original distribution.
As stated above, Inpaint can’t act as Memory Modules like CAE and VAE because
it cannot perform Compressed Replay. However, since Inpaint is a generative
model, we can still analyze its reconstruction performance in terms of FID in
Table 2 and Fig. 7.

Another interesting fact is that CAE seems to have a more stable trend than
VAE in Latent Replay strategy. In fact, it can be observed that while CAE is
worse than VAE in Single Model, it obtains better performance for Replay and
Compressed Replay.

Among the proposed models VAE is the only one to be applied under the
Generative Replay strategy. Results are shown in Table 2 and it can be seen
that, as said before, the forgetting is very high.
According to these graphs, the difference between Compressed Replay and Ideal
Replay appears to be greater for models CAE and VAE than for SR and Inpaint.
It should be noted that the compression factors for the CAE, VAE, and SR
architectures are 6, 196, and 64, respectively.
The only model that allows Generative Replay is VAE, but as seen in 2, the
performance degrades very quickly, so it is only shown in the tables and not
the plots (the same for fine-tuning).
As a final remark, it can be stated that the SR model allows for a good
compression factor while maintaining a very high image quality in memory.

###  6.3 Results of SR Model

Strategy | epochs |  |  Average FID $\downarrow$
---
| FID Increase %
---
wrt Ideal Replay
$\downarrow$
| Single
---
Model
30 | 125.15 | -
| Ideal
---
Replay
30 |  | 89.53
---
(7.74%)
-
Replay | 30 |  | 111.87
---
(40.6%)
24.95%
| Compressed
---
Degenerative
Replay
30 |  | 106.49
---
(13.58%)
18.94%
| Compressed
---
Replay
30 |  | 105.57
---
(14.66%)
17.92%
| Single
---
Model
50 | 82.06 | -
| Ideal
---
Replay
50 |  | 85.27
---
(11.18%)
-
Replay | 50 |  | 109.51
---
(59.41%)
28.43%
| Compressed
---
Degenerative
Replay
50 |  | 97.31
---
(19.08%)
14.12%
| Compressed
---
Replay
50 |  | 91.17
---
(10.88%)
6.92%
Table 3: Comparison of different strategies for training a SR model. The table
shows the results for two different values of the number of epochs (30 and
50). The FID score is shown in the third column, with the percentage of
forgetting in round brackets. The fourth column shows the increase in
performance with respect to the Ideal Replay strategy, in other words more a
value is lower and more is close to the optimal. The best results for each
number of epochs are highlighted in bold.

As the first study on Super Resolution in the context of Continual Learning,
we thoroughly examine how variables such as the number of epochs and the way
in which images are saved affect the final performance in terms of the quality
of the reconstructed images. This analysis helps us to gain insight into the
critical issues facing generative models such as GANs in Continual Learning.
Based on this analysis, we conclude that there are at least three critical
issues that arise during the training of GANs in CL.

  1. 1.

The initial quality of the learned distribution is the first critical issue.
Based on the number of epochs, we can easily demonstrate that an initial
higher quality has a long-term beneficial effect.

  2. 2.

In classic GANs, we create images only using a random vector for each new
task, producing an output image that will be used as input to the model at the
new task. However, there will be some perturbation in the final image. These
perturbations accumulate in the produced images over time, increasing their
distance from the original distribution.

  3. 3.

The third critical issue is catastrophic forgetting of the model’s weights,
which is inherent in all models in the CL setting.

To simplify the explanation we are going to define the following notation:

  * •

$X_{i}$: The original data of the $i$-th Task;

  * •

$O_{i}(X_{i})$: The output produced by SR after training the i-th Task on data
$X_{i}$;

  * •

$S_{i}$: The scaled images of the data $X_{i}$;

  * •

$M_{ij}$: The memory about the $j-$th Task after training the $i$-th Task.

Figure 8: The comparison of the SR model using 30 epochs (left) and 50 epochs
(right) with different strategies is shown in the figures. The y-axis shows
the FID metric, and the x-axis shows the index of the current training task.
It can be observed that the Compressed Replay strategy performs closer to the
Ideal Replay strategy (i.e. the optimal performance) when there is an increase
in the number of epochs from 30 to 50.

Figure 9: The figure on the left shows the best model using different
continual learning (CL) strategies, and the figure on the right shows all
models using the Compressed Replay strategy. The y-axis shows the Average f1
metric, and the x-axis shows the index of the current training task.

In Compressed Replay for SR, the scaled image $S_{i}$ is saved in memory
$M_{i,i}$ during the $i$-th Task i.e. $M_{i,i}=S_{i}$. Such scaled images will
never be changed, they will remain fixed over time or in other terms
$M_{i,i}=M_{i+1,i}=\cdots=M_{T,i}$.
Now we define a modified version called Compressed Degenerative Replay that
simulates the critical issue (ii) present in GANs, where the images used for
an old task during the training of a new task are different from the images
used for a previous task. During the $(i+1)-$th Task, we take the images saved
during the previous task $M_{i,i}=S_{i}$ and produce the new reconstructed
images as $O_{i+1}(S_{i})$. Such images are scaled and saved in memory i.e.
$M_{i+1,i}=O_{i+1}(S_{i})$. This is repeated for all the tasks and then we
have that for the $i$-th Task the memory change overtime i.e. $M_{i,i}\neq
M_{i+1,i}\neq\cdots\neq M_{T,i}$

The difference in results for each strategy based on the number of epochs (30
or 50) can be seen in Table 3 and Fig. 8. Generally, the results obtained
using a larger number of epochs (i.e., a better initial learned
representation) are better, as expected. For example, the Ideal Replay
strategy improves from 95.2 to 89.5 in terms of FID, and the Replay strategy
improves from 112 to 109.5. However, the most significant improvement is seen
with the Compressed Replay strategy, which drops from 105.5 to 91. In terms of
percentage increase compared to Replay, Compressed Replay shows a minor
decrease at 50 epochs (7%) compared to 30 epochs (18%). This suggests that the
proposed solution performs very close to the upper bound of Ideal Replay,
particularly when the initially learned representation is very good (i.e., a
higher number of epochs).

Finally, when comparing the Compressed Replay strategy vs Compressed
Degenerative Replay strategy, we can see that the positive effect of
Compressed Replay vs compressed Degenerative Replay increases with the number
of epochs. Indeed, with 50 epochs, Compressed Replay has a 7% increase over
Ideal Replay, while Compressed Degenerative Replay has a 14% increase.
Furthermore, the percentage increase in comparison to the Ideal Replay for
Compressed Degenerative Replay is lower for 50 epochs (14% instead of 18%).
This means that using more epochs and Compressed Replay instead of Replay or
compressed Degenerative Replay allows you to get closer to the Ideal Replay,
which uses the original distribution in memory.
However, while Compressed Degenerative Replay is inferior to Compressed
Replay, the table shows that Compressed Degenerative Replay with 50 epochs is
superior to Compressed Replay with 30 epochs. This means that the main factor
is the number of epochs i.e. the initial quality of the learned
representation.

##  7 Conclusions

In this research, we propose a framework for performing anomaly detection in a
setting where the model is continually learning new tasks. Our approach
combines a Memory module, which stores information from previous tasks, and an
Anomaly Detection module, which identifies anomalies in new data. By using
these modules together, our approach can detect anomalies at the pixel level
in new image classes without forgetting how to identify anomalies in
previously learned classes. To the best of our knowledge, this is the first
time that a method using a Super Resolution module to enable Compressed Replay
has been proposed for use in continual learning. We also suggest a real-world
benchmark dataset specifically designed for evaluating the performance of
anomaly detection approaches in a continual learning setting.

There are several potential future developments. Application and evaluation of
other models considered state-of-the-art in Anomaly Detection in the CL
setting should be a research direction. Furthermore, some of these models
could be modified to produce better results in the case of Anomaly Detection
for Continual Learning. While the MVTec Dataset is an excellent benchmark for
Anomaly Detection, other datasets could be used to confirm the models’
efficacy. Furthermore, a follow-up study on the SR problem should be carried
out, analyzing models closer to the state of the art and using a more complex
dataset to validate their efficacy in the Continual Learning setting.

## References

  * [1] F. Wiewel, B. Yang, Continual learning for anomaly detection with variational autoencoder, in: ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019, pp. 3837–3841.  doi:10.1109/ICASSP.2019.8682702.
  * [2] G. M. van de Ven, A. S. Tolias, Three scenarios for continual learning, CoRR abs/1904.07734 (2019).  arXiv:1904.07734.
URL http://arxiv.org/abs/1904.07734

  * [3] M. De Lange, R. Aljundi, M. Masana, S. Parisot, X. Jia, A. Leonardis, G. Slabaugh, T. Tuytelaars, A continual learning survey: Defying forgetting in classification tasks, IEEE transactions on pattern analysis and machine intelligence 44 (7) (2021) 3366–3385.
  * [4] A. ROBINS, Catastrophic forgetting, rehearsal and pseudorehearsal, Connection Science 7 (2) (1995) 123–146.  arXiv:https://doi.org/10.1080/09540099550039318, doi:10.1080/09540099550039318.
URL https://doi.org/10.1080/09540099550039318

  * [5] G. Merlin, V. Lomonaco, A. Cossu, A. Carta, D. Bacciu, Practical recommendations for replay-based continual learning methods.  doi:10.48550/ARXIV.2203.10317.
URL https://arxiv.org/abs/2203.10317

  * [6] P. Isola, J.-Y. Zhu, T. Zhou, A. A. Efros, Image-to-image translation with conditional adversarial networks, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 1125–1134.
  * [7] J. Yang, R. Xu, Z. Qi, Y. Shi, Visual anomaly detection for images: A systematic survey, Procedia Computer Science 199 (2022) 471–478.
  * [8] P. Bergmann, M. Fauser, D. Sattlegger, C. Steger, Mvtec ad — a comprehensive real-world dataset for unsupervised anomaly detection, in: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 9584–9592.  doi:10.1109/CVPR.2019.00982.
  * [9] H. Shin, J. K. Lee, J. Kim, J. Kim, Continual learning with deep generative replay, Advances in neural information processing systems 30 (2017).
  * [10] D. Rolnick, A. Ahuja, J. Schwarz, T. Lillicrap, G. Wayne, Experience replay for continual learning, Advances in Neural Information Processing Systems 32 (2019).
  * [11] X. Chen, J. Jiang, Z. Li, H. Qi, Q. Li, J. Liu, L. Zheng, M. Liu, Y. Deng, An online continual object detector on vhr remote sensing images with class imbalance, Engineering Applications of Artificial Intelligence 117 (2023) 105549\.
  * [12] J. Pomponi, S. Scardapane, V. Lomonaco, A. Uncini, Efficient continual learning in neural networks with embedding regularization, Neurocomputing 397 (2020) 139–148.
  * [13] J. Ramapuram, M. Gregorova, A. Kalousis, Lifelong generative modeling, Neurocomputing 404 (2020) 381–400.
  * [14] Y.-n. Han, J.-w. Liu, Online continual learning via the meta-learning update with multi-scale knowledge distillation and data augmentation, Engineering Applications of Artificial Intelligence 113 (2022) 104966.
  * [15] G. Sokar, D. C. Mocanu, M. Pechenizkiy, Spacenet: Make free space for continual learning, Neurocomputing 439 (2021) 1–11.
  * [16] A. Mallya, D. Davis, S. Lazebnik, Piggyback: Adapting a single network to multiple tasks by learning to mask weights, in: Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 67–82.
  * [17] Q. Yang, F. Feng, R. Chan, A benchmark and empirical analysis for replay strategies in continual learning (2022).  doi:10.48550/ARXIV.2208.02660.
URL https://arxiv.org/abs/2208.02660

  * [18] L. Pellegrini, G. Graffieti, V. Lomonaco, D. Maltoni, Latent replay for real-time continual learning, in: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2020, pp. 10203–10209.
  * [19] P. Buzzega, M. Boschini, A. Porrello, S. Calderara, Rethinking experience replay: a bag of tricks for continual learning, in: 2020 25th International Conference on Pattern Recognition (ICPR), IEEE, 2021, pp. 2180–2187.
  * [20] C. D. Kim, J. Jeong, G. Kim, Imbalanced continual learning with partitioning reservoir sampling, in: European Conference on Computer Vision, Springer, 2020, pp. 411–428.
  * [21] A. Chaudhry, M. Rohrbach, M. Elhoseiny, T. Ajanthan, P. K. Dokania, P. H. Torr, M. Ranzato, On tiny episodic memories in continual learning, arXiv preprint arXiv:1902.10486 (2019).
  * [22] O. Ostapenko, T. Lesort, P. Rodríguez, M. R. Arefin, A. Douillard, I. Rish, L. Charlin, Foundational models for continual learning: An empirical study of latent replay, arXiv preprint arXiv:2205.00329 (2022).
  * [23] T. L. Hayes, K. Kafle, R. Shrestha, M. Acharya, C. Kanan, Remind your neural network to prevent catastrophic forgetting, in: European Conference on Computer Vision, Springer, 2020, pp. 466–483.
  * [24] A. Iscen, J. Zhang, S. Lazebnik, C. Schmid, Memory-efficient incremental learning through feature adaptation, in: European conference on computer vision, Springer, 2020, pp. 699–715.
  * [25] A. Ayub, A. R. Wagner, Storing encoded episodes as concepts for continual learning, arXiv preprint arXiv:2007.06637 (2020).
  * [26] L. Caccia, E. Belilovsky, M. Caccia, J. Pineau, Online learned continual compression with adaptive quantization modules, in: International conference on machine learning, PMLR, 2020, pp. 1240–1250.
  * [27] S. K. Amalapuram, A. Tadwai, R. Vinta, S. S. Channappayya, B. R. Tamma, Continual learning for anomaly based network intrusion detection, in: 2022 14th International Conference on COMmunication Systems & NETworkS (COMSNETS), IEEE, 2022, pp. 497–505.
  * [28] B. Maschler, T. T. H. Pham, M. Weyrich, Regularization-based continual learning for anomaly detection in discrete manufacturing, Procedia CIRP 104 (2021) 452–457.
  * [29] A.-S. Collin, C. De Vleeschouwer, Improved anomaly detection by training an autoencoder with skip connections on images corrupted with stain-shaped noise (2020).  doi:10.48550/ARXIV.2008.12977.
URL https://arxiv.org/abs/2008.12977

  * [30] V. Zavrtanik, M. Kristan, D. Skočaj, Reconstruction by inpainting for visual anomaly detection, Pattern Recognition 112 (2021) 107706\.  doi:https://doi.org/10.1016/j.patcog.2020.107706.
URL https://www.sciencedirect.com/science/article/pii/S0031320320305094

  * [31] W. Liu, R. Li, M. Zheng, S. Karanam, Z. Wu, B. Bhanu, R. Radke, O. Camps, Towards visually explaining variational autoencoders (11 2019).
  * [32] S. Akcay, A. Atapour-Abarghouei, T. P. Breckon, Ganomaly: Semi-supervised anomaly detection via adversarial training (2018).  arXiv:1805.06725.
  * [33] T. Schlegl, P. Seeböck, S. M. Waldstein, U. Schmidt-Erfurth, G. Langs, Unsupervised anomaly detection with generative adversarial networks to guide marker discovery, CoRR abs/1703.05921 (2017).  arXiv:1703.05921.
URL http://arxiv.org/abs/1703.05921

  * [34] V. Zavrtanik, M. Kristan, D. Skočaj, Reconstruction by inpainting for visual anomaly detection, Pattern Recognition 112 (2021) 107706.
  * [35] N. Cohen, Y. Hoshen, Sub-image anomaly detection with deep pyramid correspondences (2021).  arXiv:2005.02357.
  * [36] S. Zagoruyko, N. Komodakis, Wide residual networks, arXiv preprint arXiv:1605.07146 (2016).
  * [37] A. Krizhevsky, I. Sutskever, G. E. Hinton, Imagenet classification with deep convolutional neural networks, Communications of the ACM 60 (6) (2017) 84–90.
  * [38] T. Defard, A. Setkov, A. Loesch, R. Audigier, Padim: a patch distribution modeling framework for anomaly detection and localization (2020).  arXiv:2011.08785.
  * [39] D. Rezende, S. Mohamed, Variational inference with normalizing flows, in: International conference on machine learning, PMLR, 2015, pp. 1530–1538.
  * [40] D. P. Kingma, P. Dhariwal, Glow: Generative flow with invertible 1x1 convolutions, Advances in neural information processing systems 31 (2018).
  * [41] Z. Mai, R. Li, J. Jeong, D. Quispe, H. Kim, S. Sanner, Online continual learning in image classification: An empirical survey, Neurocomputing 469 (2022) 28–51.
  * [42] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, et al., Overcoming catastrophic forgetting in neural networks, Proceedings of the national academy of sciences 114 (13) (2017) 3521–3526.
  * [43] T. Lesort, H. Caselles-Dupré, M. Garcia-Ortiz, A. Stoian, D. Filliat, Generative models from the perspective of continual learning, in: 2019 International Joint Conference on Neural Networks (IJCNN), IEEE, 2019, pp. 1–8.
  * [44] S. Anwar, S. Khan, N. Barnes, A deep journey into super-resolution: A survey, ACM Computing Surveys (CSUR) 53 (3) (2020) 1–34.
  * [45] M. Mirza, S. Osindero, Conditional generative adversarial nets, arXiv preprint arXiv:1411.1784 (2014).
  * [46] Z. Chen, B. Liu, Lifelong machine learning, Synthesis Lectures on Artificial Intelligence and Machine Learning 12 (3) (2018) 1–207.
  * [47] N. Díaz-Rodríguez, V. Lomonaco, D. Filliat, D. Maltoni, Don’t forget, there is more than forgetting: new metrics for continual learning, arXiv preprint arXiv:1810.13166 (2018).
  * [48] D. Lopez-Paz, M. Ranzato, Gradient episodic memory for continual learning, Advances in neural information processing systems 30 (2017).
  * [49] J. Yu, Y. Zheng, X. Wang, W. Li, Y. Wu, R. Zhao, L. Wu, Fastflow: Unsupervised anomaly detection and localization via 2d normalizing flows (2021).  arXiv:2111.07677.
  * [50] J. Pomponi, S. Scardapane, A. Uncini, Pseudo-rehearsal for continual learning with normalizing flows, arXiv preprint arXiv:2007.02443 (2020).
  * [51] P. Kirichenko, M. Farajtabar, D. Rao, B. Lakshminarayanan, N. Levine, A. Li, H. Hu, A. G. Wilson, R. Pascanu, Task-agnostic continual learning with hybrid probabilistic models, arXiv preprint arXiv:2106.12772 (2021).
  * [52] T. Saito, M. Rehmsmeier, The precision-recall plot is more informative than the roc plot when evaluating binary classifiers on imbalanced datasets, PloS one 10 (3) (2015) e0118432.
  * [53] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, S. Hochreiter, Gans trained by a two time-scale update rule converge to a local nash equilibrium, Advances in neural information processing systems 30 (2017).
  * [54] A. Chaudhry, P. K. Dokania, T. Ajanthan, P. H. Torr, Riemannian walk for incremental learning: Understanding forgetting and intransigence, in: Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 532–547.
  * [55] D. D. Pezze, D. Deronjic, C. Masiero, D. Tosato, A. Beghi, G. A. Susto, A multi-label continual learning framework to scale deep learning approaches for packaging equipment monitoring, arXiv preprint arXiv:2208.04227 (2022).


# Bayesian Model Selection on Random Networks

Marios Papamichalis Purdue University, Postdoctoral Researcher at the
Statistics Department<EMAIL_ADDRESS>
###### Abstract

A general Bayesian framework for model selection on random network models
regarding their features is considered. The goal is to develop a principle
Bayesian model selection approach to compare different fittable, not
necessarily nested, models for inference on those network realisations. The
criterion for random network models regarding the comparison is formulated via
Bayes factors and penalizing using the most widely used loss functions.
Parametrizations are different in different spaces. To overcome this problem
we incorporate and encode different aspects of complexities in terms of
observable spaces. Thus, given a range of values for a feature, network
realisations are extracted. The proposed principle approach is based on
finding random network models such that a reasonable trade off between the
interested feature and the complexity of the model is preserved, avoiding
over-fitting problems.

Key words: Model Selection, Random Networks, HM-MCMC.

##  1 Introduction

In this paper, we propose a general approach for generating and comparing
random network models regarding their features. Given a random network model,
we consider the joint distribution between the parameters and features of the
model we are interested in and generate respective models that follows this
distribution. By formulating the joint distribution we can find a part of
distribution which captures a given desired distribution for either the
parameters or the features. Different random network models capture
differently parameters or features, sometimes not at all.

The researcher might be interested in observing a feature of the random
network model (e.g. degree distribution or transitivity) and there is no
obvious model that encodes the features you are interested in making it
impossible or difficult to parametrize a model in terms of those features. He
can only use models that understand, interpret, implement and can actually
fit, because of computational, logistic tools, and carry out a secondary
analysis if this random network model is appropriate for describing this
feature. For example, SBM does not parametrize average path length but is
pretty flexible model and you have the means to fit. We want to answer the
following questions: Which model, that has the resources to fit, can capture
the feature better? How limited is our description? Maybe the model is not
powerful enough to say much things to represent the feature. We want a
criteria that says if some features are well behaving. For instance, a
researcher might be interested in degree distribution and suppose that he can
only fit an Erdös-Rényi or a SBM model with certain parameters, degree density
and number of blocks, respectively. Which one describes better a certain
degree distribution?

Our research relates to the literature on intrinsic Bayes factor, model
selection on parameters and features on random network models and the
literature on informative prior elicitation. In [14] the authors are dealing
with the elicitation of informative priors on graph space that encode
parameters and features. In [17] the author proposes a Bayesian framework for
choosing the number of blocks as well as comparing it to the more elaborate
degree-corrected block models, ultimately leading to a universal model
selection framework capable of comparing multiple modeling combinations. In
[16] they consider an approach based on the log likelihood ratio statistic and
analyze its asymptotic properties under model misspecification in order to
solve problems concerning estimating the latent node labels and the model
parameters than the issue of choosing the number of blocks. They show the
limiting distribution of the statistic in the case of underfitting is normal
and obtain its convergence rate in the case of overfitting. In [12], the
authors do not assume only community based penalties but any specific model
for the network. They make a more general structural assumption of a model
being approximately low rank, which holds for most popular network models.
This is their limitation in contrast with our method. In [6] paper, the
authors are limited in model selection under block models and its variants.
They focus on a generic idea of network cross-validation. Cross-validation is
a very popular and appealing method in many model selection problems. The
adaptation to network data is usually through a node splitting procedure and
has been considered by [9], [15], among others. In [17, 7] the author develop
a Bayesian model selection criterion for stochastic block models which is
inspired by BIC. In [4] the contribution of this paper is the development of a
fully Bayesian model selection method based on a reversible jump Markov chain
Monte Carlo algorithm which estimates the posterior probability for each
competing model. Conceptually, the closest to our research framework is that
of [10] where the authors design and implement MCMC algorithms for computing
the maximum likelihood for four popular models: a power-law random graph
model, a preferential attachment model, a small-world model, and a uniform
random graph model. However there limitation is that their method is confined
by those four models. Moreover, they do not use a loss function as in this we
do in this paper.

Here we propose a methodology for generating and comparing random network
models in such a way that the top model tend to produce posteriors that
preserve simplicity and information for a certain feature when compared to
posteriors obtained from models less favored by the comparison. The selection
of random network models is obtained by decision theory approach. We make the
case that the selection of random networks models implied by our approach are
reasonably consistent with selection implied both by the most widely used loss
functions functions for estimation and prediction.

The approach we propose is creating a joint distribution for a random network
model which encapsulate both the parameters and features of the network. For
this reason we use subsampling procedures by producing network instances for
random network models and computing their joint feature distribution. Next, we
are based on Bayesian model selection concepts and tools and we compare these
models using a Bayesian model selection approach. The criterium for random
network models regarding the comparison is formulated via Bayes factors of the
most widely used loss functions. The rationale behind the proposed approach is
to find random network models such that preserve the best possible trade of
the information provided for the feature and the complexity of the model from
the original data generating mechanism avoiding the problem of over-
fitting.Thus, computational and Bayesian statistics enable us to generate and
compare, in a principled way, random network models.

The paper proceeds as follows: In Section 2, we describe settings of the
problem, we formulate them and give notation and definitions of networks and
random networks. Furthermore, we present, briefly, subsampling and Bayes
factors tools that are useful in the next sections. Then, in section 3, we
focus in our main purpose of this paper which is how we use and compare random
network models by using decision theory in order to preserve the needed amount
of information for a certain given feature or number of features required by
Bayes factors. Conceptually and computationally, our methodology is presented.
In section 4, for many random network models data analysis that gives
experimental results involving decision theory is conducted showing the
results of our approach. Finally, in section 5, we present with more details
the limitation of the method and future work involving overlapping research
areas.

##  2 Preliminaries

###  2.1 Random Networks

We define a network as a pair $G=(V,E)$, where $V$ denotes the set of nodes,
and $E$ the set of edges $E\in V\times V$ . We denote by $A_{G}$ the adjacency
matrix of $G$. Let $N$ denote $\mid V\mid$. A network is called simple if at
most one edge exists between each pair of nodes and no self-loops are allowed.
A network is called undirected if the corresponding adjacency matrix is
symmetric. A random network (or random graph model) is a probability model on
the space of adjacency matrices. In this paper we consider random network
models in the space of simple undirected networks, i.e., a distribution on the
space of binary symmetric adjacency matrices. For the sake of simplicity, we
are using the same symbol ($G$) for a network and a random network. We use
$G(\omega)$ to denote a realization from the random network model. The
simplest example of random network is Erdös-Rényi model, where each possible
edge in the graph is included with a constant probability $p$.

###  2.2 Complexity of a Random network

$x$$y$$z$$(a_{1},a_{2},a_{3})$$x^{{}^{\prime}}$$y^{{}^{\prime}}$$z^{{}^{\prime}}$
Figure 1: Penalizing in the space of parameters is complex as we can see.
$a_{1}$ denotes the community structure. $a_{2}$ denotes the degree
distribution and $a_{3}$ the network subcounts

A mathematical framework has been proposed for analyzing Random Network Models
and for characterizing their complexity. Such framework allows the study of
several network properties or features (link density, clustering coefficient,
degree distribution, connectivity), and their relationship with the random
network model complexity. For doing so, different entropy measures have been
evaluated and their relationship has been assessed. The sample degree
distribution entropy has shown to be correlated with the random network model
entropy, providing a practical measurable indicator of complexity in real
networks. Generally, link density, clustering coefficient, degree
distributions and connectivity are computed and comparatively analyzed in
order to illustrate their relationship with random network model complexity
(Figure 1). The fundamental simulations have been performed by imposing
varying values of the link density and triangle density parameters in ERGMs.

###  2.3 Bayes Factor

One of the central quantities in Bayesian learning is the evidence (or
marginal likelihood) , the probability of the data given the model $P(D\mid
M_{i})$ computed as the integral over the parameters w of the likelihood times
the prior. The evidence is related to the probability of the model,
$P(M_{i}\mid D)$ through Bayes rule:

$P(D\mid M_{i})=\int P(D\mid w,M_{i})P(w\mid M_{i})dw,P(M_{i}\mid D)=P(D\mid
M_{i})P(M_{i})P(D)$ (1)

where it is not uncommon that the prior on models $P(M_{i})$ is flat, such
that $P(M_{i}\mid D)$ is proportional to the evidence. It is typically
impossible to compute analytically. However, the model evidence is crucial for
Bayesian model selection since it allows us to make statements about posterior
model probabilities. The evidence discourages overcomplex models, and can be
used to select the most probable model. It is also possible to understand how
the evidence discourages overcomplex models and therefore embodies Occam’s
Razor by using the following interpretation. The evidence is the probability
that if you randomly selected parameter values from your model class, you
would generate data set $D$. Models that are too simple will be very unlikely
to generate that particular data set, whereas models that are too complex can
generate many possible data sets, so again, they are unlikely to generate that
particular data set at random. To the point, on computation, we will evaluate
the different likelihood integrals in the Bayesian setting with a large-scale
Monte Carlo procedure in almost any case of practical interest.

###  2.4 Intrinsic Expected Losses

As we mentioned in the introduction, many methods for model selection between
random networks have been developed. All of them are penalizing based on one
important attribute of the network which tries to measure the complexity of
the network, as it is described in the previous subsection. Moreover, all of
them select models from the same family of models e.g. are able to select
which SBM fits better the data without overfitting them. Non of the mentioned
methods are able to perform a principled universal model selection between two
different random network models (e.g. compare a certain Barabasi-Albert model
with a certain SBM).

To address the natural question of which model is best for a particular data
set, we propose a model selection criterion for graph models and present a
more flexible principle model selection criterion inspired by Bayesian
Decision Theory penalizing with Expected Losses of one or more specific
features. The Bayes decision is simply the hypothesis with the larger
posterior probability. The posterior expected loss of two models $M_{1}$ and
$M_{2}$ are $K_{1}\times P(M_{1}\mid D)$ and $K_{2}\times P(M_{2}\mid D)$,
respectively. The Bayes decision is again treat corresponding to the smallest
posterior expected loss. In this Bayes test, the null hypothesis is rejected
when:

$\frac{K_{1}}{K_{2}}>\frac{P(M_{1}\mid D)}{P(M_{2}\mid D)}$ (2)

In statistics and decision theory a loss function or cost function is a
function that maps an event or values of one or more variables onto a real
number intuitively representing some ”cost” associated with the event. Simply,
we formalize good and bad results with a loss function. The loss function
determines the penalty for deciding how well a model is behaving in terms of
the feature which is going to penalize it. Some examples involve: i) 0-1 loss
function where $L(\hat{y},y)=I(\hat{y}\neq y)$ and $I$ is the indicator
function. ii) Quadratic Loss function where for a scalar parameter $\theta$, a
decision function whose output $\hat{\theta}$ is an estimate of $\theta$, and
a quadratic loss function $L(\theta,\hat{\theta})=(\theta-\hat{\theta})$ and
iii) absolute Loss function where for a scalar parameter $\theta$, a decision
function whose output $\hat{\theta}$ is an estimate of $\theta$, and a
quadratic loss function $L(\theta,\hat{\theta})=\mid\theta-\hat{\theta}\mid$.

Since the choice of a particular loss function strongly influences the
resulting inference, it seems necessary to rely on intrinsic losses when no
information is available about the utility function of the decision-maker,
rather than to call for classical losses like the squared error loss. Since
this setting is quite similar to the derivation of non-informative priors in
Bayesian analysis, we first recall the conditions of this derivation and
deduce from these conditions some requirements on the intrinsic losses. For
that reason intrinsic loss functions could be used.

###  2.5 Extracting Random graph information

We would like to define a prior such that, when sampling from the prior, we
generate reasonable random networks. Here, reasonable is taken to mean that
the networks should respect graph-theoretical properties that have been
inferred by network properties. A general form of informative log linear
distribution over graph is proposed by [14]. The random graph prior is defined
as a distribution:

$P(G)\propto exp(\lambda\sum_{i}w_{i}f_{i}(G))$ (3)

,in which $\lambda$ is a strength parameter and the parameters $w_{i}$ tune
the relative strengths of the individual concordance functions that capture
several graph-theoretical properties of random network realization properties.
Some examples of concordance functions are: individual edges, controlling the
in-degree of graphs, higher-level Network features, degree distributions,
priors on individual edges and priors on degrees counts.

###  2.6 Exchangeability and Concensus Monte Carlo

For estimating features from exchangeable models which are not straightforward
to relate to the parametrization at hand, we extract the information about the
feature included in the empirical graphon through the corresponding SBM of the
model (each node/edge/measure-exchangeable model ([13], [3], [5], [8]) can be
divided into parts in line with empirical graphon and described by it). This
task is high dimensional (NP-complete) and in order to make it scalable we
divide the model into $n\times n$ cells of 10 nodes each. When exchangeable or
more generally model exchangeable parametric objects, through empirical
graphon, are available then for estimating features from exchangeable models
which are not analytically parametrized e.g. random networks producing
diameters, we extract the information through [14]. Let $f(G)$ be a real-
valued function on graphs that is increasing in the degree to which graph $G$
agrees with prior beliefs (a “concordance function”). For potentially multiple
concordance functions $f_{i}(G)$, we suggest a log-linear network prior of the
form:

$P(G)\propto\exp\\{\lambda\sum_{i=1}^{n^{2}}w_{i}f_{i}(G)\\}=\prod_{i=1}^{n^{2}}\exp\\{\lambda
w_{i}f_{i}(G)\\}$ (4)

, where the $\exp\\{\lambda w_{i}f_{i}(G)\\}$ is one cell of the model which
describes a submodel (e.g one can be scale free distribution and the other
follow a constant distribution).

Then, we merge all the networks into one each time, extracting the feature
(parameter) ([18]). For this, we use concensus monte carlo ([2]), in order to
merge the observables for each feature. Finally, we use all those merged
observables using intrinsic loss functions for each feature. Obviously, this
approach is computationally expensive, though it is scalable, due to model
division in $n\times n$ squared cells (number of cores needed in Map-Reduce
framework). Depending on the number of cores used we can reach complex
networks up to 10.000 nodes.

##  3 Methodology

Figure 2: Procedure for producing observables.

###  3.1 General Concepts

Suppose we have an observed feature (e.g. centrality) and we are given two
different network models (e.g. SBM and Erdös-Rényi). What is the probability
of observing the observed feature given the first model (e.g. SBM) compared
with the probability of observing it given the second model (e.g. Erdös-Rényi
model)? More specifically:

$BF=\frac{Pr(D_{a}\mid M_{1})}{Pr(D_{a}\mid M_{2})}$ (5)

Here $D_{a}$ are networks produced from the concordance function of centrality
for a specific regimes of centralities, $M_{1}$= Centralities produced by a
SBM with parameters $K$, $z$ and $M_{2}$= Centralities produced by an Erdös-
Rényi with parameters $\theta$. $M_{1}$ and $M_{2}$ are produced in the space
of observables. We sample SBM and Erdös-Rényi models and for each realization
we produce networks. For those networks we calculate their centralities. Given
those networks we produce the distribution of centralities for this model (SBM
or Erdös-Rényi respectively). Model comes from prior predictive (Figure 2) in
random variable of centrality $a$: $Pr(a\mid K)$ for SBM and $Pr(a\mid\theta)$
for Erdös-Rényi. If the observed feature is discrete (like diameter which is
the maximum distance between nodes) then we are fine. If the observed feature
is continuous we can compute posterior probability and use a smoothing method
like a kernel density function to get the density.

Figure 3: Practitioner’s tool for tweaking priors.

Another thing we want to achieve is to provide a tool (figure 3) to formulate
a question in terms of ranges, which can be useful for prior elicitation. In
networks is very hard for practitioner to tell you information in terms of
parameters (uncertainty in terms of parameters). Practitioners that are
interested in specific ranges of features and once they have those ranges they
can set candidates for the priors and see which one is more reasonable-better
for those ranges. This tool is powerful in terms of flexibility way for
looking of ranges of features. So we are interested in answering how the prior
of the network should be? That is very hard. Networks are very complicated
objects and hard to be interpreted. Practitioners can look at the distribution
of low dimensional features, get intervals that can be used in ways to
calibrate your prior or to compare different models which could serve as a
prior. We can achieve this by calculating the Bayes Factor:

$BF=\frac{Pr(D\mid M_{1})}{Pr(D\mid M_{2})}$ (6)

Here $D$ are general network data, $M_{1}$= SBM with parameters $K$, $z$ given
$\hat{\alpha}$ and $M_{2}$=Erdös-Rényi with $\theta$ given $\hat{\alpha}$.
$M_{1}$ and $M_{2}$ are produced in the space of observables. We sample SBM
and Erdös-Rényi models and for each realization we produce networks.

For those networks we calculate their centralities. Given those networks we
produce the distribution of centralities for this model (SBM or Erdös-Rényi
respectively). Model comes from: $Pr(K,\hat{a})$ in SBM and
$Pr(\theta,\hat{a})$ in Erdös-Rényi.

###  3.2 Model Selection of networks with respect to more than one feature.

Which model is better to fit to network data $D$? Model $M_{1}$ or model
$M_{2}$? Until now we use the Bayesian factor test for constructing a
principle Bayesian model selection test. How do we combine three universal
features such as community structure, degree distribution and network
subcounts in a model selection test in a plausible principle way that makes
sense? The penalty of the one might dominate the other and we might encounter
problems where e.g. community structure is much more dominant than degree
distribution or the opposite because of their values (large, small).

The Left Hand Side in (8), (9) and (10) is a number which absorbs (cancel out)
any scaling problem from (11). You do not need to scale the axis every time to
find the right portion of inclusion of each parameter’s penalty. Q.E.L. stands
for the quadratic expected loss function in observable space. $D_{M_{1,2}}$
are the networks which are produced by the models $M_{1,2}$, respectively, and
which we want to compare. We extract their features and compare them (Number
of blocks, entropy of Degree distribution and number of Subgraph counts) which
the corresponding features of the network data $D$. Which model could be
fitted best regarding those three universal features? With this approach we do
not encounter any problem regarding a dominance feature (scaling penalties) as
in (5) which is derived if we directly use all the penalties from [1]

$\frac{Q.E.L(D_{M_{1}},D)_{Blocks}}{Q.E.L(D_{M_{2}},D)_{Blocks}}><\frac{P(M_{1}\mid
D)}{P(M_{2}\mid D}$ (7)

$\frac{Q.E.L(D_{M_{1}},D)_{D.D.}}{Q.E.L(D_{M_{2}},D)_{D.D.}}><\frac{P(M_{1}\mid
D)}{P(M_{2}\mid D}$ (8)

$\frac{Q.E.L(D_{M_{1}},D)_{Motifs}}{Q.E.L(D_{M_{2}},D)_{Motifs}}><\frac{P(M_{1}\mid
D)}{P(M_{2}\mid D}$ (9)

Summing the above 3 equations and divide by 3 we have:

$\frac{\frac{Q.E.L(D_{M_{1}},D)_{Blocks}}{Q.E.L(D_{M_{2}},D)_{Blocks}}+\frac{Q.E.L(D_{M_{1}},D)_{D.D.}}{Q.E.L(D_{M_{2}},D)_{D.D.}}+\frac{Q.E.L(D_{M_{1}},D)_{Motifs}}{Q.E.L(D_{M_{2}},D)_{Motifs}}}{3}><\frac{P(M_{1}\mid
D)}{P(M_{2}\mid D}$ (10)

##  4 Simulation Studies

We will perform three types of simulations. In all of them we will use
Barabasi-Albert model and Stochastic Block Model. For the Barabasi-Albert
model we will produce networks given the power law degree distribution
sampling. For the Stochastic Block Model we will use [11] method. The three
simulation setups are described below. The number of samples is N=100.

Random Network Produced Read Data Parameters Tested Parameters Barabasi Albert
$\alpha=3.2$ $P(\alpha=[2.9,3.1]\mid D)$ Stochastic Block Model $K$=10 blocks
$P(K=9\mid D)$

Table 1: Real networks are constructed through Barabasi-Albert and Stochastic
Block Model

Local Feature Power Law SBM Degree distribution through entropy

Table 2: Feature according to which we select a model

Loss Function Expression Type of Inference Quadratic Loss
$L(\theta,\hat{\theta})=(\theta-\hat{\theta})^{2}$ point estimation Absolute
Loss $L(\theta,\hat{\theta})=\mid\theta-\hat{\theta}\mid$ point estimation

Table 3: Loss functions used for penalties

Real Param. L.F. Ratio Tested Parameters Probabilities Local Feature $\alpha$
Q.L.=1493 $P(\alpha=[2.9,3.1\mid D)=0.67,P(K=9)=0.010$ Power Law $\alpha$
A.L=0.0328 $P(\alpha=[2.9,3.1\mid D)=0.67,P(K=9)=0.010$ Power Law $K$
Q.L.=31.082 $P(\alpha=[2.9,3.1\mid D)=0.03,P(K=9)=0.37$ SBM $K$ A.L.=8.0287
$P(\alpha=[2.9,3.1\mid D)=0.03,P(K=9)=0.37$ SBM $\alpha=3.2$ Q.L.=0.8685
$P(\alpha=[2.9,3.1\mid D)=0.02,P(K=9)=0.03$ Power law and D.D. $\alpha=3.2$
A.L.=0.241 $P(\alpha=[2.9,3.1\mid D)=0.02,P(K=9)=0.03$ Power law and D.D.

Table 4: Results of the loss functions

###  4.1 Simulation Set Up

As we can see from Table 5 when the data resemble more to a power law
distribution it much more possible to select models that follow the power law
distribution and have similar behaviour (their exponents are very close). On
the other hand, when the network data are extracted from a SBM then it is much
more likely to select models with a SBM with number of blocks close to one
that the read network date were constructed. The same happened with degree
equals to 0.4 but here the results are more ambiguous that in the other two
cases. As we can observe from the last two rows from in the case of the
combination of two loss functions, one for power law and one for the degree
obviously the power law fraction dominates the degree distribution fraction.
Still in that case power law distribution is much more possible to select
models that follow the power law distribution and have similar behavior.

##  5 Discussion

Our methodology constitutes an attempt for comparing random network models
regarding local features by using intrinsic Bayesian factor. We investigated
how the following three interact: i)feature that you have phrase your
scientific questions ii) features that are formulated by the model you are
fitting and iii) features that can be retrieved more efficiently having sample
size you have. For example: i) scientific question in terms of transitivity
ii) you do not have a model that can be generated in terms of transitivity and
iii) the compromise in terms of the feature what you want to capture and the
model you can fit.

The need to compare random networks is fundamental to many fields, from the
physical and life sciences to the social, behavioral, and economic sciences.
This need is currently particularly significant in understanding information
and incorporating complexity of random networks in network model selection.
Although all model selection methods address the inevitable trade-off between
goodness-of-fit and complexity, the manner in which they measure and penalize
model complexity can differ substantially. The main advantage of our method is
that it enables the statistician, for the first time, to compare random
network models, both nested and non-nested and overcome the complexity
problem. To this end, we have shown, through a comparative analysis of several
rich and varied examples that the results obtained suggest that our method is
both reasonable and universal.

The limitations of the approach in its current form include: i) Prior
information of the data have to be incorporated to the complexity of the model
in order to penalize the model, so it is reasonable the model to be related
with the data ii) In multidimensional random network models computational
costs might be an issue.

Future work includes: i) how can we reduce the cost of our algorithms in terms
of complexity in high dimensional random network model features in the model
selection ii) is it possible to create concordance function for every local
feature regimes? ii) is there another way to penalize models due to features
like e.g. modularity, motifs work or use two dimensional entropy of a graphon,
as a penalty? iv) is it possible to connect asymptotically (e.g. through von
Mises theorem) this approach for node-exchangeable models with the work in
[16]?

## Acknowledgement

The author thanks Simón Lunagómez from Department of Mathematics and
Statistics at Lancaster University for his valuable comments.

## References

  * [1] James O. Berger  “Statistical Decision Theory and Bayesian Analysis”  In _Springer Series in Statistics_ , 1985
  * [2] Steven L., Alexander W. Blocker and Fernando V. Bonassi  “Bayes and Big Data: The Consensus Monte Carlo Algorithm”  In _International Journal of Management Science and Engineering Management_ 11.2, 2013, pp. 78–88
  * [3] Tamara Broderick and Diana Cai  “Edge-exchangeable graphs and sparsity”  In _Advances in Neural Information Processing Systems 29 (NIPS)_ , 2016
  * [4] Alberto Caimo and Nial Friel  “Bayesian model selection for exponential random graph models”  In _Social Networks_ 35.1, 2013, pp. 11–24
  * [5] Francois Caron and Emily B. Fox  “Sparse graphs using exchangeable random measures”  In _Royal Statistical Society(B)_ , 2017
  * [6] K. Chen and J. Lei  “Network cross-validation for determining the number of communities in network data”  In _Journal of the American Statistical Association_ , 2017
  * [7] Christian Borgs, Jennifer T., Hendry Cohn and Victor Veitch  “Estimating the dimension of a model”  In _The annals of statistics_ 6.2, 1978, pp. 461–464
  * [8] Harry Crane and Walter Dempsey  “Edge exchangeable models for network data”  In _Journal of the American Statistical Association_ 113, 2016
  * [9] Airoldi E.., Blei D.., Fienberg S. E. and Xing E. P.  “Mixed membership stochastic blockmodels”  In _The Journal of Machine Learning Research_ 9, 2008, pp. 1981–2014
  * [10] Ivona Bezakova, Adam Kalai and Rahul Santhanam  “Graph Model Selection using Maximum Likelihood”  In _ICML ’06: Proceedings of the 23rd international conference on Machine learningJune_ , 2006, pp. 105–112
  * [11] Pierre Latouche and Stephane Robin  “Variational Bayes Model Averaging for Graphon Functions and Motif Frequencies Inference in W-graph Models”  In _Stat Comput_ 26, 2015, pp. 1173–1185
  * [12] Tianxi Li, Elizaveta Levina and Ji Zhu  “Network cross-validation by edge sampling”  In _arXiv:1612.04717v6_ , 2018
  * [13] L. Lovász  “Large networks and graph limits”  In _Amer. Math. Soc._ 60 Amer. Math. Soc., 2012
  * [14] Sach Mukherjee and Terence P. Speed  “Network inference using informative priors”  In _PNAS_ 105.38, 2009, pp. 14313–14318
  * [15] Neville J., Gallagher B., Eliassi-Rad T. and T. Wang  “Correcting evaluation bias of relational classifiers with network cross validation”  In _Knowledge and information systems_ 30.1, 2012, pp. 31–55
  * [16] Y.X. Wang and Peter J. Bickel  “Likelihood-Based model selection for Stochastic Block Model”  In _Annals of Statistics_ , 2016
  * [17] Xiaoran Yan  “Bayesian Model Selection of Stochastic Block Models”  In _ASONAM ’16: Proceedings of the 2016 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining_ , 2016, pp. 323–328
  * [18] Xiaowei Ying and Xintao Wu  “Graph Generation with Precribed Feature consraints”  In _SIAM_ , 2009