WGMwgm QEqe EPep PMSpms BECbec DEde

[0]E-mail<EMAIL_ADDRESS>(R. Yan)

[type=editor, auid=000, orcid=0000-0003-2798-4655, ] [1]

[style=chinese] [2] [ ] [3] [ ] [3] [ ] [3] [ ] [3] [ ] [4] [ ] [1] [4] [ ]
[4] [1] [ ] [4]

#  CAINNFlow: Convolutional block Attention modules and Invertible Neural
Networks Flow for anomaly detection and localization tasks

Ruiqing Yan  Fan Zhang  Mengyuan Huang  Wu Liu  Dongyu Hu  Jinfeng Li  Qiang
Liu  Jinrong Jiang  Qianjin Guo  Linghan Zheng

###### Abstract

Detection of object anomalies is crucial in industrial processes, but
unsupervised anomaly detection and localization is particularly important due
to the difficulty of obtaining a large number of defective samples and the
unpredictable types of anomalies in real life. Among the existing unsupervised
anomaly detection and localization methods, the NF-based scheme has achieved
better results. However, the two subnets (complex functions) $s_{i}(u_{i})$
and $t_{i}(u_{i})$ in NF are usually multilayer perceptrons, which need to
squeeze the input visual features from 2D flattening to 1D, destroying the
spatial location relationship in the feature map and losing the spatial
structure information. In order to retain and effectively extract spatial
structure information, we design in this study a complex function model with
alternating CBAM embedded in a stacked $3\texttimes 3$ full convolution, which
is able to retain and effectively extract spatial structure information in the
normalized flow model. Extensive experimental results on the MVTec AD dataset
show that CAINNFlow achieves advanced levels of accuracy and inference
efficiency based on CNN and Transformer backbone networks as feature
extractors, and CAINNFlow achieves a pixel-level AUC of $98.64\%$ for anomaly
detection in MVTec AD.

###### keywords:

Computer Version Normalization Flow Invertible Neural Networks Convolutional
Block Attention Modules Unsupervised Anomaly Detection and Location

##  1 Introduction

Artificial intelligence anomaly detection system is widely used in
manufacturing error detection Roth et al. (2021), network intrusion detection
Bergman and Hoshen (2020), and diagnosis of medical cases Wang et al.
(2021).Anomaly detection and positioning technology are used to detect and
locate anomalies from samples. In the field of industrial manufacturing, it is
used to timely find defects in industrial equipment and industrial products
that are not easily noticed by people, to ensure product quality and improve
industrial production efficiency. In traditional anomaly detection methods,
the training sample is roughly divided into samples of normal and abnormal
samples, however, in real life, it is difficult to get a lot of defective
samples Li et al. (2021), the abnormal sample distribution imbalance caused by
abnormal types at the same time uncertainly, training can’t contain all the
abnormal types, so to break through the limitations, we use the form of
distribution fitting to train the model with unlabeled data. We use a
distribution transformation structure based on a reversible neural network to
transform the distribution corresponding to the normal sample into the normal
distribution. On the contrary, if the final distribution is not normal, then
the input image is abnormal. Our method is mainly composed of a feature
extraction module and distribution estimation module, which are responsible
for image feature extraction and feature distribution transformation
respectively.

In the feature extraction module, we choose spatial structure information of
the image, in the field of computer vision, used in spatial structure
information of image extraction model mainly divided into two kinds, and based
on the Transformer based on convolution, among them, the classic model based
on convolution is the ResNet, it through the residual structure can solve the
problem of gradient disappeared, it can effectively extract spatial structure
information of images. Recently, Transformer based machine vision model has
gradually become a new paradigm, leading the traditional model based on
Convolutional Neural Network (CNN) in a variety of scenarios. For example, ViT
Dosovitskiy et al. (2020) imitates Bert’s way of processing natural language
and extracts the features of patch embedding sequence by Decoder. Although
Transformer based machine vision model has achieved great success,
convolution-based ConvNeXt still has faster processing speed and higher
accuracy than Swin Transformer Liu et al. (2021) under the same FLOPs, which
shows us that the CNN structure still has a certain potential. CAINNFlow is a
plug-in structure that can be connected to any image feature that extractor
for distribution transformation, regardless of the structure based on which
image feature extractor is formed.

In the distribution estimation module, we transform the distribution and
convert the original probability distribution of the output features of the
feature extractor into normal distribution and other distributions (or the
original probability distribution itself).In this module, we make assumptions
about the distribution of data that based on statistical methods and find out
the ”anomalies” defined under the assumptions. Recently, a method named NF has
been proposed Cunningham et al. (2020). Flow-based NF can transform an
arbitrary probability distribution into another arbitrary probability
distribution, Such as the normal distribution converts into other arbitrary
distributions, including the normal distribution itself. Improved GAN, VAE,
and other traditional methods can not accurately evaluate the probability
distribution and reasoning defects. However, the original one-dimensional
normalized flow model required flattening the two-dimensional input features
into one-dimensional vectors to estimate the distribution, which limited the
flow model’s ability and destroyed the inherent spatial position relationship
of two-dimensional images Cunningham et al. (2020).To solve this problem,
CAINNFlow is proposed, that is, CNN structure embedded in CBAM(Convolutional
Block Attention Model) Woo et al. (2018) is used to retain the spatial
location of input samples, which enables CAINNFlow to effectively extract
spatial structure information while preserving spatial structure information.
In addition, in the traditional structure of the Convolutional structure, the
Attention of each point and channel in the space is the same, which enables
the Model to focus on important features and suppress unnecessary features.
Using the structure of CBAM, effectively improve the effect of anomaly
detection.

As shown in the figure: 1, we first used the feature extractor pre-trained in
the large-scale open field to extract the spatial features of the image, and
then input the obtained spatial feature tensor into CAINNFlow to realize
anomaly detection and location. During training, CAINNFlow can transform the
distribution corresponding to the spatial features of normal images into
normal distribution through parameter updating. In the transformation,
CAINNFlow can retain the spatial structure information in the features through
the stacked CNN structure embedded with CBAM and extract it effectively.
During the test, we calculate the distance between the model input image
distribution and the standard normal distribution to obtain the anomaly score
of each pixel, to achieve pixel-level anomaly detection and location. In
general, we use a visual feature extractor based on a neural network for
visual feature extraction, and then use CAINNFlow for distribution
transformation of reserved spatial information to realize anomaly detection
and pixel-level location.

Figure 1:  An example of the proposed Flow. Flow transforms features of the
input image from the original distribution to the standard normal
distribution. The features of the normal area in the input image fall in the
center of the distribution, while the abnormal features are far away from the
center of the distribution

The main contributions of this paper are summarized as follows:

$\bullet$ The CAINNFlow proposed by us retains spatial information during
distributed transformation not only uses CNN structure embedded with CBAM to
effectively extract spatial location information but also information of each
channel to improving the accuracy of anomaly detection.

$\bullet$ CAINNFlow is a straight-in structure that acts as a distributed
converter and can be attached to any feature extractor, which can be any
computer vision model using neural networks, such as ResNet, ViT, and
ConvNeXt.

$\bullet$ The experimental results using the MVTec Bergmann et al. (2019) AD
anomaly detection data set show that the CAINNFlow method we proposed,which
indexes with fewer parameters, achieves the advanced level of image-level
AUROC(Area Under the Receiver Operating Characteristic) and pixel-level AUROC.

##  2 Related Work

###  2.1  Anomaly detection method

Anomaly detection is the most common method in industrial production. One of
the most commonly used methods is unsupervised anomaly detection and location,
which can effectively detect and locate anomalies Roth et al. (2021). Because
in anomaly detection in real life, it is difficult to get a lot of defective
sample sample distribution caused by serious imbalance, and the
unpredictability of the kinds of causes training focus to include all types of
abnormal, made with limited supervision and the application of anomaly
detection, unsupervised anomaly detection and location is particularly
important. In order to be close to the actual situation, the anomaly check
data set of MVTec Bergmann et al. (2019) was used as the main supporting data
set in the experiment in this study. It contains $5354$ high-resolution color
images of different object and texture classes. It contains normal (that is,
defect-free and unlabeled) images for training and abnormal images for
testing. Anomalies are represented by more than 70 different types of defects,
such as scratches, dents, contamination and various structural changes. This
is the first comprehensive multi-target, multi-defect anomaly detection
dataset, which is taken from real industrial scenarios and provides pixel-
level anomaly annotation.

In addition, during 2017, Y. Feng et al used PCANet to extract both appearance
and motion features from 3D gradients on two publicly available datasets, the
UCSD Ped1 Dataset and the Avenue Dataset, and video events were automatically
represented and modeled in an unsupervised manner. A deep Gaussian Mixture
Model (GMM) is constructed using the observed normal events. It is proved that
the depth model is effective in detecting abnormal events in video
surveillance Feng et al. (2017). In 2018, S. Wang et al. modeled the features
of ordinary video events on UCSD PED1, PED2, and UMN datasets, and introduced
an extreme learning machine (OC-ELM) as the data description algorithm. A new
method for automatic detection and location of video anomalies is proposed,
which can achieve the most advanced results in both video anomaly detection
and location tasks Wang et al. (2018). In 2019, Nanjun Li et al conducted
experiments on three common data sets of UCSD dataset, UMN dataset, and Avenue
dataset through Multi-variable Gaussian Full Convolution Adversarial
Autoencoder (MGFC-AAE). A video anomaly detection and location method based on
deep learning is proposed. Li and Chang (2019). M. Canizo et al. proposed an
architecture of supervised multi-time series anomaly detection method based on
deep learning, multi-head CNN-RNN(Regressive Neural Network), which combined
CNN and RNN in different ways. Different from other methods, we use
independent CNN, called convolution head, to handle anomaly detection in
multi-sensor systems. This architecture is suitable for multi-time series
anomaly detection and has achieved good results in actual industrial scenes
Canizo et al. (2019). In 2020, X. Zhang et al. proposed a method based on
scene perception combining fluid force expression and psychological theory,
introduced the line integral convolution flow field visualization technology
to segment the moving pedestrians in the scene, and proposed a clustering
strategy guided by scene perception to cluster the consistent crowd.
Experimental results show that the proposed method achieves higher accuracy in
both frames and pixel-level measurement than the existing methods. Zhang et
al. (2020). Labiba Gillani Fahad et al. executed a comprehensive activity
level assessment of the proposed method on two smart home datasets and used
probabilistic neural networks for the classification of presegmented activity
instances. It is used to recognize abnormal phenomena in multi-resident
activities Fahad and Tahir (2021). In this study, pixel-accurate Ground
Truth(GT) was provided for all anomalies, and the current advanced
unsupervised anomaly detection methods were evaluated Yu et al. (2021), and
compared with the CAINNFlow method adopted by us.

###  2.2 Feature Extraction

With the gradual integration of artificial intelligence into people’s
production and life, deep learning has been widely used in computer vision.
Currently, the commonly used methods of feature extraction are respectively
based on convolutional Neural Network (CNN) and Transformer. Some studies use
ResNet to extract object features. For example, the residual network (ResNet)
adopted by Microsoft Li et al. (2016) won the first place in ILSVRC-2015 with
an astonishing error rate of $3.6\%$. The 152-layer network they use has lower
complexity than the VGG Simonyan and Zisserman (2014) network and solves the
degradation problem by introducing residual learning of deep connections.
While Transformer has been a big success in Natural Language Processing(NLP),
Vision Transformer extends the Transformer model architecture into the realm
of computer Vision. Transformer is a good replacement for convolution
operations and can still achieve good results in computer vision tasks without
relying on convolution. The convolution operation can only consider local
feature information, while the attention mechanism in Transformer can
comprehensively consider global feature information. In order to be as
compatible as possible with Transformer related structures in the NLP domain,
Vision Transformer directly migrates Transformer from the NLP domain to the
computer Vision domain without changing the Encoder architecture in
Transformer as much as possible. Recently, DeiT Touvron et al. (2021a), which
is realized by Transformer and distillation technology without any
convolution, can get good results only by training on ImageNet with relatively
less computation. Later, based on DeiT’s work, CaiT Touvron et al. (2021b)
made Deep Vision Transformer easy to converge and improve accuracy through
LayerScale, and CaiT adopted class-attention layers. Make the model more
efficient for class token processing. Recently, however, a ConvNeXt Liu et al.
(2022) based on the CNN model design framework emerged, which surpassed the
performance of such models as ViT based on Transformer in terms of overall
performance. To sum up, both CNN-based and Transformer Based computer vision
models have their own advantages and disadvantages, so the feature extractor
modules selected in this study include Transformer Based and CNN Based, both
of which are combined with CAINNFlow for experiments.

###  2.3 Normalizing Flow

In recent years, researchers have developed many methods to learn probability
distributions of data sets, including Generative Adversarial Networks (gans),
Variational Self-encoders (VAE), and Normalizing Flow. However, both GAN and
VAE lack accurate evaluation and inference of probability distribution, which
leads to poor quality of fuzzy results in VAE, and GAN training also faces
problems such as pattern collapse. To overcome the deficiency of GAN and VAE,
relevant researchers proposed Normalizing Flow. NF based on Flow can transform
random probability distribution into another random probability distribution,
while GAN can only transform a random vector into an image, for example,
normal distribution into another random distribution including the normal
distribution itself. Compared with GAN and VAE, NF can transform the
probability distribution corresponding to complex data into a simple
probability distribution, for example, the probability distribution
corresponding to images in MNIST data into a simple normal distribution, and
vice versa, the simple normal distribution can be transformed into complex
probability distribution corresponding to images in MNIST.

Therefore, based on the advantages of NF above, we propose CAINNFlow based on
NF to achieve distributed transformation on the basis of retaining and
effectively extracting spatial structure information.

##  3 Methodology

###  3.1 Definition And Methods of NF

The general representation of Normalized Flow refers to the gradual
approximation of complex distribution to simple Gaussian distribution by using
multiple nested reversible functions Yu et al. (2021). The original
distribution has several nested reversible functions and then transforms into
any other distribution (including the original distribution itself), where the
corresponding increase or decrease in the probability area is the product of
the Jacobian determinant of all the reversible functions. Normalizing Flow
Cunningham et al. (2020) is essentially a series of reversible functions, so
Normalizing Flow is reversible. Meanwhile, the probability density
distribution of samples can be converted back to its corresponding original
distribution through the reverse process of Normalizing Flow Agnelli et al.
(2010).

In simple terms, Normalizing Flow is a set of invertible functions, or the
analytic inverse of these functions can be computed.For example, $f(x)=x+2$ is
a reversible function because every input has and only one unique output, and
vice versa, and $f(x)=x^{2}$ is not a reversible function.Such functions are
also called bijective functions.

In particular, I’m given an invertible
mapping$f:\mathbb{R}^{d}\xrightarrow{}\mathbb{R}^{d}$, and use it to put
random variables $z~{}q(z)$ Transform to a new variable $z^{{}^{\prime}}=f(z)$
After, the distribution of the new variable
is:$q\left(x^{\prime}\right)=g(z)\left|\operatorname{Let}\frac{\partial
f^{-1}}{\partial x^{2}}\right|=q(z)\left|\operatorname{det}\frac{\partial
f}{\partial z}\right|^{-1}$ .Then, in order to build a sufficiently complex
distribution, we can design the model structure with multiple similar
reversible mappings and nested sequences of mappings:

Figure 2: invertible process

$\displaystyle\mathbf{v}_{1}=\mathbf{u}_{1}\odot\exp\left(s_{2}\left(\mathbf{u}_{2}\right)\right)+t_{2}\left(\mathbf{u}_{2}\right)$
(1)
$\displaystyle\mathbf{v}_{2}=\mathbf{u}_{2}\odot\exp\left(s_{1}\left(\mathbf{v}_{1}\right)\right)+t_{1}\left(\mathbf{v}_{1}\right)$

In the figure: 2, the basic building block of a reversible neural network is
the affine coupling layer generalized by the Real NVP model. Its working
principle is to divide the input data into two parts $u_{1}$ and $u_{2}$,
which are converted by learning functions $s_{i}(u_{i})$ and $t_{i}(u_{i})$
and coupled alternately.

The specific operation is as follows: firstly, input data $u_{2}$ is
substituted into function $s_{2}$, the result of $s_{2}(u_{2})$ is substituted
into base E for exponential operation, and then the result of input data
$u_{1}$ and $exp(s_{2}(u_{2}))$ are dot multiplied to get $u_{1}\bigodot$
After $exp(s_{2}(u_{2}))$, add the result with the result of input data
$u_{2}$ into the function $T_{2}$, and finally get our output node $v_{1}$.
Similarly, we can perform a series of operations to get the output node
$v_{2}$.

Figure 3: Probability transformation process

$\mathbf{z}_{K}=f_{K}\circ\ldots\circ f_{2}\circ
f_{1}\left(\mathbf{z}_{0}\right)$ (2)

In this way,$\ln q_{K}\left(\mathbf{z}_{K}\right)=\ln
q_{0}\left(\mathbf{z}_{0}\right)-\sum_{k=1}^{K}\ln\left|\operatorname{det}\frac{\partial
f_{k}}{\partial z_{k-1}}\right|$ When you calculate the distribution after the
transformation $q_{k}$. Value is not evaluated explicitly $q_{k}$ But by the
initial distribution $q_{0}$. Through the above process, NF can transform the
original distribution into a new distribution with a series of reversible
mappings. By optimizing the variable parameters of the bijective function
during training, the bijective function can transform the basic distribution
into an arbitrary distribution. Each bijective function can be written as a
layer of a network, with the optimizer learning the parameters and finally
fitting the real data. By using the maximum likelihood estimation method, the
distribution problem of fitting real data is changed into the log-likelihood
problem of the probability after fitting transformation, and the log-
likelihood problem is also used for the stability of calculation. The
traditional Normalized Flow has the following characteristics Kobyzev et al.
(2020) :

$\bullet$ A mapping from input to output is bijective, that is, its inverse
function exists.

$\bullet$ Both forward and reverse mappings are effectively computable.

$\bullet$ The map has a tractable Jacobian determinant, so the probabilities
can be explicitly converted by variable formulas.

The proposal of NF provides an effective idea and method to solve the fitting
problem of complex distribution. Reversible functions can map the vector space
in the Cartesian coordinate system into different vector Spaces and reversely
map back to the original vector space using the inverse operation of
reversible functions. It is worth noting that the area of the vector space is
changed in the above mapping process by the Jacobian determinant of the
reversible function.

Since some information is lost in the forward process, an additional potential
output variable $z$ is introduced, which is trained to capture information
related to $x$ but not contained in $y$. In addition, the training network
needs to adjust $p(z)$ according to the Gaussian distribution. Namely,
$p(x|y)$ is adjusted to a certain function $x=g(y,z)$, this function will be
known to the distribution of $p(z)$ in the case of meet $y$ to $x$ space.

If $x\in RD$ $y\in RM$, then due to the loss of information in the forward
process, the inherent dimension m of $y$ must be less than $D$, even though
$M$ may be greater than $D$. We hope that based on the model $q(x|y)$ to
predict the $\rho(x|y)$, with the introduction of implicit variable $z$ and
$q(x|y)$ can be in $g(y,z;\theta)$ appears as follows:

$\displaystyle\mathbf{x}=g(\mathbf{y},\mathbf{z};\theta)\quad\text{ with
}\quad\mathbf{z}\sim p(\mathbf{z})=\mathcal{N}\left(\mathbf{z};0,I_{K}\right)$
(3)

Correspondingly, the forward process can also be expressed by $f(x;\Theta)$
said:

$\displaystyle\lceil\mathbf{y},\mathbf{z}]=f(\mathbf{x};\theta)$
$\displaystyle=\left[f_{\mathbf{y}}(\mathbf{x};\theta),f_{\mathbf{z}}(\mathbf{x};\theta)\right]=g^{-1}(\mathbf{x};\theta)$
(4) $\displaystyle f_{\mathbf{y}}(\mathbf{x};\theta)$ $\displaystyle\approx
s(\mathbf{x})$

Bidirectional training $f$ and $G$ can avoid the problems in GAN and Bayesian
neural networks. Since INN requires $f=G-1$, the dimensions of both sides
(whether inherent or displayed) should be the same, requiring the dimension of
variable $Z$, $K=D--m$. If it causes $M+K>D$, we need to complement the $x$
vector with the $0$ vector of the $M+k-d$ dimensions.

Combination of all these definitions, our network will be $q(x|y)$ is
expressed as:

$\displaystyle q(\mathbf{x}$
$\displaystyle=g(\mathbf{y},\mathbf{z};\theta)\mid\mathbf{y})=p(\mathbf{z})\left|J_{\mathbf{x}}\right|^{-1}$
(5) $\displaystyle J_{\mathbf{x}}$
$\displaystyle=\operatorname{det}\left(\left.\frac{\partial
g(\mathbf{y},\mathbf{z};\theta)}{\partial[\mathbf{y},\mathbf{z}]}\right|_{\mathbf{y},f_{\mathbf{z}}(\mathbf{x})}\right)$

By alternating forward and backward iterations and accumulating bidirectional
gradients before parameters are updated, the input and output errors are
reduced and training is more efficient.

$s_{i}(u_{i})$ and $t_{i}(u_{i})$ subnets (complex functions) in NF are
usually multi-layer perceptrons, which need to flatten and extrude the input
visual features from $2D$ to $1D$, thus destroying the spatial position
relationship in the feature graph and losing the spatial structure
information. To retain and effectively extract spatial structure information,
we designed a complex function model with alternating CBAM embedded in the
stacked $3\texttimes 3$ full convolution in this study, which can retain and
effectively extract spatial structure information in the Normalized Flow
model. Its model structure is the combination of two-dimensional convolution
and CBAM structure. CBAM is a convolution attention module that integrates
channel attention and spatial attention mechanism, and the tensor shape of
input and output of CBAM is consistent so that it can be seamlessly integrated
into the structure constituted by CNN. Therefore, we propose a structure named
CAINNFlow that can retain spatial information and extract spatial information
effectively.

As the characteristic distribution of abnormal samples is different from that
of normal samples, the likelihood value of abnormal samples obtained by
CAINNFlow should be lower than that of normal samples. Therefore, the
likelihood value of test samples can be used as the anomaly score to detect
and locate anomalies.

###  3.2 CBAM

Figure 4: The overview with CBAM

CBAM is a convolution attention module proposed by Sanghyun et al., which
integrates channel attention and spatial attention mechanism Woo et al.
(2018). CBAM can be seamlessly integrated into CNNs and can conduct end-to-end
training with CNNs.

The structure of CBAM is shown in the figure: 4 below, which has two sub-
modules: channel attention and spatial attention. Place the two modules
sequentially in CBAM. Given a feature graph $F\in R^{C\texttimes H\texttimes
W}$, input it into the channel attention module will generate channel
attention diagram $Mc\in R^{C\texttimes 1\texttimes 1}$, which can be obtained
by multiplying with $F$ element $F^{{}^{\prime}}\in R^{C*H*W}$. Then as the
input, the two-dimensional spatial attention graph $Ms\in R^{1\texttimes
H\texttimes W}$ is generated through the spatial attention module, and then
the sum is made the elements are multiplied to get the final output.It is
worth noting that we use a cascade structure of ”channel attention first,
space attention second”, and this stacked topology performs better in other
experiments. The calculation process of CBAM is as follows:

$\displaystyle\mathrm{F}^{\prime}$
$\displaystyle=\mathrm{M}_{\mathrm{c}}(\mathrm{F})\otimes\mathrm{F},$ (6)
$\displaystyle\mathrm{F}^{\prime\prime}$
$\displaystyle=\mathrm{M}_{\mathrm{s}}(\mathrm{F}^{\prime})\otimes\mathrm{F}^{\prime}$

where $\otimes$denotes element-wise multiplication. During multiplication, the
attention values are broadcasted (copied) accordingly: channel attention
values are broadcasted along the spatial dimension, and vice versa.
$F^{{}^{\prime\prime}}$ is the final refined output.

Figure 5: Channel Attention Module

The structure of the channel attention module is shown in the figure: 5.In the
channel attention module, the maximum pooling and average pooling operations
are used to compress each channel, so that the information on the input
feature graph $F$ is aggregated into the feature graph $F^{c}_{avg}\in
R^{C\texttimes 1\texttimes 1}$ and $F^{c}_{max}\in R^{C\texttimes 1\texttimes
1}$. $F^{c}_{avg}$ and $F^{c}_{max}$ represent Average-pooled features and
max-pooled Features, respectively. The two feature maps are then input into a
shared network of a multi-layer perceptron. To reduce parameter overhead, the
hidden activation size is set to $R^{C/r\texttimes 1\texttimes 1}$, where R is
the reduction ratio. Add the elements of the two outputs of the shared network
and use the Sigmoid activation function to generate our channel. Note the
graph $Mc\in R^{C\texttimes 1\texttimes 1}$. The calculation process of
channel attention is as follows:

$\displaystyle\mathbf{M}_{c}(\mathbf{F})$
$\displaystyle=\sigma(\operatorname{MLP}(\operatorname{Avg}\text{ Pool
}(\mathbf{F}))+\operatorname{MLP}(\operatorname{Max}\text{ Pool
}(\mathbf{F})))$ (7)
$\displaystyle=\sigma\left(\mathbf{W}_{\mathbf{1}}\left(\mathbf{W}_{0}\left(\mathbf{F}_{\text{avg
}}^{\mathrm{c}}\right)\right)+\mathbf{W}_{\mathbf{1}}\left(\mathbf{W}_{0}\left(\mathbf{F}_{\max}^{\mathrm{c}}\right)\right)\right)$

where $\sigma$ denotes the sigmoid function, $W_{0}\in R^{C/r\texttimes C}$,
and $W_{1}\in R^{C\texttimes C/r}$. Note that the MLP weights, $W_{0}$and
$W_{1}$, are shared for both inputs and the ReLU activation function is
followed by $W_{0}$.

Figure 6: Spatial Attention Module

The structure of the spatial attention module is shown in the figure: 6. In
the spatial attention module, we first perform mean pooling and max pooling
operations on the input feature map F along the channel axis to aggregate the
information on the feature map $F$ into the feature maps $F^{s}_{avg}\in
R^{1\texttimes H\texttimes W}$and $F^{s}_{max}\in R^{1\texttimes H\texttimes
W}$, $F^{s}_{avg}$ and $F^{s}_{max}$ represent average-pooled features and
max-pooled features of all channels, respectively. The two feature maps are
then concatenated and fed into a standard convolutional layer for convolution.
Finally, the sigmoid operation is performed on the output of the convolutional
layer to generate a two-dimensional spatial attention map $Ms(F)\in
R^{H\texttimes W}$. The calculation process of spatial attention is:

$\displaystyle\mathbf{M}_{s}(\mathbf{F})$
$\displaystyle=\sigma\left(f^{7\times
7}([\operatorname{AvgPool}(\mathbf{F});\text{ MaxPool }(\mathbf{F})])\right)$
(8) $\displaystyle=\sigma\left(f^{7\times 7}\left(\left[\mathbf{F}_{\text{avg
}}^{s};\mathbf{F}_{\text{max }}^{s}\right]\right)\right)$

where $\sigma$ denotes the sigmoid function and $f^{7\texttimes 7}$represents
a convolution operation with the filter size of $7\texttimes 7$.

###  3.3 Comparison of different spatial information extraction structures of
CAINNFlow

CAINNFlow has three modules to extract spatial information: CA, AC, and CAC,
as well as a CC module for ablation experiments. The structure of the CA
module is connected to a CBAM after a $3\texttimes 3$ convolution layer, as
shown in the figure: 7. The final output of the input feature graph $F$ is
obtained through CBAM after convolution. The calculation process of the CA
module is as follows:

$\displaystyle F^{\prime}$ $\displaystyle=\operatorname{Con}(F)$ (9)
$\displaystyle F^{\prime\prime}$
$\displaystyle=Mc\left(F^{\prime}\right)\otimes F^{\prime}\downarrow$
$\displaystyle F^{\prime\prime\prime}$
$\displaystyle=Ms\left(F^{\prime\prime}\right)\otimes F^{\prime\prime}$

Among them, $\otimes$ For element-level multiplication, Con for convolution
operation, $F^{{}^{\prime}}$ Represents the output of the convolution.
$F^{{}^{\prime\prime}}$ Note the output of the graph elements multiplied by
each other,$F^{{}^{\prime\prime\prime}}$ for $F^{{}^{\prime\prime}}$, And the
final output of the multiplication of the force diagram elements in two-
dimensional space.

Figure 7: CA Module

The structure of the AC module is connected with a $3\texttimes 3$ convolution
layer after a CBAM, and its structure diagram is shown in the figure: 8.The
final output of input feature graph $F$ is obtained after CBAM and
convolution.The AC module is calculated as follows:

$\displaystyle F^{\prime}$ $\displaystyle=\mathrm{M}_{c}(F)\otimes F^{\prime}$
(10) $\displaystyle F^{\prime\prime}$
$\displaystyle=M_{s}\left(F^{\prime}\right)\otimes F^{\prime}\downarrow$
$\displaystyle F^{\prime\prime\prime}$
$\displaystyle=\operatorname{Con}\left(F^{\prime\prime}\right)$

Among them $\otimes$ For element-level multiplication, and Con for convolution
operation. $F^{{}^{\prime}}$ for $F$ Notice the output when you multiply the
graph elements, $F^{{}^{\prime\prime}}$ For $F^{{}^{\prime}}$ the And the
output of the multiplication of the force diagram elements in two-dimensional
space, $F^{{}^{\prime\prime\prime}}$ for $F^{{}^{\prime\prime}}$ The final
output after convolution.

Figure 8: AC Module

The structure of the CAC module is that a $3\texttimes 3$ convolution layer is
followed by a CBAM, and then another $3\texttimes 3$ convolution layer, as
shown in the figure: 9. Input feature graph $F$ is first convolved and then
input CBAM, and then the output of CBAM is convolved to obtain the final
output. The calculation process of the CAC module is as follows:

$\displaystyle F^{\prime}$ $\displaystyle=\operatorname{Con}(F)_{\downarrow}$
(11) $\displaystyle F^{\prime\prime}$
$\displaystyle=Mc\left(F^{\prime}\right)\otimes F_{\downarrow}^{\prime}$
$\displaystyle F^{\prime\prime\prime}$
$\displaystyle=Ms\left(F^{\prime\prime}\right)\otimes F_{\downarrow}^{\prime}$
$\displaystyle F^{(4)}$
$\displaystyle=\operatorname{Con}\left(F^{\prime\prime\prime}\right)$

Among them $\otimes$ For element-level multiplication, Conv stands for
convolution operation.$F^{{}^{\prime}}$ Is the output after the first
convolution, $F^{{}^{\prime\prime}}$ for $F^{{}^{\prime}}$ Notice the output
when you multiply the graph elements, $F^{{}^{\prime\prime\prime}}$ for
$F^{{}^{\prime\prime}}$ And the output of the multiplication of the force
diagram elements in two-dimensional space, $F^{(4)}$ for
$F^{{}^{\prime\prime\prime}}$ The final output after the second convolution.

Figure 9: CAC Module

The structure of the CC module is two $3\texttimes 3$ convolution layers, and
its structure diagram is shown in the figure: 10. The final output of the
input feature graph F is obtained after two levels of convolution. The
calculation process of the CC module is as follows:

$\displaystyle F^{\prime}$ $\displaystyle=\operatorname{Con}(F)$ (12)
$\displaystyle F^{\prime\prime}$
$\displaystyle=\operatorname{Con}(F)^{\prime}$

Conv stands for convolution operation.$F^{{}^{\prime}}$ Is the output after
the first convolution and the $F^{{}^{\prime\prime}}$ output after the second
convolution.

Figure 10: CC Module

##  4 Experiments

###  4.1 The data set

The detection of abnormal structures in natural image data is very important
for many types of research in computer vision. The data set used in this study
is the MVTec Anomaly Detection dataset, which contains $5354$ high-resolution
color images of different object and texture categories, as well as normal
images for training without defects and defective anomaly images for testing.
The abnormal images show more than 70 different types of defects, such as
scratches, dents, contamination, and various structural changes.

This dataset is a new unsupervised anomaly detection dataset that simulates
real industrial detection scenarios and offers the possibility of evaluating
unsupervised anomaly detection methods for various textures and object classes
with different types of exceptions. Since this dataset provides pixel-level
accurate GT annotation for anomaly areas in the image, we can evaluate anomaly
detection methods for image-level classification and pixel-level segmentation.
At the same time, the proposed data set promotes the development of new
unsupervised anomaly detection methods. The development of unsupervised
anomaly detection methods requires data to train and evaluate new methods and
ideas, so in an unsupervised setting, we train our model with normal images
for each category and evaluate it in test images that contain both normal and
abnormal images.

Method | PatchSVDD | SPADE | DifferNet | PaDiM | Cut Paste | PatchCore | CFlow | CC(3×3) | CAINNFlow(3×3)
---|---|---|---|---|---|---|---|---|---
carpet | (92.64, 92.41) | (98.62, 97.53) | (83.96, -) | (- 98.92) | (100.00, 98.31) | (98.34, 98.62) | (100.00, 99.18) | (100.00, 99.28) | (100.00, 99.40)
grid | (94.63, 96.17) | (98.89, 93.57) | (97.06, -) | (- 97.26) | (96.21, 97.52) | (98.00, 98.67) | (97.36, 98.86) | (99.90, 98.58) | (100.00, 98.88)
leather | (90.92, 97.41) | (99.16, 97.37) | (99.42, -) | (- 99.22) | (95.05, 99.34) | (100.00, 99.29) | (97.44, 99.60) | (100.00, 99.58) | (100.00, 99.68)
tile | (97.79, 91.44) | (89.75, 87.43) | (92.73, -) | (- 94.07) | (100.00, 90.53) | (98.71, 95.62) | (96.74, 97.53) | (100.00, 97.16) | (100.00, 97.54)
wood | (96.76, 91.07) | (95.84, 88.47) | (99.83, -) | (- 94.87) | (98.75, 95.40) | (99.22, 95.04) | (99.52, 96.66) | (99.88, 95.90) | (99.78, 95.51)
bottle | (98.56, 98.05) | (98.03, 98.11) | (99.14, -) | (- 98.28) | (100.00, 97.73) | (99.92, 98.13) | (99.56, 98.26) | (100.00, 98.08) | (100.00, 98.49)
cable | (90.32, 96.76) | (93.21, 97.22) | (86.94, -) | (- 96.73) | (100.00, 90.24) | (99.12, 98.05) | (100.00, 97.61) | (100.00, 98.06) | (100.00, 98.71)
capsule | (76.37, 95.58) | (98.63, 99.02) | (88.58, -) | (- 98.49) | (98.87, 97.61) | (98.13, 98.82) | (99.30, 98.97) | (100.00, 99.00) | (99.96, 98.94)
hazelnut | (92.03, 97.52) | (98.67, 99.02) | (99.14, -) | (- 98.09) | (93.25, 97.28) | (99.67, 98.48) | (96.81, 98.94) | (100.00, 99.18) | (100.00, 99.29)
meta nut | (93.81, 97.92) | (96.92, 98.13) | (95.19, -) | (- 97.21) | (86.49, 93.32) | (100.00, 98.27) | (91.62, 98.56) | (99.94, 98.35) | (100.00, 99.11)
pill | (86.11, 95.07) | (96.77, 96.21) | (95.94, -) | (- 95.86) | (99.76, 95.66) | (96.62, 97.13) | (98.82, 97.92) | (99.18, 98.97) | (98.81, 98.51)
screw | (81.23, 95.62) | (99.54, 98.93) | (99.12, -) | (- 98.46) | (90.70, 96.66) | (98.06, 99.29) | (99.72, 98.92) | (97.74, 99.30) | (98.72, 99.68)
toothbrush | (100.00, 98.10) | (98.63, 97.80) | (96.13, -) | (- 98.64) | (97.53, 98.11) | (100.00, 98.66) | (95.47, 99.18) | (96.44, 99.16) | (97.23, 99.55)
transistor | (91.64, 97.12) | (80.98, 94.11) | (96.07, -) | (- 97.40) | (100.00, 93.18) | (100.00, 96.26) | (98.61, 97.51) | (99.76, 97.28) | (100.00, 97.58)
zipper | (97.89, 95.14) | (98.78, 96.53) | (98.56, -) | (- 98.44) | (99.57, 98.67) | (98.67, 98.60) | (98.13, 98.63) | (99.42, 98.58) | (99.60, 98.71)
AUCROC | (92.05, 95.69) | (96.16, 95.96) | (95.19, -) | (-,97.46) | (97.08, 95.97) | (98.96, 97.93) | (97.94, 98.42) | (99.48, 98.43) | (99.61, 98.64)

Table 1:  Figure 11: Feature Visualization and Generation

###  4.2 The evaluation

In this study, the method we used and other methods compared with it need to
be measured by the image or pixel-level region under the receiver Operation
Characteristic Curve (AUROC) during the comparison of their respective
performance. AUROC is an indicator used to measure the performance of a
classifier. It reflects the performance of the classifier through the area
between the receiver operating characteristic curve and the coordinate axis.
Its significance can be understood as the expectation that uniformly selected
random positive samples (positive samples) rank ahead of uniformly selected
random negative samples (negative samples). AUROC is a value between $0$ and
$1$. When the AUROC value is close to $1$, it indicates that the classifier
has a better classification of positive and negative samples. In the locating
task, we need to output an exception score for each pixel for comparison.
Meanwhile, AUROC is also a dichotomous evaluation method commonly used in
machine learning, which directly means the area under the ROC curve. A pair of
samples (one positive sample and one negative sample) are randomly selected,
and then the trained classifier is used to predict the two samples. The
probability of a positive sample is predicted to be better than the
probability of a negative sample.

There is how to calculate AUROC:

In a data set with $M$ positive samples and $N$ negative samples. There are
$M$ times $N$ pairs of samples. So let’s figure out how many of these $M$
times $N$ samples, the positive sample is more likely to predict than the
negative sample.

$AUC=\frac{\sum I\left(P_{\text{Positive samples }},P_{\text{Negative samples
}}\right)}{M*N}$ (13)

In addition, the advantages of using AUROC to measure the performance of the
classifier are as follows:

$\bullet$ It is not affected by category imbalance, and different sample
proportions will not affect the evaluation results of AUC.

$\bullet$ AUC can be directly used as a loss function during training.

Therefore, two evaluation indexes, image-level AUROC and pixel-level AUROC are
adopted in anomaly detection and location tasks. Image-level AUROC is an
evaluation indicator used to classify whether the whole image is abnormal, and
pixel-level AUROC is an evaluation indicator used to judge whether the model
accurately classifies whether a single pixel is defective.

| bottle | cable | capsule | carpet | hazelnut | leather | metal nut | pill | tile | toothbrush | transistor | wood | grid | screw | zipper
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
CAINNFlowCAC | 98.38 | 97.11 | 98.40 | 99.13 | 95.80 | 99.62 | 96.71 | 96.30 | 94.81 | 96.89 | 96.62 | 94.56 | 98.64 | 97.33 | 98.69
CAINNFlowCA | 98.03 | 96.71 | 98.51 | 98.98 | 95.98 | 99.54 | 96.53 | 95.52 | 94.41 | 96.88 | 96.31 | 94.25 | 98.06 | 97.26 | 98.19
CAINNFlowAC | 98.07 | 95.48 | 98.50 | 98.95 | 95.22 | 99.60 | 95.17 | 94.43 | 93.44 | 96.30 | 96.12 | 94.73 | 98.07 | 97.25 | 98.35
CC | 98.06 | 96.82 | 98.65 | 99.11 | 95.78 | 99.58 | 95.21 | 97.10 | 94.56 | 96.43 | 96.34 | 95.42 | 98.57 | 96.15 | 98.57

Table 2: Structure comparison of CAC, CA, AC and CC
The feature extractor of the experiment was ResNet18, the epoch was $750$, the
parameter was step2, and the learning rate LR was $5e-4$.

| bottle | cable | capsule | carpet | hazelnut | leather | metal nut | pill | tile | toothbrush | transistor | wood | grid | screw | zipper
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
CAINNFlowCAC/step5 | 97.88 | 96.65 | 97.99 | 98.83 | 96.15 | 99.58(8) | 93.48 | 95.92 | 93.50 | 94.85 | 96.53 | 93.92 | 98.85 | 95.93 | 98.31
CAINNFlowCAC/step4 | 97.99 | 96.76 | 98.17 | 98.99 | 96.02 | 99.58(5) | 93.86 | 95.10 | 93.85 | 93.34 | 96.85 | 93.80 | 98.27 | 95.99 | 98.15
CAINNFlowCAC/step3 | 98.01 | 96.91 | 98.12 | 99.11 | 95.20 | 99.65 | 95.40 | 95.77 | 94.89 | 96.69 | 96.74 | 94.45 | 98.57 | 96.70 | 98.46
CAINNFlowCAC/step2 | 98.38 | 97.11 | 98.40 | 99.13 | 95.80 | 99.62 | 96.71 | 96.30 | 94.81 | 96.89 | 96.62 | 94.56 | 98.64 | 97.33 | 98.69
CAINNFlowCAC/step1 | 98.28 | 96.33 | 98.64 | 99.02 | 96.74 | 99.55 | 97.15 | 95.73 | 94.22 | 97.82 | 95.80 | 95.03 | 98.12 | 96.93 | 98.51

Table 3:  Experimental results of CAC with different parametersResNet18 epoch
was $750$, and the LR of learning was $5e-4$.

###  4.3 contrast others

####  4.3.1 Different anomaly detection methods

Table (1): Anomaly detection and location performance of the MVTec AD dataset
(image-level AUC, pixel-level AUC). The MVTec AD dataset contains $5354$ high-
resolution color images of different object and texture classes, the training
set contains only normal samples without defects, and the test set adds
abnormal samples. We compared folw with other anomaly detection methods that
work well, such as: PatchSVDD Yi and Yoon (2020), SPADE Reiss et al. (2021),
DifferNet Rudolph et al. (2021), Cut Paste Li et al. (2021), PaDiM Defard et
al. (2021), PatchCore Roth et al. (2021), CFlow Gudovskiy et al. (2022), CC Yu
et al. (2021). The comparison results are shown in Table (1). It can be found
that in the case of Flow using fewer parameters, the image-level AUC and
pixel-level AUC of most industrial products are higher than those of other
anomaly detection methods.

####  4.3.2 Different feature extractors

As shown in Table (4), we used four different feature extractors to detect and
locate anomalies and filled the average into the table as the final result.

AUROC (Best) | Wide-ResNet-50 | ResNet18 | DeiT | CaiT
---|---|---|---|---
bottle | 99.31 | 98.20 | 96.64 | 98.02
cable | 98.44 | 96.87 | 98.59 | 98.63
capsule | 98.89 | 98.41 | 98.77 | 98.92
carpet | 99.15 | 98.76 | 99.63 | 99.31
grid | 99.62 | 98.73 | 98.34 | 97.74
hazelnut | 99.08 | 96.72 | 99.11 | 99.24
leather | 99.62 | 99.62 | 99.45 | 99.64
metal nut | 99.88 | 97.11 | 99.92 | 99.06
pill | 97.31 | 96.26 | 98.58 | 98.79
screw | 99.39 | 96.93 | 99.48 | 99.63
tile | 97.42 | 94.81 | 97.14 | 97.48
toothbrush | 99.11 | 97.76 | 99.29 | 99.53
transistor | 97.51 | 97.53 | 97.30 | 97.22
wood | 96.07 | 95.03 | 95.99 | 95.46
zipper | 98.86 | 98.04 | 97.62 | 98.61
Average | 98.64 | 97.39 | 98.39 | 98.49
Table 4:

The structure of CAINNFlow is a plug-in, and we can attach any feature
extractor before it. To observe the performance of CAINNFlow when using
different feature extractors, we designed a group of experiments, in which
RESNet18, CaiT, wide-ResNet50-2 and Deit were used as our feature extractors,
followed by CAINNFlow which used CAC structure to extract spatial information.
Under the same parameter setting. It can be found that CAINNFlow using CaiT,
Wide-ResNet50-2 and Deit feature extractor, the Pixels-level AUROC is
generally higher than CAINNFlow using RESNet18 feature extractor. The
experimental results show that the stronger the feature extractor before
CAINNFlow, the better the output effect of CAINNFlow. The effect of the
feature extractor and CAINNFlow combination structure is positively correlated
with the strength of the feature extractor in the combination structure.

###  4.4 Structure comparison of CAC, CA, AC and CC

Our CAINNFlow has four structures for extracting spatial information: CA, AC,
CAC, and CC. To compare the effects of CAINNFlow with four different
structures, we conducted experiments on CAINNFlow using these four structures
to extract spatial information, and the experimental results obtained are
shown in Table (2). In general, the CAC module adopted by CAINNFlow has a
better overall effect than the CA module and AC module in $12$ evaluation
indicators, and only the AC module is weaker than the CC module.

Figure 12: Qualitative Results

###  4.5 CAC was compared with different steps

We conducted experiments on the CAC module in $F$ under different parameters.
In the same experiment, the feature extractor used RESNet18, the epoch was
$750$, and the learning rate LR was $5e-4$. The experimental parameters were
step1, Step2, step3, step4, and step5. The experimental results are shown in
Table (3):

It was observed that the evaluation index increased with Step and reached a
better level when Step was equal to $2$, then decreased. In practice, Step can
be set to $2$ to ensure performance and accuracy.

Although the performance of CAINNFlow with the large model as a feature
extractor is better than that with the ResNet18 feature extractor, the
experimental cost brought by the number of parameters of the large model is
huge. To explore whether CAINNFlow can achieve better results when using small
models as feature extractors, we conducted experiments. In the experiment, we
used RESNet18 as the feature extractor and then modified the parameter
Settings of CAINNFlow to find the global optimal solution. It is observed that
under the condition of an invariable feature extractor, a better effect can be
achieved by tuning CAINNFlow parameters. The experimental results are shown in
Table (5).

Category | Best | Category | Best
---|---|---|---
bottle | 98.38 | pill | 96.64
cable | 97.47 | tile | 94.89
capsule | 98.64 | toothbrush | 97.82
carpet | 99.27 | transistor | 97.55
hazelnut | 96.78 | wood | 95.03
leather | 99.65 | grid | 98.85
metal nut | 97.15 | screw | 97.33
zipper | 98.69 |  |
Table 5:

###  4.6 Feature Visualization and Generation

The ability of bidirectional reversible probability distribution is one of the
indispensable functions in our design of CAINNFlow. In the forward process,
CAINNFlow can transform the feature map of the backbone network into the
standard Gaussian distribution of high availability. In its reverse process,
we also designed noise interference to prove that CAINNFlow has the visual
characteristics of visualizing the reverse generation of specific probability
sampling variables. In the figure(11) We have visualized this process.

As shown in figure(11) , as is shown in figure(11) In the forward process of
class image, the probability graph after feature extraction and transformation
has significant distribution characteristics. Moreover, in the experiment of
adding noise points to the probability graph to verify the reversibility of
CAINNFlow, obvious abnormal visual features were also obtained in the reverse
results.

###  4.7 Qualitative Results

Through experiments on the MVTecAD data set, we find the following figure(12)
Visualizes the results. We designed a control experiment containing ground
truth Mask and heatmap of abnormal location score with two kinds of reference
images. The experimental results show that our CAINNFlow architecture is
effective in anomaly recognition, whether based on normal images or abnormal
images.

##  5 Conclusion

CAINNFlow is a plug-in structure proposed in this study to identify and
localize objects with anomalies, which can be used as a plug-in connected
behind any computer vision model as a backbone feature extractor. In our
research, in order to retain and effectively extract spatial structure
information while performing distribution transformation, we constructed
CAINNFlow based on INN by using a stacked CNN structure embedded with CBAM as
a subnetwork.The results of extensive experiments on MVTec AD dataset show
that CAINNFlow can be used as a feature extractor based on CNN and Transformer
backbone network as a feature extractor, it achieves advanced level in terms
of accuracy and inference efficiency.

## Acknowledgements

This research is supported by:

$\bullet$ The National Natural Science Foundation of China (Grant
No.41931183). The numerical calculation in this study were carried out on the
SunRising-1 computing platform.

$\bullet$ The fund of the Beijing Municipal Education Commission, China, under
grant number 22019821001.

$\bullet$ The fund of Climbing Program Foundation from Beijing Institute of
Petrochemical Technology (Project No. BIPTAAI-2021007).

## References

  * Roth et al. (2021) K. Roth, L. Pemula, J. Zepeda, B. Schölkopf, T. Brox, P. Gehler,  Towards total recall in industrial anomaly detection,  arXiv preprint arXiv:2106.08265 (2021).
  * Bergman and Hoshen (2020) L. Bergman, Y. Hoshen,  Classification-based anomaly detection for general data,  arXiv preprint arXiv:2005.02359 (2020).
  * Wang et al. (2021) S. Wang, L. Wu, L. Cui, Y. Shen,  Glancing at the patch: Anomaly localization with global and local feature comparison,  in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 254–263.
  * Li et al. (2021) C.-L. Li, K. Sohn, J. Yoon, T. Pfister,  Cutpaste: Self-supervised learning for anomaly detection and localization,  in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 9664–9674.
  * Dosovitskiy et al. (2020) A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al.,  An image is worth 16x16 words: Transformers for image recognition at scale,  arXiv preprint arXiv:2010.11929 (2020).
  * Liu et al. (2021) Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, B. Guo,  Swin transformer: Hierarchical vision transformer using shifted windows,  in: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
  * Cunningham et al. (2020) E. Cunningham, R. Zabounidis, A. Agrawal, I. Fiterau, D. Sheldon,  Normalizing flows across dimensions,  arXiv preprint arXiv:2006.13070 (2020).
  * Woo et al. (2018) S. Woo, J. Park, J.-Y. Lee, I. S. Kweon,  Cbam: Convolutional block attention module,  in: Proceedings of the European conference on computer vision (ECCV), 2018, pp. 3–19.
  * Bergmann et al. (2019) P. Bergmann, M. Fauser, D. Sattlegger, C. Steger,  Mvtec ad–a comprehensive real-world dataset for unsupervised anomaly detection,  in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 9592–9600.
  * Feng et al. (2017) Y. Feng, Y. Yuan, X. Lu,  Learning deep event models for crowd anomaly detection,  Neurocomputing 219 (2017) 548–556.
  * Wang et al. (2018) S. Wang, E. Zhu, J. Yin, F. Porikli,  Video anomaly detection and localization by local motion based joint video representation and ocelm,  Neurocomputing 277 (2018) 161–175.
  * Li and Chang (2019) N. Li, F. Chang,  Video anomaly detection and localization via multivariate gaussian fully convolution adversarial autoencoder,  Neurocomputing 369 (2019) 92–105.
  * Canizo et al. (2019) M. Canizo, I. Triguero, A. Conde, E. Onieva,  Multi-head cnn–rnn for multi-time series anomaly detection: An industrial case study,  Neurocomputing 363 (2019) 246–260.
  * Zhang et al. (2020) X. Zhang, D. Ma, H. Yu, Y. Huang, P. Howell, B. Stevens,  Scene perception guided crowd anomaly detection,  Neurocomputing 414 (2020) 291–302.
  * Fahad and Tahir (2021) L. G. Fahad, S. F. Tahir,  Activity recognition and anomaly detection in smart homes,  Neurocomputing 423 (2021) 362–372.
  * Yu et al. (2021) J. Yu, Y. Zheng, X. Wang, W. Li, Y. Wu, R. Zhao, L. Wu,  Fastflow: Unsupervised anomaly detection and localization via 2d normalizing flows,  arXiv preprint arXiv:2111.07677 (2021).
  * Li et al. (2016) S. Li, J. Jiao, Y. Han, T. Weissman,  Demystifying resnet,  arXiv preprint arXiv:1611.01186 (2016).
  * Simonyan and Zisserman (2014) K. Simonyan, A. Zisserman,  Very deep convolutional networks for large-scale image recognition,  arXiv preprint arXiv:1409.1556 (2014).
  * Touvron et al. (2021a) H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, H. Jégou,  Training data-efficient image transformers & distillation through attention,  in: International Conference on Machine Learning, PMLR, 2021a, pp. 10347–10357.
  * Touvron et al. (2021b) H. Touvron, M. Cord, A. Sablayrolles, G. Synnaeve, H. Jégou,  Going deeper with image transformers,  in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021b, pp. 32–42.
  * Liu et al. (2022) Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, S. Xie,  A convnet for the 2020s,  arXiv preprint arXiv:2201.03545 (2022).
  * Agnelli et al. (2010) J. Agnelli, M. Cadeiras, E. G. Tabak, C. V. Turner, E. Vanden-Eijnden,  Clustering and classification through normalizing flows in feature space,  Multiscale Modeling & Simulation 8 (2010) 1784–1802.
  * Kobyzev et al. (2020) I. Kobyzev, S. J. Prince, M. A. Brubaker,  Normalizing flows: An introduction and review of current methods,  IEEE transactions on pattern analysis and machine intelligence 43 (2020) 3964–3979.
  * Yi and Yoon (2020) J. Yi, S. Yoon,  Patch svdd: Patch-level svdd for anomaly detection and segmentation,  in: Proceedings of the Asian Conference on Computer Vision, 2020.
  * Reiss et al. (2021) T. Reiss, N. Cohen, L. Bergman, Y. Hoshen,  Panda: Adapting pretrained features for anomaly detection and segmentation,  in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 2806–2814.
  * Rudolph et al. (2021) M. Rudolph, B. Wandt, B. Rosenhahn,  Same same but differnet: Semi-supervised defect detection with normalizing flows,  in: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2021, pp. 1907–1916.
  * Defard et al. (2021) T. Defard, A. Setkov, A. Loesch, R. Audigier,  Padim: a patch distribution modeling framework for anomaly detection and localization,  in: International Conference on Pattern Recognition, Springer, 2021, pp. 475–489.
  * Gudovskiy et al. (2022) D. Gudovskiy, S. Ishizaka, K. Kozuka,  Cflow-ad: Real-time unsupervised anomaly detection with localization via conditional normalizing flows,  in: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2022, pp. 98–107.

./6.jpg Yan Ruiqing, born in 2000, is an undergraduate student in Beijing
Institute of Petrochemical Technology. He is now working as an intern in
computer Network Information Center of Chinese Academy of Sciences, engaged in
algorithm research. His research interests include computer vision, natural
language processing, spatio-temporal modeling algorithms, and cross-modal
modeling algorithms. ./1.jpg Linghan Zheng received the Master’s degree in
Tongji University, China in 2017. He now works for Ant Group and works on
natural language understanding. His research interests is natural language
retrival and natural language representation problem. At the same time, he
also has a strong interest in multimodality content understanding. ./2.jpg
Jinfeng Li received a bachelor’s degree in software engineering from Taiyuan
University of Science and Technology in 2021 and is currently pursuing a
master’s degree in the School of Information Engineering, Beijing Institute of
Petrochemical Technology. Her research interests are in computer vision.
./3.jpg Mengyuan Huang , graduated from Heilongjiang University with a
bachelor’s degree in network engineering in 2019. From 2019 to 2020, she
worked as a software engineer in Shanghai BYD Co., Ltd. Since 2021, she has
been studying at the Artificial Intelligence Research Institute of Beijing
Institute of Petrochemical Technology as a graduate student. Her current
research interests include smart healthcare. ./65.jpg Liu Wu received his
B.Eng. degree in high polymer material from Hubei University in 2019.From 2019
to 2021,Engaged in python development. Be familiared with Django framework and
VUE. Organized and built e-commerce websites, novel websites and crawler
projects during career.His research interests include computer vision. ./4.jpg
Dongyu Hu, from 2020 to now, is studying for a bachelor’s degree in computer
science and technology at the School of Information Engineering, Beijing
Institute of Petrochemical Technology. During school, she has participated in
a number of artificial intelligence projects. Her current research interests
include artificial intelligence, computer vision and intelligent medical care.

./5.jpg Zhang Fan received the bachelor’s degree in Jiyang College of Zhejiang
agriculture and Forestry University ,China in 2020.From March 2020 to May
2020,he worked in software testing.From 2021 to now, he is a graduate student
at the Department of Information Engineering of Beijing Institute of
Petrochemical Technology. His current research interests include natural
language processing and image processing. ./7.png Jinrong Jiang received the
Ph.D. degree in enginering frim Chinese Academy of Sciences in 2007. He is a
professor at Computer Network Information Center,Chinese Academy of Sciences,
Beijing, China. His research interests include high performance computing,
computational geosciences and AI application. ./Qianjin_Guo.png Qianjin Guo
received the Ph.D. degree in Mechanical and Electronic Engineering from
Shenyang Institute of Automation (SIA), Chinese Academy of Sciences, in 2008.
From 2009 to 2021, he was a Associate Researcher with the Institute of
Chemistry,Chinese Academy of Sciences.Since 2021, he is working as a Full
Professor at the School of Academy of Artificial Intelligence, Beijing
Institute of Petrochemical Technology, Beijing , China. His current research
interests include artificial intelligent, soft computing, deep learning,
intelligent and hybrid control systems, signal processing, biomedical imaging
and chemical imaging in innovation. ./Liu_Qiang.jpg Dr. Liu Qiang, professor,
male, artificial intelligence and integrated circuit expert, CISP. September
1996 - July 2000, Majored in Computer Software and Theory, Department of
Computer Science and Technology, Peking University, bachelor’s degree.
September 2000 – July 2005, Computer System Structure, Department of Computer
Science and Technology, Peking University, PH. D. degree. He has worked in IBM
China Research Institute and China Electronics Information Industry Group and
held senior management positions. Participated in the establishment of four
high-tech enterprises, two research and development institutions, and one
investment institution.