# CFLOW-AD: Real-Time Unsupervised Anomaly Detection with Localization
via Conditional Normalizing Flows

Denis Gudovskiy1 Shun Ishizaka2 Kazuki Kozuka2
1Panasonic AI Lab, USA 2Panasonic Technology Division, Japan
<EMAIL_ADDRESS>{ishizaka.shun,
<EMAIL_ADDRESS>
###### Abstract

Unsupervised anomaly detection with localization has many practical
applications when labeling is infeasible and, moreover, when anomaly examples
are completely missing in the train data. While recently proposed models for
such data setup achieve high accuracy metrics, their complexity is a limiting
factor for real-time processing. In this paper, we propose a real-time model
and analytically derive its relationship to prior methods. Our CFLOW-AD model
is based on a conditional normalizing flow framework adopted for anomaly
detection with localization. In particular, CFLOW-AD consists of a
discriminatively pretrained encoder followed by a multi-scale generative
decoders where the latter explicitly estimate likelihood of the encoded
features. Our approach results in a computationally and memory-efficient
model: CFLOW-AD is faster and smaller by a factor of 10$\times$ than prior
state-of-the-art with the same input setting. Our experiments on the MVTec
dataset show that CFLOW-AD outperforms previous methods by 0.36% AUROC in
detection task, by 1.12% AUROC and 2.5% AUPRO in localization task,
respectively. We open-source our code with fully reproducible
experiments111Our code is available at github.com/gudovskiy/cflow-ad.

##  1 Introduction

Anomaly detection with localization (AD) is a growing area of research in
computer vision with many practical applications _e.g_. industrial inspection
[4], road traffic monitoring [20], medical diagnostics [44] _etc_. However,
the common supervised AD [32] is not viable in practical applications due to
several reasons. First, it requires labeled data which is costly to obtain.
Second, anomalies are usually rare long-tail examples and have low probability
to be acquired by sensors. Lastly, consistent labeling of anomalies is
subjective and requires extensive domain expertise as illustrated in Figure 1
with industrial cable defects.

Figure 1: An example of the proposed out-of-distribution (OOD) detector for
anomaly localization trained on anomaly-free $\mathcal{D_{\mathrm{train}}}$
(top row). Sliced cable images are from the MVTec dataset [4], where the
bottom row illustrates $\mathcal{D_{\mathrm{test}}}$ ground truth masks for
anomalies (red) and the middle row shows examples of anomaly-free patches
(green). The OOD detector learns the distribution of anomaly-free patches
${\bm{x}}$ with $p_{X}({\bm{x}})$ density and transforms it into a Gaussian
distribution with $p_{Z}({\bm{z}})$ density. Threshold $\tau$ separates in-
distribution patches from the OOD patches with $p_{\tilde{Z}}({\bm{z}})$
density.

With these limitations of the supervised AD, a more appealing approach is to
collect only unlabeled anomaly-free images for train dataset
$\mathcal{D_{\mathrm{train}}}$ as in Figure 1 (top row). Then, any deviation
from anomaly-free images is classified as an anomaly. Such data setup with low
rate of anomalies is generally considered to be unsupervised [4]. Hence, the
AD task can be reformulated as a task of out-of-distribution detection (OOD)
with the AD objective.

While OOD for low-dimensional industrial sensors (_e.g_. power-line or
acoustic) can be accomplished using a common $k$-nearest-neighbor or more
advanced clustering methods [10], it is less trivial for high-resolution
images. Recently, convolutional neural networks (CNNs) have gained popularity
in extracting semantic information from images into downsampled feature maps
[4]. Though feature extraction using CNNs has relatively low complexity, the
post-processing of feature maps is far from real-time processing in the state-
of-the-art unsupervised AD methods [8].

To address this complexity drawback, we propose a CFLOW-AD model that is based
on conditional normalizing flows. CFLOW-AD is agnostic to feature map spatial
dimensions similar to CNNs, which leads to a higher accuracy metrics as well
as a lower computational and memory requirements. We present the main idea
behind our approach in a toy OOD detector example in Figure 1. A distribution
of the anomaly-free image patches ${\bm{x}}$ with probability density function
$p_{X}({\bm{x}})$ is learned by the AD model. Our translation-equivariant
model is trained to transform the original distribution with $p_{X}({\bm{x}})$
density into a Gaussian distribution with $p_{Z}({\bm{z}})$ density. Finally,
this model separates in-distribution patches ${\bm{z}}$ with $p_{Z}({\bm{z}})$
from the out-of-distribution patches with $p_{\tilde{Z}}({\bm{z}})$ using a
threshold $\tau$ computed as the Euclidean distance from the distribution
mean.

##  2 Related work

We review models222For comprehensive review of the existing AD methods we
refer readers to Ruff  _et al_. [31] and Pang  _et al_. [26] surveys. that
employ the data setup from Figure 1 and provide experimental results for
popular MVTec dataset [4] with factory defects or Shanghai Tech Campus (STC)
dataset [22] with surveillance camera videos. We highlight the research
related to a more challenging task of pixel-level anomaly localization
(segmentation) rather than a more simple image-level anomaly detection.

Napoletano  _et al_. [25] propose to use CNN feature extractors followed by a
principal component analysis and $k$-mean clustering for AD. Their feature
extractor is a ResNet-18 [13] pretrained on a large-scale ImageNet dataset
[16]. Similarly, SPADE [7] employs a Wide-ResNet-50 [43] with multi-scale
pyramid pooling that is followed by a $k$-nearest-neighbor clustering.
Unfortunately, clustering is slow at test-time with high-dimensional data.
Thus, parallel convolutional methods are preferred in real-time systems.

Numerous methods are based on a natural idea of generative modeling. Unlike
models with the discriminatively-pretrained feature extractors [25, 7],
generative models learn distribution of anomaly-free data and, therefore, are
able to estimate a proxy metrics for anomaly scores even for the unseen images
with anomalies. Recent models employ generative adversarial networks (GANs)
[35, 36] and variational autoencoders (VAEs) [3, 38].

A fully-generative models [35, 36, 3, 38] are directly applied to images in
order to estimate pixel-level probability density and compute per-pixel
reconstruction errors as anomaly scores proxies. These fully-generative models
are unable to estimate the exact data likelihoods [6, 24] and do not perform
better than the traditional methods [25, 7] according to MVTec survey in [4].
Recent works [34, 15] show that these models tend to capture only low-level
correlations instead of relevant semantic information. To overcome the latter
drawback, a hybrid DFR model [37] uses a pretrained feature extractor with
multi-scale pyramid pooling followed by a convolutional autoencoder (CAE).
However, DFR model is unable to estimate the exact likelihoods.

Another line of research proposes to employ a student-teacher type of
framework [5, 33, 41]. Teacher is a pretrained feature extractor and student
is trained to estimate a scoring function for AD. Unfortunately, such
frameworks underperform compared to state-of-the-art models.

Patch SVDD [42] and CutPaste [19] introduce a self-supervised pretraining
scheme for AD. Moreover, Patch SVDD proposes a novel method to combine multi-
scale scoring masks to a final anomaly map. Unlike the nearest-neighbor search
in [42], CutPaste estimates anomaly scores using an efficient Gaussian density
estimator. While the self-supervised pretraining can be helpful in uncommon
data domains, Schirrmeister  _et al_. [34] argue that large natural-image
datasets such as ImageNet can be a more representative for pretraining
compared to a small application-specific datasets _e.g_. industrial MVTec [4].

The state-of-the-art PaDiM [8] proposes surprisingly simple yet effective
approach for anomaly localization. Similarly to [37, 7, 42], this approach
relies on ImageNet-pretrained feature extractor with multi-scale pyramid
pooling. However, instead of slow test-time clustering in [7] or nearest-
neighbor search in [42], PaDiM uses a well-known Mahalanobis distance metric
[23] as an anomaly score. The metric parameters are estimated for each feature
vector from the pooled feature maps. PaDiM has been inspired by Rippel  _et
al_. [29] who firstly advocated to use this measure for anomaly detection
without localization.

DifferNet [30] uses a promising class of generative models called normalizing
flows (NFLOWs) [9] for image-level AD. The main advantage of NFLOW models is
ability to estimate the exact likelihoods for OOD compared to other generative
models [35, 36, 3, 38, 37]. In this paper, we extend DifferNet approach to
pixel-level anomaly localization task using our CFLOW-AD model. In contrast to
RealNVP [9] architecture with global average pooling in [30], we propose to
use conditional normalizing flows [2] to make CFLOW-AD suitable for low-
complexity processing of multi-scale feature maps for localization task. We
develop our CFLOW-AD with the following contributions:

  * •

Our theoretical analysis shows why multivariate Gaussian assumption is a
justified prior in previous models and why a more general NFLOW framework
objective converges to similar results with the less compute.

  * •

We propose to use conditional normalizing flows for unsupervised anomaly
detection with localization using computational and memory-efficient
architecture.

  * •

We show that our model outperforms previous state-of-the art in both detection
and localization due to the unique properties of the proposed CFLOW-AD model.

Figure 2: Overview of our CFLOW-AD with a fully-convolutional translation-
equivariant architecture. Encoder $h({\bm{\lambda}})$ is a CNN feature
extractor with multi-scale pyramid pooling. Pyramid pooling captures both
global and local semantic information with the growing from top to bottom
receptive fields. Pooled feature vectors ${\bm{z}}^{k}_{i}$ are processed by a
set of decoders $g_{k}({\bm{\theta}}_{k})$ independently for each $k$th scale.
Our decoder is a conditional normalizing flow network with a feature input
${\bm{z}}^{k}_{i}$ and a conditional input ${\bm{c}}^{k}_{i}$ with spatial
information from a positional encoder (PE). The estimated multi-scale
likelihoods $p^{k}_{i}$ are upsampled to the input size and added up to
produce anomaly map.

##  3 Theoretical background

###  3.1 Feature extraction with Gaussian prior

Consider a CNN $h({\bm{\lambda}})$ trained for classification task. Its
parameters ${\bm{\lambda}}$ are usually found by minimizing Kullback-Leibler
($D_{KL}$) divergence between joint train data distribution
$Q_{{\bm{x}},{\bm{y}}}$ and the learned model distribution
$P_{{\bm{x}},{\bm{y}}}({\bm{\lambda}})$, where $({\bm{x}},{\bm{y}})$ is an
input-label pair for supervised learning.

Typically, the parameters ${\bm{\lambda}}$ are initialized by the values
sampled from the Gaussian distribution [12] and optimization process is
regularized as

$\operatorname*{arg\,min}_{{\bm{\lambda}}}D_{KL}\left[Q_{{\bm{x}},{\bm{y}}}\|P_{{\bm{x}},{\bm{y}}}({\bm{\lambda}})\right]+\alpha
R({\bm{\lambda}}),$ (1)

where $R({\bm{\lambda}})$ is a regularization term and $\alpha$ is a
hyperparameter that defines regularization strength.

The most popular CNNs [13, 43] are trained with $L_{2}$ weight decay [17]
regularization ($R({\bm{\lambda}})=\|{\bm{\lambda}}\|_{2}^{2}$). That imposes
multivariate Gaussian (MVG) prior not only to parameters ${\bm{\lambda}}$, but
also to the feature vectors ${\bm{z}}$ extracted from the feature maps of
$h({\bm{\lambda}})$ [11] intermediate layers.

###  3.2 A case for Mahalanobis distance

With the same MVG prior assumption, Lee  _et al_. [18] recently proposed to
model distribution of feature vectors ${\bm{z}}$ by MVG density function and
to use Mahalanobis distance [23] as a confidence score in CNN classifiers.
Inspired by [18], Rippel  _et al_. [29] adopt Mahalanobis distance for anomaly
detection task since this measure determines a distance of a particular
feature vector ${\bm{z}}$ to its MVG distribution. Consider a MVG distribution
$\mathcal{N}({\bm{\mu}},{\bm{\Sigma}})$ with a density function
$p_{Z}({\bm{z}})$ for random variable ${\bm{z}}\in\mathbb{R}^{D}$ defined as

$p_{Z}({\bm{z}})=(2\pi)^{-D/2}\det{\bm{\Sigma}}^{-1/2}e^{-\frac{1}{2}({\bm{z}}-{\bm{\mu}})^{T}{\bm{\Sigma}}^{-1}({\bm{z}}-{\bm{\mu}})},$
(2)

where ${\bm{\mu}}\in\mathbb{R}^{D}$ is a mean vector and
${\bm{\Sigma}}\in\mathbb{R}^{D\times D}$ is a covariance matrix of a true
anomaly-free density $p_{Z}({\bm{z}})$.

Then, the Mahalanobis distance $M({\bm{z}})$ is calculated as

$M({\bm{z}})=\sqrt{({\bm{z}}-{\bm{\mu}})^{T}{\bm{\Sigma}}^{-1}({\bm{z}}-{\bm{\mu}})},$
(3)

Since the true anomaly-free data distribution is unknown, mean vector and
covariance matrix from (3) are replaced by the estimates $\hat{{\bm{\mu}}}$
and $\hat{{\bm{\Sigma}}}$ calculated from the empirical train dataset
$\mathcal{D_{\mathrm{train}}}$. At the same time, density function
$p_{\tilde{Z}}({\bm{z}})$ of anomaly data has different $\tilde{{\bm{\mu}}}$
and $\tilde{{\bm{\Sigma}}}$ statistics, which allows to separate out-of-
distribution and in-distribution feature vectors using $M({\bm{z}})$ from (3).

This framework with MVG distribution assumption shows its effectiveness in
image-level anomaly detection task [29] and is adopted by the state-of-the-art
PaDiM [8] model in pixel-level anomaly localization task.

###  3.3 Relationship with the flow framework

Dinh  _et al_. [9] introduce a class of generative probabilistic models called
normalizing flows. These models apply change of variable formula to fit an
arbitrary density $p_{Z}({\bm{z}})$ by a tractable base distribution with
$p_{U}({\bm{u}})$ density and a bijective invertible mapping
$g^{-1}:Z\rightarrow U$. Then, the $\log$-likelihood of any ${\bm{z}}\in Z$
can be estimated by

$\log\hat{p}_{Z}({\bm{z}},{\bm{\theta}})=\log
p_{U}({\bm{u}})+\log\left|\det{\bm{J}}\right|,$ (4)

where a sample ${\bm{u}}\sim p_{U}$ is usually from standard MVG distribution
$({\bm{u}}\sim\mathcal{N}({\bm{0}},{\bm{I}}))$ and a matrix
${\bm{J}}=\nabla_{{\bm{z}}}g^{-1}({\bm{z}},{\bm{\theta}})$ is the Jacobian of
a bijective invertible flow model $({\bm{z}}=g({\bm{u}},{\bm{\theta}})$ and
${\bm{u}}=g^{-1}({\bm{z}},{\bm{\theta}}))$ parameterized by vector
${\bm{\theta}}$.

The flow model $g({\bm{\theta}})$ is a set of basic layered transformations
with tractable Jacobian determinants. For example, $\log|\det{\bm{J}}|$ in
RealNVP [9] coupling layers is a simple sum of layer’s diagonal elements.
These models are optimized using stochastic gradient descent by maximizing
$\log$-likelihood in (4). Equivalently, optimization can be done by minimizing
the reverse
$D_{KL}\left[\hat{p}_{Z}({\bm{z}},{\bm{\theta}})\|p_{Z}({\bm{z}})\right]$
[27], where $\hat{p}_{Z}({\bm{z}},{\bm{\theta}})$ is the model prediction and
$p_{Z}({\bm{z}})$ is a target density. The loss function for this objective is
defined as

$\mathcal{L}({\bm{\theta}})=\mathbb{E}_{\hat{p}_{Z}({\bm{z}},{\bm{\theta}})}\left[\log\hat{p}_{Z}({\bm{z}},{\bm{\theta}})-\log
p_{Z}({\bm{z}})\right].$ (5)

If $p_{Z}({\bm{z}})$ is distributed according to Section 3.1 MVG assumption,
we can express (5) as a function of Mahalanobis distance $M({\bm{z}})$ using
its definition from (3) as

$\mathcal{L}({\bm{\theta}})=\mathbb{E}_{\hat{p}_{Z}({\bm{z}},{\bm{\theta}})}\left[\frac{M^{2}({\bm{z}})-E^{2}({\bm{u}})}{2}+\log\frac{|\det{\bm{J}}|}{\det{\bm{\Sigma}}^{-1/2}}\right],$
(6)

where $E^{2}({\bm{u}})=\|{\bm{u}}\|_{2}^{2}$ is a squared Euclidean distance
of a sample ${\bm{u}}\sim\mathcal{N}({\bm{0}},{\bm{I}})$ (detailed proof in
Appendix A).

Then, the loss in (6) converges to zero when the likelihood contribution term
$|\det{\bm{J}}|$ of the model $g({\bm{\theta}})$ (normalized by
$\det{\bm{\Sigma}}^{-1/2}$) compensates the difference between a squared
Mahalanobis distance for ${\bm{z}}$ from the target density and a squared
Euclidean distance for ${\bm{u}}\sim\mathcal{N}({\bm{0}},{\bm{I}})$.

This normalizing flow framework can estimate the exact likelihoods of any
arbitrary distribution with $p_{Z}$ density, while Mahalanobis distance is
limited to MVG distribution only. For example, CNNs trained with $L_{1}$
regularization would have Laplace prior [11] or have no particular prior in
the absence of regularization. Moreover, we introduce conditional normalizing
flows in the next section and show that they are more compact in size and have
fully-convolutional parallel architecture compared to [7, 8] models.

##  4 The proposed CFLOW-AD model

###  4.1 CFLOW encoder for feature extraction

We implement a feature extraction scheme with multi-scale feature pyramid
pooling similar to recent models [7, 8]. We define the discriminatively-
trained CNN feature extractor as an encoder $h({\bm{\lambda}})$ in Figure 2.
The CNN encoder maps image patches ${\bm{x}}$ into a feature vectors
${\bm{z}}$ that contain relevant semantic information about their content.
CNNs accomplish this task efficiently due to their translation-equivariant
architecture with the shared kernel parameters. In our experiments, we use
ImageNet-pretrained encoder following Schirrmeister  _et al_. [34] who show
that large natural-image datasets can serve as a representative distribution
for pretraining. If a large application-domain unlabeled data is available,
the self-supervised pretraining from [42, 19] can be a viable option.

One important aspect of a CNN encoder is its effective receptive field [21].
Since the effective receptive field is not strictly bounded, the size of
encoded patches ${\bm{x}}$ cannot be exactly defined. At the same time,
anomalies have various sizes and shapes, and, ideally, they have to be
processed with the variable receptive fields. To address the ambiguity between
CNN receptive fields and anomaly variability, we adopt common multi-scale
feature pyramid pooling approach. Figure 2 shows that the feature vectors
${\bm{z}}^{k}_{i}\in\mathbb{R}^{D_{k}},i\in\\{H_{k}\times W_{k}\\}$ are
extracted by $K$ pooling layers. Pyramid pooling captures both local and
global patch information with small and large receptive fields in the first
and last CNN layers, respectively. For convenience, we number pooling layers
in the last to first layer order.

###  4.2 CFLOW decoders for likelihood estimation

We use the general normalizing flow framework from Section 3.3 to estimate
$\log$-likelihoods of feature vectors ${\bm{z}}$. Hence, our generative
decoder model $g({\bm{\theta}})$ aims to fit true density $p_{Z}({\bm{z}})$ by
an estimated parameterized density $\hat{p}_{Z}({\bm{z}},{\bm{\theta}})$ from
(1). However, the feature vectors are assumed to be independent of their
spatial location in the general framework. To increase efficacy of
distribution modeling, we propose to incorporate spatial prior into
$g({\bm{\theta}})$ model using conditional flow framework. In addition, we
model $\hat{p}^{k}_{Z}({\bm{z}},{\bm{\theta}})$ densities using $K$
independent decoder models $g_{k}({\bm{\theta}}_{k})$ due to multi-scale
feature pyramid pooling setup.

Our conditional normalizing flow (CFLOW) decoder architecture is presented in
Figure 2. We generate a conditional vector ${\bm{c}}^{k}_{i}$ using a 2D form
of conventional positional encoding (PE) [39]. Each
${\bm{c}}^{k}_{i}\in\mathbb{R}^{C_{k}}$ contains $\sin$ and $\cos$ harmonics
that are unique to its spatial location $(h_{k},w_{k})_{i}$. We extend
unconditional flow framework to CFLOW by concatenating the intermediate
vectors inside decoder coupling layers with the conditional vectors
${\bm{c}}_{i}$ as in [2].

Then, the $k$th CFLOW decoder contains a sequence of conventional coupling
layers with the additional conditional input. Each coupling layer comprises of
fully-connected layer with $(D_{k}+C_{k})\times(D_{k}+C_{k})$ kernel, softplus
activation and output vector permutations. Usually, the conditional extension
does not increase model size since $C_{k}\ll D_{k}$. For example, we use the
fixed $C_{k}=128$ in all our experiments. Our CFLOW decoder has translation-
equivariant architecture, because it slides along feature vectors extracted
from the intermediate feature maps with kernel parameter sharing. As a result,
both the encoder $h({\bm{\lambda}})$ and decoders $g_{k}({\bm{\theta}}_{k})$
have convolutional translation-equivariant architectures.

We train CFLOW-AD using a maximum likelihood objective, which is equivalent to
minimizing loss defined by

$\begin{split}\mathcal{L}({\bm{\theta}})&=D_{KL}\left[p_{Z}({\bm{z}})\right\|\hat{p}_{Z}({\bm{z}},{\bm{c}},{\bm{\theta}})]\approx\\\
&\frac{1}{N}\sum^{N}_{i=1}\left[\frac{\|{\bm{u}}_{i}\|_{2}^{2}}{2}-\log\left|\det{\bm{J}}_{i}\right|\right]+\text{const},\end{split}$
(7)

where the random variable
${\bm{u}}_{i}=g^{-1}({\bm{z}}_{i},{\bm{c}}_{i},{\bm{\theta}})$, the Jacobian
${\bm{J}}_{i}=\nabla_{{\bm{z}}}g^{-1}({\bm{z}}_{i},{\bm{c}}_{i},{\bm{\theta}})$
for CFLOW decoder and an expectation operation in $D_{KL}$ is replaced by an
empirical train dataset $\mathcal{D_{\mathrm{train}}}$ of size $N$. For
brevity, we drop the $k$th scale notation. The derivation is given in Appendix
B.

After training the decoders $g_{k}({\bm{\theta}}_{k})$ for all $K$ scales
using (7), we estimate test dataset $\mathcal{D_{\mathrm{test}}}$
$\log$-likelihoods as

$\log\hat{p}_{Z}({\bm{z}}_{i},{\bm{c}}_{i},\hat{{\bm{\theta}}})=-\frac{\|{\bm{u}}_{i}\|_{2}^{2}+D\log(2\pi)}{2}+\log\left|\det{\bm{J}}_{i}\right|.$
(8)

Next, we convert $\log$-likelihoods to probabilities
$p^{k}_{i}=e^{\log\hat{p}_{Z}({\bm{z}}^{k}_{i},{\bm{c}}^{k}_{i},\hat{{\bm{\theta}}}_{k})}$
for each $k$th scale using (8) and normalize them to be in $\left[0:1\right]$
range. Then, we upsample $p^{k}_{i}$ to the input image resolution ($H\times
W$) using bilinear interpolation ${\bm{P}}_{k}=b(p^{k})\in\mathbb{R}^{H\times
W}$. Finally, we calculate anomaly score maps ${\bm{S}}^{H\times W}$ by
aggregating all upsampled probabilities as
${\bm{S}}=\max_{\sum^{K}_{k=1}{\bm{P}}_{k}}-\sum^{K}_{k=1}{\bm{P}}_{k}$.

Table 1: Complexity estimates for SPADE [7], PaDiM [8] and our CFLOW-AD. We compare train and test complexity as well as memory requirements. All models use the same encoder $h({\bm{\lambda}})$ setup, but diverge in the post-processing. SPADE allocates memory for a train gallery ${\mathbb{G}}$ used in $k$-nearest-neighbors. PaDiM keeps large matrices $({\bm{\Sigma}}^{k}_{i})^{-1},i\in\\{H_{k}\times W_{k}\\}$ for Mahalanobis distance. Our model employs trained decoders $g_{k}({\bm{\theta}}_{k})$ for post-processing. Model | Train | Test | Memory
---|---|---|---
SPADE [7] | ${\mathbb{G}}$ | $\sum\nolimits_{{\bm{v}}\in{\mathbb{G}}}\|{\bm{v}}-{\bm{z}}_{i}\|_{2}^{2}$ | ${\bm{\lambda}}+{\mathbb{G}}$
PaDiM [8] | ${\bm{\Sigma}}^{-1}_{i}$ | $M({\bm{z}}_{i})$ | ${\bm{\lambda}}+{\bm{\Sigma}}^{-1}_{i}$
Ours | $\mathcal{L}({\bm{\theta}})$ | $\log\hat{p}_{Z}({\bm{z}}_{i},{\bm{c}}_{i},{\bm{\theta}})$ | ${\bm{\lambda}}+{\bm{\theta}}$

###  4.3 Complexity analysis

Table 1 analytically compares complexity of CFLOW-AD and recent state-of-the-
art models with the same pyramid pooling setup _i.e_.
${\bm{z}}^{k\in\\{1\ldots K\\}}=h({\bm{x}},{\bm{\lambda}})$.

SPADE [7] performs $k$-nearest-neighbor clustering between each test point
${\bm{z}}^{k}_{i}$ and a gallery ${\mathbb{G}}$ of train data. Therefore, the
method requires large memory allocation for gallery ${\mathbb{G}}$ and a
clustering procedure that is typically slow compared to convolutional methods.

PaDiM [8] estimates train-time statistics _i.e_. inverses of covariance
matrices $({\bm{\Sigma}}^{k}_{i})^{-1}\in\mathbb{R}^{D_{k}\times
D_{k}},i\in\\{H_{k}\times W_{k}\\}$ to calculate $M({\bm{z}}^{k}_{i})$ at
test-time. Hence, it has low computational complexity, but it stores in memory
$H_{k}\times W_{k}$ matrices of $D_{k}\times D_{k}$ size for every $k$th
pooling layer.

Our method optimizes generative decoders $g_{k}({\bm{\theta}}_{k})$ using (7)
during the train phase. At the test phase, CFLOW-AD simply infers data
$\log$-likelihoods
$\log\hat{p}_{Z}({\bm{z}}_{i},{\bm{c}}_{i},\hat{{\bm{\theta}}})$ using (8) in
a fully-convolutional fashion. Decoder parameters
${\bm{\theta}}_{k\in\\{1\ldots K\\}}$ are relatively small as reported in
Table 6.

##  5 Experiments

Table 2: Ablation study of CFLOW-AD using localization AUROC metric on the MVTec [4] dataset, %. We experiment with input image resolution ($H\times W$), encoder architecture (ResNet-18 (R18), WideResnet-50 (WRN50) and MobileNetV3L (MNetV3)), type of normalizing flow (unconditional (UFLOW) and conditional (CFLOW)), number of coupling (# of CL) and pooling layers (# of PL). Encoder | WRN50 | WRN50 | WRN50 | WRN50 | WRN50 | R18 | R18 | MNetV3 | MNetV3
---|---|---|---|---|---|---|---|---|---
# of CL | 4 $\quad\longrightarrow\quad$ 8 | 8 | 8 | 8 | 8 | 8 | 8 | 8
# of PL | 2 | 2 $\quad\longrightarrow\quad$ 3 | 3 | 3 | 3 | 3 | 3 | 3
H$\times$W | 256 | 256 | 256 $\quad\longrightarrow\quad$ 512 | 512 | 256 $\quad\longrightarrow\quad$ 512 | 256 $\quad\longrightarrow\quad$ 512
Type | CFLOW | CFLOW | CFLOW | CFLOW $\rightarrow$ UFLOW | CFLOW | CFLOW | CFLOW | CFLOW
Bottle | 97.28$\pm$0.03 | 97.24$\pm$0.03 | 98.76$\pm$0.01 |  98.98$\pm$0.01 | 98.83$\pm$0.01 | 98.47$\pm$0.03 |  98.64$\pm$0.01 | 98.74 | 98.92
Cable | 95.71$\pm$0.01 | 96.17$\pm$0.07 |  97.64$\pm$0.04 | 97.12$\pm$0.06 | 95.29$\pm$0.04 |  96.75$\pm$0.04 | 96.07$\pm$0.06 | 97.62 | 97.49
Capsule | 98.17$\pm$0.02 | 98.19$\pm$0.05 |  98.98$\pm$0.00 | 98.64$\pm$0.02 | 98.40$\pm$0.12 |  98.62$\pm$0.02 | 98.28$\pm$0.05 | 98.89 | 98.75
Carpet | 98.50$\pm$0.01 | 98.55$\pm$0.01 | 99.23$\pm$0.01 |  99.25$\pm$0.01 | 99.24$\pm$0.00 | 99.00$\pm$0.01 |  99.29$\pm$0.00 | 98.64 | 99.00
Grid | 93.77$\pm$0.05 | 93.88$\pm$0.16 | 96.89$\pm$0.02 |  98.99$\pm$0.02 | 98.74$\pm$0.00 | 93.95$\pm$0.04 |  98.53$\pm$0.01 | 94.75 | 98.81
Hazelnut | 98.08$\pm$0.01 | 98.13$\pm$0.02 | 98.82$\pm$0.01 |  98.89$\pm$0.01 | 98.88$\pm$0.01 |  98.81$\pm$0.01 | 98.41$\pm$0.01 | 98.88 | 99.00
Leather | 98.92$\pm$0.02 | 99.00$\pm$0.06 | 99.61$\pm$0.01 |  99.66$\pm$0.00 | 99.65$\pm$0.00 | 99.45$\pm$0.01 |  99.51$\pm$0.02 | 99.50 | 99.64
Metal Nut | 96.72$\pm$0.03 | 96.72$\pm$0.06 |  98.56$\pm$0.03 | 98.25$\pm$0.04 | 98.16$\pm$0.03 |  97.59$\pm$0.05 | 96.42$\pm$0.03 | 98.36 | 98.78
Pill | 98.46$\pm$0.02 | 98.46$\pm$0.01 |  98.95$\pm$0.00 | 98.52$\pm$0.05 | 98.20$\pm$0.08 |  98.34$\pm$0.02 | 97.80$\pm$0.05 | 98.69 | 98.44
Screw | 94.98$\pm$0.06 | 95.28$\pm$0.06 | 98.10$\pm$0.05 |  98.86$\pm$0.02 | 98.78$\pm$0.01 | 97.38$\pm$0.03 |  98.40$\pm$0.03 | 98.04 | 99.09
Tile | 95.52$\pm$0.02 | 95.66$\pm$0.06 | 97.71$\pm$0.02 |  98.01$\pm$0.01 | 97.98$\pm$0.02 | 95.10$\pm$0.02 |  95.80$\pm$0.10 | 96.07 | 96.48
Toothbrush | 98.02$\pm$0.03 | 97.98$\pm$0.00 | 98.56$\pm$0.02 |  98.93$\pm$0.00 | 98.89$\pm$0.00 | 98.44$\pm$0.02 |  99.00$\pm$0.01 | 98.09 | 98.80
Transistor | 93.09$\pm$0.28 | 94.05$\pm$0.11 |  93.28$\pm$0.40 | 80.52$\pm$0.13 | 76.28$\pm$0.14 |  92.71$\pm$0.23 | 83.34$\pm$0.46 | 97.79 | 95.22
Wood | 90.65$\pm$0.10 | 90.59$\pm$0.07 | 94.49$\pm$0.03 |  96.65$\pm$0.01 | 96.56$\pm$0.02 | 93.51$\pm$0.03 |  95.00$\pm$0.04 | 92.24 | 94.96
Zipper | 96.80$\pm$0.02 | 97.01$\pm$0.05 | 98.41$\pm$0.09 |  99.08$\pm$0.02 | 99.06$\pm$0.01 | 97.71$\pm$0.06 |  98.98$\pm$0.01 | 97.50 | 99.07
Average | 96.31 | 96.46 | 97.87 | 97.36 | 96.86 | 97.06 | 96.90 | 97.59 | 98.16
Table 3: The detailed comparison of PaDiM [8], SPADE [7], CutPaste [19] and our CFLOW-AD on the MVTec [4] dataset for every class using AUROC or, if available, a tuple (AUROC, AUPRO) metric, %. CFLOW-AD model is with the best hyperparameters from Section 5.2 ablation study. For fair comparison, we group together results with the same encoder architectures such as ResNet-18 and WideResNet-50. Task | Localization | Detection | Localization
---|---|---|---
Encoder | ResNet-18 | EffNetB4 | WideResNet-50
Class/Model | CutPaste | Ours | CutPaste | Ours | CutPaste | Ours | SPADE | PaDiM | Ours
Bottle | 97.6 | 98.64 | 98.3 | 100.00 | 100.0 | 100.0 | (98.4, 95.5) | (98.3, 94.8) | (98.98, 96.80)
Cable | 90.0 | 96.75 | 80.6 | 97.62 | 96.2 | 97.59 | (97.2, 90.9) | (96.7, 88.8) | (97.64, 93.53)
Capsule | 97.4 | 98.62 | 96.2 | 93.15 | 95.4 | 97.68 | (99.0, 93.7) | (98.5, 93.5) | (98.98, 93.40)
Carpet | 98.3 | 99.29 | 93.1 | 98.20 | 100.0 | 98.73 | (97.5, 94.7) | (99.1, 96.2) | (99.25, 97.70)
Grid | 97.5 | 98.53 | 99.9 | 98.97 | 99.1 | 99.60 | (93.7, 86.7) | (97.3, 94.6) | (98.99, 96.08)
Hazelnut | 97.3 | 98.81 | 97.3 | 99.91 | 99.9 | 99.98 | (99.1, 95.4) | (98.2, 92.6) | (98.89, 96.68)
Leather | 99.5 | 99.51 | 100.0 | 100.00 | 100.0 | 100.0 | (97.6, 97.2) | (98.9, 88.8) | (99.66, 99.35)
Metal Nut | 93.1 | 97.59 | 99.3 | 98.45 | 98.6 | 99.26 | (98.1, 94.4) | (97.2, 85.6) | (98.56, 91.65)
Pill | 95.7 | 98.34 | 92.4 | 93.02 | 93.3 | 96.82 | (96.5, 94.6) | (95.7, 92.7) | (98.95, 95.39)
Screw | 96.7 | 98.40 | 86.3 | 85.94 | 86.6 | 91.89 | (98.9, 96.0) | (98.5, 94.4) | (98.86, 95.30)
Tile | 90.5 | 95.80 | 93.4 | 98.40 | 99.8 | 99.88 | (87.4, 75.9) | (94.1, 86.0) | (98.01, 94.34)
Toothbrush | 98.1 | 99.00 | 98.3 | 99.86 | 90.7 | 99.65 | (97.9, 93.5) | (98.8, 93.1) | (98.93, 95.06)
Transistor | 93.0 | 97.69 | 95.5 | 93.04 | 97.5 | 95.21 | (94.1, 87.4) | (97.5, 84.5) | (97.99, 81.40)
Wood | 95.5 | 95.00 | 98.6 | 98.59 | 99.8 | 99.12 | (88.5, 97.4) | (94.9, 91.1) | (96.65, 95.79)
Zipper | 99.3 | 98.98 | 99.4 | 96.15 | 99.9 | 98.48 | (96.5, 92.6) | (98.5, 95.9) | (99.08, 96.60)
Average | 96.0 | 98.06 | 95.2 | 96.75 | 97.1 | 98.26 | (96.0, 91.7) | (97.5, 92.1) | (98.62, 94.60)

###  5.1 Experimental setup

We conduct unsupervised anomaly detection (image-level) and localization
(pixel-level segmentation) experiments using the MVTec [4] dataset with
factory defects and the STC [22] dataset with surveillance camera videos. The
code is in PyTorch [28] with the FrEIA library [1] used for generative
normalizing flow modeling.

Industrial MVTec dataset comprises 15 classes with total of 3,629 images for
training and 1,725 images for testing. The train dataset contains only
anomaly-free images without any defects. The test dataset contains both images
containing various types of defects and defect-free images. Five classes
contain different types of textures (carpet, grid, leather, tile, wood), while
the remaining 10 classes represent various types of objects. We resize MVTec
images without cropping according to the specified image resolution (_e.g_.
$H\times W=256\times 256,512\times 512$ _etc_.) and apply $\pm 5^{\circ}$
augmentation rotations during training phase only.

STC dataset contains 274,515 training and 42,883 testing frames extracted from
surveillance camera videos and divided into 13 distinct university campus
scenes. Because STC is significantly larger than MVTec, we experiment only
with $256\times 256$ resolution and apply the same pre-processing and
augmentation pipeline as for MVTec.

We compare CFLOW-AD with the models reviewed in Section 2 using MVTec and STC
datasets. We use widely-used threshold-agnostic evaluation metrics for
localization: area under the receiver operating characteristic curve (AUROC)
and area under the per-region-overlap curve (AUPRO) [4]. AUROC is skewed
towards large-area anomalies, while AUPRO metric ensures that both large and
small anomalies are equally important in localization. Image-level AD
detection is reported by the AUROC only.

We run each CFLOW-AD experiment four times on the MVTec and report mean
($\mu$) of the evaluation metric and, if specified, its standard deviation
($\pm\sigma$). For the larger STC dataset, we conduct only a single
experiment. As in other methods, we train a separate CFLOW-AD model for each
MVTec class and each STC scene. All our models use the same training
hyperparameters: Adam optimizer with 2e-4 learning rate, 100 train epochs, 32
mini-batch size for encoder and cosine learning rate annealing with 2 warm-up
epochs. Since our decoders are agnostic to feature map dimensions and have low
memory requirements, we train and test CFLOW-AD decoders with 8,192
(32$\times$256) mini-batch size for feature vector processing. During the
train phase 8,192 feature vectors are randomly sampled from 32 random feature
maps. Similarly, 8,192 feature vectors are sequentially sampled during the
test phase. The feature pyramid pooling setup for ResNet-18 and WideResnet-50
encoder is identical to PaDiM [8]. The effects of other architectural
hyperparameters are studied in the ablation study.

###  5.2 Ablation study

Table 2 presents a comprehensive study of various design choices for CFLOW-AD
on the MVTec dataset using AUROC metric. In particular, we experiment with the
input image resolution ($H\times W$), encoder architecture (ResNet-18 [13],
WideResnet-50 [43], MobileNetV3L [14]), type of normalizing flow
(unconditional (UFLOW) or conditional (CFLOW)), number of flow coupling layers
(# of CL) and pooling layers (# of PL).

Our study shows that the increase in number of decoder’s coupling layers from
4 to 8 gives on average 0.15% gain due to a more accurate distribution
modeling. Even higher 1.4% AUROC improvement is achieved when processing
3-scale feature maps (layers 1, 2 and 3) compared 2-scale only (layers 2, 3).
The additional feature map (layer 1) with larger scale ($H_{1}\times
W_{1}=32\times 32$) provides more precise spatial semantic information. The
conditional normalizing flow (CFLOW) is on average 0.5% better than the
unconditional (UFLOW) due to effective encoding of spatial prior. Finally,
larger WideResnet-50 outperforms smaller ResNet-18 by 0.81%. MobileNetV3L,
however, could be a good design choice for both fast inference and high AUROC.

Importantly, we find that the optimal input resolution is not consistent among
MVTec classes. The classes with macro objects _e.g_. cable or pill tend to
benefit from the smaller-scale processing (256$\times$256), which,
effectively, translates to larger CNN receptive fields. Majority of classes
perform better with 512$\times$512 inputs _i.e_. smaller receptive fields.
Finally, we discover that the transistor class has even higher AUROC with the
resized to 128$\times$128 images. Hence, we report results with the highest
performing input resolution settings in the Section 5.3 comparisons.

Table 4: Average AUROC and AUPRO on the MVTec [4] dataset, %. Both the best detection and localization metrics are presented, if available. CFLOW-AD is with WideResNet-50 encoder. Metric | AUROC | AUPRO
---|---|---
Model | Detection | Localization
DifferNet [30] | 94.9 | - | -
DFR [37] | - | 95.0 | 91.0
SVDD [42] | 92.1 | 95.7 | -
SPADE [7] | 85.5 | 96.0 | 91.7
CutPaste [19] | 97.1 | 96.0 | -
PaDiM [8] | 97.9 | 97.5 | 92.1
CFLOW-AD (ours) | 98.26 | 98.62 | 94.60
Table 5: Average AUROC on the STC [22] dataset, %. Both the best available detection and localization metrics are showed. CFLOW-AD is with WideResNet-50 encoder. Metric | AUROC
---|---
Model | Detection | Localization
CAVGA [40] | - | 85.0
SPADE [7] | 71.9 | 89.9
PaDiM [8] | - | 91.2
CFLOW-AD (ours) | 72.63 | 94.48

###  5.3 Quantitative comparison

Table 4 summarizes average MVTec results for the best published models. CFLOW-
AD with WideResNet-50 encoder outperforms state-of-the-art by 0.36% AUROC in
detection, by 1.12% AUROC and 2.5% AUPRO in localization, respectively. Table
3 contains per-class comparison for the subset of models grouped by the task
and type of encoder architecture. CFLOW-AD is on par or significantly exceeds
the best models in per-class comparison with the same encoder setups.

Table 5 presents high-level comparison of the best recently published models
on the STC dataset. CFLOW-AD outperforms state-of-the-art SPADE [7] by 0.73%
AUROC in anomaly detection and PaDiM [8] by 3.28% AUROC in anomaly
localization tasks, respectively.

Note that our CFLOW-AD models in Tables 3-4 use variable input resolution as
discussed in the ablation study: 512$\times$512, 256$\times$256 or
128$\times$128 depending on the MVTec class. We used fixed 256$\times$256
input resolution in Table 5 for the large STC dataset to decrease training
time. Other reference hyperparameters in Tables 4-5 are set as: WideResnet-50
encoder with 3-scale pooling layers, conditional normalizing flow decoders
with 8 coupling layers.

Figure 3: Examples of the input images with ground truth anomaly masks (top
row) for various classes of the MVTec. Our CFLOW-AD model from Table 4
estimates anomaly score maps (middle row) and generates segmentation masks
(bottom row) for a threshold selected to maximize F1-score. The predicted
segmentation mask should match the corresponding ground truth as close as
possible. Figure 4: Distribution of anomaly scores for the cable class from
the MVTec learned by CFLOW-AD model from Table 4. Green density represent
scores for the anomaly-free feature vectors, while region of red-color density
shows scores for feature vectors with anomalies. The threshold $\tau$ is
selected to optimize F1-score.

###  5.4 Qualitative results

Figure 3 visually shows examples from the MVTec and the corresponding CFLOW-AD
predictions. The top row shows ground truth masks from
$\mathcal{D_{\mathrm{test}}}$ including examples with and without anomalies.
Then, our model produces anomaly score maps (middle row) using the
architecture from Figure 2. Finally, we show the predicted segmentation masks
with the threshold selected to maximize F1-score.

Figure 4 presents an additional evidence that our CFLOW-AD model actually
addresses the OOD task sketched in Figure 1 toy example. We plot distribution
of output anomaly scores for anomaly-free (green) and anomalous feature
vectors (red). Then, CFLOW-AD is able to distinguish in-distribution and out-
of-distribution feature vectors and separate them using a scalar threshold
$\tau$.

###  5.5 Complexity evaluations

Table 6: Complexity comparison in terms of inference speed (fps) and model size (MB). Inference speed for CFLOW-AD models from Table 3 is measured for (256$\times$256) / (512$\times$512) inputs. Complexity metric and Model |  Inference speed, fps | Model size, MB
---|---|---
STC | MVTec
R18 encoder only | 80 / 62 | 45
PaDiM-R18 [8] | 4.4 | 210 | 170
CFLOW-AD-R18 | 34 / 12 | 96
WRN50 encoder only | 62 / 30 | 268
SPADE-WRN50 [7] | 0.1 | 37,000 | 1,400
PaDiM-WRN50 [8] | 1.1 | 5,200 | 3,800
CFLOW-AD-WRN50 | 27 / 9 | 947
MNetV3 encoder only | 82 / 61 | 12
CFLOW-AD-MNetV3 | 35 / 12 | 25

In addition to analytical estimates in Table 1, we present the actual
complexity evaluations for the trained models using inference speed and model
size metrics. Particularly, Table 1 compares CFLOW-AD with the models from
Tables 4-5 that have been studied by Defard  _et al_. [8].

The model size in Table 6 is measured as the size of all floating-point
parameters in the corresponding model _i.e_. its encoder and decoder (post-
processing) models. Because the encoder architectures are identical, only the
post-processing models are different. Since CFLOW-AD decoders do not
explicitly depend on the feature map dimensions (only on feature vector
depths), our model is significantly smaller than SPADE and PaDiM. If we
exclude the encoder parameters for fair comparison, CFLOW-AD is 1.7$\times$ to
50$\times$ smaller than SPADE and 2$\times$ to 7$\times$ smaller than PaDiM.

Inference speed in Table 6 is measured with INTEL I7 CPU for SPADE and PaDiM
in Defard  _et al_. [8] study with 256$\times$256 inputs. We deduce that this
suboptimal CPU choice was made due to large memory requirements for these
models in Table 6. Thus, their GPU allocation for fast inference is
infeasible. In contrast, our CFLOW-AD can be processed in real-time with
8$\times$ to 25$\times$ faster inference speed on 1080 8GB GPU with the same
input resolution and feature extractor. In addition, MobileNetV3L encoder
provides a good trade-off between accuracy, model size and inference speed for
practical inspection systems.

##  6 Conclusions

We proposed to use conditional normalizing flow framework to estimate the
exact data likelihoods which is infeasible in other generative models.
Moreover, we analytically showed the relationship of this framework to
previous distance-based models with multivariate Gaussian prior.

We introduced CFLOW-AD model that addresses the complexity limitations of
existing unsupervised AD models by employing fully-convolutional translation-
equivariant architecture. As a result, CFLOW-AD is faster and smaller by a
factor of 10$\times$ than prior models with the same input resolution and
feature extractor setup.

CFLOW-AD achieves new state-of-the-art for popular MVTec with 98.26% AUROC in
detection, 98.62% AUROC and 94.60% AUPRO in localization. Our new state-of-
the-art for STC dataset is 72.63% and 94.48% AUROC in detection and
localization, respectively. Our ablation study analyzed design choices for
practical real-time processing including feature extractor choice, multi-scale
pyramid pooling setup and the flow model hyperparameters.

## References

  * [1] Lynton Ardizzone, Jakob Kruse, Carsten Rother, and Ullrich Köthe.  Analyzing inverse problems with invertible neural networks.  In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
  * [2] Lynton Ardizzone, Carsten Lüth, Jakob Kruse, Carsten Rother, and Ullrich Köthe.  Guided image generation with conditional invertible neural networks.  arXiv:1907.02392, 2019.
  * [3] Christoph Baur, Benedikt Wiestler, Shadi Albarqouni, and Nassir Navab.  Deep autoencoding models for unsupervised anomaly segmentation in brain MR images.  arXiv:1804.04488, 2018.
  * [4] Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger.  MVTec AD – a comprehensive real-world dataset for unsupervised anomaly detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [5] Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger.  Uninformed students: Student-teacher anomaly detection with discriminative latent embeddings.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [6] Paul Bergmann, Sindy Löwe, Michael Fauser, David Sattlegger, and Carsten Steger.  Improving unsupervised defect segmentation by applying structural similarity to autoencoders.  Proceedings of the 14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, 2019\.
  * [7] Niv Cohen and Yedid Hoshen.  Sub-image anomaly detection with deep pyramid correspondences.  arXiv:2005.02357v3, 2021.
  * [8] Thomas Defard, Aleksandr Setkov, Angelique Loesch, and Romaric Audigier.  PaDiM: a patch distribution modeling framework for anomaly detection and localization.  In Proceedings of the International Conference on Pattern Recognition (ICPR) Workshops, 2021.
  * [9] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio.  Density estimation using real NVP.  In Proceedings of the International Conference on Learning Representations (ICLR), 2017.
  * [10] Markus Goldstein and Seiichi Uchida.  A comparative evaluation of unsupervised anomaly detection algorithms for multivariate data.  PLOS ONE, 2016.
  * [11] Ian J. Goodfellow, Yoshua Bengio, and Aaron Courville.  Deep Learning.  MIT Press, 2016.
  * [12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2015.
  * [13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [14] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, Quoc V. Le, and Hartwig Adam.  Searching for MobileNetV3.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
  * [15] Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson.  Why normalizing flows fail to detect out-of-distribution data.  In Advances in Neural Information Processing Systems, 2020.
  * [16] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  ImageNet classification with deep convolutional neural networks.  In Advances in Neural Information Processing Systems, 2012.
  * [17] Anders Krogh and John Hertz.  A simple weight decay can improve generalization.  In Advances in Neural Information Processing Systems, 1992.
  * [18] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin.  A simple unified framework for detecting out-of-distribution samples and adversarial attacks.  In Advances in Neural Information Processing Systems, 2018.
  * [19] Chun-Liang Li, Kihyuk Sohn, Jinsung Yoon, and Tomas Pfister.  CutPaste: Self-supervised learning for anomaly detection and localization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [20] Yingying Li, Jie Wu, Xue Bai, Xipeng Yang, Xiao Tan, Guanbin Li, Shilei Wen, Hongwu Zhang, and Errui Ding.  Multi-granularity tracking with modularlized components for unsupervised vehicles anomaly detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2020.
  * [21] Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel.  Understanding the effective receptive field in deep convolutional neural networks.  In Advances in Neural Information Processing Systems, 2016.
  * [22] Weixin Luo, Wen Liu, and Shenghua Gao.  A revisit of sparse coding based anomaly detection in stacked RNN framework.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2017.
  * [23] Prasanta Chandra Mahalanobis.  On the generalized distance in statistics.  Proceedings of the National Institute of Sciences (Calcutta), 1936\.
  * [24] Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan.  Do deep generative models know what they don’t know?  In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
  * [25] Paolo Napoletano, Flavio Piccoli, and Raimondo Schettini.  Anomaly detection in nanofibrous materials by CNN-based self-similarity.  Sensors, 2018.
  * [26] Guansong Pang, Chunhua Shen, Longbing Cao, and Anton Van Den Hengel.  Deep learning for anomaly detection: A review.  ACM Comput. Surv., 2021.
  * [27] George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan.  Normalizing flows for probabilistic modeling and inference.  Journal of Machine Learning Research, 2021.
  * [28] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer.  Automatic differentiation in PyTorch.  In Autodiff workshop at Advances in Neural Information Processing Systems, 2017.
  * [29] Oliver Rippel, Patrick Mertens, and Dorit Merhof.  Modeling the distribution of normal data in pre-trained deep features for anomaly detection.  arXiv:2005.14140, 2020.
  * [30] Marco Rudolph, Bastian Wandt, and Bodo Rosenhahn.  Same same but DifferNet: Semi-supervised defect detection with normalizing flows.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2021.
  * [31] Lukas Ruff, Jacob Reinhard Kauffmann, Robert Vandermeulen, Gregoire Montavon, Wojciech Samek, Marius Kloft, Thomas G. Dietterich, and Klaus-Robert Müller.  A unifying review of deep and shallow anomaly detection.  Proc of the IEEE, 2021.
  * [32] Babak Saleh, Ali Farhadi, and Ahmed Elgammal.  Object-centric anomaly detection by attribute-based reasoning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2013.
  * [33] Mohammadreza Salehi, Niousha Sadjadi, Soroosh Baselizadeh, Mohammad Hossein Rohban, and Hamid R. Rabiee.  Multiresolution knowledge distillation for anomaly detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [34] Robin Tibor Schirrmeister, Yuxuan Zhou, Tonio Ball, and Dan Zhang.  Understanding anomaly detection with deep invertible networks through hierarchies of distributions and features.  In Advances in Neural Information Processing Systems, 2020.
  * [35] Thomas Schlegl, Philipp Seeböck, Sebastian M Waldstein, Georg Langs, and Ursula Schmidt-Erfurth.  f-AnoGAN: Fast unsupervised anomaly detection with generative adversarial networks.  Medical image analysis, 2019.
  * [36] Thomas Schlegl, Philipp Seeböck, Sebastian M Waldstein, Ursula Schmidt-Erfurth, and Georg Langs.  Unsupervised anomaly detection with generative adversarial networks to guide marker discovery.  In International Conference on Information Processing in Medical Imaging, 2017.
  * [37] Yong Shi, Jie Yang, and Zhiquan Qi.  Unsupervised anomaly segmentation via deep feature reconstruction.  Neurocomputing, 2021.
  * [38] A. Vasilev, V. Golkov, M. Meissner, I. Lipp, E. Sgarlata, V. Tomassini, D.K. Jones, and D. Cremers.  q-space novelty detection with variational autoencoders.  MICCAI 2019 International Workshop on Computational Diffusion MRI, 2019.
  * [39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in Neural Information Processing Systems, 2017.
  * [40] Shashanka Venkataramanan, Kuan-Chuan Peng, Rajat Vikram Singh, and Abhijit Mahalanobis.  Attention guided anomaly localization in images.  In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
  * [41] Guodong Wang, Shumin Han, Errui Ding, and Di Huang.  Student-teacher feature pyramid matching for unsupervised anomaly detection.  arXiv:2103.04257, 2021.
  * [42] Jihun Yi and Sungroh Yoon.  Patch SVDD: Patch-level SVDD for anomaly detection and segmentation.  In Proceedings of the Asian Conference on Computer Vision (ACCV), 2020.
  * [43] Sergey Zagoruyko and Nikos Komodakis.  Wide residual networks.  In Proceedings of the British Machine Vision Conference (BMVC), 2016\.
  * [44] Kang Zhou, Yuting Xiao, Jianlong Yang, Jun Cheng, Wen Liu, Weixin Luo, Zaiwang Gu, Jiang Liu, and Shenghua. Gao.  Encoding structure-texture relation with P-Net for anomaly detection in retinal images.  In Proceedings of the European Conference on Computer Vision (ECCV), 2020.

##  Appendix A Relationship with the flow framework

The loss function for the reverse
$D_{KL}\left[\hat{p}_{Z}({\bm{z}},{\bm{\theta}})\|p_{Z}({\bm{z}})\right]$
objective [27], where $\hat{p}_{Z}({\bm{z}},{\bm{\theta}})$ is the model
prediction and $p_{Z}({\bm{z}})$ is a target density, is defined as

$\mathcal{L}({\bm{\theta}})=\mathbb{E}_{\hat{p}_{Z}({\bm{z}},{\bm{\theta}})}\left[\log\hat{p}_{Z}({\bm{z}},{\bm{\theta}})-\log
p_{Z}({\bm{z}})\right].$ (5)

The first term in (5) can be written using (4) definition for a standard MVG
prior $({\bm{u}}\sim\mathcal{N}({\bm{0}},{\bm{I}}))$ as

$\log\hat{p}_{Z}({\bm{z}},{\bm{\theta}})=\log(2\pi)^{-D/2}-E^{2}({\bm{u}})/2+\log\left|\det{\bm{J}}\right|,$
(5.1)

where $E^{2}({\bm{u}})=\|{\bm{u}}\|_{2}^{2}$ is a squared Euclidean distance
of ${\bm{u}}$.

Similarly, the second term in (5) can be written for MVG density (2) using a
square of Mahalanobis distance as

$\log
p_{Z}({\bm{z}})=\log(2\pi)^{-D/2}+\log\det{\bm{\Sigma}}^{-1/2}-M^{2}({\bm{z}})/2.$
(5.2)

By substituting (5.1-5.2) into (5), the constants $\log(2\pi)^{-D/2}$ are
eliminated and the loss is

$\mathcal{L}({\bm{\theta}})=\mathbb{E}_{\hat{p}_{Z}({\bm{z}},{\bm{\theta}})}\left[\frac{M^{2}({\bm{z}})-E^{2}({\bm{u}})}{2}+\log\frac{|\det{\bm{J}}|}{\det{\bm{\Sigma}}^{-1/2}}\right].$
(6)

##  Appendix B CFLOW decoders for likelihood estimation

We train CFLOW-AD using a maximum likelihood objective, which is equivalent to
minimizing the forward $D_{KL}$ objective [27] with the loss defined by

$\mathcal{L}({\bm{\theta}})=D_{KL}\left[p_{Z}({\bm{z}})\right\|\hat{p}_{Z}({\bm{z}},{\bm{c}},{\bm{\theta}})],$
(7)

where $\hat{p}_{Z}({\bm{z}},{\bm{c}},{\bm{\theta}})$ is a conditional
normalizing flow (CFLOW) model with a condition vector
${\bm{c}}\in\mathbb{R}^{C}$.

The target density $p_{Z}({\bm{z}})$ is usually replaced by a constant because
the parameters ${\bm{\theta}}$ do not depend on this density during gradient-
based optimization. Then by analogy with unconditional flow (4), the loss (7)
for $\hat{p}_{Z}({\bm{z}},{\bm{c}},{\bm{\theta}})$ can be written as

$\mathcal{L}({\bm{\theta}})=-\mathbb{E}_{p_{Z}({\bm{z}})}\left[\log
p_{U}({\bm{u}})+\log\left|\det{\bm{J}}\right|\right]+\text{const}.$ (7.1)

In practice, the expectation operation in (7.1) is replaced by an empirical
train dataset $\mathcal{D_{\mathrm{train}}}$ of size $N$. Using the definition
of base distribution with $p_{U}({\bm{u}})$, the final form of (7) can be
expressed as

$\mathcal{L}({\bm{\theta}})\approx\frac{1}{N}\sum^{N}_{i=1}\left[\frac{\|{\bm{u}}_{i}\|_{2}^{2}}{2}-\log\left|\det{\bm{J}}_{i}\right|\right]+\text{const},$
(7.2)

where the random variable
${\bm{u}}_{i}=g^{-1}({\bm{z}}_{i},{\bm{c}}_{i},{\bm{\theta}})$ and the
Jacobian
${\bm{J}}_{i}=\nabla_{{\bm{z}}}g^{-1}({\bm{z}}_{i},{\bm{c}}_{i},{\bm{\theta}})$
depend both on input features ${\bm{z}}_{i}$ and conditional vector
${\bm{c}}_{i}$ for CFLOW model.


# Transformers as Strong Lens Detectors- From Simulation to Surveys

Hareesh Thuruthipilly<EMAIL_ADDRESS>Margherita Grespan
<EMAIL_ADDRESS>Adam Zadrożny  National Centre for Nuclear
Research, Astrophysics division (BP4), ul. Pasteura 7, 02-093 Warszawa.

(August 27, 2024)

###### Abstract

With the upcoming large-scale surveys like LSST, we expect to find
approximately $10^{5}$ strong gravitational lenses among data of many orders
of magnitude larger. In this scenario, the usage of non-automated techniques
is too time-consuming and hence impractical for science. For this reason,
machine learning techniques started becoming an alternative to previous
methods. In our previous work [59], we have proposed a new machine learning
architecture based on the principle of self-attention, trained to find strong
gravitational lenses on simulated data from the Bologna Lens Challenge. Self-
attention-based models have clear advantages compared to simpler CNNs and
highly competing performance in comparison to the current state-of-art CNN
models. We apply the proposed model to the Kilo Degree Survey, identifying
some new strong lens candidates. However, these have been identified among a
plethora of false positives, which made the application of this model not so
advantageous. Therefore, throughout this paper, we investigate the pitfalls of
this approach, and possible solutions, such as transfer learning, are
proposed.

##  I  Introduction

Strong gravitational lensing is one of the interesting predictions of General
Relativity (GR); it describes how light rays from a distant astronomical
source are deflected due to the massive gravitational potential of a
foreground galaxy. The manifestation of this effect in nature is observed as
arcs or rings around massive galaxies or multiple images of the source.
Different lens-source configurations produce different distortions. For a
detailed review, see [60]. Strong lensing (SL) systems are employed for
probing many unique astrophysical and cosmological phenomena. For instance,
they act as "gravitational telescopes", enabling the study of distant
astronomical objects which would be otherwise too faint to observe, such as
high redshift galaxies [21, 54], dwarf galaxies [39], etc. Depending on the
source and the lens configuration, there are various kinds of strong lensing
systems, such as galaxy clusters acting as lenses, quasars or supernovas
acting as sources, etc. One particular case of gravitational lensing is
galaxy-galaxy strong lensing (GGSL), where the background source and
foreground lens are both galaxies and massive enough to produce distortion in
the images of the source.

Ever since the discovery of the first GGSL system by Hewitt et al. [26],
numerous applications of GGSL in cosmology and astrophysics have been
discovered [34, 14, 13, 7, 3]. Strong lensing has been used to derive
cosmological constraints on dark energy [57] and on the cosmic expansion
history [1, 32, 6]. Strong lensing can also be used to constrain dark matter
models [62, 18, 19, 45], as well as to detect dark-matter substructures [17].
For these reasons, the current and upcoming surveys have significantly focused
on detecting SLs. For a detailed review of the applications of strong lensing,
please refer to Blandford & Narayan [2].

However, as promising and exciting as it sounds, strong gravitational lensing
is a rare phenomenon, with an order of magnitude of one gravitational lens per
thousand galaxies observed [10, 64]. For all the applications mentioned above,
a large sample of SLs is required to obtain statistically significant results.
Unfortunately, only a few hundred lensing systems have been detected and
confirmed by the present astronomical surveys until now. With the upcoming era
of advanced missions such as the Euclid [53], and LSST [29, 63], the number of
observable SLs is expected to reach $10^{5}$ [46, 12, 41], which should be
identified among $10^{9}$ galaxies [65]. Similarly, the number of new SLs
expected to be discovered in the Square Kilometre Array (SKA) survey will also
have similar orders of magnitude [40]. Therefore, analyzing data on future
large-scale surveys with manual or semi-automatic methods is highly time-
consuming and impractical. Hence, the situation demands better and more
effective alternative approaches, which should be fast, reliable, and robust.

In recent times, advancements in deep learning and computer vision have found
applications in astronomy. In particular, Convolutional Neural Networks (CNNs)
have been able to emerge as the state-of-the-art algorithm in various sectors
of astronomy (e.g. galaxy classification by Pérez-Carrasco et al. 49,
supernova classification by Cabrera-Vives et al. 5, and galaxy merger
identification by Pearson et al. 48). Similarly, CNNs and other deep learning-
based algorithms have been found to be effective in detecting and analyzing
SLs from large-scale surveys [37, 56, 15, 11]. For instance, Jacobs et al.
[31] applied CNNs to the data from Canada-France-Hawaii Telescope Legacy
Survey (CFHTLS) to find SLs. Similarly, numerous other successful attempts to
find potential SL candidates from the Kilo Degree Survey (KiDS) have been
reported [51, 50, 52, 25, 38]. Furthermore, various groups have successfully
used CNNs to identify strong lens galaxy-scale systems from large-scale
surveys such as Dark Energy Survey (DES) [30, 55], Dark Energy Spectroscopic
Instrument Legacy Imaging Surveys [28, 27], Pan-STARRS [4] and also on
comparatively small scale surveys like VOICE [23]. Furthermore, the Strong
Gravitational Lens Finding Challenge, designed to compare and develop new
lens-finding approaches, has demonstrated that CNNs perform better than human
inspection or any other traditional methods [42].

Recently, there was a breakthrough in natural language processing (NLP) by
Vaswani et al. [61] with the introduction of self-attention-based architecture
known as the transformers. Since then, there have been attempts to adapt the
idea of self-attention to build better image processing models [47, 67, 58].
The basic idea behind the transformer architecture is the attention mechanism,
which has also found a wide variety of applications in machine learning [66,
22]. In the case of NLP, self-attention correlates the different positions of
a single sequence to calculate a representation of the sequence. Similarly,
the idea of self-attention, as the name suggests, is to give relative
significance to the input features based on the input features themselves,
which helps the network to create a representation of the input with the
relatively essential features only. Recently, Facebook Inc. [9] and Google
Brain [20] have been able to surpass the existing image recognition models
with transformer-based architectures.

Even though transformer models have overtaken the image processing sector in
the machine learning regime, it is still an unexplored area in astrophysics.
Hence, to investigate the potential capacity of attention-based models to find
strong lenses, we have implemented various self-attention-based encoder models
(transformer encoders) to find the gravitational lenses in the Bologna Lens
Challenge [59]. In our earlier work, we also compared the performance of the
encoder models to high-accuracy CNNs taking part in the challenge [59].
Attention-based models showed high reliability for identifying SL,
outperforming most of the CNNs of the Bologna lens challenge. Since the
testing on the simulated dataset was a success, we did a blind search (no data
pre-processing) of lenses on the Kilo Degree Survey (KiDS) data release 4
(DR4). Testing the model on the KiDS survey showed the difficulties of moving
from simulated data to real data, showing the limited ability of the model to
generalize SL detection. The training data are simulated with respect to the
KiDS survey, which makes it possible to apply the model trained on simulations
to the survey data. The application of the model gives some candidates which
need a visual inspection by human experts further confirm the prediction.

The paper is organized as follows. Section 2 briefly describes the data used
to train and test our models. Section 3 provides a brief overview of the
methodology used in our study, including the model’s architecture and the
procedure we adapted for human visual inspection. The results of our analysis
and a detailed discussion of our results are presented in sections 4 and 5,
respectively. Finally, we conclude in section 6.

##  II  Data

###  II.1  Simulated lenses - Bologna Lens Challenge

The novel model for SL detection presented in [59] is trained on the Bologna
Strong Gravitational Lens Finding Challenge [42]. This challenge required
participants to classify strong lenses among other kinds of sources. The
images in this dataset are mock observations based on the Kilo-Degree Survey
(KiDS). However, the simulated images do not strictly mimic the surveys; they
are only employed as references to set noise levels, pixel sizes,
sensitivities, and other parameters. During the challenge, machine learning-
based methods were able to classify the images with high accuracy where a
human would have doubt. The mock images for the challenge are created using
Millennium simulation and GLAMER lensing code [43, 44]. The ground-based
images consist of simulated images from four bands $(u,g,r,$ and $i)$, and the
reference band was the $r$ band. The noise for the mock images is simulated by
adding normally distributed numbers with the variance given by the weight maps
from the KiDS survey. The example images of a mock simulated lens and non-lens
for the challenge are shown in Fig. 1. For a detailed review of how the data
was created, please refer to Metcalf et al. [42].

Figure 1: Typical image of a mock simulated lens (above) and a non-lens
(below) for the challenge. Bands are shown in the following order: $u,g,r,i$,
and the corresponding RGB image.

###  II.2  Real Data - Kilo-Degree Survey

The Kilo-Degree Survey [16] is an ESO public large optical imaging survey of
the Southern sky. It is carried out with the OmegaCAM wide-field imager [35]
mounted on the VLT Survey Telescope [8] at the Paranal Observatory in Chile.
The OmegaCAM has a field of view of 1 deg2, and the angular scale of the
pixels is 0.21 arcsec. KiDS has surveyed $\approx$ 1350 deg2 of sky in the
four optical bands u,g,r,i of which 1006 tiles, corresponding to $\approx$ 1
deg2, are publicly available (data release 4, DR4 [36]). The survey has the
best seeing in the $r$-band, with a median point spread function (PSF) full
width at half-maximum (FWHM) values of 1.0, 0.8, 0.65, and 0.85 arcsec in the
u, g, r, and i bands, respectively. For this study, from the complete KiDS DR4
(1006 tiles of $\sim$1 deg2), we randomly select $\sim$ 200 tiles as an
initial test. For these selected tiles we further sub-sample with a redshift
cut for the lens, $z_{L}$<0.8, giving an average amount of elements per tile
of $\approx$23000\. The KiDS collaboration carried out a search for strong
lenses with a convolutional neural network finding 169 SL candidates in the
DR4 [51, 50, 52, 25, 38].

##  III Methodology

###  III.1 Self-attention and Transformer encoder

The Transformer models we propose to detect SLs are inspired by the DEtection
TRansformer (DETR) created by Facebook, based on the principle of self-
attention [9]. In general, the attention function can be defined
mathematically as
$\text{Attention}(Q,K,V)=\text{softmax}\left(QK^{T}/\sqrt{d_{k}}\right)V,$
where $Q,K,V$ are vectors and $\sqrt{d_{k}}$ is the dimension of the vector
key $K$. The softmax function, by definition, is a normalized exponential
function that takes an input vector of K real numbers and normalizes it into a
probability distribution proportional to the exponential of the input numbers.
As we compute the normalized dot product between the query ($Q$) and the key
($K$), we get a tensor ($QK^{T}$) that encodes the relative importance of the
features in the key to the query [61]. For self-attention, the vectors $Q$,
$V$, and $K$ are identical. Hence, multiplying the tensor ($QK^{T}$) by the
vector ($V$) results in a vector that encodes the relative importance of
features inside the input vector. In simple terms, the central idea of self-
attention is to assign relative importance to the features of the input based
on the input itself. As shown in Fig. 2, the transformer encoder has a very
simple architecture. The architecture consists of a CNN backbone followed by a
set of attention layers that altogether extract and weigh the relevant
features of an input. In the final layers, there is a feed-forward network
(FFN) that learns how to combine these features to predict the output.

Figure 2: Scheme of the multi-head attention layer and scheme of the
architecture of the transformer encoder

The model we used for our study is the model named ’Lens Detector 15’
mentioned in the Thuruthipilly et al. [59]. The model was first trained for
300 epochs with an initial learning rate of $\alpha=10^{-4}$ to reduce the
binary cross entropy loss and again trained for another 100 epochs starting
with a learning rate of $\alpha=10^{-5}$. We use the exponential linear unit
(ELU) function as the activation function for all the layers in the model. The
weights of the model are initialized with the Xavier uniform initializer, and
all layers are trained from scratch by the ADAM optimizer with the default
exponential decay rates [24, 33]. The accuracy and the loss of the model as a
function of epochs are plotted in Fig. 3. For a detailed description of the
models, their architecture, and how the hyper-parameters affect the
performance, please refer to Thuruthipilly et al. [59]. In the spirit of
reproducible research, our code for Lens Detector 15 is publicly available at
https://github.com/hareesht23/Lens-Detector.

Figure 3: Accuracy and loss of the model ’Lens Detector 15’ as a function of
the epoch during training.

###  III.2 Visual Inspection

One of the big differences between simulated data and real data is the
presence of labels. Real data do not come with labels, and therefore in order
to check the accuracy of a model (how many SL candidates detected by the model
are actually lenses), a visual inspection by experts is needed. We trained the
model to distinguish lenses from non-lenses. More specifically, lenses are
labelled with 1 and other objects with 0. Therefore, when an object is
presented to the model, the latter gives a prediction probability of being a
lens. For this study, we use a threshold of 0.8; if an object has a prediction
probability greater than the threshold, it is considered a lens candidate and
further inspected. In our case, four volunteers visually inspected the strong
lens candidates with the task of tagging real lenses from false positives.
However, visual inspections can be biased depending on different factors, for
this reason, multiple people and a scale of grades are needed in the process.
We give a score to the images as follows:

  * •

5 - a sure lens, clear arcs-like structures

  * •

3 - maybe a lens, arcs-like structure but not so resolved

  * •

1 - interesting candidate but most likely not a lens

  * •

0 - not a lens

Finally, candidates graded with at least one 5 or identified by at least two
people (total grade bigger than 4) were further considered and inspected.

##  IV Results

###  IV.1 Results on the simulated dataset

The Bologna Lens Challenge was intended to improve the efficiency and biases
of tools used to find strong gravitational lenses on galactic scales. During
the challenge, it was proven that automated machine learning methods are more
efficient in detecting SLs than traditional machine learning methods. Here we
present the results from the best self-attention-based encoder ’Lens Detector
15’ [59] that was able to surpass all the other methods that participated in
the challenge and the model used in this study on the KiDS survey. The
confusion matrices for the challenge dataset with three different thresholds
(0.8,0.95, and 0.999, respectively) are shown in Fig. 4. It should be noted
that even with a very high threshold, such as 0.999, the model is able to
identify more than 80% of the strong lenses. The area under the receiver
operating characteristic curve (AUROC) and the probability output for the lens
detector 15 on the challenge set are shown in Fig. 5. For a detailed
discussion and review of different encoder models created for the challenge,
please refer to Thuruthipilly et al. [59].

Figure 4: The area under the receiver operating characteristic curve (AUROC)
and the probability output for the lens detector 15 on the challenge set.
Figure 5: The area under the receiver operating characteristic curve (AUROC)
and the probability output for the lens detector 15 on the challenge set.

###  IV.2 Results on KiDS DR4

The promising results on the simulated dataset encouraged us to test the model
on the real KiDS DR4 data. However, since the simulated dataset does not
exactly mimic the real survey, the number of false positives (FP) is too high
to be eventually useful. For a threshold prediction probability = 0.8, the
lenses detected by the model are around 1000 per tile. After a visual
inspection, only a few SL candidates remain - the model output is contaminated
by FPs. For instance, the KiDS $0.0-28.2$ tile contains $\approx$25000
galaxies with $z_{L}<0.8$, of which 900 are detected as SL, but only 5 SL
candidates are high-quality candidates selected by the labellers. This gives a
rate of 0.5% True Positives. The prediction probability distribution for the
discussed tile is shown in Fig.6.

Figure 6: The output probability distribution of our model for the KiDS tile
$0.0-28.2$. The blue vertical line indicates the prediction probability
threshold.

The model is able to filter out most of the not lensed galaxies, but this
result is not good enough for an application on the entire survey; having 1000
candidates per tile means a visual inspection of 105 objects. One example of a
lens found in this first blind search is shown in Fig. 7, the model is able to
identify arc-like structures, but at the same time, it is contaminated with
false positives. In the next section, we discuss how to improve this.

Figure 7: A new SL candidate found in this work with a lens redshift = 0.31.
(KiDS tile = 0.0,-28.2, Ra= 0.06, Dec = -28.193, the information about the
candidate is taken from KiDS DR4.

,

##  V Discussion

An initial glance at the results shows that our model is able to detect strong
lens candidates along with a large fraction of false positives. As we
explained earlier, all the SL candidates identified by the model should be
inspected by human inspectors, which is a time-consuming procedure. For this
reason, reducing the number of false positives is a very important step. Hence
a detailed interpretation of the results is necessary to understand how to
increase the performance of the model when moving from simulated data to real
data.

Firstly, as mentioned above, we did not apply any selection criteria, in the
catalogue space, to the candidates (e.g. image quality, flux density, and/or
morphology). This was done to understand whether object preselection impacts
the performance. The existing literature on KiDS survey choose a sample of
luminous red galaxies (LRGs) or bright galaxies (BGs) because the probability
of finding an SLs in this sample is very high [51, 50, 52, 25, 38]. This kind
of preselection lowers the number of false positives simply because it
decreases the objects given, in total, to the model. However, we are not
planning to apply this kind of selection to be sure of not losing important SL
candidates that the previous searches missed. Finally, our results are in
favour of preselection by the image quality; during the visual inspection, we
found that the model gives a high probability of being a lens for some poorly
observed images (glitches), which could have been removed with the
preselection.

To clearly understand the nature of the False positives, we looked deeper into
the training data by visually inspecting the strong lenses in it. Some
peculiar examples are shown in Fig. 8; for this purpose, we only show the
r-band image, which is the one with the highest resolution. From the figure,
it is clear that the first two cases have visible geometrical features of an
SL, whereas the last two cases (starting from the left) do not have any
evident features. The classification of glitched images as lenses can be
explained by the presence of objects like the third lens in Fig. 8 in the
training data. The central pixel is clearly saturated - resembling a glitch in
the charge-coupled device (CCD) camera. Moreover, some point-like objects have
been classified as lenses, which is explained by the presence of objects like
the fourth lens in Fig. 8.

Figure 8: Example of images labelled as a strong lens. Each image corresponds
to a different strong lens configuration, and the images are only plotted only
in the r-band.

This brings us back to our initial problem of switching from simulated data to
real data without having a significant number of false positives (FP). Even
though the model is trained on the simulated data to achieve very high
accuracy or AUROC, it gets confused when it is shown real data. Earlier works
also faced the same problem of going from simulated to real data. For example,
among the SLs candidates identified by Li et al. [38] using a CNN, only 1.7%
passed the visual inspection. Hence this problem poses an interesting question
to the astrophysics community. We plan to tackle this problem using transfer
learning and other data-augmenting methods. Instead of training the model with
more simulations, we train the existing model on the set of 169 SL candidates
found by the KiDS collaboration (examples in Fig.9) and use data augmentation
to increase the sample size. For the non-lens class, we plan to use the false
negatives uncovered by the model. For this class, we do not need data
augmentation and to ensure the dataset is not biased toward one part of the
sky we will use the same amount of objects per tile with the highest
prediction probability of being a lens. Giving the ’most’ wrong examples
should help with the model performance. The training set will be balanced,
with the same amount of real (augmented) lenses and non-lenses.
Fine-tuning the model parameters with transfer learning on small datasets can
improve the model performance, which will tackle the issue of the lack of
large realistic datasets. We have verified this idea and found that it is a
promising approach, and the final results will be discussed in further
publications.

Figure 9: Strong Lenses found by Petrillo et al. [52]

##  VI Conclusions

In [59], we have proposed a novel machine learning approach, built on self-
attention-based encoders, to detect strong gravitational lenses. In this
follow-up study, we have explored the usage of this architecture on the KiDS
survey and investigated the pitfalls of applying a simulation-trained model on
real data. The poor performance of the model on survey data is demonstrated by
the inability of the simulated events to capture the high complexity of real
data and the presence of mislabelled events. For very rare events such as SLs,
it is challenging to have a labelled dataset, and it is imperative to look for
new methods for training models rather than sticking with simulations only. We
propose transfer learning and data augmentation as practical solutions to this
problem, and we plan to discuss them further in the upcoming publications.

###### Acknowledgements.

We would like to acknowledge the support of dr. Orest Dorosh and Marianna
Zadrożna for helping with the visual inspection. Authors and NCBJ are grateful
for financial support from MNiSW grant DIR/WK/2018/12, and NCN grants
UMO-2017/26/M/ST9/00978 and UMO-2018/30/M/ST9/00757.

## References

  * Biesiada [2006] Biesiada, M. 2006, Phys. Rev. D, 73, 023006
  * Blandford & Narayan [1992] Blandford, R. D. & Narayan, R. 1992, Annual Review of Astronomy and Astrophysics, 30, 311
  * Bonvin et al. [2016] Bonvin, V., Courbin, F., Suyu, S. H., et al. 2016, MNRAS, 465, 4914
  * Cañameras et al. [2020] Cañameras, R., Schuldt, S., Suyu, S. H., et al. 2020, Astronomy & Astrophysics, 644, A163
  * Cabrera-Vives et al. [2017] Cabrera-Vives, G., Reyes, I., Förster, F., Estévez, P. A., & Maureira, J.-C. 2017, Astrophys. J. , 836, 97
  * Caminha et al. [2016] Caminha, G. B., Grillo, C., Rosati, P., et al. 2016, Astronomy & Astrophysics, 587, A80
  * Cao et al. [2015] Cao, S., Biesiada, M., Gavazzi, R., Piórkowska, A., & Zhu, Z.-H. 2015, Astrophys. J. , 806, 185
  * Capaccioli & Schipani [2011] Capaccioli, M. & Schipani, P. 2011, The Messenger, 146, 27
  * Carion et al. [2020] Carion, N., Massa, F., Synnaeve, G., et al. 2020, in Computer Vision – ECCV 2020, ed. A. Vedaldi, H. Bischof, T. Brox, & J.-M. Frahm (Cham: Springer International Publishing), 213–229
  * Chae et al. [2002] Chae, K. H., Biggs, A. D., Blandford, R. D., et al. 2002, Phys. Rev. Lett. , 89, 151301
  * Chianese et al. [2020] Chianese, M., Coogan, A., Hofma, P., Otten, S., & Weniger, C. 2020, Monthly Notices of the Royal Astronomical Society, 496, 381
  * Collett [2015] Collett, T. E. 2015, Astrophys. J. , 811, 20
  * Collett & Auger [2014] Collett, T. E. & Auger, M. W. 2014, MNRAS, 443, 969
  * Covone et al. [2009] Covone, G., Paolillo, M., Napolitano, N. R., et al. 2009, Astrophys. J. , 691, 531
  * Davies et al. [2019] Davies, A., Serjeant, S., & Bromley, J. M. 2019, Monthly Notices of the Royal Astronomical Society, 487, 5263
  * de Jong et al. [2013] de Jong, J. T. A., Verdoes Kleijn, G. A., Kuijken, K. H., & Valentijn, E. A. 2013, Experimental Astronomy, 35, 25
  * Despali et al. [2018] Despali, G., Vegetti, S., White, S. D. M., Giocoli, C., & van den Bosch, F. C. 2018, Monthly Notices of the Royal Astronomical Society, 475, 5424
  * Diaz Rivero et al. [2018] Diaz Rivero, A., Cyr-Racine, F.-Y., & Dvorkin, C. 2018, Phys. Rev. D, 97, 023001
  * Díaz Rivero et al. [2018] Díaz Rivero, A., Dvorkin, C., Cyr-Racine, F.-Y., Zavala, J., & Vogelsberger, M. 2018, Phys. Rev. D, 98, 103517
  * Dosovitskiy et al. [2021] Dosovitskiy, A., Beyer, L., Kolesnikov, A., et al. 2021, in 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 (OpenReview.net)
  * Ebeling et al. [2018] Ebeling, H., Stockmann, M., Richard, J., et al. 2018, Astrophysical Journal, Letters, 852, L7
  * Fu et al. [2019] Fu, J., Liu, J., Tian, H., et al. 2019, in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (Los Alamitos, CA, USA: IEEE Computer Society), 3141–3149
  * Gentile et al. [2021] Gentile, F., Tortora, C., Covone, G., et al. 2021, MNRAS, 510, 500
  * Glorot & Bengio [2010] Glorot, X. & Bengio, Y. 2010, in AISTATS
  * He et al. [2020] He, Z., Er, X., Long, Q., et al. 2020, MNRAS, 497, 556
  * Hewitt et al. [1988] Hewitt, J. N., Turner, E. L., Schneider, D. P., Burke, B. F., & Langston, G. I. 1988, Nature (London), 333, 537
  * Huang et al. [2021] Huang, X., Storfer, C., Gu, A., et al. 2021, Astrophys. J. , 909, 27
  * Huang et al. [2020] Huang, X., Storfer, C., Ravi, V., et al. 2020, Astrophys. J. , 894, 78
  * Ivezić et al. [2019] Ivezić, Ž., Kahn, S. M., Tyson, J. A., et al. 2019, Astrophys. J. , 873, 111
  * Jacobs et al. [2019] Jacobs, C., Collett, T., Glazebrook, K., et al. 2019, Astrophysical Journal, Supplement, 243, 17
  * Jacobs et al. [2017] Jacobs, C., Glazebrook, K., Collett, T., More, A., & McCarthy, C. 2017, Monthly Notices of the Royal Astronomical Society, 471, 167
  * Jullo et al. [2010] Jullo, E., Natarajan, P., Kneib, J. P., et al. 2010, Science, 329, 924
  * Kingma & Ba [2015] Kingma, D. P. & Ba, J. 2015, in 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, ed. Y. Bengio & Y. LeCun
  * Koopmans et al. [2006] Koopmans, L. V. E., Treu, T., Bolton, A. S., Burles, S., & Moustakas, L. A. 2006, Astrophys. J. , 649, 599
  * Kuijken [2011] Kuijken, K. 2011, The Messenger, 146, 8
  * Kuijken et al. [2019] Kuijken, K., Heymans, C., Dvornik, A., et al. 2019, Astronomy & Astrophysics, 625, A2
  * Lanusse et al. [2017] Lanusse, F., Ma, Q., Li, N., et al. 2017, MNRAS, 473, 3895
  * Li et al. [2020] Li, R., Napolitano, N. R., Tortora, C., et al. 2020, Astrophys. J. , 899, 30
  * Marshall et al. [2007] Marshall, P. J., Treu, T., Melbourne, J., et al. 2007, Astrophys. J. , 671, 1196
  * McKean et al. [2015] McKean, J., Jackson, N., Vegetti, S., et al. 2015, in Advancing Astrophysics with the Square Kilometre Array (AASKA14), 84
  * McKean et al. [2015] McKean, J. P., Jackson, N., Vegetti, S., et al. 2015, Strong gravitational lensing with the SKA
  * Metcalf et al. [2019] Metcalf, R. B., Meneghetti, M., Avestruz, C., et al. 2019, Astronomy & Astrophysics, 625, A119
  * Metcalf & Petkova [2014a] Metcalf, R. B. & Petkova, M. 2014a, Monthly Notices of the Royal Astronomical Society, 445, 1942
  * Metcalf & Petkova [2014b] Metcalf, R. B. & Petkova, M. 2014b, Monthly Notices of the Royal Astronomical Society, 445, 1942
  * Nadler et al. [2021] Nadler, E. O., Birrer, S., Gilman, D., et al. 2021, Astrophys. J. , 917, 7
  * Oguri & Marshall [2010] Oguri, M. & Marshall, P. J. 2010, Monthly Notices of the Royal Astronomical Society, 405, 2579
  * Parmar et al. [2019] Parmar, N., Ramachandran, P., Vaswani, A., et al. 2019, in Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, 68–80
  * Pearson et al. [2022] Pearson, W. J., Suelves, L. E., Ho, S. C. C., et al. 2022, Astronomy & Astrophysics, 661, A52
  * Pérez-Carrasco et al. [2019] Pérez-Carrasco, M., Cabrera-Vives, G., Martinez-Marin, M., et al. 2019, Publications of the ASP, 131, 108002
  * Petrillo et al. [2018] Petrillo, C. E., Tortora, C., Chatterjee, S., et al. 2018, MNRAS
  * Petrillo et al. [2017] Petrillo, C. E., Tortora, C., Chatterjee, S., et al. 2017, MNRAS, 472, 1129
  * Petrillo et al. [2019] Petrillo, C. E., Tortora, C., Vernardos, G., et al. 2019, MNRAS, 484, 3879
  * R. et al. [2021] R., S., J., A., Y., M., C., B., & S., C. C. 2021, arXiv e-prints, arXiv:2108.01201
  * Richard et al. [2011] Richard, J., Jones, T., Ellis, R., et al. 2011, Monthly Notices of the Royal Astronomical Society, 413, 643
  * Rojas et al. [2021] Rojas, K., Savary, E., Clément, B., et al. 2021, arXiv e-prints, arXiv:2109.00014
  * Schaefer et al. [2018] Schaefer, C., Geiger, M., Kuntzer, T., & Kneib, J.-P. 2018, Astronomy & Astrophysics, 611, A2
  * Shiralilou et al. [2020] Shiralilou, B., Martinelli, M., Papadomanolakis, G., et al. 2020, Journal of Cosmology and Astroparticle Physics, 2020, 057
  * Tan et al. [2021] Tan, A., Nguyen, D. T., Dax, M., Nießner, M., & Brox, T. 2021, in Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, 9799–9807
  * Thuruthipilly et al. [2022] Thuruthipilly, H., Zadrozny, A., Pollo, A., & Biesiada, M. 2022, Astronomy & Astrophysics, 664, A4
  * Treu [2010] Treu, T. 2010, Annual Review of Astronomy and Astrophysics, 48, 87
  * Vaswani et al. [2017] Vaswani, A., Shazeer, N., Parmar, N., et al. 2017, in Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, 5998–6008
  * Vegetti et al. [2012] Vegetti, S., Lagattuta, D. J., McKean, J. P., et al. 2012, Nature (London), 481, 341
  * Verma et al. [2019] Verma, A., Collett, T., Smith, G. P., Strong Lensing Science Collaboration, & the DESC Strong Lensing Science Working Group. 2019, arXiv e-prints, arXiv:1902.05141
  * Wardlow et al. [2013] Wardlow, J. L., Cooray, A., De Bernardis, F., et al. 2013, Astrophys. J. , 762, 59
  * Wilde et al. [2022] Wilde, J., Serjeant, S., Bromley, J. M., et al. 2022, Monthly Notices of the Royal Astronomical Society, 512, 3464
  * Zhang et al. [2018] Zhang, H., Goodfellow, I. J., Metaxas, D. N., & Odena, A. 2018, CoRR, abs/1805.08318 [eprint 1805.08318]
  * Zhao et al. [2020] Zhao, H., Jia, J., & Koltun, V. 2020, CoRR, abs/2004.13621 [eprint 2004.13621]