# ProPanDL: A Modular Architecture for Uncertainty-Aware Panoptic Segmentation

Jacob Deery Chang Won Lee Steven L. Waslander  Institute for Aerospace Studies
University of Toronto
Toronto, Canada
{jacob.deery<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

We introduce ProPanDL, a family of networks capable of uncertainty-aware
panoptic segmentation. Unlike existing segmentation methods, ProPanDL is
capable of estimating full probability distributions for both the semantic and
spatial aspects of panoptic segmentation. We implement and evaluate ProPanDL
variants capable of estimating both parametric (Variance Network) and
parameter-free (SampleNet) distributions quantifying pixel-wise spatial
uncertainty. We couple these approaches with two methods (Temperature Scaling
and Evidential Deep Learning) for semantic uncertainty estimation. To evaluate
the uncertainty-aware panoptic segmentation task, we address limitations with
existing approaches by proposing new metrics that enable separate evaluation
of spatial and semantic uncertainty. We additionally propose the use of the
energy score, a proper scoring rule, for more robust evaluation of spatial
output distributions. Using these metrics, we conduct an extensive evaluation
of ProPanDL variants. Our results demonstrate that ProPanDL is capable of
estimating well-calibrated and meaningful output distributions while still
retaining strong performance on the base panoptic segmentation task.

###### Index Terms:

panoptic segmentation; uncertainty; probabilistic deep learning; evidential
deep learning

##  I Introduction

The desire for full scene explainability in computer vision has spurred
interest in panoptic segmentation [1]. This task, requiring semantic
segmentation of all pixels and instance discrimination for foreground pixels,
unifies semantic and instance segmentation. Panoptic segmentation is appealing
in that it provides theoretically full scene explainability: every pixel
receives a predicted class, and every object is detected with the finest
possible representation (pixel-level masks). However, at present, most
panoptic and instance segmentation methods are capable of producing only point
estimates: a single class and a single instance per pixel.

In parallel with research on panoptic segmentation, there has been a growing
interest in uncertainty-aware deep learning [2]. An uncertainty-aware network
outputs a full distribution over its predictions that implicitly captures
their uncertainty, rather than a single point estimate. Applications for
uncertainty estimation are numerous: well-calibrated uncertainty estimates can
be used for anomaly detection [3], offline for active learning [4], or passed
directly to consumers such as tracking or localization pipelines [5], which
are often based on classical, probabilistic algorithms that can easily
leverage predicted probability distributions.

Figure 1: ProPanDL outputs on a sample frame from the Cityscapes dataset.
Clockwise from top left: input image, panoptic segmentation, semantic
uncertainty, offset vector samples, spatial uncertainty.

To date, there has been little research exploring the application of
uncertainty estimation to the panoptic segmentation task. Uncertainty-aware
panoptic segmentation was introduced by Sirohi et al. [6], who introduce a
network architecture capable of producing pixel-wise uncertainties as well two
metrics, pECE and uECE, for evaluating the task. However, their approach is
not without limitations. First, while panoptic segmentation has two aspects
(semantic segmentation and instance discrimination), their proposed metrics
evaluate only the total uncertainty rather than the two components
independently. Second, their EvPSNet segmentation network is capable of only
producing scalar uncertainty scores in the range $[0,1]$, rather than full
distributions. This limitation is reflected in the uECE and pECE metrics,
which evaluate only real-valued uncertainty rather than full distributions.

To address these shortcomings, in this work, we introduce Probabilistic
Panoptic-DeepLab (ProPanDL), a family of networks that extend the Panoptic-
DeepLab architecture [7] to produce pixel-wise predictive uncertainty in
panoptic segmentation. To our knowledge, this work is the first to predict
full output distributions for either instance or panoptic segmentation.
ProPanDL uses separate semantic and spatial branches to jointly estimate
semantic segmentation, instance discrimination, and their associated pixel-
wise uncertainties. Semantic segmentation uncertainty is represented as a
calibrated categorical distribution, produced via Temperature Scaling [8] or
Evidential Deep Learning [9]. For instance discrimination, we apply Variance
Networks [10] and SampleNet [11] to estimate the distribution of pixels
relative to their corresponding instance center.

In addition, we propose to decouple the semantic and spatial uncertainty
evaluation in pECE. Our proposed new metrics, $\text{pECE}_{\text{spa}}$ and
$\text{pECE}_{\text{sem}}$, are straightforward modifications of the pECE
metric adapted for separate spatial and semantic uncertainty evaluation. We
additionally propose the use of the energy score [12], a proper scoring rule
used for robust evaluation of regression predictive uncertainty [13], for
evaluation of the distributions produced by the ProPanDL spatial branch.

We conduct an extensive evaluation of ProPanDL variants, evaluating on both
the standard Panoptic Quality metric and the uncertainty-aware metrics. We
find that SampleNet + Evidential Deep Learning is the strongest combination,
capable of estimating reliable spatial and semantic distributions while still
retaining strong segmentation perfomance.

In summary, our contributions are threefold:

  1. 1.

We introduce Probabilistic Panoptic-DeepLab (ProPanDL), a family of networks
extending the Panoptic-DeepLab architecture [7] for uncertainty-aware
segmentation.

  2. 2.

We propose two metrics, $\text{pECE}_{\text{spa}}$ and
$\text{pECE}_{\text{sem}}$, for separate evaluation of spatial and semantic
uncertainty.

  3. 3.

We conduct a thorough evaluation of ProPanDL and analyze each proposed
uncertainty estimation method, demonstrating that our approach is capable of
robust panoptic uncertainty estimation.

##  II Related Work

###  II-A Panoptic Segmentation

Panoptic segmentation methods can be categorized into one of two approaches.
Top-down methods merge the outputs of semantic segmentation and proposal-based
instance segmentation branches. These methods require additional
postprocessing to resolve overlaps between individual instances and semantic
segmentation boundaries [14] [15] [16] [17]. Conversely, bottom-up methods
predict non-overlapping segments directly.

At the time of writing, Panoptic-DeepLab (PDL) [7] is the state of the art
panoptic segmentation method, ranking at the top of the Cityscapes [18]
leaderboard. PDL is a bottom-up method that operates by predicting the centers
of instance masks, regressing every pixel to a corresponding center, and
classifying the resulting segments using a parallel semantic segmentation
branch.

###  II-B Uncertainty Estimation

Uncertainty-aware deep learning is generally used to refer to any methods
which extend deep learning to estimation of full probability distributions
rather than point estimates. Uncertainty is often classified into aleatoric
and epistemic uncertainty. Epistemic uncertainty refers to the uncertainty
present in the parameters of a neural network, and can be reduced arbitrarily
by training the network with additional data. Conversely, aleatoric
uncertainty is inherent in the data (e.g. sensor noise, quantization) and
cannot be reduced via additional training.

####  II-B1 Uncertainty in Semantic Segmentation

Due to the nature of the segmentation task, which is structured as a pixel-
wise classification problem, many of the uncertainty estimation methods
developed for image classification can be extended directly to segmentation.
The first work to investigate this application was performed by Kendall and
Gal [19], who applied Monte Carlo Dropout (MCD) [20] in conjunction with
direct variance estimation [2] to jointly estimate aleatoric and epistemic
uncertainty. MCD approximates Bayesian inference over the network weights by
sampling the same input through a fully-trained network with Dropout enabled
at inference time.

Due to its simplicity and strong performance, MCD is widely used [21] [22]
[23] as a baseline uncertainty estimation method. However, MCD requires
multiple forward passes and is therefore computationally intensive and
generally cannot meet latency requirements to run online in an autonomous
driving setting. For this reason, there has been increasing interest in
”direct modelling” methods, which directly predict the parameters of a
probability distribution in one forward pass of the network [24]. As the
target output in classification is a categorical distribution, the Maximum
Softmax Probability has been proposed as a simple baseline for confidence
estimation [25], with temperature scaling a common variant to improve network
calibration [8]. Other work investigates regressing a distribution-over-
distributions to quantify the network’s confidence in the predicted class
probabilities. Both [9] (Evidential Deep Learning) and [26] (Dirichlet Prior
Networks) adopt this approach, drawing from Dempster-Shafer evidential
reasoning with the Dirichlet distribution as the target. The major difference
between the methods is the use of out-of-distribution data for regularization
in [26].

####  II-B2 Uncertainty in Instance Segmentation

Unlike semantic segmentation, few published works explore uncertainty
estimation for instance segmentation. Morrison et al. [27] obtain instance-
level uncertainty estimates by applying Monte Carlo Dropout sampling to Mask
R-CNN. Their method is instance-centric and is not capable of estimating
pixel-wise uncertainty. Rumberger et al. [28] use a similar dropout-based
approach, but employ a bottom-up metric learning approach to produce overlap-
free masks and pixel-wise uncertainties. Both of these approaches rely on
dropout sampling and are therefore ill-suited to deployment in autonomous
driving applications.

####  II-B3 Panoptic Segmentation

Sirohi et al. introduce the uncertainty-aware panoptic segmentation task in
[6] which, to the best of our knowledge, remains the only work on this task.
The authors introduce two new metrics: the uncertainty-aware calibration error
uECE and the panoptic calibration error pECE. Using the EfficientPS network
[14] as a base network, the authors propose the new EvPSNet architecture,
applying evidential deep learning separately to the semantic segmentation and
instance prediction branches. The predicted semantic and instance segmentation
map and masks and their associated uncertainties are fused by a post-
processing panoptic fusion module to produce pixel-wise semantic class
predictions, instance IDs, and uncertainties. While this method estimates
semantic and instance uncertainty separately, it fuses them into a total
panoptic uncertainty map. We argue that separately estimating and evaluating
each component of uncertainty is valuable; downstream tasks, such as tracking
or localization, may be able to benefit greatly from knowing that the location
is certain but the class is invalid, or vice versa. Moreover, while EvPSNet
estimates only a single real-valued uncertainty for every pixel, our method is
capable of estimating full posterior distributions for both semantic and
instance predictions, which provide more information to downstream consumers.

####  II-B4 Regression Tasks

The most common approaches to uncertainty estimation in regression assume some
parametric distribution for the output and modify the network to estimate the
parameters of this distribution. Gaussian (Variance Networks [10]) or Mixture
of Gaussians (Mixture Density Networks [29]) are common assumptions and have
been widely used for bounding box uncertainty in probabilistic object
detection [30] [31] [24] and dense regression tasks such as depth prediction
[32].

More recently, Harakeh et al. [11] introduced SampleNet, a network
architecture for estimating parameter-free distributions. By modifying the
last layer of a network to estimate $M$ samples rather than a single point
estimate, the network can accurately model distributions for which parametric
assumptions, such as Gaussian, may not hold. The network requires only one
forward pass during both training and inference to produce all $M$ samples,
resulting in minimal additional overhead. The authors successfully applied
SampleNet to monocular depth estimation; we adapt this approach for pixel
offset regression in Panoptic-DeepLab, a similar dense regression task.

####  II-B5 Evaluation

The widely accepted standard metrics for uncertainty evaluation are the
Expected Calibration Error (ECE) and Maximum Calibration Error (MCE) [8].
Sirohi et al. [6] propose a modified form of the ECE which they call the uECE,
as well as an additional pECE metric used specifically for panoptic
segmentation. However, these metrics evaluate only the total uncertainty and
do not separately evaluate semantic and spatial uncertainty. Moreover, they
are incapable of evaluating the full probability distributions that many
networks are capable of providing; calibration requires uncertainty to be
represented as a single real number in the range $[0,1]$. We propose a
modified form of pECE that addresses the first limitation and employ the
alternative energy score [12] to address the second.

For evaluation of probability distributions, proper scoring rules are
typically used. A scoring rule is proper if it reaches its minimum when the
evaluated distribution is identical to the target distribution [13] and is
called strictly proper if it is greater for all other distributions. For
classification, the Brier score [33] and standard cross-entropy loss are
common proper scoring rules; for regression, the negative log-likelihood (NLL)
and energy score (ES) [12] are widely used [13]. We use energy score both as a
training loss for ProPanDL and as an evaluation metric of the pixel-wise
probability distributions predicted by the offset regression branch.

##  III Methodology

Figure 2: An overview of the ProPanDL network architecture. The shared encoder
backbone, semantic decoder, spatial decoder, and center head are unchanged
from Panoptic-DeepLab [7]. We modify the semantic head to produce calibrated
pixel-wise categorical distributions and uncertainty estimates. Similarly, we
modify the offset head to predict a distribution of pixel offset vectors
rather than point estimates.

The task of panoptic segmentation involves two distinct subtasks: pixel
classification and instance discrimination. It is logical and intuitive that
any uncertainty method should be capable of quantifying uncertainty separately
for each aspect. We call the uncertainty associated with semantic segmentation
the semantic uncertainty and the uncertainty associated with instance
discrimination the spatial uncertainty, mirroring the parallel terms used in
probabilistic object detection [34].

We select Panoptic-DeepLab (PDL) [7] as the meta-architecture for our family
of uncertainty estimation networks, which we call Probabilistic Panoptic-
DeepLab (ProPanDL). With separate branches for semantic segmentation and
instance discrimination, PDL can be easily modified to produce separate
semantic and spatial uncertainty maps. The standard semantic segmentation
branch follows the DeepLabV3+ decoder architecture [35] and produces a
$C$-dimensional vector for each pixel indicating the predicted class
probabilities. The spatial branch of PDL produces two maps: the center
prediction map and the center offset regression map. The center prediction map
is a heatmap that predicts whether each pixel is located at the centroid of an
instance, while the center offset regression map regresses the $x,y$ offset
from each pixel to the center of the instance to which it belongs.

In order to produce the final panoptic segmentation map, PDL employs a simple
postprocessing stage. First, all predicted stuff pixels are copied directly
from the semantic segmentation map. Second, thresholding and non-max
suppression are applied to the center prediction heatmap to produce individual
predicted instance centers. Third, all thing pixels are associated to the
instance center closest to their predicted offset. All pixels associated to
one center are considered to be part of the same segment. Fourth, each segment
is given a semantic class based on simple majority vote of the pixels it
contains. This results in a single class assigned to each pixel and a single
instance ID to each thing pixel.

In order to enable uncertainty-aware panoptic segmentation, our ProPanDL
modifies the semantic head and pixel offset head of PDL. The modular nature of
this approach enables us to test different spatial and semantic uncertainty
methods in different combinations. When equipped for both spatial and semantic
uncertainty, ProPanDL is capable of estimating pixel-wise calibrated
categorical distributions in the semantic branch and pixel-wise multivariate
distributions of offset vectors in the spatial branch. An overview of the
ProPanDL architecture can be seen in Figure 2.

The following sections detail the uncertainty estimation methods we apply to
each branch. In all equations, we use $i$ to denote an individual pixel, $N$
for the number of pixels in an image, and $C$ for the number of semantic
classes.

###  III-A Semantic Uncertainty Estimation

####  III-A1 Temperature Scaling

While the maximum class probability (MCP) or maximum softmax probability (MSP)
is a widely used heuristic for classification confidence, modern neural
networks tend to be overconfident. Temperature Scaling is a simple method
proposed by Guo et al. [8] for improving the calibration of a classifier that
uses the softmax function as its last operation. A single parameter $T$, known
as the temperature, is used to adjust the logits $\mathbf{z}$ before they are
passed through softmax:

$\text{softmax}(\mathbf{z})=\frac{e^{\mathbf{z}/T}}{\sum_{c=1}^{C}e^{z_{c}/T}}$
(1)

Since the same temperature is applied to all of the class logits, the maximum
element does not change; therefore, temperature scaling only affects
calibration and does not change the predicted class. The temperature value is
usually treated as a learned parameter and is tuned by minimizing the cross
entropy loss on the validation set. During this procedure, the pretrained
weights of the network remain unchanged. We perform temperature scaling by
training over 25 epochs with a learning rate of 0.001.

####  III-A2 Evidential Deep Learning

Evidential Deep Learning [9] is a sampling-free direct modeling method that
can be easily applied to any network architecture. This approach requires
modifications only to the last layer of the network. Rather than estimating
class probabilities directly, the network is modified to instead regress the
parameters $\alpha_{1\ldots C}$ of a $C$-dimensional Dirichlet distribution.
As these parameters have a lower bound of 1, we append a final softplus
nonlinearity to the classifier output and add 1 to the resulting values. While
in the original image classification setting this estimation was performed
once per image, we estimate one Dirichlet distribution per pixel $i$ in the
image. Three auxiliary quantities can then be computed:

$\displaystyle S_{i}$ $\displaystyle=\sum_{c=1}^{C}\alpha_{i}^{c}$ (2)
$\displaystyle u_{i}$ $\displaystyle=C/S_{i}$ (3)
$\displaystyle\hat{p}_{i}^{c}$ $\displaystyle=\alpha_{i}^{c}/S_{i}$ (4)

The first quantity is known as the strength of the distribution. A higher
strength indicates higher confidence in the predicted per-class evidence. The
corresponding uncertainty $u_{i}$ is inversely proportional to the strength
and always falls in the range $(0,1]$. The estimated class probabilities
$\hat{p}_{i}^{c}$ can be calculated by dividing the corresponding Dirichlet
parameter $\alpha_{i}^{c}$ by the distribution strength. Although the
evidential formulation of this method provides a direct definition of
uncertainty, previous work has observed that this quantity often provides
worse estimates than using the maximum class probability or entropy of the
resulting categorical distribution [9] [36]. We evaluate and compare both
approaches.

Rather than training with the standard classification cross-entropy loss, the
authors of [9] propose several different loss terms that can be used for EDL.
In this work, we use the Type II Maximum Likelihood form of the loss, shown
below:

$\displaystyle\mathcal{L}_{\text{EDL}}$
$\displaystyle=\sum_{i=1}^{N}\sum_{c=1}^{C}y_{i}^{c}\left(\log\left(S_{i}\right)-\log\left(\alpha_{i}^{c}\right)\right)$
(5)

We additionally apply the KL divergence regularization term, which penalizes
the Kullback-Liebler divergence between two Dirichlet distributions $D$. We
first define $\boldsymbol{\tilde{\boldsymbol{\alpha}}_{i}}$ as the vector of
predicted Dirichlet parameters $\boldsymbol{\alpha}_{i}$ with the true class
parameter $\alpha_{i}^{c}$ replaced with 1.
$D_{A}=D(\mathbf{p}_{\mathbf{i}}\mid\tilde{\boldsymbol{\alpha}}_{\boldsymbol{i}})$
is then the Dirichlet distribution parameterized by
$\boldsymbol{\tilde{\boldsymbol{\alpha}}_{i}}$. The loss term penalizes the KL
divergence between $D_{A}$ and the Dirichlet distribution with uniform
parameters, $D_{1}=D\left(\mathbf{p}_{i}\mid\langle 1,\ldots,1\rangle\right)$.

$\displaystyle\mathcal{L}_{\text{KL}}$
$\displaystyle=\lambda_{t}\sum_{i=1}^{N}KL\left[D\left(\mathbf{p}_{\mathbf{i}}\mid\tilde{\boldsymbol{\alpha}}_{\boldsymbol{i}}\right)\|D\left(\mathbf{p}_{i}\mid\langle
1,\ldots,1\rangle\right)\right]$ (6)

The Kullback-Liebler divergence $KL$ between the two Dirichlet distributions
can be computed as follows, where $\Gamma:\mathbb{R}\rightarrow\mathbb{R}$ is
the gamma function and $\psi:\mathbb{R}\rightarrow\mathbb{R}$ is the digamma
function:

$\displaystyle KL[$ $\displaystyle
D\left(\mathbf{p}_{i}\mid\tilde{\boldsymbol{\alpha}}_{i}\right)\|D\left(\mathbf{p}_{i}\mid\mathbf{1}\right)]$
(7)
$\displaystyle=\log\left(\frac{\Gamma\left(\sum_{c=1}^{C}\tilde{\alpha}_{i}^{c}\right)}{\Gamma(C)\prod_{c=1}^{C}\Gamma\left(\tilde{\alpha}_{ik}\right)}\right)$
$\displaystyle+\sum_{c=1}^{C}\left(\tilde{\alpha}_{i}^{c}-1\right)\left[\psi\left(\tilde{\alpha}_{i}^{c}\right)-\psi\left(\sum_{c=1}^{C}\tilde{\alpha}_{ic}\right)\right]$

Since a value of 1 indicates no evidence, this term incentivizes the network
to remove all evidence for every class besides the true one. While the
$\mathcal{L}_{\text{EDL}}$ loss previously described rewards this as well, the
authors of [9] found that the KL divergence term resulted in improved
performance when the annealing coefficient $\lambda_{t}$ was scheduled
correctly. For our experiments, we found $\lambda_{t}=\min(0.1,t/60)$ to work
effectively, similar to the values proposed by [6].

###  III-B Spatial Uncertainty Estimation

Unlike the semantic segmentation subtask, which is pixel-wise classification,
the offset estimation subtask is a pixel-wise regression problem. We seek to
estimate, for every pixel, the distribution of offset vectors rather than a
single point estimate. In this section, we use $v_{i}$ to denote the ground
truth offset vector at pixel $i$ and use $\hat{v}_{i}$ to denote the predicted
offset vector for this pixel.

####  III-B1 Variance Network

Our first approach is to augment Panoptic-DeepLab with a Variance Network
(VarNet) [10]. VarNets assume that the output is drawn from a multivariate
normal distribution and seek to estimate the mean and covariance of this
distribution. In applying VarNet to ProPanDL, we assume a diagonal covariance
matrix, a common assumption in similar settings [30] [11]. We change the
dimension of the last layer from $H\times W\times 2$ to $H\times W\times 4$,
estimating both the mean offset vector and the diagonal elements of the
covariance matrix. We train VarNet with the standard negative log-likelihood
(NLL) loss [10]:

$\displaystyle\mathcal{L}_{\text{NLL}}$
$\displaystyle=\sum_{i=1}^{N}\left(\mathcal{L}_{\text{NLL},x}(i)+\mathcal{L}_{\text{NLL},y}(i)\right)$
(8) $\displaystyle\mathcal{L}_{\text{NLL},x}(i)$
$\displaystyle=\frac{1}{2}\ln\left[\sigma^{2}\left(\hat{v}_{i,x}\right)\right]+\frac{\left[v_{i,x}-y\left(\hat{v}_{i,x}\right)\right]^{2}}{2\hat{\sigma}_{i,x}^{2}\left(\hat{v}_{i,x}\right)}$

At inference time, we feed the estimated mean forward to the Panoptic-DeepLab
postprocessing stage. In order to produce the real-valued uncertainty map for
pECE evaluation, we calculate the total variance for each pixel (sum of the
variance in $x$ and $y$) and divide by the maximum variance across the
training set to scale the data between 0 and 1.

####  III-B2 SampleNet

While normally distributed data is a common assumption in uncertainty-aware
deep learning and is likely reasonable for the offset vectors of most pixels,
we expect that it may not hold for certain difficult cases in offset
regression. In particular, a pixel on the boundary of two instances will
possess uncertainty not just about the precise location of an instance center
but also about which instance it should be associated with. The true
distribution in this case is likely to be multimodal. In order to estimate
cases for which simple parametric distributions may be inadequate, we adapt
SampleNet [11]. Rather than predicting a single offset vector, we modify the
last layer of the offset head to predict $M$ different offset vectors
$\hat{v}_{i,1...M}$, which we treat as samples from the underlying
distribution. We train SampleNet using the pixel-wise energy score loss,
defined below for a single image with ground truth offset vectors $v_{i}$:

$\displaystyle\mathcal{L}_{\text{ES}}=\sum_{i=1}^{N}$
$\displaystyle\left(\frac{1}{M}\sum_{j=1}^{M}\left\|\hat{v}_{i,j}-{v}_{i}\right\|\right.$
(9)
$\displaystyle\left.-\frac{1}{2M^{2}}\sum_{j=1}^{M}\sum_{k=1}^{M}\left\|\hat{v}_{i,k}-\hat{v}_{i,j}\right\|\right)$

Due to memory constraints, we cap the number of samples at $M=10$ for all
experiments. In order to produce panoptic segmentation results at inference
time, we calculate predicted centers for all sample vectors in the same way as
Panoptic-DeepLab, and assign centers to pixels via majority among all samples.
We reduce the set of samples to scalar uncertainties with a similar approach
to VarNet: we calculate the sample variance for every pixel and divide by the
maximum variance seen for every pixel in the training set.

###  III-C Evaluation Metrics

Sirohi et al. [6] introduce two metrics for evaluating uncertainty-aware
panoptic segmentation, defined as follows:

uECE
$\displaystyle=\sum_{r=1}^{R}\frac{|B_{r}|}{N}|\text{acc}(B_{r})-\text{conf}(B_{r})|$
(10)

pECE
$\displaystyle=\frac{1}{S}\left(\sum_{(f,g)}\text{uECE}(f,g)+\sum_{\tilde{f}}\text{uECE}(\tilde{f})\right)$
(11)

The uECE, a modification of the standard ECE, is calculated by partitioning
pixels into $R$ equally-spaced bins based on confidence, where
$\text{conf}(x)=1-\text{unc}(x)$. Within each bin $B_{r}$, $\text{acc}(B_{r})$
is the average pixel accuracy and $\text{conf}(B_{r})$ is the average pixel
confidence.

While uECE is task-agnostic and can be applied to any uncertainty-aware
problem, pECE is specific to the panoptic segmentation task and is calculated
by averaging uECE over all $S$ predicted segments. For true positive segments
$f$ associated with ground-truth segments $g$, $\text{acc}(i)=1$ if the
semantic class agrees with the ground truth and the pixel $i$ lies in both $f$
and $g$. For false positive segments $\tilde{x}$, $\text{acc}(i)=0$ for all
pixels.

By averaging uECE over all segments, pECE ensures that calibration of all
pixels is evaluated. Moreover, averaging over segments implicitly places a
higher weight on segments with fewer pixels. This emphasizes performance on
small thing instances such as pedestrians and distant vehicles, which networks
often struggle to properly segment.

While these metrics are useful for evaluating the total uncertainty, they do
not allow for independent evaluation of spatial and semantic uncertainties. We
argue that evaluating each quantity separately provides much more insight into
the performance of the network, especially for modular architectures like
ProPanDL that support different uncertainty estimation methods in each branch.
Therefore, we propose the following modified metrics:

$\displaystyle\text{uECE}_{\text{spa}}$
$\displaystyle=\sum_{r=1}^{R}\frac{|B_{r}|}{N}|\text{acc}_{\text{spa}}(B_{r})-\text{conf}_{\text{spa}}(B_{r})|$
(12) $\displaystyle\text{pECE}_{\text{spa}}=$
$\displaystyle\frac{1}{S}\Big{(}\sum_{(f,g)}\text{uECE}_{\text{spa}}(f,g)$
(13)
$\displaystyle+\sum_{\tilde{f}}\text{uECE}_{\text{spa}}(\tilde{f})\Big{)}$
$\displaystyle\text{uECE}_{\text{sem}}$
$\displaystyle=\sum_{r=1}^{R}\frac{|B_{r}|}{N}|\text{acc}_{\text{sem}}(B_{r})-\text{conf}_{\text{sem}}(B_{r})|$
(14) $\displaystyle\text{pECE}_{\text{sem}}=$
$\displaystyle\frac{1}{S}\Big{(}\sum_{(f,g)}\text{uECE}_{\text{sem}}(f,g)$
(15)
$\displaystyle+\sum_{\tilde{f}}\text{uECE}_{\text{sem}}(\tilde{f})\Big{)}$

As the equations suggest, we simply separate the uECE and pECE metrics into
separate spatial and semantic metrics. Semantic accuracy
$\text{acc}_{\text{sem}}(i)=1$ if pixel $i$ is given the correct semantic
label, regardless of its instance association. Likewise, spatial accuracy
$\text{acc}_{\text{spa}}(i)=1$ if pixel $i$ belongs to both a true positive
segment $f$ and its associated ground truth segment $g$.

In order to evaluate against the pECE metric proposed by Sirohi et al. [6], we
are required to compute a total uncertainty map for every pixel. We define the
simple heuristic:

$\text{unc}=\max(\text{unc}_{\text{spa}},\text{unc}_{\text{sem}})$ (16)

for all methods. For methods which do not estimate spatial uncertainty, we set
$\text{unc}_{\text{spa}}=0$. We use the Maximum Class Probability [25] as
semantic uncertainty metric for all methods except EDL, where we use the
evidential uncertainty defined in Eq. (3).

A notable shortcoming of both the original and modified metrics is that they
operate only on scalar-valued uncertainty maps where uncertainty is
represented by a single real number. In contrast, ProPanDL seeks to model full
distributions for both semantic class and pixel offset. While evaluating a
categorical distribution via Shannon entropy or maximum softmax probability is
a standard approach [2], reducing a sample-modelled distribution to a single
real number discards much of the valuable information present in the samples.
We therefore use the energy score as an additional metric to directly evaluate
the quality of the predicted offset distributions.

##  IV Experimental Results

TABLE I: Panoptic segmentation performance on Cityscapes | Uncertainty | Performance
---|---|---
Method | Spatial | Semantic | PQ $\uparrow$ | pECE $\downarrow$ |  $\text{pECE}_{\text{spa}}$ $\downarrow$ |  $\text{pECE}_{\text{sem}}$ $\downarrow$
Panoptic-DeepLab [7] | $-$ | $-$ | 60.7 | 32.5 | 32.7 | 22.1
EvPSNet [6] | ✓ | ✓ | 63.7 | 19.3 | $-$ | $-$
SampleNet Only | ✓ | $-$ | 59.3 | 30.2 | 27.6 | 22.5
VarNet Only | ✓ | $-$ | 54.2 | 38.4 | 33.3 | 22.5
EDL Only | $-$ | ✓ | 56.1 | 25.2 | 33.0 | 18.0
TS Only | $-$ | ✓ | 60.7 | 29.3 | 32.7 | 20.1
SampleNet + TS | ✓ | ✓ | 59.3 | 30.4 | 27.4 | 20.3
VarNet + TS | ✓ | ✓ | 54.2 | 38.0 | 33.3 | 20.3
SampleNet + EDL | ✓ | ✓ | 58.8 | 29.2 | 29.2 | 19.5
VarNet + EDL | ✓ | ✓ | 48.0 | 35.1 | 32.8 | 17.3

###  IV-A Training

We use the Detectron2 111https://github.com/facebookresearch/detectron2
implementation of Panoptic-DeepLab with a ResNet-50 backbone as the starting
point for our experiments. The energy score used to train SampleNet is
implemented using the GeomLoss library [37]. We initialize all experiments
from the Detectron2 weights trained on Cityscapes with ImageNet pretraining on
the backbone. We leave the backbone frozen during training for all
experiments.

We use a batch size of 10 and crop of 512 $\times$ 1024 for all experiments
due to memory constraints. We train using the Adam optimizer [38] with a
polynomial learning rate schedule and no weight decay.

We independently train and evaluate all combinations of {SampleNet [11],
VarNet [10]} and {TS [8], EDL [9]}.

###  IV-B Quantitative Results

Table I shows the results of all methods evaluated on the Cityscapes
validation set.

In terms of panoptic segementation performance, we observe that all ProPanDL
methods drop in PQ relative to the PDL baseline. We note that the original
Panoptic-DeepLab paper finds a batch size of 32 and no crop to be optimal,
which may explain some of the performance drops that we observe. While all
methods fall short of the PQ achieved by EvPSNet in [6], we reiterate that our
goal was not to develop a state-of-the-art panoptic segmentation network;
rather, our objective was to demonstrate that it is possible to estimate
output distributions while not overly compromising performance on the base
task.

Performance on the uncertainty estimation task varies by method and by metric.
In terms of semantic uncertainty, both TS and EDL outperform the PDL baseline,
with EDL outperforming TS in all settings. In terms of spatial uncertainty,
SampleNet consistently outperforms VarNet. We further evaluate and analyze the
spatial uncertainty methods in the following section.

We note that none of our methods perform similarly to EvPSNet on the pECE
metric. We conjecture that this is due to the simple maximum heuristic which
we use to calculate the pixelwise total uncertainty, in contrast to the
panoptic fusion postprocessing stage which they adopt. However, as described
earlier, we believe that scalar-valued uncertainty maps are inadequate for
capturing all of the relevant uncertainty information, especially when
semantic and scalar uncertainty are conflated.

####  IV-B1 Offset distribution evaluation

TABLE II: Offset head characteristics Method | Avg. length | RMSE $\downarrow$ | ES $\downarrow$
---|---|---|---
Ground truth | 79.04 | $-$ | $-$
Panoptic-DeepLab [7] | 74.22 | 18.95 | 17.84
SampleNet [11] | 66.12 | 38.95 | 3.47
VarNet [10] | 32.49 | 65.51 | 46.84

In Table I, we can observe that while SampleNet variants perform well in terms
of PQ, VarNet performance drops by over 10% in the worst case. We believe that
this is due to the tendency of variance networks to produce high variance when
trained with NLL, as described in [13] and observed when training variance
networks for depth estimation [11]. In order to evaluate the predicted offset
probability distributions independently of the center and semantic
predictions, we calculate the energy score across all pixels. We calculate ES
for VarNet by sampling from each pixelwise multivariate Gaussian distribution,
similarly to [11].

Table II outlines the energy score for each model as well as the average
length and RMS error of the predicted offset vectors. We observe that VarNet
predicted sample vectors tend to be much shorter and have a significantly
higher RMSE. Due to the postprocessing stage of Panoptic-DeepLab, pixels with
shorter offset vectors are far more likely to end up clustered into an
incorrect instance, which can result in significant PQ drops. Figure 3 shows a
representative example, where the predicted offset variance is highest inside
vehicle instances, even in areas without overlap or ambiguity.

While SampleNet’s RMSE is also notably worse than vanilla Panoptic-DeepLab,
this is not unexpected; we calculate the RMSE over all samples, and the ES
loss used by SampleNet gives it the freedom to predict samples that have high
error in attempt to model the underlying distribution (i.e. multimodal
predictions near instance boundaries).

In terms of energy score, we observe that SampleNet attains the lowest value,
followed by PDL and then by VarNet. The high ES of VarNet is consistent with
our observation that the predicted distributions appear to have very high
variance. While NLL and ES are both proper scoring rules and thus attain their
global minimum at the same value, their behaviour outside of the minimum is
complementary; networks trained with NLL have a tendency to overestimate
rather than underestimate the variance, which is harshly penalized by ES [13].

Figure 3: An example figure produced by VarNet+EDL, demonstrating poor
instance discrimination performance. Clockwise from top left: input image,
panoptic segmentation, semantic uncertainty, predicted offset variance
(spatial uncertainty). Brighter colours in uncertainty maps indicate higher
uncertainty. While most ”car” pixels are given the correct semantic class, we
observe that the instance discrimination, as seen in the panoptic segmentation
image, is poor.

####  IV-B2 Oracle test

TABLE III: Oracle study on ProPanDL performance Offsets | Centers | Sem. seg | PQ | $\text{pECE}_{\text{spa}}$ | $\text{pECE}_{\text{sem}}$
---|---|---|---|---|---
SampleNet [11] | GT | EDL | 59.0 | 29.1 | 19.6
SampleNet [11] | PDL | GT | 79.4 | 20.3 | $-$
SampleNet [11] | GT | GT | 82.8 | 19.1 | $-$
VarNet [10] | GT | EDL | 48.6 | 33.2 | 17.5
VarNet [10] | PDL | GT | 74.0 | 33.4 | $-$
VarNet [10] | GT | GT | 78.2 | 36.4 | $-$
PDL | PDL | PDL | 60.7 | 32.7 | 22.1
PDL | GT | GT | 83.3 | $-$ | $-$

We conduct an oracle study to investigate the impact of each panoptic
segmentation aspect on the performance of ProPanDL. This is accomplished by
providing ground truth instance centers and/or semantic segmentation during
evaluation instead of the network predictions, and allows us to determine the
performance upper bound when using different offset branches. The results can
be seen in Table III. We see that in all cases, using ground truth semantic
segmentation provides the biggest increase in performance, a trend previously
observed in [7] and [39]. We note that while SampleNet remains competitive
with PDL, VarNet performance lags well behind, demonstrating the limitations
of this method when applied to panoptic segmentation.

##  V Conclusion

In this work, we introduced Probabilistic Panoptic-DeepLab, a family of
networks capable of uncertainty-aware panoptic segmentation. ProPanDL variants
model spatial and semantic uncertainty separately, producing calibrated
distributions for every pixel in the image. Possible extensions to this work
would leverage the rich information available in the predicted distributions
and could include extension to panoptic tracking or uncertainty-guided domain
adaptation. We hope this work motivates further research in the estimation and
application of predictive uncertainty for panoptic segmentation.

## References

  * [1] A. Kirillov, K. He, R. Girshick, C. Rother, and P. Dollár, “Panoptic segmentation,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 9404–9413.
  * [2] Y. Gal, “Uncertainty in deep learning,” Ph.D. dissertation, University of Cambridge, 2016.
  * [3] H. Blum, P.-E. Sarlin, J. Nieto, R. Siegwart, and C. Cadena, “The fishyscapes benchmark: Measuring blind spots in semantic segmentation,” _International Journal of Computer Vision_ , vol. 129, no. 11, pp. 3119–3135, sep 2021.
  * [4] S. Ebrahimi, M. Elhoseiny, T. Darrell, and M. Rohrbach, “Uncertainty-guided continual learning with bayesian neural networks,” in _International Conference on Learning Representations (ICLR)_ , 2020. [Online]. Available: https://openreview.net/forum?id=HklUCCVKDB
  * [5] N. Akai, T. Hirayama, and H. Murase, “Semantic localization considering uncertainty of object recognition,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 3, pp. 4384–4391, 2020.
  * [6] K. Sirohi, S. Marvi, D. Büscher, and W. Burgard, “Uncertainty-aware panoptic segmentation,” 2022. [Online]. Available: https://arxiv.org/abs/2206.14554
  * [7] B. Cheng, M. D. Collins, Y. Zhu, T. Liu, T. S. Huang, H. Adam, and L.-C. Chen, “Panoptic-deeplab: A simple, strong, and fast baseline for bottom-up panoptic segmentation,” in _CVPR_ , 2020.
  * [8] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” in _34th International Conference on Machine Learning (ICML)_ , ser. ICML’17. JMLR.org, 2017, p. 1321–1330.
  * [9] M. Sensoy, L. Kaplan, and M. Kandemir, “Evidential deep learning to quantify classification uncertainty,” _Advances in neural information processing systems_ , vol. 31, 2018.
  * [10] D. Nix and A. Weigend, “Estimating the mean and variance of the target probability distribution,” in _1994 IEEE International Conference on Neural Networks (ICNN’94)_ , vol. 1, 1994, pp. 55–60 vol.1.
  * [11] A. Harakeh, J. Hu, N. Guan, S. L. Waslander, and L. Paull, “Estimating regression predictive distributions with sample networks,” in _37th AAAI Conference on Artificial Intelligence_ , 2023.
  * [12] T. Gneiting and A. E. Raftery, “Strictly proper scoring rules, prediction, and estimation,” _Journal of the American statistical Association_ , vol. 102, no. 477, pp. 359–378, 2007.
  * [13] A. Harakeh and S. L. Waslander, “Estimating and evaluating regression predictive uncertainty in deep object detectors,” in _International Conference on Learning Representations_ , 2021.
  * [14] R. Mohan and A. Valada, “EfficientPS: Efficient panoptic segmentation,” _International Journal of Computer Vision_ , vol. 129, no. 5, pp. 1551–1579, 2021.
  * [15] L. Porzi, S. R. Bulo, A. Colovic, and P. Kontschieder, “Seamless scene segmentation,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 8277–8286.
  * [16] Y. Xiong, R. Liao, H. Zhao, R. Hu, M. Bai, E. Yumer, and R. Urtasun, “UPSNet: A unified panoptic segmentation network,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 8818–8826.
  * [17] A. Kirillov, R. Girshick, K. He, and P. Dollár, “Panoptic feature pyramid networks,” in _IEEE/CVF conference on computer vision and pattern recognition (CVPR)_ , 2019, pp. 6399–6408.
  * [18] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for semantic urban scene understanding,” in _IEEE conference on computer vision and pattern recognition (CVPR)_ , 2016, pp. 3213–3223.
  * [19] A. Kendall and Y. Gal, “What uncertainties do we need in bayesian deep learning for computer vision?” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [20] Y. Gal and Z. Ghahramani, “Dropout as a bayesian approximation: Representing model uncertainty in deep learning,” in _33rd International Conference on Machine Learning_. PMLR, 2016, pp. 1050–1059.
  * [21] J. Mukhoti and Y. Gal, “Evaluating bayesian deep learning methods for semantic segmentation,” _arXiv preprint arXiv:1811.12709_ , 2018.
  * [22] J. Mukhoti, J. van Amersfoort, P. H. S. Torr, and Y. Gal, “Deep deterministic uncertainty for semantic segmentation,” 2021. [Online]. Available: https://arxiv.org/abs/2111.00079
  * [23] S. Isobe and S. Arai, “Deep convolutional encoder-decoder network with model uncertainty for semantic segmentation,” in _2017 IEEE International Conference on INnovations in Intelligent SysTems and Applications (INISTA)_ , 2017, pp. 365–370.
  * [24] D. Feng, A. Harakeh, S. L. Waslander, and K. Dietmayer, “A review and comparative study on probabilistic object detection in autonomous driving,” _IEEE Trans. Intell. Transp. Syst._ , 2021.
  * [25] D. Hendrycks and K. Gimpel, “A baseline for detecting misclassified and out-of-distribution examples in neural networks,” _International Conference on Learning Representations_ , 2017.
  * [26] A. Malinin and M. Gales, “Predictive uncertainty estimation via prior networks,” _Advances in neural information processing systems_ , vol. 31, 2018.
  * [27] D. Morrison, A. Milan, and E. Antonakos, “Uncertainty-aware instance segmentation using dropout sampling,” in _The Robotic Vision Probabilistic Object Detection Challenge (CVPR 2019 Workshop)_ , 2019.
  * [28] J. L. Rumberger, L. Mais, and D. Kainmueller, “Probabilistic deep learning for instance segmentation,” in _European Conference on Computer Vision Workshop on BioImage Computing_. Springer, 2020, pp. 445–457.
  * [29] C. M. Bishop, “Mixture density networks,” 1994.
  * [30] A. Harakeh, M. Smart, and S. L. Waslander, “BayesOD: A Bayesian approach for uncertainty estimation in deep object detectors,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2020, pp. 87–93.
  * [31] Y. He and J. Wang, “Deep mixture density network for probabilistic object detection,” in _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2020, pp. 10 550–10 555.
  * [32] F. Tosi, Y. Liao, C. Schmitt, and A. Geiger, “SMD-nets: Stereo mixture density networks,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 8942–8952.
  * [33] G. W. Brier _et al._ , “Verification of forecasts expressed in terms of probability,” _Monthly weather review_ , vol. 78, no. 1, pp. 1–3, 1950.
  * [34] D. Hall, F. Dayoub, J. Skinner, H. Zhang, D. Miller, P. Corke, G. Carneiro, A. Angelova, and N. Sünderhauf, “Probabilistic object detection: Definition and evaluation,” in _IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2020, pp. 1031–1040.
  * [35] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoder-decoder with atrous separable convolution for semantic image segmentation,” in _European conference on computer vision (ECCV)_ , 2018, pp. 801–818.
  * [36] L. Grannas, “Real-time uncertainty estimation for semantic segmentation: Improving uncertainty estimates with temperature scaling and predicted dirichlet distributions,” 2020.
  * [37] J. Feydy, T. Séjourné, F.-X. Vialard, S.-i. Amari, A. Trouve, and G. Peyré, “Interpolating between optimal transport and MMD using sinkhorn divergences,” in _The 22nd International Conference on Artificial Intelligence and Statistics_ , 2019, pp. 2681–2690.
  * [38] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _International Conference on Learning Representations (ICLR)_ , 2015.
  * [39] Z. Zhou, Y. Zhang, and H. Foroosh, “Panoptic-polarnet: Proposal-free lidar point cloud panoptic segmentation,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 13 194–13 203.


# Industrial and Medical Anomaly Detection Through
Cycle-Consistent Adversarial Networks

Arnaud Bougaham111 These authors contributed equally to this work.
<EMAIL_ADDRESS>Valentin Delchevalerie222 These authors contributed
equally to this work. Mohammed El Adoui  Benoît Frénay

###### Abstract

In this study, a new Anomaly Detection (AD) approach for industrial and
medical images is proposed. This method leverages the theoretical strengths of
unsupervised learning and the data availability of both normal and abnormal
classes. Indeed, the AD is often formulated as an unsupervised task, implying
only normal images during training. These normal images are devoted to be
reconstructed, through an autoencoder architecture for instance. However, the
information contained in abnormal data, when available, is also valuable for
this reconstruction. The model would be able to identify its weaknesses by
better learning how to transform an abnormal (respectively normal) image into
a normal (respectively abnormal) one, helping the entire model to learn better
than a single normal to normal reconstruction. To address this challenge, the
proposed method uses Cycle-Generative Adversarial Networks (Cycle-GAN) for
(ab)normal-to-normal translation. After an input image has been reconstructed
by the normal generator, an anomaly score quantifies the differences between
the input and its reconstruction. Based on a threshold set to satisfy a
business quality constraint, the input image is then flagged as normal or not.
The proposed method is evaluated on industrial and medical datasets. The
results demonstrate accurate performance with a zero false negative constraint
compared to state-of-the-art methods. The code is available at
https://github.com/ValDelch/CycleGANS-AnomalyDetection.

###### keywords:

Cycle-GAN, Industry 4.0, Industrial Images, Medical Images, Anomaly Detection,
Zero False Negative

††journal: Neurocomputing

[aff1]organization=Faculty of Computer Science, NaDI institute,
addressline=Rue Grandgagnage 21, city=Namur, postcode=5000, country=Belgium
[aff2]naXys institute

##  1 Introduction

Image anomaly detection involves identifying anomalies within visual data,
playing a crucial role in highlighting unexpected patterns in images. This
work proposes a new approach with a Generative Adversarial Networks (GAN)
architecture for the task of Anomaly Detection (AD), which aims to combine the
advantages of both unsupervised learning and the data availability of the
normal and abnormal classes. Indeed, AD is often formulated as an unsupervised
task due to the frequent high imbalance between normal and abnormal data, and
the need for generalization across a wide range of anomalies. A common
practice is to use an autoencoder architecture to encode / decode normal
images. Only the normal class is taken into account, and the reconstruction
from the abnormal to the normal class is not included in this training
process. Yet, this is precisely the task we are expecting from a
reconstruction-based AD method during the inference step. The proposed method
seeks to overcome this limitation by learning how to transform an abnormal
image into a normal one by exploiting samples from both classes. The objective
is to generate a reconstructed image where any abnormal pixel is replaced by a
normal one in a visually-coherent manner. During the training step, both a
“normal” and an “abnormal” generator are tied together in an adversarial
framework, using Cycle-Generative Adversarial Networks (Cycle-GAN) (proposed
by Zhu et al. [32]). Also, reconstructing the abnormal data during the
training step yields to a better normal generator than the classical methods
using only the normal class. Even if the abnormal datasets can be small as it
is usually the case in the AD context, the normal generator performs better,
because its performance is also constrained by the abnormal generator,
resulting in a good reconstruction. To the best of our knowledge, this is the
first time that Cycle-GAN has been studied for this purpose. We still consider
this as an unsupervised learning task because the abnormal data used during
training is not necessarily representative of all anomalies that could occur.
Abnormal data are just given to help during the training phase by giving more
feedback to the generators. Therefore, the generalization is guaranteed as it
is the case in a classical GAN context, except that the normal reconstruction
is less noisy.

Cycle-GAN is a well-known architecture proposed a few years ago. It
constitutes an elegant way to learn conditional mappings from two different
domains $\mathcal{X}$ and $\mathcal{Y}$ (for image-to-image translation) by
applying a cycle-consistent constraint on the transformations. The popularity
of Cycle-GAN lies in the fact that they only need a dataset of unpaired images
to learn the mappings. In other words, they do not need the one-to-one
correspondence between data from $\mathcal{X}$ and $\mathcal{Y}$ and from
$\mathcal{Y}$ and $\mathcal{X}$, but only two independent sets of data
$\\{x_{i}\in\mathcal{X}\\}$ and $\\{y_{i}\in\mathcal{Y}\\}$. As an example,
one can consider two unpaired datasets $\\{x_{i}\\}$ and $\\{y_{i}\\}$ made of
unpaired aerial images and Google maps, respectively. A Cycle-GAN can be
trained to learn meaningful mappings from $\mathcal{X}$ to $\mathcal{Y}$ and
$\mathcal{Y}$ to $\mathcal{X}$. Figure 1 presents an example generated with
this Cycle-GAN.

Figure 1: Example generated from a Cycle-GAN (see Section 5 for training
details) that learns mappings between aerial photos $\mathcal{X}$ and Google
maps $\mathcal{Y}$ (dataset from [1]). The initial image $x\in\mathcal{X}$ can
be mapped to $\tilde{y}\in\mathcal{Y}$ thanks to a first generator $G$. The
second one $F$ can then go back from $\tilde{y}\in\mathcal{Y}$ to
$\tilde{x}\in\mathcal{X}$. A cycle-consistent constraint enforces $\tilde{x}$
to be close to $x$.

Numerous studies have demonstrated the versatility of Cycle-GAN in various
image analysis applications. Nonetheless, Cycle-GAN remains seldom used in
practice to solve problems in the industrial and medical domains. It is for
example the case of AD where no prior work directly exploits Cycle-GAN for
other purposes than data augmentation. Compared to the state-of-the-art
methods for AD in images as shown by [3], [10] or [24], Cycle-GAN seems to be
more suitable than concurrent methods that rely on traditional methods.
Furthermore, we show that the formalism behind Cycle-GANs makes them efficient
and well-suited for AD. Specifically, it introduces an identity loss for the
reconstruction of normal-to-normal (and abnormal-to-abnormal) images. This
makes the generators much less noisy than traditional GANs, making it possible
to better discriminate normal and abnormal images. To illustrate this, we
focus our experiments on several industrial and medical problems of AD. This
is motivated by (i) the abundance of AD problems in these domains, and (ii)
the positive societal impact of developing efficient AD algorithms for them.

The main contributions of our work could be listed as follow:

  * 1.

Use abnormal data in the training process by leveraging a Cycle-GAN
architecture for AD, thus considering an identity loss that allows a better
discrimination between normal and abnormal images.

  * 2.

Characterize and discuss the performances of the method for diverse industrial
and medical AD problems.

  * 3.

Conduct an extensive benchmark to compare the proposed approach with state-of-
the-art methods.

  * 4.

Discuss why Cycle-GAN is well-suited for AD in specific image types and
explore its potential in industrial and medical domains.

In the following, Section 2 presents the theoretical prerequisites to
understand Cycle-GAN. After that, Section 3 presents the previous works, and
highlights that most of them only use simple architectures up to the
traditional GAN ones, by training with only normal data. The proposed AD
method with Cycle-GAN is described in Section 4. Section 5 then introduces the
considered datasets, the experimental setup as well as the results. Finally a
discussion is presented in Section 6 before concluding with Section 7.

##  2 Background on Cycle-GAN

This section introduces the formalism for Cycle-Generative Adversarial Network
(Cycle-GAN); the building blocks and the loss functions are described.

###  2.1 Building Blocks

Cycle-GAN learns image-to-image mappings from an unpaired dataset composed of
two types of images from domains $\mathcal{X}$ and $\mathcal{Y}$. Cycle-GAN is
obtained by tying together two distinct conditional GANs with a cycle-
consistent constraint. The first GAN is made of a generator

$G:\mathcal{X}\cup\mathcal{Y}\longrightarrow\mathcal{Y}:G\left(z\right)=\tilde{y},$

and a discriminator $D_{Y}(\cdot)$, and the other is made of a generator

$F:\mathcal{X}\cup\mathcal{Y}\longrightarrow\mathcal{X}:F\left(z\right)=\tilde{x},$

and a discriminator $D_{X}(\cdot)$. For convenience, let us already consider
an AD task where $\mathcal{X}$ are abnormal images, while $\mathcal{Y}$ are
normal ones. On the one hand, the aim of $G$ is to generate from
$x\in\mathcal{X}\cup\mathcal{Y}$ an image such that $D_{Y}$ cannot distinguish
it from real normal images in $\mathcal{Y}$. On the other hand, $F$ aims to
generate images such that $D_{X}$ is fooled and cannot distinguish it from
real abnormal images in $\mathcal{X}$. To achieve this, Cycle-GAN is trained
with a combination of different losses that are described in the next section.

###  2.2 Objective Function

A Cycle-GAN is made of two GANs that are tied together with a cycle-consistent
constraint. The loss can be broken down into three parts so that

$G^{*},F^{*}=\text{arg}\min_{F,G}\max_{D_{X},D_{Y}}\mathcal{L}_{\text{adv}}+\lambda_{\text{cyc}}\mathcal{L}_{\text{cyc}}+\lambda_{\text{ide}}\mathcal{L}_{\text{ide}},$
(1)

where $\lambda_{\text{cyc}}$ and $\lambda_{\text{ide}}$ are meta-parameters
that constraint the different parts of the loss.

The first part of (1) is made of two classical adversarial losses [12]

$\mathcal{L}_{\text{adv}}=\mathcal{L}_{\text{GAN}}\left(G,D_{Y}\right)+\mathcal{L}_{\text{GAN}}\left(F,D_{X}\right),$

where,

$\displaystyle\mathcal{L}_{\text{GAN}}\left(G,D\right)$
$\displaystyle=\mathbb{E}_{y}\left[\log\left(D\left(y\right)\right)\right]$
$\displaystyle+\mathbb{E}_{x}\left[\log\left(1-D\left(G\left(x\right)\right)\right)\right].$

On the one side, by enforcing $G$ (resp. $F$) to minimize
$\mathcal{L}_{\text{adv}}$, the generator will try to generate images that
look similar to images from $\mathcal{Y}$ (resp. $\mathcal{X}$). On the other
side, by enforcing $D_{Y}$ (resp. $D_{X}$) to maximize
$\mathcal{L}_{\text{adv}}$, the discriminator will try to distinguish between
images coming from the generator $G$ (resp. $F$) and real images in
$\mathcal{Y}$ (resp. $\mathcal{X}$).

The second part of (1) is motivated by the fact that the reconstructed images
$F\left(G\left(x\right)\right)$ and $G\left(F\left(y\right)\right)$ should be
close to $x$ and $y$, respectively. In other words, the pair of GANs should be
cycle-consistent. This is achieved by the cycle-consistent loss

$\mathcal{L}_{\text{cyc}}=\mathbb{E}_{x}\left[\|F\left(G\left(x\right)\right)-x\|_{1}\right]+\mathbb{E}_{y}\left[\|G\left(F\left(y\right)\right)-y\|_{1}\right],$

where a L1 norm is used in the original work on Cycle-GAN [32].

In addition to the $\mathcal{L}_{\text{adv}}$ and $\mathcal{L}_{\text{cyc}}$,
an identity loss is added to constrain the generators to leave the images
unmodified if they are already in the desired output domain, defined as

$\mathcal{L}_{\text{ide}}=\mathbb{E}_{x}\left[\|F\left(x\right)-x\|_{1}\right]+\mathbb{E}_{y}\left[\|G\left(y\right)-y\|_{1}\right],$

so as to enforce $F\left(x\right)=x$ and $G\left(y\right)=y$. In other words,
$F$ should not add anomalies if the input image is already abnormal, and $G$
should not make any modification if it is already normal. Although the
identity loss is present in the implementation of Cycle-GAN from the original
paper, it is not discussed and seldom used in practice. However, in the
context of AD, the use of the identity loss is particularly relevant. Indeed,
it is expected from $G$ that it erases any abnormal pixel from the image.
Nonetheless, in the case the image does not contain any of them, it should
learn to leave it unmodified. This important property is enforced by the
identity loss.

##  3 Related Works

AD has long been an area of great concern in a wide range of fields such as
biomedicine [27], industry [6] and security [19, 2]. Furthermore, a
significant number of works have been published to characterize the AD
approaches in the literature. The scope of this section is focused on previous
works based mainly on GANs and Cycle-GAN, applied to the industrial and
medical domains. GANs are used for many image-related tasks, specifically in
AD [28, 3, 30, 7], segmentation, data augmentation, etc. However, in the AD
context, Cycle-GAN has been mostly used for data augmentation only [22, 17,
11]. Since our research is focused on AD in medical and industrial
applications, in this section, we review the most relevant researches applied
to these two fields. An in-depth analysis of the state-of-the-art methods
associated with our research issue shows that recent AD methods are mainly
using only GANs. Yet, Cycle-GAN can be highly useful for AD thanks to the
combination of unsupervised learning and the data availability of the normal
and abnormal classes.

Regarding the industrial studies, Bougaham et al. [6] propose to use
intermediate patches (i.e., parts of image) for the inference step after a
Wasserstein GAN training process. The objective is to produce an efficient
approach for AD on real industrial images of electronic Printed Circuit Board
Assembly (PCBA). The technique can be used to assist current industrial image
processing algorithms and to avoid tedious manual processing. Nevertheless,
due to the wide variety of possible anomalies in a PCBA and the high
complexity of autoencoder architectures, a real-world implementation remains a
challenging task, specifically for small anomalies, even if the method evolved
to overcome some limitations in the work of Bougaham et al. [7]. Akcay et al.
[3] use an autoencoder with skip connections, culminating in a GAN
discriminator, which proves to be an effective means of training the model for
the normal class, in the context of AD. Different from GAN techniques, Roth et
al. [24] recently suggest a patch-features encoding AD method applied to
industrial images, modifying the prediction threshold to ensure a 100% recall
rate with no false negatives. Also, Defard et al. [10] use a pre-trained CNN
for patch embedding and employ multivariate Gaussian distributions to obtain a
probabilistic representation of the normal class. Rippel et al. [23] and Zhang
et al. [31], as for them, suggest to use Cycle-GAN to perform data
augmentation by generating synthetic images for industrial inspection.

Regarding the medical field, Schlegl et al. [28] proposed an unsupervised AD
framework GANs (f-AnoGAN), that can detect the unseen anomalies of medical
subjects after being trained on healthy tomography images. Among the previous
studies in medical imaging, works of Hammami et al. [13] and Sandfort et al.
[26] could be cited. These authors use Cycle-GAN to perform the data
augmentation task for Magnetic Resonance Imaging (MRI) and Computed Tomography
(CT) scan images, respectively. Again, they show that using Cycle-GAN for data
augmentation leads to better segmentation performances afterwards.

Despite being better than the previous approaches, the above AD approaches use
unsupervised deep learning techniques, such as autoencoders, pre-trained CNN
and GANs, to characterize the normal class, without using the insights given
by the anomalies. In contrast, in this work, the anomaly images are leveraged
immediately in the training phase, using it as prior knowledge to strengthen
the model at recognizing anomalies. Furthermore, we assess our method in both
industrial and medical images while enforcing zero false negative (ZFN), which
is the most useful in these fields where missed detections have large impacts
on customers or patients.

In a concise way, a thorough analysis of the most important studies in the
literature shows that in the industrial and medical domains, Cycle-GAN has
been mainly used for data augmentation. This paper demonstrates the
suitability of Cycle-GAN for AD, in particular for industrial and medical
images, which has never been covered in the literature.

##  4 Methods

This section introduces the developed approach and shows its relevance for AD
with Cycle-GAN. The training and inference steps are illustrated in Figure 2.

The basic idea behind the use of Cycle-GAN for AD is to exploit the
conditional mapping learned by one of the two generators: the one that goes
from abnormal to normal images. Indeed, by forward-propagating an abnormal
image in this generator, it is expected to obtain a new image where the
anomaly is erased. Nonetheless, and thanks to the identity loss, if a normal
image is forward-propagated in the generator, it is expected to remain
unchanged. Therefore, by comparing the output of the generator with its input,
anomalies in the input images can be located. The other generator (normal-to-
abnormal) is not used for AD. It is only useful to jointly train the first
one, similarly to the two discriminators, but not for the AD inference step.

Figure 2: (Color online) Architecture for the training (left side) and the
inference (right side) steps (inspired from [32]). During the training step,
the first generator $G$ tries to map abnormal to normal images by fooling the
discriminator $D_{Y}$ that should not detect fake images. $F$ and $D_{X}$
follow the same idea but for normal images as input. During the inference
step, only $G$ is used even if the input can either be normal or abnormal.

To perform AD, the normal and abnormal test images are given to the learned
abnormal-to-normal generator. Then, an anomaly score is computed to measure
the distance between the original test image and the reconstructed one. In
this paper, two metrics are considered: a per-pixel sum of the squared
differences (SSE), and a Frechet Inception Distance (FID) [15]. The FID
anomaly score is more elaborated and focuses on perceptual differences thanks
to the use of a pre-trained Inception V3 network [29]. Two different
thresholds are considered for the anomaly detector. The first threshold is set
by minimizing the number of classification errors, which yields an anomaly
detector with maximum accuracy (ACC). The second one is set so that all true
positives are detected (only false alarms can be raised, but no anomaly can be
missed). This setting yields an anomaly detector with zero false negative
(ZFN), which is the most useful in the critical business applications of the
industrial or medical fields, where false negatives have large consequences
for customers or patients. In summary, four anomaly detectors are built, i.e.,
one for each pair of metrics and thresholds. In addition, the receiver
operating characteristic curve (AUCROC) [8] is also considered, in order to
quantify the performance at different threhsolds. Notice that the thresholds
are set on the test sets, and make it possible to assess how much the two
distributions (SSE and FID for normal and abnormal data) are discriminated.
The use of an additional validation set should be preferred, but it would have
been too costly in terms of abnormal data for several datasets (due to the
scarcity of abnormal data). Therefore, the accuracy values are an overestimate
of the classification performances, and should be seen as a metric to quantify
how much the method highlights abnormal images compared to normal images on
the test sets. Our goal is indeed to measure the discriminative power of
Cycle-GAN for AD, so as to prospectively validate the practical interest of
our idea in the industrial and medical domains.

##  5 Experiments

This section presents the experiments carried out to evaluate the proposed AD
method. First, the datasets and the data preprocessing steps are introduced.
Next, the model architecture is detailed. Finally, qualitative and
quantitative results are presented, and then discussed in Section 6.

###  5.1 Datasets

Seven datasets are used for both the industrial and medical domains, where
several types of anomalies may occur. To assess the strengths and weaknesses
of our method, we defined four categories of anomalies: small / large object-
shaped, or small / large textured-shaped anomalies. Indeed, the anomaly can
either arise on a specific object characterized by abrupt changes in pixels
intensity (a picture of a screw with a broken head for instance), or where the
pixels intensity changes are more progressive and more homogeneous (an
abnormal color in a wood picture for instance).

To cover the industrial side, the public MVTEC-AD dataset [5] is used. It
consists of different high resolution industrial images from 15 different
categories of object and texture-shaped products with and without anomalies.
In this work, 4 datasets were selected from MVTEC-AD to cover the different
natures of images: the Hazelnut (large object-shaped images), the Screw (small
object-shaped images), the Tile (large texture-shaped images) and the Wood
(small texture-shaped images) dataset, which are made of $501$, $480$, $347$
and $326$ images, respectively. All of these datasets are clearly imbalanced
with a minority of abnormal images.

To investigate the medical side, three datasets of object and texture-shaped
images are used, coming from healthy and unhealthy subjects. The first dataset
is made of $253$ Brain MRI images (large object-shaped images) [9]. Second,
Breast Ultrasound (large texture-shaped images) [4] is made of $789$ images.
One should mention that for this dataset, many images were manually labeled by
experts by highlighting the tumor on the images. Therefore, to avoid any bias
during training, we removed all those annotated images from the dataset,
resulting in a dataset of $654$ images. Finally, the Retina OCT dataset (small
and large texture-shaped anomalies) [18] contains $83,600$ images of Optical
Coherence Tomography. All of these medical datasets are imbalanced with a
minority of normal images. Table 1 provides an overview of the number of
normal and abnormal images for each dataset.

Table 1: Sizes of the training and test sets for each dataset, regarding the number of normal and abnormal data. A particular split to guarantee that the test sets are balanced is always chosen, even if the initial datasets are imbalanced. | Training set | Test set
---|---|---
Dataset | # Normal | # Abnormal | # Normal | # Abnormal
Hazelnut | 396 | 35 | 35 | 35
Screw | 301 | 59 | 60 | 60
Tile | 222 | 43 | 41 | 41
Wood | 236 | 30 | 30 | 30
Brain MRI | 49 | 106 | 49 | 49
Breast Ultrasound | 67 | 455 | 66 | 66
Retina OCT | 13,172 | 44,084 | 13,172 | 13,172

Some of the aforementioned datasets come with different types of anomalies. In
this case, a single abnormal class is created by aggregating all the abnormal
classes together. Also, the normal and abnormal classes are sometimes
imbalanced. Therefore, in order to avoid the use of an imbalanced metric to
evaluate the results, a specific split of the dataset is applied to ensure
that the test sets are fully balanced. This is obtained by keeping half of the
minority class images for the test set, as well as the same number of randomly
picked images from the majority class. All the remaining images are left to
the training set. Given that AD may sometimes be a very imbalanced predictive
problem, one of the two classes is generally overpopulated in the training
sets. However, the test sets are perfectly balanced, which allows us to assess
the performances with a simple accuracy metric. Furthermore, even if using
imbalanced training sets may hurt the performances for most of the supervised
machine learning algorithms, the training process of Cycle-GAN is less
sensitive to this. The task of Cycle-GAN differs from a simple label
prediction, and each image gives feedback to the two generators, either
directly or indirectly. The same data preprocessing steps are performed for
all the datasets. Images are resized to a resolution of $256\times 256$ pixels
by using a bicubic interpolation method. Data augmentation is also performed
so that objects and textures are rotated and flipped along both axes (except
for Retina OCT where only flipping along the horizontal axis is pertinent).

###  5.2 Network Architecture and Training Procedure

For convenience and practical purposes, the architectures used in this work as
well as the training procedures are similar for the different applications,
and follow the experimental setup presented in the initial paper on Cycle-GAN
[32]. The generators are formed by three convolution layers, several residual
blocks [14], two fractionally-strided convolution layers and one final
convolution layer. We use 9 residual blocks for images resized at $256\times
256$ resolution, and instance normalization. For the discriminators, we use
$70\times 70$ PatchGANs [16, 20, 21]. All the models are trained through 200
iterations of the Adam optimizer with a learning rate of $2\times 10^{-4}$. A
linear learning decay is introduced at the middle of the training. The meta-
parameters $\lambda_{\text{cyc}}$ and $\lambda_{\text{ide}}$ are fixed to 10
and 5, respectively. To give an idea of the computation time, training the
Cycle-GAN on 500 images (4500 images with the data augmentation) with a
$256\times 256$ resolution for 200 iterations roughly takes 12 hours on a
single Nvidia A100 GPU. All experiments have been performed with CUDA 11.3 and
Pytorch 1.10.2.

###  5.3 Experimental Results

This subsection showcases the qualitative and quantitative assessments.

####  5.3.1 Qualitative Assessments

The quality of the reconstruction, as well as the highlighting of anomalies
are presented in Figure 3. It shows the original image, the normal (generated)
version and their squared pixel difference image, for the selected industrial
and medical datasets.

Figure 3: (Color online) Industrial and medical image examples. For each
dataset, the left green-framed block presents normal images and the right red-
framed block shows abnormal images, with the original image (1st column), the
normal version generated (2nd column), and their squared pixel-wise difference
image (3rd column).

Notice that the difference images (3rd column) of the abnormal data highlight
the anomaly, unlike the ones of the normal data. These images have been
specifically chosen to illustrate different typical cases. The following
quantitative assessment evaluates the global performances on all the test
sets, which are in agreement with the qualitative examples presented below.

####  5.3.2 Quantitative Assessments

Table 2 shows the quantitative results of our method (CycleGAN-AD-256). For
comparison purposes, the same architecture is assessed with a lower input
image resolution of $64\times 64$, and only 6 residual blocks for the
generator (CycleGAN-AD-64). Three different methods are also compared to ours,
namely Ganomaly [3], Padim [10] and PatchCore [24].

Table 2: Quantitative Performance Metrics for all the different models on all
the different industrial (4 first) and medical (3 last) datasets. For each
model, the accuracy with the zero false negative constraint (ZFN), the maximum
accuracy (ACC) and the AUCROC (AUC) metrics are presented, considering FID
(when applicable) and SSE metrics. All the metrics are the mean ± the standard
deviation (both in percent) over 5 runs for which random train and test splits
were generated. The mean performance is computed for each model in the last
line w.r.t. their most profitable setting (SSE or FID) when available, and
bold values are the best of each metric.

CycleGAN-AD-256 (ours) CycleGAN-AD-64 (ours) Ganomaly [3] Padim [10] PatchCore
[25] ZFN ACC AUC ZFN ACC AUC ZFN ACC AUC ZFN ACC AUC ZFN ACC AUC Hazelnut FID
98.00 ± 2.14 99.14 ± 0.70 99.89 ± 0.12 74.86 ± 12.89 94.57 ± 2.10 97.32 ± 0.91
51.14 ± 1.67 66.57 ± 3.33 63.31 ± 4.73 / / / / / / SSE 96.29 ± 2.94 98.29 ±
1.07 99.67 ± 0.25 95.43 ± 2.29 98.29 ± 0.57 99.71 ± 0.14 80.29 ± 10.32 87.14 ±
7.17 92.16 ± 4.90 53.43 ± 1.71 95.71 ± 0.00 92.02 ± 0.29 54.29 ± 0.00 95.71 ±
0.00 92.16 ± 0.00 Screw FID 52.03 ± 2.18 57.63 ± 4.25 54.34 ± 8.10 50.51 ±
0.68 57.46 ± 3.36 53.08 ± 5.96 51.19 ± 1.15 57.63 ± 1.42 54.04 ± 3.14 / / / /
/ / SSE 52.37 ± 3.53 57.97 ± 5.46 52.81 ± 6.29 52.03 ± 3.24 57.63 ± 2.89 51.31
± 5.06 62.71 ± 10.39 77.12 ± 5.22 80.86 ± 3.49 55.59 ± 4.57 66.27 ± 5.42 54.02
± 11.91 59.32 ± 0.00 90.68 ± 0.00 84.83 ± 0.00 Tile FID 91.43 ± 7.88 98.33 ±
1.21 99.30 ± 0.73 58.81 ± 6.37 78.81 ± 1.90 83.53 ± 2.61 57.62 ± 3.07 70.48 ±
2.05 72.47 ± 1.56 / / / / / / SSE 78.10 ± 7.85 89.76 ± 3.42 95.40 ± 2.52 52.86
± 2.21 75.24 ± 3.64 78.32 ± 2.67 50.48 ± 0.58 53.57 ± 2.61 42.39 ± 5.50 61.90
± 0.00 88.10 ± 0.00 81.86 ± 0.00 61.90 ± 0.00 88.10 ± 0.00 81.86 ± 0.00 Wood
FID 91.33 ± 6.78 97.00 ± 1.94 99.04 ± 0.79 71.00 ± 14.85 88.67 ± 1.63 92.82 ±
2.09 53.33 ± 3.80 56.00 ± 2.71 43.62 ± 3.76 / / / / / / SSE 97.00 ± 3.71 97.67
± 2.91 98.89 ± 1.37 92.33 ± 6.96 96.33 ± 2.87 98.48 ± 1.51 61.00 ± 4.29 71.67
± 7.67 75.09 ± 7.24 94.67 ± 10.67 95.33 ± 9.33 95.02 ± 9.96 100.00 ± 0.00
100.00 ± 0.00 100.00 ± 0.00 Brain MRI FID 78.57 ± 4.65 87.76 ± 3.35 93.19 ±
2.56 73.27 ± 4.05 79.39 ± 2.99 80.85 ± 3.92 54.90 ± 2.08 60.20 ± 3.48 58.88 ±
4.83 / / / / / / SSE 84.49 ± 3.95 86.94 ± 3.73 91.50 ± 2.92 84.08 ± 4.81 88.37
± 3.63 91.96 ± 3.16 61.02 ± 4.63 68.37 ± 1.12 69.98 ± 2.78 50.41 ± 0.50 92.45
± 13.06 91.58 ± 12.78 50.41 ± 0.50 98.98 ± 0.00 97.98 ± 0.02 Breast Ultrasound
FID 83.23 ± 6.53 91.38 ± 1.23 95.45 ± 0.91 84.92 ± 3.29 85.85 ± 3.46 89.62 ±
3.52 57.69 ± 2.48 70.15 ± 4.83 74.41 ± 5.29 / / / / / / SSE 85.23 ± 2.98 89.38
± 3.31 92.53 ± 2.41 86.46 ± 3.85 87.85 ± 3.01 91.23 ± 2.31 61.23 ± 5.19 68.62
± 2.45 71.81 ± 2.29 50.62 ± 0.31 96.77 ± 4.92 97.27 ± 2.42 50.62 ± 0.31 99.23
± 0.00 98.48 ± 0.01 Retina OCT FID 50.73 ± 0.59 97.23 ± 0.05 98.81 ± 0.08
50.19 ± 0.08 92.10 ± 0.22 96.87 ± 0.09 50.09 ± 0.07 69.92 ± 5.23 76.25 ± 6.61
/ / / / / / SSE 50.29 ± 0.26 96.74 ± 0.07 98.49 ± 0.10 51.02 ± 0.75 96.45 ±
0.04 98.33 ± 0.07 50.37 ± 0.40 79.33 ± 1.34 86.86 ± 1.47 50.01 ± 0.01 93.90 ±
3.03 98.24 ± 1.51 50.01 ± 0.01 99.95 ± 0.05 99.97 ± 0.00 MEAN 79.89 89.93
91.43 74.31 86.25 88.05 62.03 74.89 78.83 59.52 89.79 87.14 60.94 96.09 93.61

For CycleGAN-AD-256, CycleGAN-AD-64 and Ganomaly, an anomaly score (SSE or
FID) is computed to measure the distance between the original test image and
the reconstructed one. As for Padim and PatchCore, only the SSE of the
embeddings are considered. Notice that Ganomaly being a GAN method competitor,
its anomaly score differs from the one developed by the authors (the
difference between the embedding of original and reconstructed images is
considered). This is justified by the importance of the reconstructed image
quality, which has to bring the pixel-wise difference information needed for
the business experts. This is however not possible for the Padim and PatchCore
methods that do not generate any reconstructed image. For each methods and
datasets, the accuracy (for the FID, when applicable, and SSE anomaly scores)
under the zero-false-negative constraint (ZFN columns), and, in a more
standard way, without this constraint (ACC columns), as well as the AUCROC
(AUC columns) are reported in Table 2. All the metrics are the mean ± the
standard deviation (both in percent) over 5 runs for which random train and
test splits were generated. Each dataset splits are the same over the methods.
The last line gives the mean of each metrics over all the datasets.

The anomaly score distributions of the first run for the normal and abnormal
test sets of each dataset are shown in Figure 4, including the accuracy
calculated for the threshold set with the ZFN constraint, or without it.

Figure 4: (Color online) Anomaly score distributions of normal (solid-green
line and bars) and abnormal (dashed-red line and bars) images for the test
datasets, with the threshold value in the ZFN setting (vertical dashed line in
grey) or in the ACC setting (vertical dashed line in black).

##  6 Discussion

This section discusses the use of Cycle-GAN, for AD with industrial and
medical images.

###  6.1 Qualitative Discussion

We observe from the qualitative results presented in Figure 3 that the anomaly
reconstruction strongly depends on the nature of the anomaly and the image
itself. Indeed, we can see in the red-framed block (abnormal data) that for
Hazelnut, Wood and Brain datasets that small cracks, holes or spots are
perfectly erased in the reconstructed images, which faithfully highlights the
anomalies. These ones present a much higher contrast with the normal image
than other datasets like the Screw or Tile ones, where we still observe the
anomalies (small scratches or large cracks) after reconstruction.

Unlike the Tile images where the difference is still well highlighted on the
pixelwise difference image, the Screw difference images struggle to highlight
the anomaly. For this specific dataset, it can be explained, in addition to
the low contrast observed by the anomaly, by the size of the screw in the
entire image, which is relatively small compared to the objects or textures
considered in the other images. The model correctly reconstructs the
background and the shadow, being the majority surface of the image, and the
scratches could be interpreted as a reflect instead of an anomaly, in this
very small surface. Another explanation could be that the latent space is not
small enough to avoid overfitting, preventing to learn efficiently the subtle
features in the screw object. It contains too much input information and
another architecture could have fixed this issue. On the other hand, the Tile
cracks have much less probabilities to be interpreted as an original feature,
because it is visually very different that the normal features (colorization,
continuity of the texture, etc.).

Good reconstruction is also observed for the Breast Ultrasound or Retina OCT
datasets, where the object-shaped anomalies have a large abnormal and well
contrasted structure. They are not well erased (specifically for the Breast
image) but enough attenuated, which provides sufficient information in the
difference image to detect and localize the anomaly.

We can also observe in the green-framed block (normal data) that the
reconstructions (middle images) are more or less identical to the original
input (left images), resulting in an almost zero difference image (right
images). The model has extracted the features of the normal distributions, and
is able to restore normal images without changing the pixels value, thanks to
the identity loss. Pixel areas with high discontinuity, as shown in the Screw
images, the Brain MRI or the Breast Ultrasound dataset, do not fully follow
this observation, resulting in slight differences in the generated image that
disturb the anomaly score, and make it difficult to obtain good separable
thresholds, in the quantitative step.

From the qualitative results, it is noticeable that the method can reach the
anomaly at the pixel level, showing its exact location in the difference
images, in spite of the image level labeling the dataset only offers.

###  6.2 Quantitative Discussion

For the quantitative assessment, we conclude from Table 2 that our method
gives an average accuracy of 79.89% under the zero false negative constraint
(ZFN column of CycleGAN-AD-256 method), being at least about 17% better than
other methods in this setup. Without this constraint, it reaches the second
position behind the PatchCore method, which is also the case for the AUCROC
metric. Notice that even the CycleGAN-AD-64 version is still 12% better in
this setting compared to others. From this statement, we can explain the good
results with the ZFN constraint by its ability to reconstruct pretty small or
subtle anomalies, still giving a high anomaly score for these challenging
images. In other words, unlike other methods, the lowest anomaly score for the
abnormal test set is sufficiently distant from the anomaly score population of
the normal test set, bringing the threshold so that the false positives are
well contained. It means that incorporating abnormal images into the training
set, through the cycle consistency framework of Cycle-GAN, helps to better
reconstruct the anomalies. This is the case compared to a GAN technique (like
Ganomaly) or an embedding technique (like Padim or PatchCore). This
observation is particularly interesting for the critical applications in
industrial or medical businesses, where this ZFN setup is much more important
than the standard accuracy or the AUCROC metric.

In the detail, as expected from the qualitative assessment, the Hazelnut,
Tile, Wood, Brain MRI and Breast Ultrasound datasets explain the good average
first position of the method, performing better than any other methods.
Despite the challenging anomalies that images in Tile or Breast Ultrasound
datasets can contain (anomalies not well erased but still sufficiently to be
highlighted in the difference images), their abnormal images reach a
sufficient anomaly score to be discriminated compared to the normal ones. The
Screw and Retina OCT datasets, as for them, reach a quasi 50% ZFN accuracy,
yielding a tool with a quasi zero detection capability, even if the metrics
are satisfying for the Retina OCT when we release this constraint (97.23%).
This is typically the sign of the presence of one sample (at least) where the
reconstruction struggles, which pushes the ZFN threshold towards a low anomaly
score, yielding many false positives. However one can notice that these poor
results are also seen in the other methods, where Screw gets its best score of
62.71% with Ganomaly, and Retina OCT reaches also a near-random discrimination
with almost 50% ZFN accuracy.

The comparison between our method in a resolution of 256 $\times$ 256 and the
same in 64 $\times$ 64 shows the importance of getting higher resolution
images, specifically for the industrial datasets (the higher resolution images
bring better ZFN accuracy). It seems consistent, when we consider small
anomalies consisting of a few pixels or low contrast, that the better the
definition, the better the reconstruction and all the processing to find the
anomaly score. This statement is not valid for the medical datasets, where the
method still performs well at a 64 $\times$ 64 input image resolution. In
contrast to the industrial images (Tile is the best example with a drop of
about 30%), the well contrasted anomalies shown in these datasets explain this
observation. Another explanation is the presence of larger structures observed
in the considered medical images.

For Hazelnut and Tile in the ZFN setup, we can also conclude that the FID
anomaly score improves the accuracy by getting rid of the noisy pixel-by-pixel
reconstruction described above. For the Wood, the Brain MRI and the Breast
Ultrasound datasets, it appears that for the ZFN constraint, the anomaly score
based on SSE leads to better results. This could come from the fact that the
Inception V3 model is not pre-trained on many medical-like images or highly
homogeneous textured images, leading in a poor feature extraction. Depending
on the domain, the FID or the SSE anomaly score have to be chosen adequately.
However, these scores cannot avoid poor accuracy with datasets like Screw or
Retina OCT, where small anomalies are not well captured by the model and the
generated images still show the anomalies.

##  7 Conclusion

In this work, we propose and characterize for the first time an approach using
Cycle-Generative Adversarial Networks (Cycle-GAN) for Anomaly Detection (AD)
on industrial and medical images. This method reaches 79.89% accuracy under a
zero false negative constraint, being about 17% better than other state-of-
the-art methods. It exploits the abnormal images at our disposal to refine its
representation of normal data, by giving more insights on what is normal or
abnormal. Furthermore, thanks to the use of the identity loss, we show that
the formalism of Cycle-GAN is naturally well-adapted to perform AD. Particular
attention has been given to industrial and medical applications, due to the
societal impact it may offer, and motivated by the lack of studies for such
kind of work in these areas to date. The proposed method differs from previous
work by exploiting both normal and abnormal images to learn mappings that can
generate new matched data from one domain to another, under a cycle
consistency constraint. The mapping of interest for our AD method is the one
that can generate normal images. From this perspective, any differences
between the test image and its normal (generated) version can be easily
identified. Qualitatively, the pixel squared difference image is used to
locate abnormal areas, and then quantitatively, an anomaly score is created to
indicate whether the image contains abnormal areas, based on a pre-selected
threshold. Ultimately, the method identifies anomalies at the pixel level
while the labels are initially at the image level, i.e., without the
requirement for tedious annotation at the pixel level.

The achieved results demonstrate that, independent of the application, images
presenting anomalies with a sufficient contrast compared to the pixels
composing the object or texture considered tend to benefit from higher domain
change mapping than those with a low contrast. For low contrasted anomalies,
an exception is observed for images of objects with coarse defects where the
localization of anomalies always meet expectations, even with a difficult
reconstruction. Issues still remain for object filling a small part of the
entire image and with small defects. We argue in this work that when both
normal and abnormal data are available for training, the use of Cycle-GAN
architectures should be considered as an approach by the community, mainly
when the anomalies are known to be of textures or coarse objects.

New applications may also be explored for future work, such as object
segmentation or object counting for industrial and medical fields using the
same type of cycle-consistent models. This work is a first step and a proof-
of-concept for Cycle-GAN in AD for industrial and medical domains.

## Acknowledgment

V.D. benefits from the support of the Walloon region with a Ph.D. grant from
FRIA (F.R.S.-FNRS). M.E. benefits from the support of the Belgian Walloon
region for funding SMARTSENS project which is part of Win2WAL program
(agreement 2110108). The present research benefited from computational
resources made available on Lucia, the Tier-1 supercomputer of the Walloon
Region, infrastructure funded by the Walloon Region under the grant agreement
n°1910247. The authors thank Charline Dardenne, Jérôme Fink, Géraldin Nanfack
and Pierre Poitier for their insightful comments and discussions on this
paper.

## References

  * Abadi et al. [2015] Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur, M., Levenberg, J., Mané, D., Monga, R., Moore, S., Murray, D., Olah, C., Schuster, M., Shlens, J., Steiner, B., Sutskever, I., Talwar, K., Tucker, P., Vanhoucke, V., Vasudevan, V., Viégas, F., Vinyals, O., Warden, P., Wattenberg, M., Wicke, M., Yu, Y., Zheng, X., 2015.  TensorFlow: Large-scale machine learning on heterogeneous systems.  URL: https://www.tensorflow.org/. software available from tensorflow.org.
  * Abdallah et al. [2016] Abdallah, A., Maarof, M.A., Zainal, A., 2016.  Fraud detection system: A survey.  Journal of Network and Computer Applications 68, 90–113.
  * Akcay et al. [2019] Akcay, S., Atapour-Abarghouei, A., Breckon, T.P., 2019.  GANomaly: Semi-supervised Anomaly Detection via Adversarial Training, in: Jawahar, C.V., Li, H., Mori, G., Schindler, K. (Eds.), Computer Vision – ACCV 2018, Springer International Publishing. pp. 622–637.  doi:10.1007/978-3-030-20893-6_39.
  * Al-Dhabyani et al. [2020] Al-Dhabyani, W., Gomaa, M., Khaled, H., Fahmy, A., 2020\.  Dataset of breast ultrasound images.  Data in Brief 28, 104863\.
  * Bergmann et al. [2019] Bergmann, P., Fauser, M., Sattlegger, D., Steger, C., 2019\.  Mvtec ad–a comprehensive real-world dataset for unsupervised anomaly detection, in: CVPR, IEEE. pp. 9592–9600.
  * Bougaham et al. [2021] Bougaham, A., Bibal, A., Linden, I., Frenay, B., 2021\.  Ganodip-gan anomaly detection through intermediate patches: a pcba manufacturing case, in: LIDTA, PMLR. pp. 104–117.
  * Bougaham et al. [2023] Bougaham, A., El Adoui, M., Linden, I., Frénay, B., 2023\.  Composite score for anomaly detection in imbalanced real-world industrial dataset.  Machine Learning , 1–26.
  * Bradley [1997] Bradley, A.P., 1997.  The use of the area under the roc curve in the evaluation of machine learning algorithms.  Pattern recognition 30, 1145–1159.
  * [9] Chakrabarty, N., .  Brain MRI images for brain tumor detection, version 1\.  Retrieved May 9, 2021 from https://www.kaggle.com/navoneel/brain-mri-images-for-brain-tumor-detection.
  * Defard et al. [2021] Defard, T., Setkov, A., Loesch, A., Audigier, R., 2021\.  PaDiM: A Patch Distribution Modeling Framework for Anomaly Detection and Localization, in: Del Bimbo, A., Cucchiara, R., Sclaroff, S., Farinella, G.M., Mei, T., Bertini, M., Escalante, H.J., Vezzani, R. (Eds.), Pattern Recognition. ICPR International Workshops and Challenges, Springer International Publishing. pp. 475–489.  doi:10.1007/978-3-030-68799-1_35.
  * Dirvanauskas et al. [2019] Dirvanauskas, D., Maskeliūnas, R., Raudonis, V., Damaševičius, R., Scherer, R., 2019.  Hemigen: human embryo image generator based on generative adversarial networks.  Sensors 19, 3578\.
  * Goodfellow et al. [2014] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y., 2014\.  Generative adversarial nets, in: NeurIPS.
  * Hammami et al. [2020] Hammami, M., Friboulet, D., Kéchichian, R., 2020.  Cycle gan-based data augmentation for multi-organ detection in ct images via yolo, in: ICIP, IEEE. pp. 390–393.
  * He et al. [2016] He, K., Zhang, X., Ren, S., Sun, J., 2016.  Deep residual learning for image recognition, in: CVPR, IEEE. pp. 770–778.
  * Heusel et al. [2017] Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S., 2017.  Gans trained by a two time-scale update rule converge to a local nash equilibrium, in: NeurIPS.
  * Isola et al. [2017] Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A., 2017\.  Image-to-image translation with conditional adversarial networks, in: CVPR, IEEE. pp. 5967–5976.
  * Kerepecky et al. [2022] Kerepecky, T., Liu, J., Ng, X.W., Piston, D.W., Kamilov, U.S., 2022.  Dual-cycle: Self-supervised dual-view fluorescence microscopy image reconstruction using cyclegan.  ArXiv:2209.11729 .
  * Kermany et al. [2018] Kermany, D.S., Goldbaum, M., Cai, W., Valentim, C.C., Liang, H., Baxter, S.L., McKeown, A., Yang, G., Wu, X., Yan, F., Dong, J., Prasadha, M.K., Pei, J., Ting, M.Y., Zhu, J., Li, C., Hewett, S., Dong, J., Ziyar, I., …, Zhang, K., 2018.  Identifying medical diagnoses and treatable diseases by image-based deep learning.  Cell 172, 1122–1131.e9.
  * Kiran et al. [2018] Kiran, B.R., Thomas, D.M., Parakkal, R., 2018.  An overview of deep learning based methods for unsupervised and semi-supervised anomaly detection in videos.  Journal of Imaging 4, 36\.
  * Ledig et al. [2017] Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., Shi, W., 2017.  Photo-realistic single image super-resolution using a generative adversarial network, in: CVPR, IEEE. pp. 105–114.
  * Li and Wand [2016] Li, C., Wand, M., 2016.  Precomputed real-time texture synthesis with markovian generative adversarial networks, in: ECCV, Springer. pp. 702–716.
  * Pandey et al. [2020] Pandey, S., Singh, P.R., Tian, J., 2020.  An image augmentation approach using two-stage generative adversarial network for nuclei image segmentation.  Biomedical Signal Processing and Control 57, 101782.
  * Rippel et al. [2020] Rippel, O., Müller, M., Merhof, D., 2020.  Gan-based defect synthesis for anomaly detection in fabrics, in: ETFA, IEEE. pp. 534–540.
  * Roth et al. [2022a] Roth, K., Pemula, L., Zepeda, J., Schölkopf, B., Brox, T., Gehler, P., 2022a.  Towards total recall in industrial anomaly detection, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14318–14328.
  * Roth et al. [2022b] Roth, K., Pemula, L., Zepeda, J., Schölkopf, B., Brox, T., Gehler, P., 2022b.  Towards total recall in industrial anomaly detection, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14318–14328.
  * Sandfort et al. [2019] Sandfort, V., Yan, K., Pickhardt, P.J., Summers, R.M., 2019\.  Data augmentation using generative adversarial networks (cyclegan) to improve generalizability in ct segmentation tasks.  Scientific reports 9, 1–9.
  * Schlegl et al. [2017] Schlegl, T., Seeböck, P., Waldstein, S.M., Schmidt-Erfurth, U., Langs, G., 2017.  Unsupervised anomaly detection with generative adversarial networks to guide marker discovery, in: Information Processing in Medical Imaging: 25th International Conference, IPMI 2017, Boone, NC, USA, June 25-30, 2017, Proceedings, Springer. pp. 146–157.
  * Schlegl et al. [2019] Schlegl, T., Seeböck, P., Waldstein, S.M., Langs, G., Schmidt-Erfurth, U., 2019.  f-AnoGAN: Fast unsupervised anomaly detection with generative adversarial networks.  Medical Image Analysis 54, 30–44.  doi:10.1016/j.media.2019.01.010.
  * Szegedy et al. [2016] Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z., 2016.  Rethinking the inception architecture for computer vision, in: CVPR, IEEE. pp. 2818–2826.
  * Zenati et al. [2019] Zenati, H., Foo, C.S., Lecouat, B., Manek, G., Chandrasekhar, V.R., 2019.  Efficient GAN-Based Anomaly Detection.  URL: http://arxiv.org/abs/1802.06222, doi:10.48550/arXiv.1802.06222. arXiv:1802.06222 [cs, stat].
  * Zhang et al. [2021] Zhang, G., Cui, K., Hung, T.Y., Lu, S., 2021.  Defect-GAN: High-Fidelity Defect Synthesis for Automated Defect Inspection, in: WACV, IEEE. pp. 2523–2533.
  * Zhu et al. [2017] Zhu, J.Y., Park, T., Isola, P., Efros, A.A., 2017\.  Unpaired image-to-image translation using cycle-consistent adversarial networks, in: ICCV, IEEE.