# Unsupervised Pixel-level Road Defect Detection
via Adversarial Image-to-Frequency Transform

Jongmin Yu1,2, Duyong Kim3, Younkwan Lee2, Moongu Jeon2,† 1Curtin University,
Kent St, Bently WA 6102, Australia 1,2jm.andrew.yu@gmailcom2Gwangju Institute
of Science and Technology (GIST), Gwangju 61005, South Korea {2brightyoun,
<EMAIL_ADDRESS>University, 124 La Trobe St, Melbourne VIC 3000,
Australia<EMAIL_ADDRESS>
###### Abstract

In the past few years, the performance of road defect detection has been
remarkably improved thanks to advancements on various studies on computer
vision and deep learning. Although a large-scale and well-annotated datasets
enhance the performance of detecting road defects to some extent, it is still
challengeable to derive a model which can perform reliably for various road
conditions in practice, because it is intractable to construct a dataset
considering diverse road conditions and defect patterns. To end this, we
propose an unsupervised approach to detecting road defects, using Adversarial
Image-to-Frequency Transform (AIFT). AIFT adopts the unsupervised manner and
adversarial learning in deriving the defect detection model, so AIFT does not
need annotations for road defects. We evaluate the efficiency of AIFT using
GAPs384 dataset, Cracktree200 dataset, CRACK500 dataset, and CFD dataset. The
experimental results demonstrate that the proposed approach detects various
road detects, and it outperforms existing state-of-the-art approaches.

##  I INTRODUCTION

Road defect detection is one of the important studies to prevent vehicle
accidents and manage the road condition effectively. All over the United
States, road conditions contribute to the frequency and severity of motor
vehicle accidents. Almost of third of all motor vehicle crashes are related to
poor road conditions, resulting in more than two million injuries and 22,000
fatalities [1]. Over time, as road infrastructure ages, the condition of that
infrastructure steadily declines, and the volumes and severity of defects
increase [2]. Therefore, the need for the development of a method for
detecting road defects within this area only increases [3], and numerous
studies have been being proposed in the literature.

Over the past decades, diverse studies have considered the use of image
processing and machine learning approaches with hand-crafted features [4, 5,
6, 7]. Statistical analysis [4, 6] is the oldest one and also the most
popular. Acosta et al. [4] and Deutschl et al. [7] have proposed vision-based
methods based on partial differential techniques. Chambon et al. [6] have
presented a method based on Markovian modelling to take into account the local
geometrical constraints about road cracks. Bray et al. [5] have utilized the
classification approach using neural networks for identifying road defects.
These approaches usually identify road defects using the contrast of texture
information on a road surface.

However, the contrast between roads and the defects on the roads may be
reduced due to the illumination conditions and the changes in weather [8].
Additionally, the specification of cameras for capturing the surface of the
roads also can affect the detection accuracies. Hense, it is still challenging
to develop a defect detection method which can cover various road conditions
in the real world using a simple image processing or machine learning methods
alone [9].

Recently, various approaches [10, 11] based on deep learning have been
proposed to overcome these drawbacks. Pauly et al. [10] have proposed a method
for road defect detection employing convolutional neural networks (CNNs). Fan
et al. [11] have proposed segmentation method based on CNNs and apply an
adaptive. These approaches need a well-annotated dataset for road defects, and
also their performance may depend on scale of the given dataset. Regrettably,
it is problematic in practice to construct such a dataset containing various
patterns of road defects.

Developing an unsupervised method which does not need annotations for road
defects in the training step, is an issue that has been noticed for a long
time in this literature. Various unsupervised approaches based on image
processing and machine learning were proposed [12, 13]. However, these
approaches still have an inherent weakness which is detection performances are
highly dependent on camera specifications and image qualities. Recently, among
the approaches based on deep learning, several studies [14, 15] have presented
unsupervised methods using autoencoder [16]. These approaches take normal road
images as their training samples and optimize their models in a way to
minimize reconstruction errors between their input and output. These
approaches recognize defects if the reconstruction errors of inputted samples
are larger than a predefined threshold.

However, according to Perera et al. [17] and Pidhorskyi et al. [18], even
though a model based on the reconstruction setting obtains a well-optimized
solution, there is a possibility that the model can reconstruct samples which
have not appeared in the training step. It could be a significant disadvantage
in detecting road defects using the model. Due to this disadvantage, the model
may produce lower error than the expectation even if it takes defect samples
as their input, and it can make hard to distinguish whether this sample
contains defects or not.

Figure 1: Architectural detail of the adversarial image-to-frequency
transform. The blue objects denote the operation units including the generator
$G$ and the discriminators $\mathcal{D}^{I}$ and $\mathcal{D}^{F}$. The red
circles indicate the loss functions corresponded to the each operation unit.
The red arrow lines show the work flow for the image-to-frequency cycle
$G^{+}:\mathcal{X}^{I}\rightarrow{}\bar{\mathcal{X}}^{F}$, and the blue arrow
lines represent the process of the frequency-to-image cycle
$G^{-1}:\mathcal{X}^{F}\rightarrow{}\bar{\mathcal{X}}^{I}$. The dotted arrow
lines represent the correlations of each component to the loss functions.

To tackle this issue, we present an unsupervised approach, which exploits
domain transformation based on adversarial learning, to detecting road
defects. The proposed approach called Adversarial Image-to-Frequency Transform
(AIFT) is trained by normal road images only and needs no annotations for
defects. In contrast to other approaches [14, 15] optimizing their models by
minimize reconstruction errors, AIFT is concentrated on deriving mapping
function between an image-domain and a frequency-domain using adversarial
manner. To demonstrate the efficiency of the proposed approach for road defect
detection, we compare the proposed approach with various state-of-the-art
approaches, including supervised and unsupervised methods. The experimental
results show that the proposed approach can outperform existing state-of-the-
art methods.

The main contributions of our work are summarized as follows:

  * •

An unsupervised method for detecting road defects, which can provide
outstanding performance without a well-annotated dataset for road defects.

  * •

The adversarial learning for deriving the image-to-frequency mapping function.
Our approach can derive the more optimal transform model than typical
approaches such as reconstruction or classification settings.

  * •

The extensive experiments about road defect detection. The experiments include
ablation analysis depending on the loss functions and comprehensive comparison
with the existing state-of-the-art methods.

In the further sections, we describe the details of our approach and provide
the experimental results and analysis it. We conclude this paper by
summarizing our works.

##  II THE PROPOSED METHOD

###  II-A Adversarial Image-to-Frequency Transform

It is essential to derive a robust model invariant to environments in order to
detect a great number of defect patterns on roads. Our method is inspired by
novelty detection studies [17, 18], which derive a model using inlier samples
only and recognize outliers by computing a likelihood or an reconstruction
error. The proposed method, called Adversarial Image-to-Frequency Transform
(AIFT), initially derives a transform model between image-domain and
frequency-domain using normal road pavement images only. The frequency-domain
corresponding to the image-domain is generated by applying Fourier transform
to the given image-domain. Detecting road defects is conducted by comparing
given and generated samples of each domain.

AIFT is composed of three components: Generator $G$, Image discriminator
$\mathcal{D}^{I}$, Frequency discriminator $\mathcal{D}^{F}$, for applying
adversarial learning. The original intention of adversarial learning is to
learn generative models while avoiding approximating many intractable
probabilistic computations arising in other strategies e.g., maximum
likelihood estimation. This intention is suitable to derive an optimal model
for covering the various visual patterns of road defects. The workflow of AIFT
is illustrated in Fig 1.

The generator $G$ plays as a role for the mapping function between image-
domain $\boldsymbol{\mathcal{X}}^{I}=\\{\mathcal{X}^{I}_{i}\\}_{i=1:n}$ to
frequency-domain
$\boldsymbol{\mathcal{X}}^{F}=\\{\mathcal{X}^{F}_{i}\\}_{i=1:n}$ as follows,
$G:\mathcal{X}^{I}\longleftrightarrow{}\mathcal{X}^{F}$. For the convenience
of notation, we distinguish the notations of mappings for image-to-frequency
$G^{+}:\boldsymbol{\mathcal{X}}^{I}\rightarrow{}\boldsymbol{\mathcal{X}}^{F}$
and frequency-to-image
$G^{-}:\boldsymbol{\mathcal{X}}^{F}\rightarrow{}\boldsymbol{\mathcal{X}}^{I}$,
separately. $G$ generate the transformed results from each domain as follows,

$\begin{split}&G^{+}(\mathcal{X}^{I})=\bar{\mathcal{X}}^{F},\\\
&G^{-}(\mathcal{X}^{F})=\bar{\mathcal{X}}^{I},\end{split}$ (1)

where $\bar{\mathcal{X}}^{F}$ and $\bar{\mathcal{X}}^{I}$ indicate the
transformed results from $\mathcal{X}^{I}$ and $\mathcal{X}^{F}$,
respectively. $\bar{\mathcal{X}}^{I}$ and $\bar{\mathcal{X}}^{F}$ are conveyed
to the two discriminators $\mathcal{D}^{I}$ and $\mathcal{D}^{F}$ for
computing an adversarial loss. For computational-cost-effective
implementation, weight sharing has employed.

Figure 2: Structural details of the network models in the generator $G$ and
the discriminators $\mathcal{D}^{I}$ and $\mathcal{D}^{F}$. (a) and (b) denote
the structural details of the generator $G$ and the two discriminators
$\mathcal{D}^{I}$ and $\mathcal{D}^{F}$, respectively. The green, blue, and
red boxes denote the convolutional layers, the deconvolutional layers, and the
fully-connected layers, respectively.

The discriminators $\mathcal{D}^{I}$ and $\mathcal{D}^{F}$ are defined as
follows,

$\begin{split}&\mathcal{D}^{*}(\mathcal{X}^{*})=o^{*},\quad{}o^{*}\in{}R^{1},\end{split}$
(2)

where $*$ denotes the indicator to assign the discriminators
$\mathcal{D}^{*}\in\\{\mathcal{D}^{I},\mathcal{D}^{F}\\}$ depending on the
types of inputs
$\mathcal{X}^{*}\in\\{\mathcal{X}^{I},\mathcal{X}^{F},\bar{\mathcal{X}}^{I},\bar{\mathcal{X}}^{F}\\}$.
$\mathcal{D}^{I}$ takes $\mathcal{X}^{I}$ and $\bar{\mathcal{X}}^{I}$ as an
input, and $\mathcal{D}^{I}$ takes $\mathcal{X}^{F}$ and
$\bar{\mathcal{X}}^{F}$ as an input, respectively. $o^{*}$ indicates the
outputs $o^{I}$ and $o^{F}$ according to the types of the inputs and the
discriminators. The value of $o^{*}$ can be regarded by as a likelihood to
discriminate whether a given sample is truth or generated. Each component is
compiled by CNNs and fully-connected neural networks and the structural
details of these components are shown in Fig 2.

###  II-B Adversarial transform consistency learning

As the workflow of AIFT shown in Fig 1, the generator $G$ plays a role as a
bidirectional mapping function between image-domain
$\boldsymbol{\mathcal{X}}^{I}$ and corresponding frequency-domain
$\boldsymbol{\mathcal{X}}^{F}$ generated from $\boldsymbol{\mathcal{X}}^{I}$.
The underlying assumption for detecting road defects using AIFT is as follows.
Since AIFT is only trained with normal road pavement images, if AIFT takes
images containing defect patterns as an input, the error between the given
samples and the transformed results would be larger than normal ones. Given
this assumption, the prerequisite for precise road defect detection on AIFT is
deriving a strict transform model between the image-domain and the frequency-
domain from a given dataset for normal image samples for road pavement.

To end this, we present an adversarial transform consistency loss for training
AIFT. Adversarial transform consistency loss is defined by,

$\begin{split}\mathcal{L}_{\text{ATCL}}(G,\mathcal{D}^{I},\mathcal{D}^{F})&=E_{\mathcal{X}^{I}\sim{}p_{\boldsymbol{\mathcal{X}^{I}}}}[\text{log}\mathcal{D}^{I}(\mathcal{X}^{I})]\\\
&+E_{\mathcal{X}^{F}\sim{}p_{\boldsymbol{\mathcal{X}^{F}}}}[\text{log}\mathcal{D}^{F}(\mathcal{X}^{F})]\\\
&+E_{\bar{\mathcal{X}}^{F}\sim{}p_{G^{{}^{+}}(\boldsymbol{\mathcal{X}^{I}})}}[\text{log}(1-\mathcal{D}^{F}(G^{+}(\mathcal{X}^{I})))]\\\
&+E_{\bar{\mathcal{X}}^{I}\sim{}p_{G^{{}^{-}}(\boldsymbol{\mathcal{X}}^{F})}}[\text{log}(1-\mathcal{D}^{I}(G^{-}(\mathcal{X}^{F})))],\end{split}$
(3)

where $G$ tries to generate images $\bar{\mathcal{X}}^{I}$ and frequency
samples $\bar{\mathcal{X}}^{F}$ via $G^{+}$ and $G^{-}$ that look similar to
given images $\mathcal{X}^{I}$ and frequencies $\mathcal{X}^{F}$, while
$\mathcal{D}^{I}$ and $\mathcal{D}^{F}$ aim to distinguish between given
samples ($\mathcal{X}^{I}$ and $\mathcal{X}^{F}$) and transformed results
($\bar{\mathcal{X}}^{I}$ and $\bar{\mathcal{X}}^{F}$).

Figure 3: Comparison of the given and generated samples for the road pavement
image and the corresponding frequency.

Adversarial learning can, in theory, learn mappings $G$ that produce outputs
identically distributed as image and frequency domains, respectively [19].
However, with large enough capacity, $G$ can map the same samples of an input
domain to any random permutation of samples in the different domain, where any
of the learned mappings can induce an output distribution that matches the
target distribution. Thus, adversarial transform consistency loss alone may
not guarantee that the learned function can map an individual input to the
desired output.

To further reduce the space of possible mapping functions, we utilize the
reconstruction loss to optimize the generator $\mathcal{G}$. It is a common
way to enforce the output of the generator to be close to the target through
the minimization of the reconstruction error based on the pixel-wise mean
square error (MSE) [20, 21, 22, 23]. It is calculated in the form

$\begin{split}\mathcal{L}_{\text{re}}(G)&=\mathbb{E}_{\mathcal{X}^{I}\sim{}p_{\boldsymbol{\mathcal{X}^{I}}}}[\|\mathcal{X}^{F}-G^{+}(\mathcal{X}^{I})\|^{2}_{2}]\\\
&+E_{\mathcal{X}^{F}\sim{}p_{\boldsymbol{\mathcal{X}^{F}}}}[\|\mathcal{X}^{I}-G^{-}(\mathcal{X}^{F})\|^{2}_{2}].\end{split}$
(4)

Consequently, the total loss function is:,

$\begin{split}\mathcal{L}_{\text{total}}(G,\mathcal{D}^{I},\mathcal{D}^{F})=\mathcal{L}_{\text{ATCL}}(G,\mathcal{D}^{I},\mathcal{D}^{F})+\lambda\mathcal{L}_{\text{re}}(G)\end{split}$
(5)

where $\lambda$ indicates the balancing parameter to take the weight for the
reconstruction loss.

Given the definition of above loss functions, the discriminators and the
generator are trained by maximizing or minimizing corresponding loss terms
expressed by,

$\begin{split}\text{arg}\mathop{\mathrm{min}}\limits_{\theta^{G}}\mathop{\mathrm{max}}\limits_{\theta^{I},\theta^{F}}\mathcal{L}_{\text{total}}(G,\mathcal{D}^{I},\mathcal{D}^{F}),\end{split}$
(6)

where $\theta^{G}$, $\theta^{I}$,and $\theta^{F}$ denote the parameters
corresponded to the generator $\mathcal{G}$, the image discriminators
$\mathcal{D}^{I}$, and the frequency discriminator $\mathcal{D}^{F}$. Fig 3
illustrates the examples of the given samples and the transformed results for
image and frequency domains. We have conducted the ablation studies to observe
the effect of each loss term in learning AIFT.

(a)

(b)

Figure 4: The trends of AIU over the training epochs. (a) show the AIU trend
over the training epochs on GAPs384 dataset, and (b) illustrate the AIU trend
with respect to the training epochs on CFD dataset. The red-coloured curve
(AIFT${}_{\text{total}}$) denotes the AIU trend of AIFN trained by the total
loss (Eq 5). The green-colored curve (AIFT${}_{\text{GAN}}$) indicates the AIU
trend of AIFN trained by the ATCL loss (Eq 3) only. The blue-colored curve
(AIFT${}_{\text{re}}$) shows the AIU trend of AIF trained by the
reconstruction loss (Eq 4).

###  II-C Road defect detection

Detecting defects on a road is straightforward. Initially, AIFT produces the
frequency sample $\mathcal{X}^{F}$ using given an image samples
$\mathcal{X}^{I}$. Secondly, AIFT transforms $\mathcal{X}^{F}$ into the image
samples $\bar{\mathcal{X}}^{I}$ via $G^{-}$. Road defects are defected by
comparing the given image sample $\mathcal{X}^{I}$ with the transformed result
$\bar{\mathcal{X}}^{I}$.

Similarity metric for comparing the two samples $\mathcal{X}^{I}$ and
$\bar{\mathcal{X}}^{I}$, is defined as follows,

$\begin{split}d(\mathcal{X}^{I},\bar{\mathcal{X}}^{I})&=\sum_{i,j}(\bar{x}^{I}_{i,j}\text{log}\frac{\bar{x}^{I}_{i,j}}{m_{i,j}}-x^{I}_{i,j}\text{log}\frac{x^{I}_{i,j}}{m_{i,j}}),\end{split}$
(7)

where $m_{i,j}$ is expectation of ${x}^{I}_{i,j}$ and $\bar{x}^{I}_{i,j}$.
Above similarity metric is based on Jeffery divergence, which is a modified
KL-divergence to take symmetric property. Euclidean distances such as
$l1$-norm and $l2$-normal are not suitable as a similarity metric for images
since neighboring values are not considered [24]. Jeffrey divergence is
numerically stable, symmetric, and invariant to noise and input scale [25].

TABLE I: Quantitative performance comparison of the detection performance on
AIFT using GAPs384 dataset and CFD dataset depending on the loss functions
$\mathcal{L}_{\text{re}}$ (Eq 4), $\mathcal{L}_{\text{ATCL}}$ (Eq 3), and
$\mathcal{L}_{\text{total}}$ (Eq 5). The bolded figures indicate the best
performances on the experiments.

Model | GAPs384 dataset [26] | CFD dataset [27]
---|---|---
AIU | ODS | OIS | AIU | ODS | OIS
AIFT${}_{\text{re}}$ | 0.052 | 0.181 | 0.201 | 0.152 | 0.562 | 0.572
AIFT${}_{\text{GAN}}$ | 0.081 | 0.226 | 0.234 | 0.187 | 0.642 | 0.659
AIFT${}_{\text{total}}$ | 0.083 | 0.247 | 0.249 | 0.203 | 0.701 | 0.732

##  III EXPERIMENT

###  III-A Experiment setting and dataset

To evaluation the performance of the proposed method on road defect detection,
we employ the best F-measure on the dataset for a fixed scale (ODS), the
aggregate F-measure on the dataset for the best scale in each image (OIS), and
AIU, which is proposed by Yang et al. [28]. AIU is computed on the detection
and ground truth without non-max suppression (NNS) and thinking operation,
defined by,
$\frac{1}{N_{t}}\sum_{t}\frac{N^{t}_{pg}}{N^{t}_{p}+N^{t}_{g}-N^{t}_{pg}}$,
where $N_{t}$ denotes the total number of thresholds $t\in\\{0.01,0.99\\}$
with interval 0.01; for a given $t$, $N^{t}_{pg}$ is the number of pixels of
intersected region between the predicted and ground truth crack area;
$N^{t}_{p}$ and $N^{t}_{g}$ denote the number of pixels of predicted and
ground truth crack region, respectively. The proposed method has been
evaluated on four publicly available datasets. The details of the datasets are
described as follows.

GAPs384 dataset is German Asphalt Pavement Distress (GAPs) dataset presented
by Eisenbach et al. [26], and it is constructed to address the issue of
comparability in the pavement distress domain by providing a standardized
high-quality dataset of large scale. The dataset contains 1,969 gray scaled
images for road defects, with various classes for defects fsuch as cracks,
potholes, and inlaid patches. The resolution of images is 1,920$\times$1,080.

Cracktree200 dataset [29] contains 206 road pavement images with
800$\times$600 resolution, which can be categorized to various types of
pavement defects. The images on this dataset are captured with some
challenging issues such as shadows, occlusions, low contrast, and noise.

CRACK500 dataset is constructed by Yang et al. [28]. The dataset is composed
of 500 images wity 2,000$\times$1,500, and each image has a pixel-level
annotation. The dataset is seperated by training dataset and test dataset. The
training dataset consists of 1,896 images, and the test dataset is composed of
1,124 images.

CFD dataset [27] contains 118 images with 480$\times$320 resolution. Each
image has pixel-level annotation and captured by Iphone 5 with focus of 4mm
aperture of $f/2.4$ and exposure time of 1/135s.

The hyperparameter setting for the best performance is as follows. The epoch
size and the batch size are 50 and 64, respectively. The balancing weight for
the reconstruction loss $E_{re}$ is 0.1, and the critic iteration is set by
10. The networks are optimized by Adam et al. [30]. The proposed approach has
implemented with Pytorch library 111Source codes are publicly available on
https://github.com/andreYoo/Adversarial-IFTN.git, and the experiments have
conducted with GTX Titan XP and 32GB memory.

Figure 5: Visualization of the road defect detection results. The images on the first row represent the input images. The second row’s images illustrate the ground-truths. The images on the third row denote the detection results for road defects. Methods | S/U | GAPs384 [26] | Cracktree200 [29] | CRACK500 [28] | CFD [27] | FPS(s) |
---|---|---|---|---|---|---|---
AIU | ODS | OIS | AIU | ODS | OIS | AIU | ODS | OIS | AIU | ODS | OIS |  |
HED [31] | S | 0.069 | 0.209 | 0.175 | 0.040 | 0.317 | 0.449 | 0.481 | 0.575 | 0.625 | 0.154 | 0.683 | 0.705 | 0.0825 |
RCF [32] | S | 0.043 | 0.172 | 0.120 | 0.032 | 0.255 | 0.487 | 0.403 | 0.490 | 0.586 | 0.105 | 0.542 | 0.607 | 0.079 |
FCN [33] | S | 0.015 | 0.088 | 0.091 | 0.008 | 0.334 | 0.333 | 0.379 | 0.513 | 0.577 | 0.021 | 0.585 | 0.609 | 0.114 |
CrackForest [27] | U | - | 0.126 | 0.126 | - | 0.080 | 0.080 | - | 0.199 | 0.199 | - | 0.104 | 0.104 | 3.971 |
FPHBN [28] | S | 0.081 | 0.220 | 0.231 | 0.041 | 0.517 | 0.579 | 0.489 | 0.604 | 0.635 | 0.173 | 0.683 | 0.705 | 0.237 |
AAE [34] | U | 0.062 | 0.196 | 0.202 | 0.039 | 0.472 | 0.491 | 0.371 | 0.481 | 0.583 | 0.142 | 0.594 | 0.613 | 0.721 |
SVM [35] | S | 0.051 | 0.132 | 0.162 | 0.017 | 0.382 | 0.391 | 0.362 | 0.418 | 0.426 | 0.082 | 0.3R52 | 0.372 | 0.852 |
ConvNet [35] | S | 0.079 | 0.203 | 0.211 | 0.037 | 0.472 | 0.499 | 0.431 | 0.591 | 0.609 | 0.152 | 0.579 | 0.677 | 0.921 |
AIFT${}_{\text{total}}$ |  | 0.083 | 0.247 | 0.249 | 0.045 | 0.607 | 0.642 | 0.478 | 0.549 | 0.561 | 0.203 | 0.701 | 0.732 | 1.1330 |
TABLE II: Quantitative performance comparison about road defect detection
using GAPs384 [26], Cracktree200 [29], CRACK500 [28], and CFD [27]. ”-” means
the results are not provided. The bolded figures indicate that the best
performance among them. ’S/U’ denotes whether a model focuses on ’supervised’
or ’unsupervised’ approaches. FPS indicates the execution speed of each
method, and it is computed by averaging the execution speeds about all
datasets.

###  III-B Ablation study

We have conducted an ablation study to observe the effect of the loss function
terms on the performance of AIFT. We have trained AIFT using the three loss
functions $\mathcal{L}_{\text{re}}$ (Eq 4), $\mathcal{L}_{\text{ATCL}}$ (Eq
3), and $\mathcal{L}_{\text{total}}$ (Eq 5) using GAPs384 dataset and CFD
dataset, and observed AIU at every two epochs. The hyperparameter settings
applied to train each model, are all same, and only the loss functions are
different. Fig 4 shows the AIU trends of AIFTs trained by the three loss
functions. Table I contains AIUs, ODSs, and OISs on GAPs384 dataset and CFD
dataset. The experimental results show that AIFT trained by the total loss
(AIFT${}_{\text{total}}$) achieves the best performance on this experiments.
As shown in Table I, AIFT${}_{\text{total}}$ achieves 0.083 of AIU, 0.247 of
OIS, and 0.249 of ODS for GAPs384 dataset. These figures show that
AIFT${}_{\text{total}}$ can produce approximately 7% better performance than
others. In the experiments using CFD dataset, AIFT${}_{\text{total}}$ achieves
0.203 of AIU, 0.701 of OIS, and 0.732 of ODS, and these figure are all higher
than that of the others.

Notably, the overall experimental results demonstrate that the AIFTs trained
by adversarial learning, can outperform the AIFT based on the reconstruction
setting (AIFT${}_{\text{re}}$). Not only AIFT${}_{\text{total}}$, but also
AIFT${}_{\text{ATCL}}$ obtains the improved achievement than
AIFT${}_{\text{re}}$. The AIU Trends (Fig 4) also justify that the AIFT learnt
by adversarial manners can outperform the AIFT trained by the reconstruction
setting. The experimental results justify adversarial learning can improve the
robustness of AIFT for detecting road defects.

###  III-C Comparison with existing state-of-the-arts

We have carried out the comparison with existing state-of-the-art methods for
the crack detection [31, 27, 28] and the road defect detection [35]. For the
efficiency of the experiments, only AIFT${}_{\text{total}}$ is compared with
other methods. Table II contains AIUs, OISs, and ODSs on Cracktree200,
GAPs384, Cracktree200, and CFD datasets. AIFT${}_{\text{total}}$ has achieved
state-of-the-art performance for GAPs384 dataset, Cracktree200 dataset, and
CFD dataset. In the experiments using GAPs384 dataset, AIFT${}_{\text{total}}$
achieves 0.083 of AIU, 0.247 of ODS, and 0.249 of OIS. These figures show that
AIFT${}_{\text{total}}$ outperforms than the previous state-of-the-art
performance that achieved by FPHBN [28]. FPHBN obtains 0.081 of AIU, 0.220 of
ODS, and 0.231 of OIS. AIFT${}_{\text{total}}$ shows 3% better performances
than FPHBN. The experiments on Cracktree200 dataset and CFD dataset also show
that AIFT${}_{\text{total}}$ surpasses other methods. AIFT${}_{\text{total}}$
produces 0.045 of AIU, 0.607 of ODS, and 0.642 of OIS in the experiments using
Cracktree200 dataset. Additionally, AIFT${}_{\text{total}}$ achieves 0.203 of
AIU, 0.701 of ODS, and 0.732 of OIS on CFD dataset. These figures are 8.8% and
3% better than the previous state-of-the-art methods.

However, AIFT${}_{\text{total}}$ could not obtain the highest performance on
CRACK500 dataset. The state-of-the-art performance on CRACK500 dataset is
achieved by FPHBN [28], and it produces 0.489 of AIU, 0.604 of ODS, and 0.635
of OIS, respectively. AIFT${}_{\text{total}}$ has 0.478 of AIU, 0.549 of ODS,
and 0.561 of OIS. The gaps between FPHBN and AIFT${}_{\text{total}}$ are 0.011
on AIU, 0.055 on ODS, and 0.074 on OIS. However, FPHBN exploits a supervised
approach, and it needs predetermined pixel-level annotations for road defects.
Also, the network architecture applied to their approach is much deeper than
Ours. These are the great advantages of detecting road defects.

The overall experiments show that AIFT${}_{\text{total}}$ can outperform
existing state-of-the-art methods. As shown in Table II, the detection
performance of AIFT${}_{\text{total}}$ surpasses other unsupervised methods
[27, 34]. Additionally, AIFT${}_{\text{total}}$ achieves outstanding detection
performance in detecting defects than others based on supervised learning
approaches, even AIFT${}_{\text{total}}$ does not need an annotation for road
defects in the training step. This may be thought that AIFT${}_{\text{total}}$
is enabled to apply various practical situations in which a large-scale and
well-annotated dataset can not be used. Consequently, the experimental results
demonstrate that AIFT${}_{\text{total}}$ can outperform existing state-of-the-
art methods.

##  IV CONCLUSIONS

In this paper, we have proposed an unsupervised approach to detecting road
defects, based on adversarial image-to-frequency transform. The experimental
results demonstrate the proposed approach can detect various patterns of road
defects without explicit annotations for road defects in the training step,
and it outperforms existing state-of-the-art methods in most of the cases for
experiments of road defect detection.

## ACKNOWLEDGMENT

This work was partly supported by the ICT R&D program of MSIP/IITP.
(2014-0-00077, Development of global multi target tracking and event
prediction techniques based on real-time large-scale video analysis).

## References

  * [1] E. Zaloshnja and T. R. Miller, “Cost of crashes related to road conditions, united states, 2006,” in Annals of Advances in Automotive Medicine/Annual Scientific Conference, vol. 53, p. 141, Association for the Advancement of Automotive Medicine, 2009.
  * [2] T. A. Carr, M. D. Jenkins, M. I. Iglesias, T. Buggy, and G. Morison, “Road crack detection using a single stage detector based deep neural network,” in 2018 IEEE Workshop on Environmental, Energy, and Structural Monitoring Systems (EESMS), pp. 1–5, IEEE, 2018.
  * [3] Z. Hadavandsiri, D. D. Lichti, A. Jahraus, and D. Jarron, “Concrete preliminary damage inspection by classification of terrestrial laser scanner point clouds through systematic threshold definition,” ISPRS International Journal of Geo-Information, vol. 8, no. 12, p. 585, 2019.
  * [4] J. A. Acosta, J. L. Figueroa, and R. L. Mullen, “Low-cost video image processing system for evaluating pavement surface distress,” Transportation research record, no. 1348, 1992.
  * [5] J. Bray, B. Verma, X. Li, and W. He, “A neural network based technique for automatic classification of road cracks,” in The 2006 IEEE International Joint Conference on Neural Network Proceedings, pp. 907–912, IEEE, 2006.
  * [6] S. Chambon, C. Gourraud, J. M. Moliard, and P. Nicolle, “Road crack extraction with adapted filtering and markov model-based segmentation: introduction and validation,” 2010.
  * [7] E. Deutschl, C. Gasser, A. Niel, and J. Werschonig, “Defect detection on rail surfaces by a vision based system,” in IEEE Intelligent Vehicles Symposium, 2004, pp. 507–511, IEEE, 2004.
  * [8] Y. Sun, E. Salari, and E. Chou, “Automated pavement distress detection using advanced image processing techniques,” in 2009 IEEE International Conference on Electro/Information Technology, pp. 373–377, IEEE, 2009.
  * [9] M. Baygin and M. Karakose, “A new image stitching approach for resolution enhancement in camera arrays,” in 2015 9th International Conference on Electrical and Electronics Engineering (ELECO), pp. 1186–1190, IEEE, 2015.
  * [10] L. Pauly, D. Hogg, R. Fuentes, and H. Peel, “Deeper networks for pavement crack detection,” in Proceedings of the 34th ISARC, pp. 479–485, IAARC, 2017.
  * [11] R. Fan, M. J. Bocus, Y. Zhu, J. Jiao, L. Wang, F. Ma, S. Cheng, and M. Liu, “Road crack detection using deep convolutional neural network and adaptive thresholding,” in 2019 IEEE Intelligent Vehicles Symposium (IV), pp. 474–479, IEEE, 2019.
  * [12] I. Abdel-Qader, S. Pashaie-Rad, O. Abudayyeh, and S. Yehia, “Pca-based algorithm for unsupervised bridge crack detection,” Advances in Engineering Software, vol. 37, no. 12, pp. 771–778, 2006.
  * [13] H. Oliveira and P. L. Correia, “Automatic road crack detection and characterization,” IEEE Transactions on Intelligent Transportation Systems, vol. 14, no. 1, pp. 155–168, 2012.
  * [14] A. Mujeeb, W. Dai, M. Erdt, and A. Sourin, “One class based feature learning approach for defect detection using deep autoencoders,” Advanced Engineering Informatics, vol. 42, p. 100933, 2019.
  * [15] G. Kang, S. Gao, L. Yu, and D. Zhang, “Deep architecture for high-speed railway insulator surface defect detection: Denoising autoencoder with multitask learning,” IEEE Transactions on Instrumentation and Measurement, 2018.
  * [16] P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, and P.-A. Manzagol, “Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion,” Journal of machine learning research, vol. 11, no. Dec, pp. 3371–3408, 2010.
  * [17] P. Perera, R. Nallapati, and B. Xiang, “Ocgan: One-class novelty detection using gans with constrained latent representations,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2898–2906, 2019.
  * [18] S. Pidhorskyi, R. Almohsen, and G. Doretto, “Generative probabilistic novelty detection with adversarial autoencoders,” in Advances in Neural Information Processing Systems, pp. 6822–6833, 2018.
  * [19] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in Proceedings of the IEEE international conference on computer vision, pp. 2223–2232, 2017.
  * [20] X. Ying, H. Guo, K. Ma, J. Wu, Z. Weng, and Y. Zheng, “X2ct-gan: Reconstructing ct from biplanar x-rays with generative adversarial networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 10619–10628, 2019.
  * [21] Y. Bai, Y. Zhang, M. Ding, and B. Ghanem, “Finding tiny faces in the wild with generative adversarial network,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 21–30, 2018.
  * [22] W. Liu, W. Luo, D. Lian, and S. Gao, “Future frame prediction for anomaly detection–a new baseline,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6536–6545, 2018.
  * [23] M. Sabokrou, M. Fayyaz, M. Fathy, Z. Moayed, and R. Klette, “Deep-anomaly: Fully convolutional neural network for fast anomaly detection in crowded scenes,” Computer Vision and Image Understanding, vol. 172, pp. 88–97, 2018.
  * [24] Y. Rubner, C. Tomasi, and L. J. Guibas, “The earth mover’s distance as a metric for image retrieval,” International journal of computer vision, vol. 40, no. 2, pp. 99–121, 2000.
  * [25] J. Puzicha, T. Hofmann, and J. M. Buhmann, “Non-parametric similarity measures for unsupervised texture segmentation and image retrieval,” in Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 267–272, IEEE, 1997.
  * [26] M. Eisenbach, R. Stricker, D. Seichter, K. Amende, K. Debes, M. Sesselmann, D. Ebersbach, U. Stoeckert, and H.-M. Gross, “How to get pavement distress detection ready for deep learning? a systematic approach,” in 2017 international joint conference on neural networks (IJCNN), pp. 2039–2047, IEEE, 2017.
  * [27] Y. Shi, L. Cui, Z. Qi, F. Meng, and Z. Chen, “Automatic road crack detection using random structured forests,” IEEE Transactions on Intelligent Transportation Systems, vol. 17, no. 12, pp. 3434–3445, 2016.
  * [28] F. Yang, L. Zhang, S. Yu, D. Prokhorov, X. Mei, and H. Ling, “Feature pyramid and hierarchical boosting network for pavement crack detection,” IEEE Transactions on Intelligent Transportation Systems, 2019.
  * [29] Q. Zou, Y. Cao, Q. Li, Q. Mao, and S. Wang, “Cracktree: Automatic crack detection from pavement images,” Pattern Recognition Letters, vol. 33, no. 3, pp. 227–238, 2012.
  * [30] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
  * [31] S. Xie and Z. Tu, “Holistically-nested edge detection,” in Proceedings of the IEEE international conference on computer vision, pp. 1395–1403, 2015\.
  * [32] Y. Liu, M.-M. Cheng, X. Hu, K. Wang, and X. Bai, “Richer convolutional features for edge detection,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3000–3009, 2017.
  * [33] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431–3440, 2015.
  * [34] A. Makhzani, J. Shlens, N. Jaitly, I. Goodfellow, and B. Frey, “Adversarial autoencoders,” arXiv preprint arXiv:1511.05644, 2015.
  * [35] L. Zhang, F. Yang, Y. D. Zhang, and Y. J. Zhu, “Road crack detection using deep convolutional neural network,” in 2016 IEEE international conference on image processing (ICIP), pp. 3708–3712, IEEE, 2016.


# Efficiently Moving Instead of Reweighting Collider Events with Machine
Learning

Radha Mastandrea
Department of Physics, University of California, Berkeley
Berkeley, CA 94720
Physics Division, Lawrence Berkeley National Laboratory, Berkeley
Berkeley, CA 94720
<EMAIL_ADDRESS>&Benjamin Nachman
Department of Physics, University of California, Berkeley
Berkeley, CA 94720
Physics Division, Lawrence Berkeley National Laboratory, Berkeley
Berkeley, CA 94720
Berkeley Institute for Data Science, University of California, Berkeley
Berkeley, CA 94720
<EMAIL_ADDRESS>
###### Abstract

There are many cases in collider physics and elsewhere where a calibration
dataset is used to predict the known physics and / or noise of a target region
of phase space. This calibration dataset usually cannot be used out-of-the-box
but must be tweaked, often with conditional importance weights, to be
maximally realistic. Using resonant anomaly detection as an example, we
compare a number of alternative approaches based on transporting events with
normalizing flows instead of reweighting them. We find that the accuracy of
the morphed calibration dataset depends on the degree to which the transport
task is set up to carry out optimal transport, which motivates future research
into this area.

##  1 Introduction

Many tasks in collider physics depend on the calibration of auxiliary
datasets. Data from a well-understood region of phase space are chosen as a
reference to model the known physics in a target region of phase space. When
the reference and known physics in the target are identically distributed,
differences between the reference and target would indicate the presence of
new phenomena. However, the reference data may be a distorted version of the
known physics in the target and so corrections need to be applied. The
reference is chosen to be as similar as possible to the known physics in the
target so that the corrections applied should be small.

Traditionally, this calibration task has been performed using importance
weights estimated from ratios of histograms, either using data-driven
approaches like the control region method or fully data-based approaches like
the ABCD or Matrix Method techniques (see e.g. Ref. [1] for a review). These
methods can be generalized to the unbinned (and high-dimensional) case with
machine learning-based likelihood ratio estimation [2, 3, 4]. An alternative
approach that may be more precise, especially when the reference and target
probability densities have regions of non-overlapping support, is to morph the
features themselves. Given a reference distribution $X_{R}\sim p_{X_{R}}$ and
a target distribution $X_{T}\sim p_{X_{T}}$, a mapping function
$f:X_{R}\rightarrow X_{T}$ is chosen so that the probability density of the
transformed reference $f(X_{R})$ is as close a match as possible to the
probability density of the target.

Optimal transport (OT) has been studied in collider physics to solve the
calibration problem [5]. In this paper, we focus on the case of conditional
morphing, where $X_{R}$ and $X_{T}$ are conditioned on an observable $M$
(often a mass, thus the symbol), and we set $f(\cdot|M):X_{R}|M\rightarrow
X_{T}|M$. The reason for this setup is that the morphing function is typically
learned in a background-dominated region and then interpolated or extrapolated
in $M$ to a signal-sensitive region. This setup was first explored in collider
physics in Ref. [6] and uses normalizing flows (NFs) [7]. These NFs are not
specifically tasked with making the morphing minimal. This our goal in this
paper is to explore how minimal these moves are and examine if tweaks to the
setup can bring these moves closer to the OT solution and thus improve model
fidelity for downstream inference tasks. As in Ref. [6], we use anomaly
detection as our numerical example.

This paper is organized as follows. In Sec. 2, we introduce the application of
normalizing flows to the problem of resonant anomaly detection and describe
our dataset and training procedure. Numerical results are presented in Sec. 3,
and we discuss future directions in Ref. Sec. 4.

##  2 Methods

Resonant or group anomaly detection (see e.g. Ref. [8]) in collider physics
begins with a resonant feature $M$. A potential signal will have
$|M-M_{0}|\lesssim c$ (which defines the signal region) for some unknown
$M_{0}$ and often knowable $c$. The value of $M_{0}$ is usually found through
a scan. Additional features $X\in\mathbb{R}^{N}$ are chosen which can be used
to distinguish signal from background. Weakly supervised methods learn a
classifier acting on $X$ that can distinguish signal region events in the
target from events in the reference [9, 10, 11, 2, 12, 13, 6]. Approaches vary
on how they construct the reference, but most use sideband information
($|M-M_{0}|\gtrsim c$) to some degree. The first use of NFs in this context
was in Ref. [11], which used NFs as a generative model. In contrast, the idea
of Ref. [6] is to use NFs as the morphing functions described in the previous
section.

Normalizing flows are neural networks that can approximate the density of
complex data through a composition of relatively simpler functions with a
tractable Jacobian: $L=\log p(z)+\sum\log J_{i}$, where $L$ is the loss,
$p(z)$ is the base distribution, and $J_{i}$ is the Jacobian when transforming
from the $i^{\text{th}}$ function to the $(i+1)^{\text{th}}$ function in the
composition. We consider multiple ways of training the flow:

  1. 1.

Double Base. We learn two conditional NFs, one that maps from a standard
normal $z\sim\mathcal{N}(0,1)^{N}$ to the target
$f_{T}(\cdot|M):z|M\rightarrow X_{T}|M$ and one from a standard normal to the
reference $f_{R}(\cdot|M):z|M\rightarrow X_{R}|M$. The morphing function is
then $f_{T}\circ f_{R}^{-1}$.

  2. 2.

Base to Data. We learn a conditional NF for the reference
$f_{R}(\cdot|M):z|M\rightarrow X_{R}|M$ and then use this as the base density
to learn a second flow from the reference to the target
$f(\cdot|M):X_{R}|M\rightarrow X_{T}|M$.

  3. 3.

Identity Initialization. Same as (2), but we initialize the mapping at the
identity function. This is done by first learning the mapping
$f(\cdot|M):X_{R}|M\rightarrow X_{R}|M$, then using transfer learning to adapt
this flow to map $X_{R}$ to $X_{T}$.

  4. 4.

Movement penalty. Same as (2), but we add a term to the loss function that
explicitly penalizes movement in the parameter space, $L_{2}=\alpha\lVert
X_{R}-f(X_{R}|M)\rVert^{2}$ for a hyperparameter $\alpha$.

To evaluate the performance of the models, we study how far the events were
moved, $|X_{R}-f(X_{R}|M)|$ as well as the fidelity of the move, quantified by
the (ideally poor) performance of a post-hoc classifier trained to distinguish
the interpolated reference from the target in the signal region.

###  2.1 Dataset

We use the LHC 2020 Olympics R&D dataset [14, 15] which consists of 1M
background (Standard Model) events and 100k signal/anomaly events. Our setup
is nearly the same as in many previous resonant anomaly detection studies with
the same dataset [9, 10, 11, 2, 12, 13, 6]. The events naturally live in a
high-dimensional space (each containing hundreds of particles with a
momentum), but we consider a five-dimensional compressed version that has been
extensively studied in collider analyses. These five features, along with the
resonant feature, are displayed in Fig. 1. The reference dataset consists of
1M events from one simulator while the target consists of 1M background events
from a different simulator. As we are investigating the background estimation,
no anomalous events are part of the training or testing in this paper.

Figure 1: Reference and target distributions used in this study The feature
space is comprised of the resonant feature $M$ and five other features
$m_{J_{1}}$, $\Delta m_{JJ}$, $\tau^{21}_{J_{1}}$, $\tau^{21}_{J_{2}}$, and
$\Delta R_{JJ}$. A description of these observables can be found in [16]. The
signal region is defined by $|M-M_{0}|<c$ for $M_{0}=3500$ and $c=100$.

###  2.2 Training procedure

To generate flows that map from $z\sim\mathcal{N}(0,1)^{N}$ to a reference /
target distribution, we use the highly expressive MADE autoregressive (AR)
module [17]. We use 8 modules consisting of 8 stacked piecewise rational
quadratic transformations, interleaved with a reverse permutation of all
dimensions. Each masked linear layer of the MADE modules has dimensions (64,
64). The resonant feature $M$ is embedded in a linear network of size (1, 64).
We train for 60 epochs with a batch size of 128, a a cosine annealing learning
rate initialized at $10^{-4}$, and a weight decay of $10^{-4}$.

To generate flows that map between the reference and target distributions
(which are relatively similar to each other), we use a more lightweight
architecture consisting of a coupling model with piecewise rational quadratic
transformations. These flows contains 2 stacked layers interleaved with a
reverse permutation, with each masked linear layer having dimensions (16, 16).
We train for 40 epochs with a batch size of 256, a cosine annealing learning
rate initialized at $4\times 10^{-4}$, and a weight decay of $10^{-4}$.

The training dataset is comprised from the sidebands of the resonant feature
$M$, which ensures that the signal region is kept blinded until testing. All
flows are constructed using the nflows package [18], trained using PyTorch
[19], and optimized using Adam [20]. All hyperparameters are optimized through
grid search. For each training method, we repeat the flow training 5 times
with a different random seed.

##  3 Results

We evaluate the learned mappings from reference $X_{R}$ to target $X_{T}$ for
the four training procedures proposed in Sec. 2 in two ways: (1) we observe
how far in parameter space points from the reference travel as they are mapped
to the target distribution, and (2) we evaluate the fidelity of the mapping
through training a classifier to discriminate $X_{T}$ from $f(X_{R}|M)$.

The mappings are evaluated in the signal region ($|M-M_{0}|<100$), in training
sidebands regions ($100<|M-M_{0}|<300$), and additionally in outer sidebands
regions ($300<|M-M_{0}|<500$). This allows us to gauge the performance of the
mapping in both interpolation and extrapolation tasks.

###  3.1 Distanced moved in parameter space

In Fig. 2, we show the distances traveled in feature space for each data point
$X_{R}$ that is mapped to the target distribution by $f(X_{R}|M)$. In Sec. A,
we show the distances traveled for features $X_{2}$ and $X_{3}$, which
according to Fig. 1 are the ones that are most different between the reference
and target distributions.

Figure 2: Distance traveled in the full parameter space under a mapping from
reference $X_{R}$ to target $X_{T}$. The data is preprocessed so feature lies
in the interval (-3, 3), so the maximum possible distance traveled is $\sim
13$. We provide the mean distances traveled $\pm 1\sigma$ errors.

The Double Base method produces a vastly larger range of distances traveled
than the other three methods, which is to be expected as the training
procedure contains no explicit link between the reference and target
distributions. The Movement Penalty method has the smallest mean distance
traveled in the entire and individual feature space(s), although by a narrow
margin.

###  3.2 Fidelity of transform

In Table 1, we provide the ROC scores for a binary classification neural net
trained to discriminate $f(X_{R}|M)$ from $X_{T}$. The classifiers are dense
networks of three hidden layers and 32 nodes, all with ReLu activation. We
train the binary classifiers for 20 epochs with a batch size of 128 and a
cosine annealing learning rate initialized at $10^{-3}$,

For the purpose of this report, we define a “random" classifier to have a ROC
score $<0.51$. A successful mapping between the reference and target
distributions would result in the trained binary classifier performing no
better than random.

Band | Double Base | Base to Data | Identity Init. |  $L_{2}$ ($\alpha$ = $10^{-2}$)
---|---|---|---|---
OB1 | 0.630 $\pm$ 0.024 | 0.511 $\pm$ 0.003 | 0.508 $\pm$ 0.004 | 0.507 $\pm$ 0.002
SB1 | 0.501 $\pm$ 0.000 | 0.502 $\pm$ 0.001 | 0.501 $\pm$ 0.000 | 0.502 $\pm$ 0.001
SR | 0.553 $\pm$ 0.011 | 0.503 $\pm$ 0.001 | 0.503 $\pm$ 0.001 | 0.503 $\pm$ 0.000
SB2 | 0.501 $\pm$ 0.000 | 0.503 $\pm$ 0.001 | 0.503 $\pm$ 0.001 | 0.502 $\pm$ 0.001
OB2 | 0.594 $\pm$ 0.030 | 0.506 $\pm$ 0.002 | 0.507 $\pm$ 0.004 | 0.507 $\pm$ 0.003

Table 1: ROC scores for a binary classifier trained to discriminate mapped
reference $f(X_{R}|M)$ from target $X_{T}$ . The scores are calculated
separately for different regions of the resonant parameter. (OB1, OB2) = (low,
high) mass outer bands; (SB1, SB2) = (low, high) mass sidebands; SR = signal
region. We provide the mean ROC scores $\pm 1\sigma$ errors.

All four training procedures show good fidelity in the sidebands regions
(recall that this comprises the training dataset region). All methods except
the Double Base method also show good fidelity when evaluated in the signal
region. All methods exhibit a drop in performance when evaluated in the outer
bands regions, but only the Double Base method drops enough to lose fidelity.

##  4 Future areas of study

In this work, we have explored modifications to the training procedure for a
normalizing flow tasked with learning a mapping between a reference and a
target dataset. Our explored application uses reference and target datasets
that are similar, as might be expected in the collider physics calibration
problem of modeling known physics in a specific region of phase space.

The Double Base method is clearly suboptimal in terms of optimal transport and
mapping fidelity, likely due to the fact that it transports between the
reference and the target through an uncorrelated standard normal distribution.
The other three methods (Base to Data, Identity Initialization, and Movement
Penalty) show comparable performances in both metrics, and in fact all meet
the requirement of being a faithful mapping due to the indiscriminability of
$X_{T}$ from $f(X_{R}|M)$. In this situation where the reference and target
sets are similar, using the Base to Data method is acceptable. However, this
method may not be adequate for more complex (transformations between)
datasets.

As an avenue for future investigations, we may consider exchanging a
normalizing flow for a continuous normalizing flow (CNF) [21]. Such a flow
restricts transformations to be continuous, such that each point can be
assigned a trajectory with a velocity vector. Building upon this, the OT-Flow
method [22] adds to the CNF loss both an $L_{2}$ movement penalty and a
penalty that encourages the mapping to transport points along the minimum of
some potential function. Such alternatives might be explored for situations
when the reference and target distributions are significantly different.

##  5 Potential Broader Impacts

In this work, we have explored methods to augment the training of normalizing
flows that learn mappings between probability distributions derived from LHC-
like particle collision datasets. However, these modifications could be used
to improve the performance for any normalizing flow-like architecture,
regardless of the physical origin of the reference and target datasets. Our
work could then be beneficial to any physical problem that relies on the
creation or transformations of probability distributions. Since our work
serves as a method to improve an existing architecture, rather than to define
a new analysis procedure, it is our belief that this work does not present any
foreseeable societal consequence.

## Code availability

The code for all experiments in this report can be found at
https://github.com/rmastand/FETA.

## Acknowledgments and Disclosure of Funding

We thank Samuel Klein for his useful discussions relating to these
experiments. B.N. and R.M. were supported by the Department of Energy, Office
of Science under contract number DE-AC02-05CH11231. This material is based
upon work supported by the National Science Foundation Graduate Research
Fellowship Program under Grant No. DGE 2146752. Any opinions, findings, and
conclusions or recommendations expressed in this material are those of the
authors and do not necessarily reflect the views of the National Science
Foundation.

## References

  * [1] Georgia Karagiorgi et al.  “Machine Learning in the Search for New Fundamental Physics”  In _Nature Reviews Physics_ 4, 2021, pp. 399  DOI: 10.1038/s42254-022-00455-1
  * [2] Anders Andreassen, Benjamin Nachman and David Shih  “Simulation Assisted Likelihood-free Anomaly Detection”  In _Phys. Rev. D_ 101.9, 2020, pp. 095004  DOI: 10.1103/PhysRevD.101.095004
  * [3] Georges Aad  “Search for resonant pair production of Higgs bosons in the $b\bar{b}b\bar{b}$ final state using $pp$ collisions at $\sqrt{s}$ = 13 TeV with the ATLAS detector”  In _Phys. Rev. D_ 105.9, 2022, pp. 092002  DOI: 10.1103/PhysRevD.105.092002
  * [4] “Anomaly detection search for new resonances decaying into a Higgs boson and a generic new particle $X$ in hadronic final states using $\sqrt{s}$ = 13 TeV $pp$ collisions with the ATLAS detector”  In _ATLAS-CONF-2022-045_ , 2022  URL: https://cds.cern.ch/record/2816323/
  * [5] Chris Pollard and Philipp Windischhofer  “Transport away your problems: Calibrating stochastic simulations with optimal transport”, 2021  arXiv:2107.08648 [physics.data-an]
  * [6] John Andrew Raine, Samuel Klein, Debajyoti Sengupta and Tobias Golling  “CURTAINs for your Sliding Window: Constructing Unobserved Regions by Transforming Adjacent Intervals”, 2022  arXiv:2203.09470 [hep-ph]
  * [7] Danilo Jimenez Rezende and Shakir Mohamed  “Variational Inference with Normalizing Flows”  arXiv, 2015  DOI: 10.48550/ARXIV.1505.05770
  * [8] Gregor Kasieczka, Benjamin Nachman and David Shih  “New Methods and Datasets for Group Anomaly Detection From Fundamental Physics”  In _Conference on Knowledge Discovery and Data Mining_ , 2021  arXiv:2107.02821 [stat.ML]
  * [9] Jack H. Collins, Kiel Howe and Benjamin Nachman  “Anomaly Detection for Resonant New Physics with Machine Learning”  In _Phys. Rev. Lett._ 121.24, 2018, pp. 241803  DOI: 10.1103/PhysRevLett.121.241803
  * [10] Jack H. Collins, Kiel Howe and Benjamin Nachman  “Extending the search for new resonances with machine learning”  In _Phys. Rev._ D99.1, 2019, pp. 014038  DOI: 10.1103/PhysRevD.99.014038
  * [11] Benjamin Nachman and David Shih  “Anomaly Detection with Density Estimation”  In _Phys. Rev. D_ 101, 2020, pp. 075042  DOI: 10.1103/PhysRevD.101.075042
  * [12] Kees Benkendorfer, Luc Le Pottier and Benjamin Nachman  “Simulation-Assisted Decorrelation for Resonant Anomaly Detection”, 2020  arXiv:2009.02205 [hep-ph]
  * [13] Anna Hallin et al.  “Classifying Anomalies THrough Outer Density Estimation (CATHODE)”, 2021  arXiv:2109.00546 [hep-ph]
  * [14] Gregor Kasieczka, Benjamin Nachman and David Shih  “Official Datasets for LHC Olympics 2020 Anomaly Detection Challenge (Version v6) [Data set].” https://doi.org/10.5281/zenodo.4536624 Zenodo, 2019
  * [15] Gregor Kasieczka  “The LHC Olympics 2020 a community challenge for anomaly detection in high energy physics”  In _Rept. Prog. Phys._ 84.12, 2021, pp. 124201  DOI: 10.1088/1361-6633/ac36b9
  * [16] Jesse Thaler and Ken Van Tilburg  “Identifying Boosted Objects with N-subjettiness”  In _JHEP_ 03, 2011, pp. 015  DOI: 10.1007/JHEP03(2011)015
  * [17] Mathieu Germain, Karol Gregor, Iain Murray and Hugo Larochelle  “MADE: Masked Autoencoder for Distribution Estimation”  arXiv, 2015  DOI: 10.48550/ARXIV.1502.03509
  * [18] Conor Durkan, Artur Bekasov, Iain Murray and George Papamakarios  “nflows: normalizing flows in PyTorch”  Zenodo, 2020  DOI: 10.5281/zenodo.4296287
  * [19] Adam Paszke et al.  “PyTorch: An Imperative Style, High-Performance Deep Learning Library”  In _Advances in Neural Information Processing Systems 32_ Curran Associates, Inc., 2019, pp. 8024–8035  URL: http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf
  * [20] Diederik P. Kingma and Jimmy Ba  “Adam: A Method for Stochastic Optimization”  arXiv, 2014  DOI: 10.48550/ARXIV.1412.6980
  * [21] Ricky T.. Chen, Yulia Rubanova, Jesse Bettencourt and David Duvenaud  “Neural Ordinary Differential Equations”  arXiv, 2018  DOI: 10.48550/ARXIV.1806.07366
  * [22] Derek Onken, Samy Wu Fung, Xingjian Li and Lars Ruthotto  “OT-Flow: Fast and Accurate Continuous Normalizing Flows via Optimal Transport”  In _CoRR_ abs/2006.00104, 2020  arXiv: https://arxiv.org/abs/2006.00104

##  Appendix A Appendix

Here, we provide supplementary plots for the distances traveled in $X_{2}$ and
$X_{3}$ space for each data point $X_{R}$ that is mapped to the target
distribution by $f(X_{R}|M)$. The other three features did not show
appreciable movement after training, likely due to the similarity of their
distributions between the reference and the target datasets.

(a) Feature $X_{2}$, corresponding to $\tau^{21}_{J_{1}}$.

(b) Feature $X_{3}$, corresponding to $\tau^{21}_{J_{2}}$.

Figure 3:  Distance traveled in a single dimension of the parameter space
under a mapping from reference $X_{R}$ to target $X_{T}$. The maximum possible
distance traveled is 6. We provide the mean distances traveled $\pm 1\sigma$
errors.