# Efficient Anomaly Detection Using Self-Supervised Multi-Cue Tasks

Loïc Jézéquel, Ngoc-Son Vu, Jean Beaudet, and Aymeric Histace

###### Abstract

Anomaly detection is important in many real-life applications. Recently, self-
supervised learning has greatly helped deep anomaly detection by recognizing
several geometric transformations. However these methods lack finer features,
usually highly depend on the anomaly type, and do not perform well on fine-
grained problems. To address these issues, we first introduce in this work
three novel and efficient discriminative and generative tasks which have
complementary strength: (i) a piece-wise jigsaw puzzle task focuses on
structure cues; (ii) a tint rotation recognition is used within each piece,
taking into account the colorimetry information; (iii) and a partial re-
colorization task considers the image texture. In order to make the re-
colorization task more object-oriented than background-oriented, we propose to
include the contextual color information of the image border via an attention
mechanism. We then present a new out-of-distribution detection function and
highlight its better stability compared to existing methods. Along with it, we
also experiment different score fusion functions. Finally, we evaluate our
method on an extensive protocol composed of various anomaly types, from object
anomalies, style anomalies with fine-grained classification to local anomalies
with face anti-spoofing datasets. Our model significantly outperforms state-
of-the-art with up to 36% relative error improvement on object anomalies and
40% on face anti-spoofing problems.

###### Index Terms:

Anomaly detection, fine grained classification, self-supervised learning,
multi-task learning, one-class learning

##  I Introduction

One of the most fundamental challenge in machine learning is detecting an
observation as anomalous compared to a normal baseline. Properly solving such
problem with high predictability and robustness has been essential in many
fields. To mention a few, in intrusion detection [1] where we wish to detect
untrustworthy entries on a network, fraud detection [2] where a forged item or
transaction must be rejected, in medical imaging [3] where abnormalities in a
captured image must be located, video surveillance [4, 5] where abnormal
events are detected, and in manufacturing defect detection [6, 7].

With the advent of deep learning, many tasks on image data including binary
classification and anomaly detection (AD) have greatly improved. Nevertheless
classical binary classification still generally lacks robustness and
reliability outside its training domain. Many anomaly detection methods try to
solve this problem by only learning the normal class boundary, rather than
directly discriminating anomalies from normal samples. Any observation defined
outside is then deemed as anomalous. This decision rule is especially useful
when the anomaly class boundary is ill-defined or continually evolving and
only few anomalous training samples are available.

The recent explosion of self supervision further improves unsupervised
learning abilities and reduces the needed amount of labeled data. It enables
to discriminate anomalies from normal samples by learning to solve simple
tasks such as geometric transformation classification. However, although deep
anomaly detection can achieve interesting performance, it still suffers from
limitations on more challenging problems with local and fine-grained
differences between anomalies and normal samples. Indeed, existing self-
supervised anomaly detection algorithms evaluated their performance on
datasets like CIFAR10 or CIFAR100 but not on fine-grained ones like Caltech-
Birds or face anti-spoofing. Moreover, these methods usually have an high
inference time, making them impractical for real-life anomaly detection
problems.

In this given context, our main contributions in this paper are the following:

  * •

We introduce a new way to efficiently exploit the benefits of discriminative
and generative auxiliary tasks in self-supervised anomaly detection. Using the
two-branch network, we are among the first to reach high-quality results with
auxiliary tasks on fine-grained anomaly detection and face anti-spoofing in a
one-class setting.

  * •

We carefully design and optimize three novel specialized auxiliary tasks
according to loss functions, anomaly scores as well as complexity. This allows
our model to learn very rich and complementary representations which better
encompass image structure (Section III-A), colorimetry (Section III-B) and
texture (Section III-D). With these tasks, we also explore different out-of-
distribution (OOD) detection methods and fusion functions.

  * •

We compare our method with state-of-the-art using an exhaustive protocol for
anomaly detection covering object, style and local anomalies, and even more
challenging task of face anti-spoofing.

  * •

The proposed method obtains high-quality results with up to 36% AUROC relative
improvement on object anomalies and 53% on face anti-spoofing from state-of-
the-art anomaly detection methods.

This paper follows the motivation of our work presented in [8]. In [8], we
improved the anomaly detection by simultaneously solving in a self-supervised
fashion a high-scale geometric task and a low-scale jigsaw puzzle task. It is
worth noting that the differences of this paper compared to [8] are
significant: all pretext tasks are novel and more efficient. In this paper, we
address the inference complexity issue and considerably improve the anomaly
detection performance.

First, we give an overview of anomaly detection related work in Section II.
Then we present our new pretext tasks in Section III, and our study of OOD
methods with fusion in Section IV. Our complete model is summarized in Section
V which we give a general overview in Fig. 3. In a first stage, a jigsaw
puzzle task with intra-piece tint rotation detection and a partial
colorization are performed. Then in a second stage, a set of OOD scores is
computed for each task and is aggregated into a single anomaly score using a
fusion function. In addition, we extensively compare our model with state-of-
the-art in Section VI, and provide several experiments on the influence of our
model parameters in Section VII. Finally, we discuss future work in Section
VIII.

##  II Related work

We first review several common classical and deep anomaly detection methods in
Section II.A and Section II.B. We then present self-supervised learning and
how they are applied for AD in Section II.C and Section II.D, respectively.
Readers are refereed to [9, 10, 11] for more in-depth surveys on AD or self-
supervised learning.

###  II-A Classical anomaly detection

The main goal in anomaly detection is to classify a sample as normal or
anomalous. Formally, we predict $P({\mathbf{x}}\in\mathcal{X}_{\text{norm}})$
for an observation ${\mathbf{x}}$ and a normal (or positive) class
$\mathcal{X}_{\text{norm}}$. The anomalous (or negative) class is then defined
implicitly as the complementary of the normal class in image space. We can
generally categorize anomalies into three families:

  1. 1.

Object anomaly: any object which is not included in the positive class, e.g.,
a cat is an object anomaly in regards to dogs.

  2. 2.

Style anomaly: observations representing the same object as the positive class
but with a different style or support, e.g., a realistic mask or a printed
face represent faces but with a visible different style.

  3. 3.

Local anomaly: observations representing and sharing the same style as the
positive class, however a localized part of the image is different. Most of
the time, these anomalies are the superposition of two generative processes,
e.g., a fake nose on a real face is a local anomaly.

Usually, we assume in anomaly detection that only normal samples are available
during training, meaning that methods are in one-class setting. Traditionally,
one-class Support Vector Machine [12] (OC-SVM) or its extension the Support
Vector Data Description [13] (SVDD) were used for anomaly detection. The
anomaly score of an observation ${\mathbf{x}}$ is given by its distance to a
parameterized boundary $\Omega$. OC-SVM defines $\Omega$ as an hyper-plan
separating the origin from the normal samples with the maximum margin, whereas
SVDD uses an hyper-sphere containing all normal samples with the minimum
radius (see Fig. 1(a,b)).

Fully-unsupervised methods which learn from a set of unlabeled data containing
normal samples and anomalies were also used. Such non-deep methods include
Robust Principal Component Analysis [14] (RPCA) or the Isolation Forest (IF)
[15]. Rather than modeling the normal samples, the IF algorithm tries to
isolate anomalies from normal samples via successive random partitions of the
feature space. If the sample can be entirely isolated (i.e. be the only point
in a region) in a few partitions, then it is more likely to be anomalous (see
Fig. 1(c)).

(a) OC-SVM

(b) SVDD

(c) IF

Figure 1: Overview of classical methods where green circles are normal samples
and red cross anomalies. In (a) and (b) the anomalies are not part of the
training dataset. In (c) the sample on the right is predicted as anomalous
since it only required a single partition, while the blue circle is deemed
normal.

These classical methods have shown great success on low-dimensional data such
as tabular data, but usually fail on higher dimension inputs such as images.

###  II-B Deep anomaly detection

The introduction of neural networks as feature extractors gave birth to
several hybrid methods where a pre-trained neural network is used to extract
features, on which a classical algorithm such as OC-SVM or isolation forest is
trained. It ultimately led to the first end-to-end anomaly detection neural
network, the one-class Neural Network (OC-NN) [16] which integrates the OC-SVM
loss in the network training. More recent methods include different dedicated
approach to anomaly detection. In [17, 18, 19], a binary classification is
used with pseudo negative images or latent vectors to represent the anomaly
class. Another approach is to use the error of a generative model
reconstruction [20, 21, 22, 23] or the gradient of the error given that the
image is normal [24]. Finally, the self-supervision framework can be used to
learn normal class representations and subsequently form an anomaly score as
presented in Section II-D.

There also have been semi-supervised anomaly detection methods such as DeepSAD
[25] or deviation networks [26] where we assume some of the anomalies
representing a few modes are available. These methods can achieve better
accuracy on borderline cases given enough diverse anomalies, which is often
less manageable in practice. In particular, these two methods directly learn
representations by minimizing the distance of normal sample features to an
hypersphere center, while maximizing the distance to the anomalies. It follows
the compactness principle, where the normal class representations variance is
minimized and the inter-class representations variance is maximized.

###  II-C Self-supervised learning

Self supervised learning (SSL) is a part of representation learning, where
useful and general representations are learned from an unlabeled dataset. The
learned features are then used through transfer learning for a different task
such as classification.

In this manner, representations are learned by solving from the data an
auxiliary task $\mathcal{T}$, which is often unrelated to the final one. The
pretext task can either be discriminative, usually resulting in a multi-class
classification setting or generative where a regression loss is often
utilized. Any SSL is defined by its pretext objective loss $\mathcal{L}$ and
its pretext data generation function
$DG_{\mathcal{T}}:\mathcal{P}(\mathcal{X})\mapsto\mathcal{P}(\mathcal{X}\times
K)$ which yields a labeled set from an unlabeled set $\mathcal{X}$. In the
case of discriminative tasks, it is usually done via $n$ images
transformations $T_{1},\cdots,T_{n}$:

$DG_{\mathcal{T}}\left(\\{{\mathbf{x}}_{i}\\}_{i\in\llbracket
1,N\rrbracket}\right)=\\{(T_{j}({\mathbf{x}}_{i}),j)\\}_{i\in\llbracket
1,N\rrbracket,j\in\llbracket 1,n\rrbracket}$ (1)

where the ${\mathbf{x}}_{i}$ are images from the unlabeled training dataset.

In other words, SSL consists of two steps: (1) generating a labeled set
$\mathcal{X_{T}}=DG_{\mathcal{T}}(\mathcal{X})$, (2) training a classification
or regression network on this generated labeled set. One of the final layers
are thus used as a feature extractor. Some commonly used tasks are: 90°
rotation prediction [27], jigsaw puzzle [28], distortions [29], colorization
[30], image inpainting [31] or relative patches prediction [32].

More recently, the contrastive learning framework [33] has been extensively
used for self-supervised representation learning. Unlike the methods above, it
does not rely on an explicit pretext task and directly formulates losses on
the representations. The most effective contrastive method is instance
discrimination [34, 35] where the objective is to maximize similarity between
augmented versions of a same image (positive samples) while minimizing
similarity with any other images (negative samples). The instance
discrimination can be seen as a pretext task where the pretext data generation
function maps samples to the set of positive pairs and negative pairs and the
objective function is to discriminate positive from negative pairs using
cosine similarity in representation space.

###  II-D SSL anomaly detection

In this section, we first present how to apply SSL for AD and then discuss
some state-of-the-art methods exploiting SSL for AD.

Very recently, SSL has been adapted to the one-class anomaly detection
framework. First we learn to solve an auxiliary task in a SSL fashion. Then, a
measure of how well the network can solve the task on the generated dataset
$DG_{\mathcal{T}}(\mathcal{X})$ is used to classify at inference time an
observation ${\mathbf{x}}$ as anomalous or normal. The main assumption is that
the network will perform relatively well on normal samples but will fail on
anomalies. The goals of representation learning and AD are different. In
representation learning we try to maximize the performance of the
representation on as many downstream tasks and data as possible; whereas in
AD, we want a clear discrimination through performance on normal and anomalous
data.

Any SSL anomaly detector is composed of three steps (see Fig. 2):

  1. 1.

The representation learning on the normal class, carried out in a self-
supervised manner. In our case this is done by solving a pretext task
$\mathcal{T}$, but other methods employ other mechanisms such as contrastive
learning.

  2. 2.

During inference of an unseen sample ${\mathbf{x}}$, an out-of-distribution
(OOD) detection method is applied on the generated labeled samples
$DG_{\mathcal{T}}(\\{{\mathbf{x}}\\})$. The goal of OOD method is to detect
whether or not an observation has been sampled from the same distribution as
the training set. Given a pre-trained model $\Psi$ on a distribution
$F_{\mathcal{X}_{\text{train}}}$, it estimates $P({\mathbf{x}}\sim
F_{\mathcal{X}_{\text{train}}})$. The normal training set is assumed to be
close enough to the real distribution of normal samples, and since we have
access to the correct task label $y$, the following approximations hold:

$s_{OOD}(({\mathbf{x}},y);\Psi)\approx P({\mathbf{x}}\sim
F_{\mathcal{X}_{\text{train}}})\approx
P({\mathbf{x}}\in\mathcal{X}_{\text{norm}})$ (2)

where $s_{OOD}(({\mathbf{x}},y);\Psi)$ is the OOD score for an image
${\mathbf{x}}$ with its label $y$ given the pre-trained network $\Psi$.

  3. 3.

The fusion of the OOD scores into a single anomaly score $s_{a}$ using a
fusion function $M$.

Figure 2: The three steps of pretext task based self-supervised learning
anomaly detection: (1) the pretext task is solved on the normal dataset, (2)
OOD detection functions are applied during inference on a pretext dataset
generated via the data generative function on the unseen sample, and (3) these
OOD scores are aggregated into a single anomaly score.

In the rest of this section, we detail several state-of-the-art self-
supervised anomaly detection algorithms that are the most closely related to
our work.

In GeoTrans [36], the auxiliary task is to classify which geometrical
transformation has been applied to the input from a set $\\{T_{i}\\}$ of 72
random composition of translations, rotations and symmetries. At the end of
training, a Dirichlet distribution parameterized by $\tilde{\bm{\alpha}}_{i}$
is fitted over the softmax responses of each transformation on the normal
class
$\mathbf{y}\left(T_{i}({\mathbf{x}})\right)={\operatorname{smax}}(\phi\circ
f({\mathbf{x}}))$; then its log-likelihood is used during inference.

$s_{a}({\mathbf{x}})=\sum_{i=1}^{72}\left(\tilde{\bm{\alpha}}_{i}-1\right)\cdot\log\mathbf{y}\left(T_{i}({\mathbf{x}})\right)$
(3)

In MHRot [37], the task is to simultaneously classify 90° rotations,
horizontal translations ($\operatorname{VTrans}$), and vertical translation
($\operatorname{HTrans}$), each modeled by a softmax head. Accordingly, the
pretext data generation function is the composition
$T_{r,s,t}=\operatorname{Rot}(r)\circ\operatorname{HTrans}(s)\circ\operatorname{VTrans}(t)$,
where $r\in\\{0\degree,90\degree,180\degree,270\degree\\}$,
$s\in\\{0,-t_{x},+t_{x}\\}$ and $t\in\\{0,-t_{y},+t_{y}\\}$. During inference,
the three softmax of the known transformations for each of the 36
transformation compositions are summed as anomaly score:

$s_{a}({\mathbf{x}})=\sum_{r}\sum_{s}\sum_{t}\quad\mathbf{y}(T_{\scriptscriptstyle
r,s,t}({\mathbf{x}}))_{r,s,t}$ (4)

Another class of models, called two-stage anomaly detectors [38], does not use
the representation learning task during inference, but rather directly apply
OOD methods on the representation space [39, 40, 41, 42]. For example, in SSD
[40] the representation learning step is performed through contrastive
learning, then OOD detection is applied on the representation space induced by
the encoder $\phi$. The training data representations are clustered around
several centroids using K-means. The Mahalanobis distance is used to compute
the anomaly score:

$s_{a}({\mathbf{x}})=\min_{m}(\phi({\mathbf{x}})-\mu_{m})^{T}\Sigma_{m}^{-1}(\phi({\mathbf{x}})-\mu_{m})$
(5)

Similarly, DROC-contrastive (Deep Representation One-class Classification)
[38] first learn self-supervised representations from one-class data, and then
build one-class classifiers on learned representations. Contrastive learning
with distribution augmentation is used for the self-supervised representation
learning, and a OC-SVM for the one-class classification.

Finally, it is interesting noting that some SSL anomaly detectors solve the
more specific task of anomaly segmentation like CutPaste [43], SOMAD [44].
Those anomaly segmentation consists in predicting a heatmap where the anomaly
score is computed on each pixels of the input image. They usually consider
very minute and local AD, such as defect detection, while in this work we
focus on image-level anomaly detection.

Figure 3: Method overview. Our model consists of discriminative (upper
U-branch) and generative (lower L-branch) tasks. All the discriminative tasks
share the same encoder. TABLE I: Overview of the loss function and OOD score
for each proposed task. Upper U-branch consists of piece-wise puzzle, tint
rotation tasks while lower L-branch consists of partial colorization task.

---
Task (type) | Loss | Anomaly score
Piece-wise puzzle (Cross-entropy) |  $\mathcal{L}_{\text{pzl}}(I)\propto\sum\limits_{i=1}^{n}\mathcal{L}_{\text{CE}}(\phi\circ f_{i}(I);\Pi_{i})$ (Eq.6) |  $s_{a}(I)=\frac{1}{n}\sum\limits_{i=1}^{n}s_{OOD}((\Pi(I),\Pi_{i}),\phi\circ f_{i})$ (Eq.7)
Tint rotation (Expected L1 error) |  $\mathcal{L}_{\text{tint}}(I)\propto\operatorname{E}_{\Theta\sim\phi\circ f(I)}\left[\|I-\gamma(I,\theta-\Theta)\|_{1}\right]$ (Eq.8) |  $s_{a}(\gamma(I,\theta))=\sum\limits_{i=1}^{c}{\operatorname{smax}}{(\phi\circ f(\gamma(I,\theta)))}_{i}\left(\frac{\|I-\gamma(I,\theta-i\cdot\frac{2\pi}{c})\|_{1}}{W\times H\times 255}\right)$ (Eq.9)
Partial colorization (Expectation Max.) |  $\mathcal{L}_{\text{col}}(I)=\sum\limits_{ij}\sum\limits_{k=1}^{K}Q_{\text{EM}}\left({\pi_{ij}^{(k)}},{\mu_{ij}^{(k)}},{\Sigma_{ij}^{(k)}}\right)$ (Eq.21) |  $s_{a}(A_{ij},B_{ij}|I_{\text{part}})=\sum\limits_{k=1}^{K}{\pi_{ij}^{(k)}}\mathcal{N}\left(A_{ij},B_{ij};{\mu_{ij}^{(k)}},{\Sigma_{ij}^{(k)}}\right)$ (Eq.22)

##  III Novel pretext tasks

In the rest of the paper, we note $s_{OOD}((z,y);\phi\circ f)$ a score
obtained by any out-of-distribution method from an observation $z$, its label
$y$ and a pre-trained network $\phi\circ f$. We gradually detail the proposed
pretext tasks for anomaly detection which focus on different visual cues:
structure, colorimetry and texture. The tasks of piece-wise puzzle, tint
rotation and their combination are discriminative (Sections III-A, III-B,
III-C) whereas the colorization task is generative (Section III-D). An
overview of the loss function and OOD score for each proposed task is shown in
Table I.

###  III-A Piece-wise puzzle task

The puzzle task has been successfully used as a pretext task for
representation learning [28, 45]. First an image is separated into
$n=n_{w}\times n_{h}$ pieces, with some random margin between them. Then given
the an image generated by shuffling pieces, a deep encoder is trained to
predict which permutation has been applied. It is therefore formulated as a
classification task where the prediction label corresponds to the index of the
permutation among the $n!$ total possibilities. When the number of pieces
becomes too large, the full task is not conceivable and the model should only
learn to classify a smaller random subset of all permutations. This
formulation of the jigsaw puzzle task, used in our previous work [8] along
with geometrical transformation recognition, enables our model to learn low-
scale fine features. In the rest of the paper, we call this formulation the
partial puzzle task. It is worth noting that regarding to our previous work
[8], this paper reconsiders only the puzzle task which is further optimized in
both term of time and performance, as will be described in the rest of this
section, while other tasks including tint rotation and partial colorization
have never been used for visual anomaly detection in the literature, to the
best of our knowledge.

The partial puzzle task [8] has several limitations: (i) the quality of the
representation highly depends on the chosen permutations. Indeed if the
sampled permutations are too hard (e.g. swapping two corners) or too easy, the
learned representations will suffer; (ii) Moreover from an anomaly detection
perspective, all mispredicted permutations are equally penalized regardless of
the number of misplaced pieces.

To address these limitations, we propose here an improved piece-wise puzzle
task. Rather than predicting the permutation index, we train a deep encoder to
predict the original position of each piece. By assuming each piece is
independent, we can now cover all the permutations with only $n^{2}$ outputs
instead of $n!$. Thereby we separate the output layer $f$ into $n$ functions
$f_{1},\cdots,f_{n}$, each corresponding to a piece.

Let $\Pi$ be a random permutation, $\Pi(I)$ corresponds to the image $I$ where
each piece has been moved according to $\Pi$, and $\Pi_{i}$ corresponds to the
new position of the ith piece. The task is learned using the cross-entropy
loss $\mathcal{L}_{\text{CE}}$ on every piece predictions:

$\mathcal{L}_{\text{pzl}}(\Pi(I))=\frac{1}{n}\sum_{i=1}^{n}\mathcal{L}_{\text{CE}}(\phi\circ
f_{i}(\Pi(I));\Pi_{i})$ (6)

The full task is illustrated in Fig. 4. In practice, we sample during every
training epoch a random subset of $n_{\text{tsp}}$ permutations for each
normal image. In order to have as many different permutations as possible in
the training set, we define $n_{\text{tsp}}=\frac{n!}{N_{train}\cdot ep}$,
where $N_{train}$ is the size of the training set and $ep$ the number of
training epochs.

During inference we also consider a random subset of $n_{\text{sp}}$
permutation, and compute an anomaly score for each of them:

$s_{a}(\Pi(I))=\frac{1}{n}\sum_{i=1}^{n}s_{OOD}((\Pi(I),\Pi_{i}),\phi\circ
f_{i})$ (7)

where $s_{OOD}$ is an OOD score function which is presented in more detail in
Section IV. In fact, we try different OOD functions and find out the best one.
While $n_{\text{tsp}}$ permutations are randomly used during training, it is
important to note that the $n_{\text{sp}}$ permutations are fixed for all
tests in the final model.

Figure 4: Piece-wise puzzle task for $3\times 3$ pieces, where $\Pi$ is a
random piece permutation and $\tilde{\Pi}_{i}$ is the prediction vector for
the jth piece (Section III-A).

With this new piece-wise puzzle task, lower anomaly detection errors can be
reached while keeping the same inference complexity as the partial puzzle task
(see results in Fig. 11).

###  III-B Tint rotation task

High-scale object colorimetry is a simple but powerful clue to discriminate
anomalies, especially in spoof detection. To explore this rich information
that is not considered yet in the literature, we present a novel tint rotation
recognition task which focuses on the normal class colorimetry. Given an RGB
image $I$ and a transformation $\gamma$ where $\gamma(I,\theta)$ adds an
offset $\theta$ to the hue channel (in HSV space) of $I$; we try to predict
the distribution of $\Theta$ from $\gamma(I,\Theta)$. For practical reasons,
we limit the possible tint rotation angles to $c$ distributed angles and our
task becomes to distinguish angles which are multiples of $\frac{2\pi}{c}$.

Tackling the colorimetry task with a rotation detection task allows us to
discriminatively learn high-scale and general colorimetry clues while keeping
a low computational cost. In addition, we note that contrary to the
geometrical rotation recognition task where a number of angles different from
four would leave visual artifacts, our task does not have any limitation on
$c$.

$\theta=0$°

$\theta=90$°

$\theta=180$°

$\theta=270$°

Figure 5: Tint rotation task for $c=4$ (Section III-B).

Nevertheless it is impossible to detect any tint rotation inside areas without
any original color information. To prevent high anomaly scores on desaturated
images, we need to give a lower weight on those regions. To this end, instead
of working on the angle distribution we use the expected $L_{1}$ error in RGB
space between the original image and the predicted one. Since we are computing
a pixel wise RGB error, only large areas of colorful pixels will impact the
anomaly score. The tint rotation task training loss is:

$\mathcal{L}_{\text{tint}}(\gamma(I,\theta))=\operatorname{E}_{\Theta|\gamma(I,\theta)}\left[\frac{\|I-\gamma(I,\theta-\Theta)\|_{1}}{W\times
H\times 255}\right]$ (8)

where $W\times H$ is the dimension of the image. As for the anomaly score, we
use the same error as the loss function which becomes in its developed form:

$s_{a}(\gamma(I,\theta))=\sum\limits_{i=1}^{c}{\operatorname{smax}}{(\phi\circ
f(\gamma(I,\theta)))}_{i}\left(\frac{\|I-\gamma(I,\theta-i\cdot\frac{2\pi}{c})\|_{1}}{W\times
H\times 255}\right)$

(9)

where ${\operatorname{smax}}(\cdot)$ is the softmax function.

By introducing this task we force our encoder to fully represent the normal
class colorimetry, which could be potentially ignored by the puzzle task in
case of salient geometrical features.

###  III-C Intra-piece tasks

On top of the piece-wise puzzle task, we further propose to add pretext sub-
tasks inside each puzzle piece. Given an intra-piece task
$\mathcal{T}_{piece}$ and an image composed of $n$ pieces images
$R_{1},\cdots,R_{n}$, we first sample a random augmented piece using the
pretext data generation function on each piece $(I^{(aug)}_{i},y_{i})\sim
DG_{\mathcal{T}_{piece}}(\\{R_{i}\\})$. Then our network tries to solve
simultaneously the puzzle task and the intra-piece tasks by minimizing the
loss

$\mathcal{L}(I)=\frac{1}{n}\sum_{i}^{n}\left(\mathcal{L}_{\text{CE}}(\phi\circ
f_{i}(I);\Pi_{i})+\mathcal{L}_{\text{piece}}(R_{i})\right)$ (10)

where the first term is from the piece-wise puzzle loss defined in Equation 6
and $\mathcal{L}_{\text{piece}}$ is the loss of the intra-piece task. A
summary of the intra-piece task model is given in Fig. 6.

Figure 6: Example of intra-piece tasks with tint rotation detection with $c$
possible rotations (Section III-C). The only additional cost of this task when
compared to the piece-wise task is a specialized dense layer.

By adding these intra-piece tasks, we essentially consider $n$ new tasks
during inference without increasing the number of forward pass in our encoder.
The only cost is the additional specialized dense layer for the pretext task.
Each intra-piece task will allow our network to focus on specific image
patches.

One issue with this method is that we can potentially mix object pieces and
background pieces. Solving tasks on background pieces would enable the model
to generalize on image distribution far from the normal class object. As a
result, we introduce a weight map for each piece learned during training where
higher weights are given for pieces covering the object. We could see this map
as a rough segmentation of the normal object in the image. These are computed
in a similar fashion as visual attention mechanism, which have previously
successfully been used for learning weight maps for each pixels [46].

Figure 7: Intra-piece tasks with attention (Section III-C).

First, we compute from the encoder representation $z$ a weight map
$(w_{ij})_{0\leq i+j\leq n}$, which we normalize into attention weights using
the $L_{1}$ normalized sigmoid $P_{ij}=\frac{\sigma(w_{ij})}{\|w\|_{1}}$. This
normalization function produces smoother maps than the classical softmax
activation, preventing very sparse maps where only one piece has a non-null
activation. To further prevent these cases, we include an additional term to
the loss encouraging spread matrices:

$\displaystyle\mathcal{L}_{\text{density}}(P)$
$\displaystyle=\sum_{ij}\left\|\scriptsize{\begin{pmatrix}i\\\
j\end{pmatrix}}-\mu\right\|_{2}P_{ij}$ (11)

where $\mu=\sum_{ij}P_{ij}\scriptsize{\begin{pmatrix}i\\\ j\end{pmatrix}}$.

Our final loss of intra-piece tasks taking into account the attention map
(upper branch in Fig. 3) is:

$\mathcal{L}_{\text{U-branch}}=\mathcal{L}_{\text{pzl}}(I)+\mathcal{L}_{\text{density}}(P)+\sum_{i,j}P_{ij}\cdot\mathcal{L}_{\text{piece}}(R_{i,j})$
(12)

and the corresponding anomaly score is

$s_{a}(I)=M(\left\\{s_{OOD}((z,y);\phi\circ f)\middle|(z,y)\in
DG_{\mathcal{T}}(\\{I\\})\right\\})$ (13)

where $M$ is the fusion function which is detailed in Section IV.

We can see in Table IX that the attention mechanism increases anomaly
detection performances.

###  III-D Partial colorization task

We present in this section a novel generative pretext task for anomaly
detection which is highly texture oriented. In the colorization task commonly
used in the literature [47, 30, 48], the main objective is to predict the
$(A,B)$ color channels from the luminance channel $L$ of an image in LAB
space.

One big challenge with this task is to colorize the background since it can
vary a lot inside the training normal set. The re-colorization will be
naturally poorer for unseen background during inference of new observations.
Therefore the object itself should have more impact on the AD algorithms than
the background, making the anomaly detector more object-oriented than scene-
oriented. In addition, several issues arise when considering the typical
framework of colorization through regression [47] where
$\operatorname{E}\left[(A_{ij},B_{ij})|L\right]$ is directly estimated for
each pixel $(i,j)$. First, the colorimetry of the normal class can potentially
be multi-modal. In other words, the normal class objects can have several
plausible set of colors called modes. For example, horses could have more than
one fur color yet still being part of the same class. In this case a
regression network can not model the multi-modality and will end up predicting
the mean of all modes. Second, even if one of the object mode is correctly
predicted, any error function will yield high values if the mode of the
current observation is different.

To tackle these limitations, we establish a novel method to learn colorization
well-suited to anomaly detection. First, we augment the available inputs with
the color values of the image inside a border of size $\alpha$ to make the
background re-colorization easier. Our partial colorization task thus consists
in predicting $(A,B)$ from the image with partial color channels
$I_{\text{part}}=(L,A\odot M_{\alpha},B\odot M_{\alpha})$ where $M_{\alpha}$
is a binary mask consisting of 1 in the border of size $\alpha$ and 0 in the
center. Moreover, different to existing regression methods, we estimate the
posterior density $p(A_{ij},B_{ij}|I_{\text{part}})$ of each pixel. For
density estimation, we explore two different ideas: (1) quantize the colors
into a low-range discrete variable and perform multi-class classification; (2)
parameterize the density with a gaussian mixture model and perform maximum
likelihood estimation.

####  III-D1 Color bin classification

By quantizing each color value into $K$ bins and assuming the two colors
planes to be independent, we can define the resulting categorical variables by
$2K$ probabilities:
$P(A_{ij}=1),\cdots,P(A_{ij}=K),P(B_{ij}=1),\cdots,P(B_{ij}=K)$. We thus
estimate a map $y$ of dimension $H\times W\times 2K$, where

$\displaystyle y_{i,j,2k}$ $\displaystyle=P(A_{ij}=k|I_{\text{part}})$
$\displaystyle y_{i,j,2k+1}$ $\displaystyle=P(B_{ij}=k|I_{\text{part}})$ (14)

Inspired by the label smoothing idea [49], a gaussian smoothing is applied to
the output distributions in order to propagate our model confidence to
neighbor color bins. Indeed we do not want to entirely penalize close color
bins. As such the final estimated density $\hat{P}(A_{ij}|I_{\text{part}})$
for a network $\phi$ is

$\hat{P}(A_{ij}=k|I_{\text{part}})=\left({\operatorname{smax}}(\phi(I_{\text{part}})_{ij})\star
G_{\sigma}\right)_{k}$ (15)

where $G_{\sigma}$ is the gaussian kernel of standard deviation $\sigma$.

####  III-D2 Gaussian Mixture Model MLE

Our second approach is to parameterize the densities with Gaussian Mixture
Models. Accordingly, we have for each pixel a sum of $K$ gaussian densities:

$p(A_{ij},B_{ij}|I_{\text{part}})=\sum_{k=1}^{K}{\pi_{ij}^{(k)}}\mathcal{N}\left(A_{ij},B_{ij};{\mu_{ij}^{(k)}},{\Sigma_{ij}^{(k)}}\right)$
(16)

where ${\pi_{ij}^{(k)}}\in{\mathbb{R}}$ is the prior probability of the $k$th
cluster, ${\mu_{ij}^{(k)}}\in{\mathbb{R}}^{2}$ is the mean color of the kth
cluster and ${\Sigma_{ij}^{(k)}}\in{\mathbb{R}}^{2\times 2}$ is the covariance
color matrix of the $k$th cluster.

Rather than predicting the full $2\times 2$ matrix ${\Sigma_{ij}^{(k)}}$, we
only predict the three free parameters $\bm{\sigma}$. We can then reconstruct
the positive definite covariance matrix using Cholesky decomposition [50]:

${\Sigma_{ij}^{(k)}}=\left(\begin{matrix}1&0\\\
l&1\end{matrix}\right)\text{Diag}\left(e^{d}\right)\left(\begin{matrix}1&0\\\
l&1\end{matrix}\right)^{T}$ (17)

where $d\in{\mathbb{R}}^{2}$ and $l\in{\mathbb{R}}$. This decomposition
ensures strictly positive eigen values from the exponential and a semi-
positive matrix from the Cholesky decomposition. All the possible covariance
matrices are thus parameterized by $(d,l)$. It also introduces better
numerical stability for determinant computation with the simple formula
$\log|\Sigma|=\log\left|\text{Diag}\left(e^{d}\right)\right|=\sum_{i}d_{i}$.

To train this model, we could use as the loss function the log-likelihood
which considers all pixels independent:

$\mathcal{L}(\mu,\Sigma|A,B)=\sum_{ij}\log\left(\sum_{k=1}^{K}{\pi_{ij}^{(k)}}\mathcal{N}\left(A_{ij},B_{ij};{\mu_{ij}^{(k)}},{\Sigma_{ij}^{(k)}}\right)\right)$
(18)

However this function turns out to be very hard to directly optimize for each
pixel and does not lead to any meaningful colorization. We use instead the
classical Expectation Maximization algorithm. As for details, we carry out the
three following steps:

  1. (step 1)

Compute Mahalanobis distances:

${\Delta_{ij}^{(k)}}=\left(I_{ij}-{\mu_{ij}^{(k)}}\right)^{T}{\Sigma_{ij}^{(k)}}^{-1}\left(I_{ij}-{\mu_{ij}^{(k)}}\right)$
(19)

  2. (step 2)

Compute posterior cluster probabilities:

$\gamma_{ij}(k)=\frac{{\pi_{ij}^{(k)}}\exp\left(-\frac{1}{2}\left(\sum_{l}{d_{l}^{(k)}}+{\Delta_{ij}^{(k)}}\right)\right)}{\sum_{\kappa=1}^{K}{\pi_{ij}^{(\kappa)}}\exp\left(-\frac{1}{2}\left(\sum_{l}{d_{l}^{(\kappa)}}+{\Delta_{ij}^{(\kappa)}}\right)\right)}$
(20)

  3. (step 3)

Fix the $\gamma_{ij}(k)$ and minimize loss (lower branch):

$\mathcal{L}_{\text{L-branch}}(\pi,\mu,\Sigma|I)=\sum\limits_{ij}\sum\limits_{k=1}^{K}\gamma_{ij}(k)\left({\Delta_{ij}^{(k)}}+\sum\limits_{l}{d_{l}^{(k)}}-\log{\pi_{ij}^{(k)}}\right)$

(21)

Once the training is finished, we compute the anomaly score as the likelihood
of the color channels under the predicted ${\pi_{ij}^{(k)}}$,
${\mu_{ij}^{(k)}}$ and ${\Sigma_{ij}^{(k)}}$:

$s_{a}(A_{ij},B_{ij}|I_{\text{part}})=\sum_{k=1}^{K}{\pi_{ij}^{(k)}}\mathcal{N}\left(A_{ij},B_{ij};{\mu_{ij}^{(k)}},{\Sigma_{ij}^{(k)}}\right)$
(22)

In order to choose the number of gaussians $K$, we apply beforehand a K-means
color clusterization [51] on the cropped down-sampled images of the normal
class. Then by using the elbow method, we can find the optimal $K$ inside
$\llbracket 1,10\rrbracket$.

Advantages. The GMM approach has three advantages over the bin classification:
(i) its density support is not bounded, and is continuous thus not needing any
gaussian smoothing, (ii) it can fully model the dependence between the color
channels with the full covariance matrix, and (iii) it can reach the same
quality of colorization with fewer parameters. The quality of colorization is
here measured using the mean pixel color likelihood.

Figure 8: Scheme of the partial colorization with GMM estimation and a UNet
network (Section III-D). The model predicts $6K$ parameters per pixel:
$\pi\in{\mathbb{R}}$, $\mu\in{\mathbb{R}}^{2}$ and
$\bm{\sigma}\in{\mathbb{R}}^{3}$ for each of the $K$ clusters.

##  IV OOD methods and fusion

We try two different out-of-distribution methods for each pretext task: the
softmax truth and the Mahalanobis distance. In the case of a self-supervised
classification task, the most commonly used OOD function is the likelihood of
the label given that the image is normal, which we call the “ _softmax truth_
”:

$\displaystyle s_{OOD}((z,y);\phi\circ f)$
$\displaystyle=p(y|z,z\in\mathcal{X}_{train})$
$\displaystyle\approx{\operatorname{smax}}(\phi\circ f(z))_{y}$ (23)

However, this softmax truth criterion takes into account only one component of
the softmax vector. For easy tasks, we usually have a high probability on the
correct class, however for harder, multi-issue task, we can have several
typical highly activated classes for the normal class. As such, another idea
is to look at the likelihood of the raw score vector given its label and given
that the image is normal:

$s_{OOD}((z,y);\phi\circ f)=p(\phi\circ
f(z)|y,z\in\mathcal{X}_{\text{train}})$ (24)

To approximate this conditional probability, the training dataset is first
partitioned on samples sharing the same label value $l$, i.e.
$\\{(z,y)|(z,y)\in\mathcal{X}_{\text{train}}\text{ and }y=l\\}$. The
distribution of the normal class raw score vectors given $y$ can then be
separately estimated on each partition after convergence of the network
weights.

For a given classification problem with $C$ classes and a training set
$\mathcal{X}_{\text{norm}}$, we estimate the mean scores $\mu_{c}$ and
covariance matrices $\Sigma_{c}$ for each class $c$:

$\displaystyle\mu_{c}$
$\displaystyle=\frac{1}{|\mathcal{Z}_{c}|}\sum_{z\in\mathcal{Z}_{c}}\phi\circ
f(z)$ $\displaystyle\Sigma_{c}$
$\displaystyle=\frac{1}{|\mathcal{Z}_{c}|}\sum_{z\in\mathcal{Z}_{c}}(\phi\circ
f(z)-\mu_{c})^{2}$ (25)

where $\mathcal{Z}_{c}=\\{z|(z,y)\in
DG_{\mathcal{T}}(\mathcal{X}_{\text{norm}})\text{ and }y=c\\}$. The OOD score
is approximated by the Mahalanobis distance [52] with the mode corresponding
to the truth label:

$s_{OOD}((z,y);\phi\circ f)\approx(\phi\circ
f(z)-\mu_{y})^{T}\Sigma_{y}^{-}1(\phi\circ f(z)-\mu_{y})$ (26)

We also explore different fusion functions to combine all the OOD scores into
a single anomaly score. We first use the mean, but observe heavy biases from
outlier OOD scores (very easy sub-task or harder sub-task). We then try
different order statistics including the median and the 25th percentile and
compare the results in Table VII.

##  V Full method overview

Figure 9: Examples of anomalies when considering one-vs-all on CIFAR-10. We
indicate if each task detects it as an anomaly (✓) or as normal ($-$).

This section summarizes our full method (Fig. 3). Our model is made of two
independent branches. The first discriminative branch (upper branch in Fig. 3)
solves the piece-wise puzzle task with intra-piece tint rotation detection
task. The second generative branch (lower branch in Fig. 3) performs the
partial re-colorization task. We share the same encoder network for all of the
discriminative tasks, including the attention mechanism. The re-colorization
task is modeled with _GMM_ , and we include the _attention mechanism_ for the
intra-piece task. To detect whether or not an observation ${\mathbf{x}}$ is an
anomaly, we produce the OOD scores of the re-colorization and the
$n_{\text{sp}}$ sampled permutations along with tint rotation tasks. The
chosen OOD function for every task is the _softmax truth_. All of these scores
are then combined into a single anomaly score using the _median_. Presented in
Fig. 9 are examples of anomalies detected by our three different tasks using
different visual cues. As can be seen, our detectors are of complementary
strength.

##  VI Results

###  VI-A Evaluation protocol

Our evaluation protocol is made of three types of anomaly detection
challenges: object anomalies, fine-grained style anomalies, and face
presentation attacks. First, to detect object anomalies we use general coarse
object recognition datasets. The one-vs-all protocol is used, where we
consider one class of a multi-classification dataset as the normal class. All
the other classes are then considered as anomalous, and we can obtain a set of
runs for each possible normal class. The final reported result is the mean of
all runs.

However, these datasets have become far from real anomaly detection
applications and might not be enough to fully evaluate AD methods. Thus we
include a second evaluation group where we try to detect style anomalies using
fine-grained classification datasets. Fine-grained datasets have been
introduced to tackle the recognition of classes, usually part of a same
category, with slight differences. We use here the one-vs-all protocol as
well.

Finally, we consider a real anomaly detection problem which incorporates
object anomalies, style anomalies and local anomalies. In particular we choose
a dataset from face presentation attack detection (FPAD), where the goal is to
discriminate real faces from fake representations of someone’s face. Due to
the constantly evolving frauds and high variability, anomaly detection seems a
very appealing solution to this problem.

We use the following datasets:

(i) For object anomalies:

  * •

F-MNIST [53]: has been introduced as a harder version of MNIST with 10
different classes of fashion items. All images are grayscale meaning no color
information can be used to discriminate anomalies.

  * •

CIFAR-10 [54]: object recognition dataset composed of 10 wide classes with
6000 images per class.

  * •

CIFAR-100 [54]: extended version of CIFAR-10 with 100 classes each containing
600 images.

(ii) For style anomalies:

  * •

Caltech-UCSD Birds 200 [55]: fine-grained classification dataset of 200 birds
species with approximately 30 images per class.

  * •

FounderType-200 [56]: font recognition dataset containing 200 fonts with 6700
images per class. It has been introduced for novelty detection and even though
these images lie on a low dimensional manifold compared to natural images,
they still provide insight into how well the model can capture small shape
hints.

(iii) For the face presentation attack detection, we use the WMCA dataset [57]
which contains more than 1900 short videos of real faces and presentation
attacks. It contains several modalities such as infra-red or depth, but here
we only use RGB. There are 72 real identities along with several types of
attacks: paper print, screen replay, masks and partial attacks where only a
localized area of the face is fake. The masks are composed of paper masks,
rigid mask and flexible masks. An example of each type of attack is given in
Fig. 10.

TABLE II: Summary of evaluation datasets. 5 |  |  |  |
---|---|---|---|---
| Dataset | Anomaly type
| Object | Style | Local
{31.7cm[  Obj.classif ] | F-MNIST | ✓ | - | -
CIFAR-10 | ✓ | - | -
CIFAR-100 | ✓ | - | -
{21.7cm[  Fine-grained ] | Caltech-Birds | ✓ | ✓ | -
FounderType | - | ✓ | -
{11.77cm[  FPAD ] | WMCA | ✓ | ✓ | ✓
5 |  |  |  |

In all evaluations, the metric used is the area under the ROC curve (AUROC) or
the error 1-AUROC, averaged over all possible normal classes in the case of
one-vs-all datasets. We additionally include for anti-spoofing datasets
metrics more adapted to biometric presentation attack detection:

  * •

The equal error rate (EER[58]), which is the location in the ROC curve where
the false reject rate (or Bona-fide Presentation Classification Error Rate
BPCER) is equal to the false acceptance rate (or Attack Presentation
Classification Error Rate APCER).

  * •

The Attack Presentation Classification Error Rate for the Bona-fide
Presentation Classification Error Rate fixed at 5% (APCER@5%BPCER [58]).

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

Figure 10: Types of spoof attacks in WMCA dataset: 347 bonafide, 75 glasses
(a)(b), 200 print (c), 348 replay (d), 122 fake head (e), 137 rigid mask
(f)(g)(h), 379 flexible mask (i), 71 paper mask (j).

###  VI-B Implementation details

TABLE III: Comparison with the state-of-the-art AUROC over several datasets, underline indicates best result, bold indicates best one-class learning result. For the sake of fair comparison, we re-evaluated by ourselves all methods, except the one-class methods in the first block (results are from [59, 60, 38]). DROC-contrastive [38] combines different techniques: contrastive learning, distribution augmentation and OC-SVM. 8 |  |  |  |  |  |  |
---|---|---|---|---|---|---|---
| Model | CIFAR-10 | CIFAR-100 | F-MNIST | CUB-200 | FounderType | WMCA
{11.9cm[  Supervised ] | 16-4 WideResNet [61] | 99.3 | 96.3 | 99.2 | - | - | 82.4
{31.9cm[  Semi-Supervised ] | Deep-SAD (75%) [25] | 92.5 | 88.7 | 98.1 | 73.6 | 99.8 | 83.2
Deep-SAD (25%) | 90.8 | 87.9 | 95.4 | 70.9 | 99.4 | 79.8
Deep-SAD (10%) | 86.0 | 89.1 | 88.2 | 66.1 | 98.0 | 72.6
{111.9cm[  One-class ] | ADGAN [59] | 62.4 | 54.7 | 88.4 | - | - | -
GANomaly [21] | 69.5 | 56.5 | 80.9 | - | - | -
ARNet [60] | 86.6 | 78.8 | 93.9 | - | - | -
DROC-contrastive [38] | 92.5 | 86.5 | 94.8 | - | - | -
2-8 | OCSVM [12] | 58.5 | - | 74.2 | 76.3 | - | -
| IF [15] | 73.4 | - | 84.0 | 74.2 | - | -
| OC-CNN [18] | 66.5 | - | 75.4 | - | - | -
| PIAD [22] | 79.9 | 78.8 | 94.3 | 63.5 | 90.8 | 76.4
| GeoTrans [36] | 85.4 | 84.7 | 92.6 | 66.6 | 92.3 | 79.8
| MHRot [37] | 89.5 | 83.6 | 92.5 | 77.6 | 96.7 | 81.3
| PuzzleGeom [8] | 88.2 | 85.8 | 92.8 | 83.2 | 96.9 | 85.6
2-8 | Ours | 92.5 | 88.2 | 93.7 | 83.2 | 97.4 | 91.4
8 |  |  |  |  |  |  |
TABLE IV: Detailed comparison with one-class state-of-the-art AUROC on the
CIFAR-10 dataset.
---
Model | Airplane | Automobile | Bird | Cat | Deer | Dog | Frog | Horse | Ship | Truck | Avg
VAE [62] | 70.0 | 38.6 | 67.9 | 53.5 | 74.8 | 52.3 | 68.7 | 49.3 | 69.6 | 38.6 | 58.3
OCSVM [12] | 63.0 | 44.0 | 64.9 | 48.7 | 73.5 | 50.0 | 72.5 | 53.3 | 64.9 | 50.8 | 58.5
AnoGAN [20] | 67.1 | 54.7 | 52.9 | 54.5 | 65.1 | 60.3 | 58.5 | 62.5 | 75.8 | 66.5 | 61.8
PixelCNN [63] | 53.1 | 99.5 | 47.6 | 51.7 | 73.9 | 54.2 | 59.2 | 78.9 | 34.0 | 66.2 | 61.8
Deep-SVDD [64] | 61.7 | 65.9 | 50.8 | 59.1 | 60.9 | 65.7 | 67.7 | 67.3 | 75.9 | 73.1 | 64.8
OCGAN [65] | 75.7 | 53.1 | 64.0 | 62.0 | 72.3 | 62.0 | 72.3 | 57.5 | 82.0 | 55.4 | 65.6
Puzzle-AE [66] | 78.9 | 78.0 | 69.9 | 54.8 | 75.4 | 66.0 | 74.7 | 73.3 | 83.3 | 69.9 | 72.4
DROCC [19] | 81.7 | 76.7 | 66.7 | 67.1 | 73.6 | 74.4 | 74.4 | 71.4 | 80.0 | 76.2 | 74.2
AnoNAGN [67] | 96.2 | 63.8 | 72.5 | 64.3 | 87.3 | 63.8 | 88.3 | 58.4 | 93.5 | 64.5 | 75.01
GeoTrans [36] | 74.7 | 95.7 | 78.1 | 72.4 | 87.8 | 87.8 | 83.4 | 95.5 | 93.3 | 91.3 | 86.0
PuzzleGeom [8] | 75.1 | 96.3 | 84.8 | 74.2 | 91.1 | 89.9 | 88.7 | 95.5 | 94.7 | 91.9 | 88.2
SSD [40] | 82.7 | 98.5 | 84.2 | 84.5 | 84.8 | 90.9 | 91.7 | 95.2 | 92.9 | 94.4 | 90.0
Ours | 85.9 | 97.9 | 88.7 | 81.2 | 95.4 | 94.2 | 92.1 | 96.9 | 96.5 | 95.4 | 92.5

For the piece-wise puzzle task, we use a margin of half the size of the pieces
and find best results with $n_{\text{sp}}=18$. Generally we use
$n_{w}=n_{h}=3$ pieces for most datasets, except face anti-spoofing where
$n_{w}=3$ and $n_{h}=4$. We observe better results with more vertical pieces
on faces, since they are always upright and need finer vertical analysis. For
the tint rotation recognition we use $c=4$ and for the re-colorization task,
we use a contextual border $\alpha$ of two pixels.

Regarding network architecture, we use a 16-4 WideResNet [61] ($\approx 10M$
parameters with a depth of 16) for the feature extractor network $\phi$, along
with three dense layers respectively of size $n^{2}$ for the piece-wise puzzle
task, size $n\cdot c$ for the tint rotation task and size $n$ for the
attention. Each of these dense layers have a dropout rate of 0.3 [68]. As for
the re-colorization task, we use a UNet network [69]. It was originally
introduced for image segmentation, using a down-sample / up-sample strategy
reintroducing the intermediate maps at each step of the down-sample branch
into the up-sample branch. It is in fact generally well suited for any
prediction task where the output is aligned with the input pixels (in our case
a vector of GMM parameters for each pixel). Training is performed under SGD
optimizer with Nesterov momentum [70], using a batch size of 32 and a cosine
annealing learning rate scheduler [71].

###  VI-C Comparison to the state-of-the-art

A comparison of our method with other state-of-the-art (SOTA) anomaly
detection models is performed on all three protocols. We choose to include
three families of SOTA methods: _one-class learning_ methods which only learn
using the normal class, _semi-supervised learning_ methods where a small set
of anomalies is used during training and _supervised learning_. The considered
one-class methods can be categorized into (1) reconstruction error-based
methods with ADGAN [59], GANomaly [21] and PIAD [22], (2) hybrid methods with
OCSVM [12], IF [15] , OC-CNN [18], (3) pretext tasks-based methods with ARNet
[60], GeoTrans [36], MHRot [37] and PuzzleGeom [8] and (4) two-stage anomaly
detection using contrastive learning with SSD [40] and DROC-contrastive [38].
GeoTrans uses various geometrical transformations as SSL pretext task, MHRot
adds on top 90° rotations and our previous model PuzzleGeom [8] includes a
basic jigsaw puzzle task. Regarding semi-supervised methods, we evaluate
DeepSAD [25] trained on the same normal samples but with three different ratio
of the anomaly sub-classes: 10%, 25% and 75%. For the fully supervised
baseline we simply use the same backbone as our one-class method (the 16-4
WideResNet) extended with a dense layer representing the two normal and
anomaly classes. It is important to note that its training is performed with
classical binary cross-entropy loss on the normal/anomaly label, without any
class balancing mechanism.

The experiment results are displayed in Table III and a detailed evaluation on
the CIFAR-10 dataset is included in Table IV. We note that for the sake of
fair comparison in the same conditions, we re-evaluate almost all methods
ourselves using existing implementations.

Our method maintains among the best accuracies on coarse object and fine-
grained anomaly detection. It improves upon PuzzleGeom, and closes the gap
toward semi-supervised performances with a small AUC difference of 0.5% on
CIFAR-100. We also show that our method greatly improves anti-spoofing
detection performance on WMCA. It even outperforms the supervised model and
semi-supervised anomaly detection methods which have access up to 75% of the
anomalous data.

In general we can notice that hybrid methods, although efficient for smaller
problems, do not extend well to high-dimensional data. The evaluated
reconstruction-based methods also tend to fall behind pretext-task oriented
models. On the other hand, two-stage contrastive methods like DROC-contrastive
produce very competitive performance. This model combines different techniques
including contrastive representation learning, distribution augmentation and
OC-SVM. It performs slightly better than ours on the F-MNIST dataset and
reaches the same AUC on CIFAR-10 but on the more challenging one, CIFAR-100,
we obtain a gain of nearly 2%. Moreover, we note that distribution
augmentation and OC-SVM could also be used on the concatenation of our learned
representations to reach better accuracy.

Overall, our model keeps a good balance between coarse object anomaly
detection and finer style anomaly detection, and even outperforms semi-
supervised anomaly detection methods on CUB-200 and WMCA. It achieves a
relative error improvement of 36% on CIFAR-10 and 40% on WMCA compared to
PuzzleGeom.

TABLE V: AUROC, EER and APCER at 5% BPCER on WMCA dataset, best result is in
bold.
---
Models | AUROC | EER |  | APCER
---
(5%BPCER)
MHRot [37] | 81.3 | 23.9 | 72.6%
PuzzleGeom [8] | 85.6 | 19.7 | 33.8%
Ours | 91.4 | 16.1 | 27.3%

Lastly, we compare in Table V our method with the two second best self-
supervised methods MHRot and PuzzleGeom on WMCA. Using our method the
APCER@5%BPCER drops from 33.8% to 27.3%. This also shows promising usage of
anomaly detection methods in fraud detection.

##  VII Parameter study

In this section, we evaluate the parametrization of pretext tasks in Sections
VII-A, VII-B, VII-C, the choice of OOD function in Section VII-D and perform
an ablation study in Section VII-E.

###  VII-A Puzzle task complexity

We start by comparing in Fig. 11 the two approaches on the CIFAR-10 dataset
for the jigsaw puzzle task introduced in Section III-A. The piece-wise puzzle
task greatly improves performances for all CIFAR-10 classes even though the
same permutations are tested during inference. Moreover, we confirm that the
partial puzzle task is more sensitive to the choice of $n_{\text{sp}}$, since
its representation quality also depends on this factor. We choose to fix
$n_{\text{sp}}=18$ out of $9!$ possible permutations as a good compromise
between complexity of inference and accuracy.

912183264$n_{\text{sp}}$$\displaystyle{55}$$\displaystyle{60}$$\displaystyle{65}$$\displaystyle{70}$$\displaystyle{75}$$\displaystyle{80}$$\displaystyle{85}$$\displaystyle{90}$$\displaystyle{95}$AUCModelPartialPiece-
wise

Figure 11:  Comparison of AUC with different number of tested permutations
$n_{\text{sp}}$ for 3x3 partial and piece-wise puzzle on CIFAR-10 dataset.

The influence of the number of puzzle pieces $n_{w}$ and $n_{h}$ for
$n_{\text{sp}}\in\\{9,18\\}$ is reported in Fig. 12 on CIFAR-10. We can see
that for both $n_{\text{sp}}=9$ and $n_{\text{sp}}=18$, the best value for
general one-vs-all problem is $n_{w}=n_{h}=3$.

3x24x23x34x34x4n$\displaystyle{70}$$\displaystyle{80}$$\displaystyle{90}$AUC$n_{\text{sp}}$918

Figure 12:  Comparison of the number of pieces on CIFAR-10 dataset with two
different amounts of permutations during inference.

###  VII-B Tint rotation task complexity

We measure the AUC of the isolated tint rotation task for different number of
tint rotations $c$ on the CIFAR-10 dataset in Fig. 13. The best value of $c$
across several normal classes is 4.

$\displaystyle{3}$$\displaystyle{4}$$\displaystyle{5}$$\displaystyle{6}$c$\displaystyle{75}$$\displaystyle{80}$$\displaystyle{85}$$\displaystyle{90}$AUC

Figure 13: Comparison of AUC with different number of tint rotation $c$ on
CIFAR-10 dataset.

###  VII-C Colorization task parametrization

The two colorization parametrizations using Gaussian Mixture Model and bin
classification are compared on the normal class full colorization task. Our
evaluation metric is directly the likelihood of the colorization, which is
respectively for classification and GMM

$\mathcal{L}(A,B)=\prod_{i,j}{\operatorname{smax}}(\phi(I)_{ij})_{\lfloor\frac{A_{ij}}{K}\rfloor}\cdot{\operatorname{smax}}(\phi(I)_{ij})_{\lfloor\frac{B_{ij}}{K}\rfloor}$
(27)

and

$\mathcal{L}(A,B)=\prod_{i,j}\sum_{k=1}^{K}{\pi_{ij}^{(k)}}\mathcal{N}\left(A_{ij},B_{ij};{\mu_{ij}^{(k)}},{\Sigma_{ij}^{(k)}}\right)$
(28)

Overall, we can reach higher likelihoods with GMM than bin classification.
Moreover, a better separation of the different modes can be achieved using
GMM, where bin classification usually mixes the different modes and produces
dull colors (see Fig. 14).

original

bin class.

GMM

original bin class. GMM

Figure 14: Colorization comparison on faces. The first row displays the
original images, while the second represents the re-colorization of two
methods. As we can see, the bin classification approach produces dull colors
and mixes the skin color modes, producing grayish colors.

###  VII-D Choice of OOD and fusion functions

To evaluate the effect of Mahalanobis distance as an anomaly score, we compare
it with the softmax truth and its improved form, the ODIN method [72] which
adds temperature scaling during training, and the input pre-processing
$\tilde{{\mathbf{x}}}={\mathbf{x}}-\varepsilon\operatorname{sign}(-\nabla_{{\mathbf{x}}}\log{\operatorname{smax}}({\mathbf{x}};T))$.

The results are presented in Fig. 15 for different number of puzzle pieces $n$
and $n_{\text{sp}}=18$ permutations tested. The AUC increases with the number
of pieces when using the Mahalanobis distance, whereas it decreases with the
softmax truth. In addition, the AUC of the most difficult class is always
higher when using the Mahalanobis distance. This shows that despite a lower
average anomaly detection performance, it has less variance in its predictions
and provides more robust OOD scores to different normal classes. Even though
the ODIN method provides sensible improvement for more than $3\times 3$
pieces, it greatly increases computational complexity during training and
inference. In our tests, we observe an inference time increase of more than
three times with the ODIN method. We provide in Table VI further comparisons
between the softmax truth and the Mahalanobis distance on the puzzle task with
$n_{\text{sp}}=9$.

$\displaystyle 3\times 3$$\displaystyle 4\times 2$$\displaystyle 4\times
3$$\displaystyle 4\times
4$n$\displaystyle{65}$$\displaystyle{70}$$\displaystyle{75}$$\displaystyle{80}$$\displaystyle{85}$$\displaystyle{90}$$\displaystyle{95}$AUCOOD
MethodSoftmax truthMahalanobisODIN

Figure 15:  Comparison of OOD methods AUC with different number of pieces $n$
for $n_{\text{sp}}=18$ tested permutations on CIFAR-10 dataset. TABLE VI:
Comparison of AUC with different OOD methods for the piece-wise puzzle task
with $n_{\text{sp}}=9$ on CIFAR-10 dataset.
---
n | OOD Method | $\mu_{AUC}$ | maxAUC | minAUC
$3\times 3$ | Softmax truth | 86.39 | 96.35 | 71.95
Mahalanobis | 83.44 | 95.30 | 80.60
$4\times 2$ | Softmax truth | 82.82 | 96.11 | 65.49
Mahalanobis | 83.21 | 95.74 | 80.10
$4\times 3$ | Softmax truth | 84.13 | 96.34 | 66.57
Mahalanobis | 84.41 | 96.08 | 81.22
$4\times 4$ | Softmax truth | 80.61 | 93.48 | 61.87
Mahalanobis | 86.58 | 96.29 | 80.00

Finally, we evaluate the choice of different fusion functions on the WMCA
dataset in Table VII. The evaluated fusion functions are simple order
statistics commonly found among ensemble learning decision fusion strategies.
We observe overall better performances regarding AUC and APCER with the median
fusion function.

TABLE VII: Comparison of AUC and APCER@5%BPCER with different fusion functions
for the puzzle task on WMCA dataset.
---
Function | AUC |  | APCER
---
(5%BPCER)
Mean | $90.12\pm 0.42$ | 30.3
25th percentile | $91.63\pm 0.50$ | 29.2
Median | $91.41\pm 0.45$ | 27.3

###  VII-E Ablation study

We evaluate the impact of each pretext task on the final anomaly detection
AUROC. In Table VIII, we compare on CIFAR-10 the basic partial puzzle model
with the addition of the piece-wise puzzle task, colorization task, intra-
piece tint rotation detection task with and without the attention map. While
the piece-wise puzzle and colorization give our model great discrimination
power with an AUC of 89.12, the intra-piece task with attention further
refines our model.

TABLE VIII: Ablation study of each component on CIFAR-10 using the AUROC. The
baseline is the partial puzzle task.
---
Ablation Settings | AUC
Piece-wise puzzle | Colorization |  Intra-piece tint rotation | Attention
- | - | - | - | 75.44
✓ | - | - | - | 86.97
✓ | ✓ | - | - | 89.12
✓ | ✓ | ✓ | - | 90.94
✓ | ✓ | ✓ | ✓ | 92.48

We also investigate on more datasets how the addition of attention in the
intra-piece task improves anomaly detection in Table IX. By including
attention weights for each piece, we can further improve the mean AUC on all
datasets, although marginally increasing the prediction variances on different
normal classes.

TABLE IX: Ablation study of the intra-piece task attention using the AUROC.
---
Att. | AUC
CIFAR10 | CIFAR100 | WMCA
- | $90.94\pm 0.51$ | $88.06\pm 0.84$ | $90.29\pm 0.34$
✓ | $92.48\pm 0.52$ | $88.21\pm 0.83$ | $91.43\pm 0.35$

##  VIII Conclusion and Future Work

We explore in this paper more efficient pretext tasks and show that a
combination of a colorization and a puzzle task with intra-piece tint rotation
subtasks provides the best anomaly detection performances. We also show the
importance of different out-of-distribution functions along with their fusion
functions. Finally, we provide a more comprehensive evaluation protocol than
previously used datasets in the anomaly detection literature. It presents more
challenging datasets and covers object, style and local anomalies. Our method
outperforms state-of-the-art, including a semi-supervised method, on most of
the fine-grained datasets.

For future work we could explore other generative pretext tasks such as image
reconstruction. As in the colorization task, only a part of the image mostly
covering the normal object would be destroyed. Moreover we could reframe our
method into a two-stage anomaly detection. In a first step, representations
would be learned solving our pretext re-colorization, jigsaw puzzle and intra-
piece tint rotation detection tasks. Then we could separately train a OC-SVM
on the concatenation of representations from the puzzle and colorization
encoder. Finally, we could further evaluate our model with differently sized
backbones and measure the impact on each of our three pretext tasks.

## References

  * [1] D. Kwon, H. Kim, J. Kim, S. C. Suh, I. Kim, and K. J. Kim, “A survey of deep learning-based network anomaly detection,” _Cluster Computing_ , vol. 22, pp. 949–961, 2019.
  * [2] Z. Zhang, X. Zhou, X. Zhang, L. Wang, and P. Wang, “A model based on convolutional neural network for online transaction fraud detection,” _Security and Communication Networks_ , vol. 2018, pp. 1–9, 2018.
  * [3] N. Kumar and S. P. Awate, “Semi-supervised robust mixture models in RKHS for abnormality detection in medical images,” _IEEE Trans. Image Process._ , vol. 29, pp. 4772–4787, 2020.
  * [4] H. Lv, C. Zhou, Z. Cui, C. Xu, Y. Li, and J. Yang, “Localizing anomalies from weakly-labeled videos,” _IEEE Trans. Image Process._ , vol. 30, pp. 4505–4515, 2021.
  * [5] R. Leyva, V. Sanchez, and C.-T. Li, “Video anomaly detection with compact feature sets for online performance,” _IEEE Trans. Image Process._ , vol. 26, no. 7, pp. 3463–3478, 2017.
  * [6] Z. Zeng, B. Liu, J. Fu, and H. Chao, “Reference-based defect detection network,” _IEEE Trans. Image Process._ , vol. 30, pp. 6637–6647, 2021.
  * [7] Q. Zou, Z. Zhang, Q. Li, X. Qi, Q. Wang, and S. Wang, “DeepCrack: Learning hierarchical convolutional features for crack detection,” _IEEE Trans. Image Process._ , vol. 28, no. 3, pp. 1498–1512, 2019.
  * [8] L. Jézéquel, N.-S. Vu, J. Beaudet, and A. Histace, “Fine-grained anomaly detection via multi-task self-supervision,” in _17th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS)_ , 2021, pp. 1–8.
  * [9] L. Ruff, J. R. Kauffmann, R. A. Vandermeulen, G. Montavon, W. Samek, M. Kloft, T. G. Dietterich, and K.-R. Muller, “A unifying review of deep and shallow anomaly detection,” _Proceedings of the IEEE_ , vol. 109, no. 5, pp. 756–795, 2021.
  * [10] M. Salehi, H. Mirzaei, D. Hendrycks, Y. Li, M. H. Rohban, and M. Sabokrou, “A unified survey on anomaly, novelty, open-set, and out-of-distribution detection: Solutions and future challenges,” 2021.
  * [11] A. Jaiswal, A. R. Babu, M. Z. Zadeh, D. Banerjee, and F. Makedon, “A survey on contrastive self-supervised learning,” _Technologies_ , vol. 9, no. 1, 2021\.
  * [12] B. Schölkopf, R. Williamson, A. Smola, J. Shawe-Taylor, and J. Platt, “Support vector method for novelty detection,” in _Proceedings of the 12th NIPS_ , ser. NIPS’99, 1999, pp. 582–588.
  * [13] D. M. Tax and R. P. Duin, “Support Vector Data Description,” _Machine Learning_ , vol. 54, no. 1, pp. 45–66, 2004.
  * [14] E. J. Candès, X. Li, Y. Ma, and J. Wright, “Robust principal component analysis?” _Journal of The Acm_ , vol. 58, no. 3, 2011.
  * [15] F. T. Liu, K. Ting, and Z.-H. Zhou, “Isolation forest,” in _Eighth IEEE International Conference on Data Mining_ , 2009, pp. 413–422.
  * [16] R. Chalapathy, A. K. Menon, and S. Chawla, “Anomaly detection using one-class neural networks,” _CoRR_ , vol. abs/1802.06360, 2018.
  * [17] P. C. Ngo, A. A. Winarto, C. K. L. Kou, S. Park, F. Akram, and H. K. Lee, “Fence GAN: Towards Better Anomaly Detection,” in _IEEE 31st International Conference on Tools with Artificial Intelligence (ICTAI)_ , 2019, pp. 141–148.
  * [18] P. Oza and V. M. Patel, “One-Class Convolutional Neural Network,” _IEEE Signal Processing Letters_ , vol. 26, no. 2, pp. 277–281, 2019.
  * [19] S. Goyal, A. Raghunathan, M. Jain, H. V. Simhadri, and P. Jain, “DROCC: Deep robust one-class classification,” in _International Conference on Machine Learning_ , 2020.
  * [20] T. Schlegl, P. Seeböck, S. M. Waldstein, U. Schmidt-Erfurth, and G. Langs, “Unsupervised anomaly detection with generative adversarial networks to guide marker discovery,” in _International Conference on Information Processing in Medical Imaging_ , 2017, pp. 146–157.
  * [21] S. Akcay, A. Atapour-Abarghouei, and T. P. Breckon, “Ganomaly: Semi-supervised anomaly detection via adversarial training,” in _Asian Conference on Computer Vision_ , 2018.
  * [22] N. Tuluptceva, B. Bakker, I. Fedulova, and A. Konushin, “Perceptual Image Anomaly Detection,” in _Pattern Recognition_ , 2020.
  * [23] M. Ivanovska and V. Struc, “Y-GAN: Learning dual data representations for efficient anomaly detection,” _CoRR_ , vol. abs/2109.14020, 2021.
  * [24] G. Kwon, M. Prabhushankar, D. Temel, and G. AlRegib, “Backpropagated Gradient Representations for Anomaly Detection,” in _ECCV_ , 2020, pp. 206–226.
  * [25] L. Ruff, R. A. Vandermeulen, N. Görnitz, A. Binder, E. Müller, K.-R. Müller, and M. Kloft, “Deep semi-supervised anomaly detection,” in _ICLR_ , 2020\.
  * [26] G. Pang, C. Shen, and A. van den Hengel, “Deep anomaly detection with deviation networks,” in _ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD_, 2019.
  * [27] S. Gidaris, P. Singh, and N. Komodakis, “Unsupervised representation learning by predicting image rotations,” in _ICLR_ , 2018.
  * [28] M. Noroozi and P. Favaro, “Unsupervised learning of visual representations by solving jigsaw puzzles,” in _ECCV_ , 2016, pp. 69–84.
  * [29] A. Dosovitskiy, P. Fischer, J. T. Springenberg, M. A. Riedmiller, and T. Brox, “Discriminative unsupervised feature learning with exemplar convolutional neural networks,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 38, no. 9, pp. 1734–1747, 2016.
  * [30] R. Zhang, P. Isola, and A. A. Efros, “Colorful image colorization,” in _ECCV_ , vol. 9907, 2016, pp. 649–666.
  * [31] D. Pathak, P. Krähenbühl, J. Donahue, T. Darrell, and A. A. Efros, “Context encoders: Feature learning by inpainting,” in _IEEE CVPR_ , 2016, pp. 2536–2544.
  * [32] C. Doersch, A. Gupta, and A. A. Efros, “Unsupervised visual representation learning by context prediction,” in _IEEE ICCV_ , 2015, pp. 1422–1430.
  * [33] P. Le-Khac, G. Healy, and A. Smeaton, “Contrastive Representation Learning: A Framework and Review,” _IEEE access : practical innovations, open solutions_ , vol. 8, pp. 193 907–193 934, 2020.
  * [34] T. Chen, S. Kornblith, M. Norouzi, and G. E. Hinton, “A simple framework for contrastive learning of visual representations,” in _Proceedings of the 37th International Conference on Machine Learning_ , vol. 119, 2020, pp. 1597–1607.
  * [35] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, B. Piot, k. kavukcuoglu, R. Munos, and M. Valko, “Bootstrap your own latent - a new approach to self-supervised learning,” in _NeurIPS_ , vol. 33, 2020, pp. 21 271–21 284.
  * [36] I. Golan and R. El-Yaniv, “Deep anomaly detection using geometric transformations,” in _NeurIPS_ , 2018, pp. 9758–9769.
  * [37] D. Hendrycks, M. Mazeika, S. Kadavath, and D. Song, “Using self-supervised learning can improve model robustness and uncertainty,” in _NeurIPS_ , 2019, pp. 15 637–15 648.
  * [38] K. Sohn, C.-L. Li, J. Yoon, M. Jin, and T. Pfister, “Learning and evaluating representations for deep one-class classification,” in _ICLR_ , 2021.
  * [39] J. Tack, S. Mo, J. Jeong, and J. Shin, “CSI: Novelty detection via contrastive learning on distributionally shifted instances,” _NeurIPS_ , 2020\.
  * [40] V. Sehwag, M. Chiang, and P. Mittal, “SSD: A unified framework for self-supervised outlier detection,” in _ICLR_ , 2021.
  * [41] T. Reiss and Y. Hoshen, “Mean-shifted contrastive loss for anomaly detection,” _CoRR_ , vol. abs/2106.03844, 2021.
  * [42] S. Han, H. Song, S. Lee, S. Park, and M. Cha, “Elsa: Energy-based learning for semi-supervised anomaly detection,” _CoRR_ , vol. abs/2103.15296, 2021\.
  * [43] C.-L. Li, K. Sohn, J. Yoon, and T. Pfister, “CutPaste: Self-supervised learning for anomaly detection and localization,” in _IEEE CVPR_ , 2021, pp. 9664–9674.
  * [44] N. Li, K. Jiang, Z. Ma, X. Wei, X. Hong, and Y. Gong, “Anomaly detection via self-organizing map,” in _2021 IEEE International Conference on Image Processing, ICIP 2021, Anchorage, AK, USA, September 19-22, 2021_. IEEE, 2021, pp. 974–978.
  * [45] F. M. Carlucci, A. D’Innocente, S. Bucci, B. Caputo, and T. Tommasi, “Domain generalization by solving jigsaw puzzles,” in _IEEE CVPR_ , 2019, pp. 2229–2238.
  * [46] W. Wang and J. Shen, “Deep visual attention prediction,” _IEEE Trans. Image Process._ , vol. 27, no. 5, pp. 2368–2378, 2018.
  * [47] S. Iizuka, E. Simo-Serra, and H. Ishikawa, “Let there be color! Joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification,” _ACM Trans. Graph._ , vol. 35, no. 4, 2016.
  * [48] J.-W. Su, H.-K. Chu, and J.-B. Huang, “Instance-Aware Image Colorization,” in _IEEE CVPR_ , 2020, pp. 7965–7974.
  * [49] R. Müller, S. Kornblith, and G. E. Hinton, “When does label smoothing help?” in _NeurIPS_ , 2019, pp. 4696–4705.
  * [50] N. Higham, “Cholesky Factorization,” _Wiley Interdisciplinary Reviews: Computational Statistics_ , vol. 1, pp. 251–254, 2009.
  * [51] X. Jin and J. Han, “ _K_ -Means clustering,” in _Encyclopedia of Machine Learning_ , 2010, pp. 563–564.
  * [52] G. J. McLachlan, “Mahalanobis distance,” _Resonance_ , vol. 4, no. 6, pp. 20–26, 1999.
  * [53] H. Xiao, K. Rasul, and R. Vollgraf, “Fashion-mnist: A novel image dataset for benchmarking machine learning algorithms,” _CoRR_ , vol. abs/1708.07747, 2017\.
  * [54] A. Krizhevsky, “Learning multiple layers of features from tiny images,” 2009.
  * [55] P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona, “Caltech-ucsd birds 200,” 2010.
  * [56] J. Liu, Z. Lian, Y. Wang, and J. Xiao, “Incremental kernel null space discriminant analysis for novelty detection,” in _IEEE CVPR_ , 2017, pp. 4123–4131.
  * [57] A. George, Z. Mostaani, D. Geissenbuhler, O. Nikisins, A. Anjos, and S. Marcel, “Biometric Face Presentation Attack Detection With Multi-Channel Convolutional Neural Network,” _IEEE Transactions on Information Forensics and Security_ , vol. 15, pp. 42–55, 2020.
  * [58] I. Chingovska, A. Mohammadi, A. Anjos, and S. Marcel, “Evaluation methodologies for biometric presentation attack detection,” in _Handbook of Biometric Anti-Spoofing - Presentation Attack Detection, Second Edition_ , ser. Advances in Computer Vision and Pattern Recognition, 2019, pp. 457–480.
  * [59] L. Deecke, R. Vandermeulen, L. Ruff, S. Mandt, and M. Kloft, “Image Anomaly Detection with Generative Adversarial Networks,” in _Machine Learning and Knowledge Discovery in Databases_ , ser. Lecture Notes in Computer Science, 2019, pp. 3–17.
  * [60] Y. Fei, C. Huang, C. Jinkun, M. Li, Y. Zhang, and C. Lu, “Attribute Restoration Framework for Anomaly Detection,” _IEEE Transactions on Multimedia_ , pp. 1–1, 2020.
  * [61] S. Zagoruyko and N. Komodakis, “Wide residual networks,” in _Proceedings of the British Machine Vision Conference_ , 2016.
  * [62] D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” in _ICLR_ , 2014.
  * [63] A. Van den Oord, N. Kalchbrenner, L. Espeholt, O. Vinyals, A. Graves _et al._ , “Conditional image generation with pixelcnn decoders,” in _NeurIPS_ , 2016, pp. 4790–4798.
  * [64] L. Ruff, R. A. Vandermeulen, N. Görnitz, L. Deecke, S. A. Siddiqui, A. Binder, E. Müller, and M. Kloft, “Deep one-class classification,” in _Proceedings of the 35th International Conference on Machine Learning_ , vol. 80, 2018, pp. 4393–4402.
  * [65] P. Perera, R. Nallapati, and B. Xiang, “Ocgan: One-class novelty detection using gans with constrained latent representations,” in _IEEE CVPR_ , 2019, pp. 2898–2906.
  * [66] M. Salehi, A. Eftekhar, N. Sadjadi, M. H. Rohban, and H. R. Rabiee, “Puzzle-ae: Novelty detection in images through solving puzzles,” _CoRR_ , vol. abs/2008.12959, 2020.
  * [67] C. Chen, W. Yuan, Y. Xie, Y. Qu, Y. Tao, H. Song, and L. Ma, “Novelty detection via non-adversarial generative network,” _CoRR_ , vol. abs/2002.00522, 2020.
  * [68] N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: A simple way to prevent neural networks from overfitting,” _Journal of Machine Learning Research_ , vol. 15, no. 1, pp. 1929–1958, 2014\.
  * [69] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _Medical Image Computing and Computer-Assisted Intervention_ , vol. 9351, 2015, pp. 234–241.
  * [70] I. Sutskever, J. Martens, G. Dahl, and G. Hinton, “On the importance of initialization and momentum in deep learning,” in _Proceedings of the 30th International Conference on Machine Learning_ , vol. 28, no. 3, 2013, pp. 1139–1147.
  * [71] I. Loshchilov and F. Hutter, “SGDR: Stochastic gradient descent with warm restarts,” in _ICLR_ , 2017.
  * [72] S. Liang, Y. Li, and R. Srikant, “Enhancing the reliability of out-of-distribution image detection in neural networks,” in _ICLR_ , 2018\.


Class I/F targets identified by Peterson et al. (2011) and Dunham et al.
(2015) which are lacking Gaia observations. For this sample of 325 members,
only 275 have sufficient IR data for our analysis. We collect IR data for 31
members from Spitzer (Peterson et al., 2011; Dunham et al., 2015) and 244
other members from WISE. The IR data distribution of the 275 members is of 11%
from Spitzer and 89% from WISE.

#### TW Hydra

The TW Hydra star-forming region, also referred to as TW Hya in the the text,
was well constrained by Gagné et al. (2017). The membership is conducted using
low- and high-resolution, optical and near-infrared spectroscopy, radial
velocity, and Hipparcos data to ascertain the probability of membership (Gagné
et al., 2017). From the sample, we select targets that are classified as
‘bonafide’ and ‘high likelihood’ members resulting in a sample of 40 members.
All the IR data for this sample is from WISE and available within Gagné et al.
(2017).

####  $\eta$ Chamaeleontis

The $\eta$ Chamaeleontis star-forming region, also referred to as $\eta$ Cha
in the text, was constrained and studied by (Sicilia-Aguilar et al., 2009)
using IR spectroscopy. The sample of 18 members is observed by Spitzer for
which we collect complete IR data for 14 targets (Sicilia-Aguilar et al.,
2009); the remaining 4 have incomplete IRAC/MIPS1 data but complete WISE1-4
bands thus favoring WISE data. The IR distribution is of 78% from Spitzer and
22% from WISE.

#### Upper Scorpius

The Upper Scorpius star-forming region, also referred to as Upper Sco in the
text, was re-characterized by Luhman & Esplin (2020) to having 1761 members
using Gaia data. IR data in the form of both IRAC/MIPS1 and WISE bands are
provided by Luhman & Esplin (2020) for most of the members (1712). There are
483 targets with sufficient Spitzer data which we preferentially use and have
sufficient WISE data for another 1229 targets. The IR data distribution is of
28% from Spitzer and 72% from WISE.

The ratio of IR data obtained from Spitzer to WISE is plotted as a bar graph
per star-forming region in Figure 9. We show this to illustrate the origin of
the IR photometry that is used in the classification outlined below in Section
2.2. Note, when a target has been previously identified as a YSO in membership
studies but is lacking complete Gaia data, the average distance of the
expected star-forming region is used and the target is still considered to be
a member.

##  Appendix B Fractional disk luminosity SED example

The manner by which $L_{\text{fract}}$ is calculated is illustrated in Figures
10 and 11. In both Figures, the left column of panels show the stellar flux
and disk flux. The right column of panels show the disk flux contributions per
wavelength between 1.66 $\mu$m and the millimeter flux, which are interpolated
in the far infrared regime. The disk flux between 1.66 $\mu$m at 110 $\mu$m
provides a $\mu$m-sized dust luminosity emanating from the disk. The shaded
yellow area thus represents the disk flux and once divided by the stellar flux
yields $L_{\text{fract}}$. The 70 $\mu$m Spitzer MIPS2 flux is undetected in
most Class III targets (81/85). When we consider the upper limits for the 70
$\mu$m data point (available for 15/85 Class III targets), the calculated
$L_{\text{fract}}$ value is at most a factor of 2 larger which makes its
inclusion both uncertain and largely insignificant compared to the larger
spread in $L_{\text{fract}}$ values between Class II, III and debris disk
objects. For consistency across the Class III disk sample, we have chosen to
exclude the upper limits and non-detections from MIPS2 fluxes in
$L_{\text{fract}}$. Furthermore, the MIPS2 70 $\mu$m flux is likely to be
contaminated by nearby sources when observing YSOs in clusters, due to the
large beam size; in considering IRAS-60 $\mu$m and IRAS-100 $\mu$m fluxes, the
problem would be further exacerbated by the larger beam sizes and thus these
are not used.

Figure 10: Left column SED plots for a Class II (a), Sz 91 (c), and Class III
(e) object. The shaded yellow demonstrates the disk flux contribution and the
shaded red is the expected stellar luminosity. $L_{\text{fract}}$ is
calculated from dividing the calculated disk flux by the literature stellar
luminosity. Right column Disk flux contributions per wavelength. (a): Class II
disk in Upper Sco. (b): Disk flux for the Class II disk. (c): Sz 91 in Lupus,
a Class III disk according to the $\alpha_{\mathrm{Lada}}$ classification
scheme for which there is a detected and evident 70 $\mu$m excess. (d): Disk
flux contributions for Sz 91. Although there is a significant 70 $\mu$m
excess, its particular contribution to the disk flux, and consequently the
$L_{\text{fract}}$ parameter is low.  Figure 11: Same SED plots as Figure 10
(a): Class III disk in Cr A. (b): Disk flux contribution for the Class III
disk. (c): Class III disk with a 70 $\mu$m upper limit in Taurus. (d): Disk
flux contribution for the Class III disk. Even though there is a large upper
limit for the evident 70 $\mu$m, it’s impact on $L_{\text{fract}}$ is low
changing it from $1.53\cdot 10^{-2}$ to $<1.53\cdot 10^{-1}$ which is an
uncertain upper limit.

##  Appendix C Independence of $L_{\text{fract}}$ and $\dot{M}_{\text{acc}}$
from $L_{*}$

We compute the correlation between $L_{\text{fract}}$ and $L_{*}$ to
investigate and show that the derived $L_{\text{fract}}$ \-
$\dot{M}_{\text{acc}}$ correlation is significant, shown in Figure 12. We use
an uncertainty of 20% for Class II $L_{*}$ based on similar extinction and
objects as in van der Marel et al. (2019); Francis & van der Marel (2020).

Figure 12: Both of these plots include Class II objects which have
$\dot{M}_{\text{acc}}$ measurements for which we conduct a brief analysis as
to the parameter dependencies. Left: $L_{\text{fract}}$ vs $L_{*}$
relationship for which we find a mediocre correlation $r_{\rm corr}=0.53\pm
0.12$. Right: $\dot{M}_{\text{acc}}$ vs vs $L_{*}$ relationship for which we
find a stronger correlation $r_{\rm corr}=0.81\pm 0.2$. The legend in the
right panel (b) applies to both plots.

##  Appendix D SEDs for Class III objects with mm-observations

Class III objects SED plots

[]Class III objects SED plots (cont.)

[]Class III objects SED plots (cont.)

[]Class III objects SED plots (cont.)

##  Appendix E Class II and III data

Table 3: Class III targets with millimeter-flux or with $\dot{M}_{\text{acc}}$
measurements.

Column label | Description
---|---
Region | Star-forming region object belongs to
2MASS | 2MASS Point Source Catalog source name
Gaia |  Gaia DR2 source name
$d$ | Distance to target(a) based on the inversion of Gaia parallax (pc)
SpT | Spectral type
ref_SpT | Spectral type reference
$A_{v}$ | Extinction in $V$-band (mag)
ref_$A_{v}$ | Extinction reference
$L_{*}$ | Luminosity ($L_{\odot}$)
ref_$L_{*}$ | Stellar luminosity reference
$M_{*}$ | Stellar mass ($M_{\odot}$)
ref_$M_{*}$ | Stellar mass evolutionary isochrone model used
$\dot{M}_{\text{acc}}$ | Mass accretion rate ($M_{\odot}$ yr-1)
ref_$\dot{M}_{\text{acc}}$ | Mass accretion rate reference
$L_{\text{fract}}$ | Fractional disk luminosity
$F_{mm}$ | ALMA millimeter flux (mJy)
e_$F_{mm}$ | Uncertainty in ALMA millimeter flux
ref_$F_{mm}$ | ALMA millimeter flux reference
Band | ALMA band used for millimeter flux observations
$M_{mmdust}$ | Disk dust mass ($M_{\oplus}$)
e_$M_{mmdust}$ | Uncertainty in disk dust mass
Fit | Flag when $A_{v}$ or $L_{*}$ used are beyond literature reported uncertainties

(a) When no distance is reported in the table due to a lack of Gaia result, we
adopt the average distance to the region found in Table 1.

References: 1) Pecaut et al. (2012); 2) Uyama et al. (2017); 3) Siess et al.
(2000) 4) Lieman-Sifry et al. (2016); 5) Martin et al. (1998): 6) Wahhaj et
al. (2010); 7) Baraffe et al. (2015); 8) Hardy et al. (2015); 9) Cheetham et
al. (2015); 10) Huber et al. (2016); 11) This work; 12) Williams et al.
(2019); 13) Luhman & Mamajek (2012); 14) Barenfeld et al. (2016); 15) Dunham
et al. (2015); 16) Erickson et al. (2011); 17) Prato et al. (2003); 18) Bowler
et al. (2017); 19) Wu et al. (2020); 20) Kohn et al. (2016); 21) Wilking et
al. (2005); 22) McClure et al. (2010); 23) Najita et al. (2015); 24) Bouvier &
Appenzeller (1992); 25) Kraus et al. (2014); 26) Nuernberger et al. (1998);
27) Ribas et al. (2017); 28) Luhman (2004); 29) Bulger et al. (2014); 30)
Akeson et al. (2019); 31) Ward-Duong et al. (2018); 32) Kraus et al. (2017);
33) Rebull et al. (2010); 34) Zhang et al. (2018); 35) Esplin et al. (2014);
36) Csépány et al. (2017); 37) Kraus et al. (2015); 38) Herczeg & Hillenbrand
(2014); 39) Luhman et al. (2009); 40) Scelsi et al. (2008); 41) Cieza et al.
(2012); 42) van der Marel et al. (2016); 43) Hartigan & Kenyon (2003); 44)
Manara et al. (2017); 45) Long et al. (2018); 46) Luhman (2007); 47) Daemgen
et al. (2016); 48) Frasca et al. (2015); 49) Daemgen et al. (2013); 50) Long
et al. (2017); 51) Pascucci et al. (2016); 52) Lada et al. (2006); 53) Ruíz-
Rodríguez et al. (2018); 54) Alcalá et al. (2017); 55) van der Marel et al.
(2018); 56) ; 57) Comerón et al. (2009); 58) Lovell et al. (2021); 59) Mortier
et al. (2011); 60) Galli et al. (2015); 61) Manara et al. (2013); 62) Merín et
al. (2008); 63) Rugel et al. (2018); 64) Sicilia-Aguilar et al. (2008); 65)
Currie & Sicilia-Aguilar (2011); 66) Harvey et al. (2014); 67) Cazzoletti et
al. (2019); 68) Luhman et al. (2017); 69) Liu et al. (2015); 70) Rodriguez et
al. (2015); 71) Gagné et al. (2017); 72) Rizzuto et al. (2015); 73) Luhman &
Esplin (2020); 74) Luhman et al. (2018); 75) Preibisch et al. (2001); 76)
Pecaut & Mamajek (2016); 77) Walter et al. (1994); 78) Preibisch et al.
(2002); 79) Esplin et al. (2018); 80) Houk & Smith-Moore (1988); 81) Manara et
al. (2020); 82) Venuti et al. (2019); 83) Fang et al. (2017); 84) Romero et
al. (2012); 85) Esplin & Luhman (2020); 86) Mužić et al. (2012); 87) Cieza et
al. (2010); 88) Evans et al. (2009).

Table 4: Class II targets (in Lupus, $\eta$ Cha, TW Hydra, and Upper Sco) with
millimeter-flux or with $\dot{M}_{\text{acc}}$ measurements used in Figures 2,
3, and 4.

Column label | Description
---|---
Region | Star-forming region object belongs to
2MASS | 2MASS Point Source Catalog source name
Gaia |  Gaia DR2 source name
Other name | Alternate name
$d$ | Distance to target(a) based on the inversion of Gaia parallax (pc)
SpT | Spectral type
ref_SpT | Spectral type reference
$A_{v}$ | Extinction in $V$-band (mag)
ref_$A_{v}$ | Extinction reference
$L_{*}$ | Luminosity ($L_{\odot}$)
ref_$L_{*}$ | Stellar luminosity reference
$M_{*}$ | Stellar mass ($M_{\odot}$)
ref_$M_{*}$ | Stellar mass evolutionary isochrone model used
$\dot{M}_{\text{acc}}$ | Mass accretion rate ($M_{\odot}$ yr-1)
ref_$\dot{M}_{\text{acc}}$ | Mass accretion rate reference
$L_{\text{fract}}$ | Fractional disk luminosity
$F_{mm}$ | ALMA millimeter flux (mJy)
e_$F_{mm}$ | Uncertainty in ALMA millimeter flux
ref_$F_{mm}$ | ALMA millimeter flux reference
Band | ALMA band used for millimeter flux observations
$M_{mmdust}$ | Disk dust mass ($M_{\oplus}$)
e_$M_{mmdust}$ | Uncertainty in disk dust mass
Fit | Flag when $A_{v}$ or $L_{*}$ used are beyond literature reported uncertainties
Structured | Label indicating whether a disk is considered as structured based on ALMA observations.

(a) When no distance is reported in the table due to a lack of Gaia result, we
adopt the average distance to the region found in Table 1.

References: 1) Alcalá et al. (2017); 2) Manara et al. (2018a) ; 3) Baraffe et
al. (2015); 4) Ansdell et al. (2018); 5) Siess et al. (2000); 6) Rugel et al.
(2018); 7) Venuti et al. (2019); 8) Luhman & Esplin (2020); 9) Barenfeld et
al. (2016); 10) Manara et al. (2020); 11) Luhman & Mamajek (2012); 12)
Fairlamb et al. (2015); 13) Andrews et al. (2018); 14) Rigliaco et al. (2015);
15) Evans et al. (2009); 16) Williams et al. (2019); 17) van der Marel et al.
(2016); 18) Ribas et al. (2017); 19) Ruiz-Rodriguez et al., in prep